Large-batch data exporting system and exporting method thereof

Abstract

The invention relates to a large-batch data exporting system and an exporting method thereof. The exporting method comprises the following steps: after obtaining a user request, distributing, adding into a queue space based on a priority rule, deconstructing the request to obtain a query data volume, setting the query data volume in a request annotation, and carrying out preprocessing judgment onthe request at the queue head of a first row; and if the characteristics of the large-batch data are met, allocating multiple threads to export the data, otherwise, directly exporting the data, and updating and adjusting the queue. According to the method, the problems of memory overflow, associated service system collapse, poor user experience and the like in the exporting process of a large amount of data are solved, and under limited server resources, the throughput rate of the system is increased, the stability of the system is improved, and the user experience is improved.

Description

technical field [0001] The invention relates to the technical field of electrical digital data processing, in particular to a large-volume data export system and an export method thereof. Background technique [0002] In recent years, the wave of digitalization with big data, Internet of Things, artificial intelligence, and 5G as its core features is sweeping the world. The massive data brought about by this, as well as the personalized requirements of new services such as high bandwidth and low latency, make traditional general-purpose The server and storage architecture are overwhelmed, and the import and export of data is a regular requirement of the system. [0003] At present, when exporting large batches of data, a synchronous data export method based on a single thread is usually used, namely: [0004] The user sends a data export request through the client; [0005] After receiving the request, the server queries the data from the database and returns it to the ser...

AI Technical Summary

Problems solved by technology

[0007] However, with the use of the system, the amount of data stored in the database will increase, and the export speed will become slower and slower. The performance degradation is an inevitable result. The traditional export method can no longer meet the actual application requirements.

[0008] When the amount of data stored in the database reaches a certain level, when the full amount of data is exported, operations such as data query and result set assembly will become slower and slower, and at the same time cause a huge loss of server memory; When a large amount of data at the level is exported, the system responds slowly and takes a long time to process. Users often need to wait for a long time after submitting a request, even as long as tens of minutes. The user experience is poor; the more serious problem is that when the response time of the request is too long , there will be problems such as server response timeout and request being discarded, resulting in data export failure, core business exception, and affecting multiple associated parties

Images