Method for downloading mass data on data platform

Abstract

The invention discloses a method for downloading mass data on a data platform, which is suitable for downloading the mass data. The method comprises the following steps of a data query step that a system server queries a number of data items which conform to the conditions in a database; a TXT file computing step that the system server pages the data items and determines a file format of TXT files according to the total number of the data items and the number of the TXT files corresponding to each page; a TXT file generation step that the TXT files are generated on a local WEB application server according to the file format of TXT files, the system server writes the data items into the TXT files and releases the memory resources of the system server after writing; an output flow writing step that an output stream is formed in an internal memory of the system server, the output stream is data flow, the data items in the TXT files are circularly read from the local WEB application server and the output stream is written; and a data export step that the data items are exported from the output stream, and the internal memory resources of the system server are released after exporting.

Description

technical field [0001] The invention relates to network data transmission technology, in particular to a method for downloading a large amount of data on a data platform. Background technique [0002] For large enterprises or network platforms, there are often a large number of data reports that need to be transmitted, especially for enterprises using data platforms such as ERP, users often need to download a large number of data reports to the local machine for subsequent processing. [0003] The currently used transmission method is that the user initiates a request from his own client, and the system server processes the request in its memory, queries the database according to the request, caches the data items obtained from the query in the memory of the system server, and waits for the query to be completed before retrieving it from the The system server's memory is sent to the user's client. During this process, all the work is done in the memory of the system server....

AI Technical Summary

Problems solved by technology

At this time, the available memory will be reduced, which will affect other work of the system server. When the amount of data is too large, it will cause memory overflow and cause the system to crash.

One solution is to configure larger memory, but this will increase the cost, and, as the amount of data continues to increase, only relying on increasing memory capacity cannot solve the problem of downloading large amounts of data

Images