Method, device and system for high-performance distributed data conversion

Abstract

The invention provides a method, device and system for converting high-performance distributed data. Task scheduling can be carried out based on time scales, tasks about to execute mass data are split, the task executing efficiency is improved, the error-tolerant rate is increased, an efficient file conversion template is utilized, and the high performance of data conversion can be guaranteed. The method comprises the steps that the data conversion template is configured and stored; the tasks to be executed are acquired, and then the total data amount to be executed by the tasks to be executed is acquired according to the tasks to be executed; the tasks to be executed are split into a plurality of sub-tasks according to the total data amount, and then sub-task data is acquired according to the sub-tasks; the sub-task data is converted according to the data conversion template, target conversion data is obtained, and target conversion data is stored in a target conversion file.

Description

technical field [0001] The invention relates to the field of computer technology and software, in particular to a method, device and system for high-performance distributed data conversion. Background technique [0002] In the process of connecting information systems between different companies, the scene of exchanging data through files is often encountered, which requires the data existing in the database to generate corresponding data files in a certain format. When the amount of interactive data is large, a fast, efficient and stable function module for generating data files is needed. [0003] The existing common data conversion to realize the file generation scheme roughly includes the following steps: [0004] 1. Use the scheduled task to start the task of generating the data file at the specified time point; [0005] 2. Query the corresponding data from the database according to the task parameters; [0006] 3. Load all the queried data into the memory and conver...

AI Technical Summary

Problems solved by technology

However, if all the data is loaded into the memory at one time, it is easy to cause memory overflow; and a large amount of data will cause the task execution time to be too long, and only a single machine can be used to generate files, resulting in poor conversion performance and high risk; in addition, task execution fails Afterwards, all data needs to be retrieved again. A large amount of data tasks cannot be split into multiple subtasks. The cost of re-executing tasks is higher than that of re-executing sub-tasks after splitting.

[0012] 3. Existing file upload strategy: Overwrite method is adopted, that is, data overwrites the same file, resulting in the re-covering of the valid data that was successfully uploaded before, resulting in waste of data transmission

[0013] 4. In addition, the commonly used data conversion templates in the prior art include Freemarker, Velocity, HTTP, etc., but templates based on Freemarker, Velocity, and HTTP have poor conversion performance, and it takes a long time to generate data files of massive data. Labeling is cumbersome

Images