Multi-process processing method, device and system

Abstract

The invention discloses a multi-process processing method, device and system. The multi-process processing method includes numbering local data files; inputting number M of the data files to be processed during each process; acquiring total number N of the local data files; according to modular results of N and M, determining number k of the processes to be started; starting k processes and distributing at last M data file numbers to each process; subjecting the processes to loading the data files, corresponding to the data file numbers, for processing. By the use of the multi-process processing method, device and system, flexible speed adjustment is achieved with the lower workload, and manual editing amount is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing of data processing equipment, and in particular to a multi-process processing method, device and system. Background technique [0002] With the popularization of Internet services and the growth of user data, it is common for devices in the network to process hundreds of gigabytes or tons of data. In order to be able to adapt to the processing tasks of huge amounts of data, the industry usually uses distributed multi-process processing of huge amounts of data. [0003] Such as figure 1 It is a schematic diagram of the existing distributed multi-process processing technology; see figure 1 , where different machines are divided by vertical dotted lines, and each area divided by the vertical dotted lines is the deployment situation on a machine, such as the deployment situation on the first machine in area 101, and the deployment situation on the second machine in area 102 Deploy...

AI Technical Summary

Problems solved by technology

If there are 100 machines processing at the same time, and each machine starts 10 processes, 1,000 lines of data need to be edited, the amount of editing is too large, and the labor cost is high

[0006] (2) In the actual operation process, it is often necessary to adjust the speed of importing data on each machine according to the operation needs, which requires modification of the number of processes and the names and paths of the data files processed by the processes, so if adjustments are required , then all configurations must be readjusted accordingly, the workload is huge, and it is extremely error-prone

Therefore, the speed of importing data in the existing technical solution is inconvenient to change at will, and the operation is not flexible

Images