Parallel decision-making system and method for distributed data processing

Abstract

The invention discloses a parallel decision-making system and method for distributed data processing. The system comprises an initial logic node generation component, a logic node traversal component,a predetermined configuration cost calculation component and a parallel decision-making component. The initial logic node generation component receives task configuration data input by a user to generate an initial logic node topological graph for the distributed data processing system. The logical node traversal component traverses the initial logical node topological graph to obtain a predetermined configuration in the initial logical node topological graph. The predetermined configuration cost calculation component calculates the sum of the transmission cost and the calculation cost of each predetermined configuration. The predetermined configuration transformation component reduces the initial logic node and the connection edge according to the result of the predetermined configuration and merges the initial logic node and the connection edge, thereby obtaining a transformation result logic node topological graph after the initial logic node is reduced so as to reduce the solutionspace of the parallel decision. The parallel decision component obtains a parallel scheme with the total minimum cost through a local greedy strategy for the transformation result logic node topological graph.

Description

technical field [0001] The present disclosure relates to a data processing technique. More specifically, the present disclosure relates to a parallel decision system and method for distributed data processing. Background technique [0002] Now that deep learning is popular, more and more models and larger and larger data make it impossible for deep learning training to be implemented on a single computing device. To this end, people have proposed distributed computing. With the popularization of distributed computing, large-scale jobs or large tensors will deploy different parts of data to various computing devices of different distributed data processing systems for processing through segmentation, and the calculation process of each part needs to be intermediated. Parameter interaction. In this way, during the processing of a specific job, the calculation intermediate parameters or results deployed on one computing device will become the input data of the computing task...

AI Technical Summary

Problems solved by technology

The manual adjustment method of parallel decision-making will first increase the workload of technicians and consume manpower of technicians. On the other hand, when applying and implementing different parallel modes on different logical nodes, people also need to consider the limitation of memory and running time. Consumption, will and may not bring the ideal parallel effect (compress the total running time to the extreme or close to the extreme)

Images