Distributed training network system with storage network and service network separated and communication method

Abstract

The invention provides a distributed training network system with separated storage and service networks and a communication method. A storage / service network controller is coupled with an operating system of a working node, and is coupled with a storage network interface and a service network interface; the storage network and the service network are transmitted in parallel in the physical network and are transmitted in parallel in an isolated manner logically; and therefore, the problem of mutual interference between data transmission (during parallel transmission in the same physical network) in training processes of sample set cache data transmission, training parameter transmission and the like in distributed training is solved through further network communication management, and the distributed training efficiency is further improved.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence model training, cloud computing, and data center network, in particular, to a distributed training network system and communication method that separates storage and business networks. Background technique [0002] Thanks to the development of algorithms, data and hardware computing capabilities, artificial intelligence is currently in its third development climax. In terms of algorithms, the introduction of the concept of deep learning and the development of related algorithms have greatly improved the ability of machine learning, followed by breakthroughs in algorithm research represented by deep learning and reinforcement learning, and the continuous optimization of algorithm models has greatly improved the application of artificial intelligence. Accuracy (such as speech recognition and image recognition, etc.). In terms of data, with the technological advancement and p...

AI Technical Summary

Problems solved by technology

In this way, although the caching of the next round of sample sets and the synchronous execution (overlap) of the current round of training can theoretically avoid waiting between rounds of iterations and improve training efficiency; however, in fact, storage flow and business flow Jointly occupy the same physical network bandwidth; the overlap of the two inevitably leads to disorderly competition and preemption of network bandwidth. Due to mutual interference, the time required for network data transmission becomes unpredictable, and communication becomes the performance bottleneck of distributed training.

This kind of interference is not only because the storage stream usually occupies a large bandwidth, and then interferes with the communication of parameters transfer, training cluster management and other services during the distributed training process, but also includes the parameter server communication under the PS communication model during the distributed training process. The network congestion formed by the bottleneck interferes with the storage network communication, that is, the adverse effect on the sample set caching process

Images