A Method of Writing Checkpoint Data in Massively Parallel Systems Based on Random Latency Alleviating I/O Bottleneck

Abstract

The invention discloses a random delay-based large-scale parallel system checkpoint data writing method for relieving an I / O bottleneck. According to the method, checkpoint data is temporarily cachedin a memory, and a checkpoint main process can immediately return to realize separation of a writing process, so that the time of global stop in a checkpoint data operation process is shortened. A preset delay writing time is determined by using a random delay checkpoint file processing method, and writing operation is scattered in time, so that I / O writing peak values at the same moment are reduced and the purpose of relieving the I / O bottleneck is achieved. Before a large-scale parallel system executes I / O operation, associated data information of the large-scale parallel system is periodically detected, and if running of applications is influenced, delay operation is given up and the writing operation is immediately executed, so that the influence on normal running of the applications due to long-time occupation of shared resources is avoided; and on the contrary, the writing continues to be performed according to the determined delay writing time. The pressure on an I / O subsystem caused by the applications in different system platforms compared with a conventional centralized writing mode can be reduced, thereby obtaining higher throughput rate and shorter global blocking time.

Description

technical field [0001] The present invention relates to a processing method for dynamically adjusting the optimal write timing of checkpoint (Checkpoint) data in the field of high-performance computing, more particularly, refers to a method related to alleviating the centralized writing of checkpoint data in a large-scale parallel system that causes I Checkpoint data write control method for / O bottleneck. Background technique [0002] Most high-performance computing adopts a large-scale parallel computing model. The hardware infrastructure of high-performance computing systems includes three parts: computing nodes, network interconnection, and storage systems. The computing nodes are responsible for running computing tasks, and the network interconnection realizes the interconnection between computing nodes and between computing nodes and storage systems. The storage system includes multiple I / O nodes and external storage devices. I / O nodes run parallel file systems. Respo...

AI Technical Summary

Problems solved by technology

After the operation of collecting checkpoint data is completed, the checkpoint software will write the checkpoint data directly to the external storage system by default to deal with possible node downtime failures (such as a node downtime, and the current node is not If the checkpoint data of the node is written to the stable storage, the corresponding checkpoint data of the node will be lost, and the operation of the node-related processes cannot be restored)

Since the number of I / O nodes in the system is far smaller than the number of computing nodes, the centralized writing of checkpoint data by a large number of computing nodes will have an impact on the I / O system, thereby forming a system bottleneck. become more prominent as the size increases

Images