Multi-layer digest file generation method and file correctness verification method for massively parallel system

Abstract

The invention provides a multi-layer digest file generation method and a file correctness verification method for a massively parallel system. The multi-layer digest file generation method comprises the steps as follows: an object file is split into multiple first-level files according to the size of an original Hash block as a unit; a first-level interlayer digest is generated for each first-level file by using a message digest algorithm; and the total digest is generated by using the message digest algorithm for at least one time on the basis of the first-level interlayer digests. The file correctness verification method comprises the steps as follows: a multi-layer digest file corresponding to a file is read, and the size of the original Hash block is obtained; the file is split into multiple first-level files by using the size of the original Hash block as a unit; and a first-level interlayer digest is generated for each first-level file by using a message digest algorithm, and then compared with a corresponding first-level interlayer digest in the multi-layer digest file, and if the two are not consistent, wrong information is output. After the multi-layer digest file generation method and the file correctness verification method are adopted, the correctness of a large number of files can be rapidly checked on the massively parallel system.

Description

technical field [0001] The invention relates to the field of network security, in particular to a method for generating a multi-layer summary file under a large-scale parallel system and a method for verifying correctness of a file based on a multi-layer summary file under a large-scale parallel system. Background technique [0002] On a large-scale parallel system composed of tens of thousands of CPUs, the number and capacity of files to be processed by the application program are very large, and the contents of the files on the disk array are damaged, or the program is read from the disk array into the memory during the running process. Or data content errors caused by sporadic I / O errors when writing files from memory are difficult to detect quickly. [0003] At present, the traditional means of verifying the correctness of files is to use hash algorithms such as MD5, sha1, and des to digest files. By comparing the digests, it is possible to find out whether the contents ...

AI Technical Summary

Problems solved by technology

Therefore, in a large-scale parallel system, when the file data capacity is very large, using the MD5 algorithm to generate a summary or perform verification becomes a very time-consuming task.

[0006] In addition, the existing summary tools under linux or windows are all static, and do not support real-time detection of file I / O operations in applications, and cannot detect I / O errors in time and avoid error expansion as soon as possible.

Images