Concurrent program developing method for processing of large-scale data based on small memory

Abstract

The invention discloses a concurrent program developing method for processing of large-scale data based on a small memory; the method comprises the following steps: developing data spaces in a memory and a disk, dividing the data spaces into a plurality of data blocks and reading task data into the divided disk data blocks; setting a reading thread, a calculating thread and a writing thread; and utilizing the disk as the cache of the memory to store data which is not calculated temporarily. The entire calculating data is divided into proper small data blocks and stored in the disk. Only when the data block is required to be calculated, the data is dispatched to the memory; therefore, the dynamic dispatching of the data is realized and the requirement of processing large-scale data based on the small memory is satisfied. Compared with the prior art, the concurrent program developing method for processing of large-scale data based on the small memory has the advantages: the large volume of the disk is utilized sufficiently, the calculating data is dynamically dispatched to a calculation core and the balance of the calculated load is achieved; meanwhile, the communication and the calculation are executed asynchronously, the entire performance of the system is improved and the requirement of processing the large-scale data based on the small memory is satisfied.

Description

technical field [0001] The invention relates to the field of high-performance computing, in particular to a highly practical parallel program development method for processing large-scale data based on small memory. Background technique [0002] In the current social data explosion of human beings, there are more and more information data, and people have higher and higher requirements for information data processing capabilities. Not only oil exploration, weather forecast, aerospace defense, scientific research, etc. require high-performance computing, but also financial, The demand for high-performance computing in a wider range of fields such as government informatization, education, enterprises, and online games is growing rapidly. [0003] Computing speed is particularly important for high-performance computing, which is developing towards multi-core and many-core. However, due to the relatively large scale of computing data in some applications, limited by network ban...

AI Technical Summary

Problems solved by technology

However, due to the relatively large scale of computing data in some applications, limited by network bandwidth, system memory, etc., the system cannot process data far beyond the tolerance of the system memory, resulting in the inability of the computing core to be fully utilized.

For systems with insufficient memory, the method of importing all the data into the memory at one time for reprocessing can no longer meet the current needs. It is necessary to further design a method that can process large-scale data in a single machine based on the existing limited hardware equipment, and will not A method to reduce computing efficiency and make full use of computing cores

Images