High-speed extranuclear graph processing method and system based on random and continuous disk access

Abstract

The invention discloses a high-speed extranuclear graph processing method and system based on random and continuous disk access, belonging to the technical field of graphic calculation and processing in the big data field. The high-speed extranuclear graph processing method comprises a high-speed indexing mechanism which is applied to extranuclear data of an I / O (Input / output) operation submodule and used for processing the extranuclear data at high speed by adopting a memory mapping manner; a synchronous and asynchronous model-combined hybrid scheduling strategy which is applied to an algorithm scheduling submodule and is based on bitmap management, for a purpose of reducing the number of iterations and saving a memory; an intranuclear laying and processing strategy which is applied to intermediate calculation data of a data management submodule, for a purpose of reducing extra I / O expenditure of the intermediate calculation data, wherein random I / O access is converted into continuous I / O access by means of the strategy, so that the idling of a disk is reduced and the operation efficiency of the system is improved.

Description

technical field [0001] The invention belongs to the technical field of graph computing for big data processing, and more specifically relates to a high-speed out-of-core graph processing method and system based on random and continuous disk access. Background technique [0002] With the rapid development of social network, biological information network and information technology, the graph data with these information objects is increasing day by day. The information and potential value contained in these graph data are very attractive, but how to deal with these graph data is a challenging problem. [0003] Graph data in the real world has Power-Law characteristics. Some vertices have many outgoing and incoming edges, while others have few. This distribution characteristic is also called long-tail characteristic. [0004] Existing graph processing methods include distributed processing and out-of-core processing. Distributed systems are not suitable for ordinary people due...

AI Technical Summary

Problems solved by technology

[0009] Aiming at the above defects or improvement needs of the prior art, the present invention provides a high-speed out-of-core graph processing method and system based on random and continuous disk access, the purpose of which is to design a high-speed out-of-core graph that maximizes memory usage efficiency Data processing method, thereby solving the problem of low memory usage caused by the continuous read and write method and the storage medium limitation caused by the random read and write method in the existing out-of-core graph processing system

Images