Graph data distributed processing system based on CPU-GPU heterogeneous architecture

Abstract

The invention discloses a graph data distributed processing system based on a CPU-GPU heterogeneous architecture, and the system comprises the steps: a graph summary generation method which is used for generating a summary graph of large-scale graph data and accelerating the convergence or operation of a graph algorithm; a runtime two-stage load balancing system which is used for balancing loads among the computing nodes and loads on a CPU and a GPU in each heterogeneous computing node; a message processing method improves the communication efficiency by compressing and combining messages; anddividing the large-scale graph data, and processing the graph data in a distributed manner on a plurality of computing nodes by adopting a BSP synchronization mode. The graph computing system based on the CPU-GPU heterogeneous architecture is realized, the efficiency and scale of processing graph data can be improved by the distributed processing system, and the system performance can be furtherimproved by utilizing the powerful computing capability of the GPU.

Description

technical field [0001] The invention belongs to the field of graph data processing and high-performance computing, and in particular relates to a distributed graph data processing system based on a CPU-GPU heterogeneous architecture. Background technique [0002] Graph data structure representations are widely used in various fields of science and engineering to represent various networks. Some problems are mapped as very large-scale graphs, containing millions or even billions of vertices and edges. For example, VLSI chip layout, logistics network analysis, vehicle route planning and scheduling, social network analysis and data mining often require running graph algorithms on large-scale graph data. In this era of big data, with the continuous growth of social networks and e-commerce networks, the explosive growth of graph data has been brought about. The World Wide Web currently has over 4.84 billion pages and over one trillion URLs. Furthermore, the graph of the social...

AI Technical Summary

Problems solved by technology

[0006] In view of the fact that the current stand-alone graph processing system cannot handle large-scale graph data, and the distributed graph processing system cannot use the GPU coprocessor in the computing node to accelerate calculations, the present invention proposes a distributed CPU-GPU heterogeneous architecture-based The graph processing system divides large-scale graph data into graph fragments, distributes them to multiple computing nodes for parallel processing, adopts the synchronous mode for algorithm iteration, and performs dynamic load balancing and message compression processing during the operation of the algorithm, fully Utilize the powerful computing capabilities of the CPU and GPU of the heterogeneous cluster in the data center to greatly improve the performance of large-scale graph data processing

Images