Method, device and system for processing massive data of graph structure

Abstract

The invention discloses a method, device and system for processing massive data of a graph structure. Data computational efficiency can be improved, and operational reliability of a system can be improved. The method for processing the massive data of the graph structure comprises the steps of using a slave node to read the graph structure data to a memory; preprocessing the graph structure data in the memory to obtain at least one data piece in the graph structure data, wherein every two adjacent vertexes of the data piece are arranged on the same data piece; mapping the data piece obtained through preprocessing to the slave node; using the slave node to compute the data piece which is mapped to the slave node through an iterative algorithm. Due to the fact that information of every two adjacent vertexes is mapped to the same slave node instead of different slave nodes, when the slave node computes the data piece which is mapped to the slave node through the iterative algorithm, the slave node does not need to be in communication with other slave nodes, the traffic is reduced, the requirement for bandwidth resources in a cluster is reduced, and computational efficiency of a cluster system and computational efficiency of nodes of the cluster system are improved.

Description

technical field [0001] The invention relates to the field of distributed computing, in particular to a large-scale graph structure data processing method, device and system. Background technique [0002] Graph is a data structure formed by a collection of vertices and the relationship between vertices, that is, a collection of edges. A graph can also be called a network. Many structures in practical applications can be expressed in the form of graphs. For example, in a matrix, each row and column corresponds to a vertex, and the value at the intersection of a row and a column is not zero. The edge exists, and the weight of the edge is the size of the value; another example is a city map, each city (or resident address) is a vertex, and the routes connecting different cities (or resident addresses) are called edges. A graph structure is an abstract data structure that is a model of objects (vertices, nodes) and their relationships (edges). In the real world, the distributio...

AI Technical Summary

Problems solved by technology

[0005] Giraph relies on the effective partitioning of graph-structured data to reduce the amount of communication (mainly sending and receiving messages) and maintain load balance. However, the partitioning of graph-structured data is based on random partitioning of hash functions, thus destroying the locality of computation.

For example, Giraph needs to frequently exchange the information of its adjacent vertices, and at this time the adjacent vertices are divided into different machines, which leads to a sharp increase in communication traffic and a serious reduction in computing efficiency

Images