Compression and storage method for large-scale image data

Abstract

The invention discloses a compression and storage method for large-scale image data. The method comprises the following steps of (1) storing the data of original images by a binary adjacent matrix M in a behavior unit; (2) according to the deviation value of each row in the adjacent matrix M, establishing a hash index; (3) ascending the starting points of each row in the adjacent matrix M according to the out-degree; (4) recording nodes with in-degree of 0 as root nodes, descending the root nodes according to the out-degree, and recording as a root node list; (5) for each node in the root node list, using each root node as the starting node, and sequentially distributing ID (identity) according to the depth-first strategy; (6) traversing the adjacent matrix M, converting the matrix according to the newly distributed ID, and storing in an edge list format; (7) sequencing the data with the edge list format; (8) compressing and storing the data with the edge list format in a line mode. The method has the advantages that the required data storage space is little, the number of random reading times is little, and the parallel degree of threads is high.

Description

technical field [0001] The invention relates to the field of high-performance computing, and more specifically, relates to a method for compressing and storing large-scale graph data. Background technique [0002] Processing and mining large real-world relational graphs and designing scalable systems has become an extremely pressing problem today. For example, social network graphs, web page pointing graphs, and protein interaction graphs are particularly challenging. Because they cannot really be divided into small pieces that can be processed in parallel, this defect in parallelism has made distributed computing attract the attention of many people in the industry. [0003] In recent years, some models based on graph computing have been proposed, such as Pregel and GraphLab and other vertex-centered vector models. In this model, users define a program that can run each vertex locally; in addition, some graph data is high The performance computing system is based on key-v...

AI Technical Summary

Problems solved by technology

[0005] In view of this, the object of the present invention is to provide a method for compressing and storing large-scale graph data, which expresses the relational graph data into a tree structure, and performs row compression and column compression on the graph data according to the characteristics of the tree structure, It aims to solve the problems of large data storage space, large number of random reads and low thread parallelism existing in existing methods

Images