Method and device for importing duplicate removal data into gallery based on rocksdb and storage medium

Abstract

The invention provides a method and device for importing data into a gallery based on rocksdb and a storage medium, and the method comprises the steps: classifying the data to be imported into the gallery to obtain a plurality of data categories, setting a category identifier for each data category, and setting an edge relation between the data in each data category; based on a rocksdb vertex duplicate removal database, performing duplicate removal on data in each data category, and inserting the data into the image library as a vertex; and based on the rocksdb edge duplicate removal database,performing duplicate removal on the edge relationship among the data, and inserting the edge relationship into the image library as an edge. According to the method, firstly, a rocksdb vertex duplicate removal database and a rocksdb edge duplicate removal database which correspond to data types are constructed; the method comprises the following steps of: importing data into a gallery; accordingto the technical scheme, the vertex importing is firstly carried out, and then the vertex importing is carried out after the vertex importing is completed, so that the problem that the actual data isrepeated and unavailable due to the fact that id is not duplicated when the Titan/Janus Graph image library is imported is solved, and due to the fact that the duplicate removal library of multiple types is established, the duplicate removal efficiency is improved, and the user experience is improved.

Description

technical field [0001] The invention relates to the technical field of computer program processing, in particular to a method, device and storage medium for importing deduplication data into a library based on rocksdb. Background technique [0002] The graph theory method is called the graph theory method which takes the graph as the research object. A graph can be represented as a graph composed of some points and lines connecting these points, and can also be abstractly defined as G=(V, E, Φ), where V and E are the vertices and edge sets of the graph respectively, and Φ means A certain functional relationship between V and E. In this way, all systems related to binary relations can be described by graphs, so graph theory can be used for research. When using graph theory to study problems, we only pay attention to whether two vertices are connected by a line, but the position of the vertices and the way of connection are irrelevant. Euler solved the famous Königsberg bri...

AI Technical Summary

Problems solved by technology

According to the design of the Titan / JanusGraph library, when importing data, the program automatically calculates and generates Long-type ids, and the library does not automatically sort the data. If the same piece of data is imported multiple times, different ids will be generated. Therefore, when making data When importing, you need to do a deduplication. It is easy to do deduplication for a small amount of data, but when the amount of data reaches hundreds of millions or billions, deduplication will become difficult, especially when the amount of data reaches tens of billions, one thousand Data deduplication will become more difficult

[0004] In addition, when the data is inserted into the gallery, an id will be generated first. Only after the commit is submitted, the data will be actually inserted into the underlying storage database. If the points and edges are imported at the same time, when the storage machine fails, the data will generate an id. However, the point and edge data are not inserted into the underlying storage database, resulting in an insertion error

Images