Parallel data processing method based on distributed structure

Abstract

The invention relates to a parallel data processing method based on a distributed structure. The storing comprises steps as follows: (1) a data master key value is extracted from master nodes according to types of master key values, directed slave nodes distributed by data are determined according to data attribute values and a section comparison result in the master nodes, and simultaneously, a global keyword B+ tree index is established; (2), the data are distributed to the slave nodes corresponding to the master key values according to the global keyword B+ tree index on the basis of a share-nothing principle; and (3), the slave nodes receive a data distributing request, and the data are stored in child nodes locally on the basis of the share-nothing principle. According to the method, an effective index mechanism is combined, and the storage and management efficiency of system data is improved; on one hand, the reasonable data distribution is guaranteed, the storage throughput of the slave nodes is reduced, the local query performance is improved, and the system flexibility is guaranteed by utilizing high expandability of the slave nodes; and on the other hands, local transcript safety is guaranteed through local duplication of multiple transcripts.

Description

technical field [0001] The invention is oriented to the fields of geographic information system, spatio-temporal data management, location-related services, large-scale sensor stream data management, etc., and proposes a set key-value The RDB-KV parallel cloud database storage and retrieval method with the advantages of both the database (Key-Value Store) and the relational database realizes a mass data storage technology that combines the efficient access characteristics of key-value storage and the integrity of the database. Background technique [0002] Cloud computing is an important direction of current information technology development. Computing and storage services based on cloud platforms have undergone major changes in application modes, application scope, and technical requirements due to changes in the underlying infrastructure. Cloud storage is a new concept extended and developed on the concept of cloud computing. It refers to the integration of a large numbe...

AI Technical Summary

Problems solved by technology

From a global point of view, it cannot support the unified key value of all table objects corresponding to the data ontology query; this contradiction makes it difficult to combine RDB query and KV query;

[0009] 3) The challenge of massive data analysis to high-performance query: service computing involves complex access to a large number of complex elements, object attributes, monitoring data, multimedia data, remote sensing data, and various unstructured data

However, in the data management system that the traditional GIS platform relies on, due to the inherent deficiencies in the architecture, there is an insurmountable mechanism limitation in terms of high-performance query

This causes the GIS platform to have defects in basic queries in the face of increasingly expanding geographic data

Images