A data import, query and processing method based on bitmap structure

Abstract

The invention discloses a data import, query and handling method, computing device and data handling system based on a bitmap structure, and the data import method is suitable for executing in a dataimport server. The data import method comprises the steps that a data import instruction given from a coordinator server is responded to import one piece or a plurality of pieces of original data; referring to each piece of original data, the piece of original data is conversed to obtain corresponding bitmap structure data according to a preset data handling rule; compression treatment is carriedout on each piece of acquired bitmap structure data to generate a corresponding data block, and each data block is submitted to a storage server for storage; a storage status message fed back from thestorage server is received; if the storage status message indicates that data block storage succeeds, a message of successful storage is sent to the coordinator server, so that the coordinator servercan instruct a cache server to load the data blocks from the storage server, and the failure handling is carried out on each piece of bitmap structure data.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a data import, query and processing method based on a bitmap structure, a computing device and a data processing system based on a bitmap structure. Background technique [0002] In the field of big data technology, it is a very challenging task to perform duplicate statistics on massive data sets and to perform multidimensional cross-statistics on data from multiple massive data sets. At present, there are usually two methods for counting data sets: one is to import the data into a Set data structure, and use the feature of the Set data structure that does not allow duplicate elements to do the ranking. A large amount of storage space is required, but generally all massive data cannot be loaded into the memory. Even if external storage is used, a large number of I / O (Input / Output input / output) operations are required when writing and querying, resulting in poor performance. On...

AI Technical Summary

Problems solved by technology

At present, there are usually two methods for counting data sets: one is to import the data into a Set data structure, and use the feature of the Set data structure that does not allow repeated elements to perform weight ranking. However, this method A large amount of storage space is required, but generally all massive data cannot be loaded into the memory. Even if external storage is used, a large number of I / O (Input / Output input / output) operations are required when writing and querying, resulting in poor performance. One is to use HyperLogLog, DataSketches and other estimation algorithms for weight ranking statistics, which have greatly improved storage and performance, but cannot meet the needs of accurate weight ranking

[0003] Furthermore, when it is necessary to perform cross-statistics on multiple massive data sets, the traditional method is to use a relational database to perform Join (connection) operations on data stored in multiple tables, but when the amount of data is particularly large, this This approach cannot meet the demand in terms of storage or performance

Images