Time series data storage method, device and equipment based on Hbase

Abstract

The invention discloses an Hbase-based time series data storage method, device and equipment. The method comprises the steps of obtaining a piece of time series data generated by a data source; sorting the plurality of data labels and the at least one monitoring index to obtain a label sorting result and an index sorting result, and associating the label sorting result and the index sorting result with the data names to obtain a sorting relationship; dividing timestamps according to a preset time interval, and recording the timestamps corresponding to the data generation time; forming a row key by the data name, the identification tag and the timestamp corresponding to the data generation time, forming a column of a tag column cluster by the data tags except the identification tag in the plurality of data tags according to a tag sorting result, and forming a column of an attribute column cluster by the at least one monitoring index according to an index sorting result, obtaining a multi-valued model storage structure; and storing the multi-valued model storage structure and the sorting relationship into an HBase database. By applying the method, the data label and the row key can be decoupled, and redundant storage is avoided.

Description

technical field [0001] The present invention relates to the technical field of data storage, in particular to an Hbase-based time series data storage method, device and equipment. Background technique [0002] With the advent of the Internet of Things and Industry 4.0, all data in different industries can be networked, and a large amount of time-organized data is generated every moment, which is called time series data. Time series data needs to be stored for later query, statistics and analysis. Common time-series data models are divided into single-value and multi-value models. In the single-value model, a row of data corresponds to only one monitoring indicator of time-series data. Taking the data generated by wind power generation as an example, the storage structure of the single-value model is shown in Table 1. . [0003] Table 1 Single value model storage structure [0004] metric timestamp manufacturer wind field model value power 2019-01...

AI Technical Summary

Problems solved by technology

In this single-value model storage structure, the number of tags contained in tags is limited. If there are too many tags, the rowkey will be very long, which will affect the retrieval performance

Moreover, when this single-value model storage structure is applied to the Hbase database, how many columns the column cluster contains and how many times the rowkey must be stored means how many times the tags must be stored repeatedly. There is a problem of redundant storage of tags. In the face of massive When timing data, the impact of this redundancy will be very large

Images