Method for storing massive time series data on basis of Hadoop technologies

Abstract

The invention discloses a method for storing massive time series data on the basis of Hadoop technologies. The method includes steps of building Hadoop primary key storage and primary key design scheme models; building HBase data row storage structural design scheme models; accessing the time series data and the like. The method has the advantages that the method is low in implementation cost, storage spaces and memory occupation required for storing the data can be reduced to a great extent, and the storage efficiency can be improved.

Description

technical field [0001] The invention relates to the field of centralized storage and calculation processing of massive continuous process parameter data generated by the production site of a large-scale production process automation industrial enterprise / group, belongs to the technical field of time series data storage, and specifically relates to a massive time series data storage based on Hadoop technology Implementation. Background technique [0002] Electricity, petroleum, chemical industry, new energy and other asset-heavy production process automation industries are the main production sectors related to the national economy and people's livelihood, and their main production processes need to be carried out continuously. Due to the complexity of the production process, there are many process parameters (time-series data) that need to be controlled and measured in the production process, and the amount of time-series data that needs to be recorded is very large, and the...

AI Technical Summary

Problems solved by technology

However, due to breaking the continuity of time series data, it is very inefficient to access the data of a single measuring point or a group of temporarily organized measuring points; if the time series data of the same measuring point is used in multiple calculations or applications, multiple copies need to be stored data

There are many production processes on the production site of the enterprise, and the sources of time series data are relatively scattered. It is difficult to ensure that many control systems collect time series data synchronously. The collected time series data is not synchronized at all times. The requirements of the enterprise production site time series data platform can only be used as time series data storage for special calculations or applications, and cannot be used as a general enterprise time series data storage platform

Images