Method and device for processing massive real-time generated data of thermal power plant

Abstract

The invention belongs to the field of power plant operation data processing, and particularly relates to a method and device for processing mass real-time generated data of a thermal power plant. On the basis of modeling and storing massive time series data generated in the production process of a thermal power plant based on a key value data model, the time series data is sliced according to timedimensions and stored in an HBase, and meanwhile, a SparkSQL execution process is fused to inquire the massive time series data. The Spark SQL is a module of the spark for processing the structured data, and provides a programming abstraction called DataFrame and used as a distributed SQL query engine. Data query is converted into Spark RDD, then the Spark RDD is submitted to a cluster to be executed, and extremely high data query efficiency is achieved.

Description

technical field [0001] The invention belongs to the field of power plant operation data processing, and in particular relates to a processing method and device for massive real-time generated data of thermal power plants. Background technique [0002] Time series data is data collected at different times, this type of data is collected in chronological order for situations where the described phenomenon changes over time. This type of data reflects the state or degree of a certain thing, phenomenon, etc. over time. In industrial production fields such as thermal power generation, by installing sensors in industrial equipment and production environments, sensors will collect a large amount of time-series data. Thermal power plants usually have a lot of equipment, and the sensors that collect equipment status information are large in scale. Therefore, the time series data collected during the production process generally have the characteristics of fast collection frequency a...

AI Technical Summary

Problems solved by technology

When querying data, there may be a large amount of data and real-time changes that make it impossible to write effective SQL statements for querying, or the speed of SQL statements is too slow during query execution

Although the NoSQL database storage solution that has emerged in recent years has alleviated the pressure of massive data access to a certain extent, the storage solution for time-series data is still immature, especially for thermal power plants with massive time-series data. In the production field, there is still an urgent need for efficient data storage and query solutions

Images