Graphical data quality evaluation device and method based on big data technology

Abstract

Disclosed are a graphical data quality evaluation device and method based on a big data technology. The device comprises a data acquisition device, an equipment monitoring device, a distributed memory, a spark memory calculation engine, a calculation unit, a data quality checking unit and a data preprocessing unit, wherein the data acquisition device is connected with the equipment monitoring device connected with the distributed memory, the distributed memory is connected with the data preprocessing unit, the data preprocessing unit comprises the spark memory calculation engine, the calculation unit and the data quality checking unit, mass heterogeneous data can be processed rapidly, efficiently and in time, and at the same time, the safe, stable and efficient operation of the device is ensured.

Description

technical field [0001] The invention relates to the application field of equipment monitoring and analysis, in particular to a device and method for graphical data quality evaluation based on big data technology. Background technique [0002] With the rapid development of smart grid, the power system has begun to move towards the era of energy Internet and "big data". A large amount of operating data in the power industry is increasingly characterized by large volume, multiple types, and high value. Data analysis and processing capabilities are lagging behind and data is growing rapidly. The contradiction between them will become more prominent; with the continuous increase of data volume and data types, there will also be problems such as data analysis performance bottlenecks, lack of advanced methods for data analysis and mining, and lack of effective use of unstructured data, which restricts the power industry. Informatization is developing from digitization to intelligen...

AI Technical Summary

Problems solved by technology

Common open source technology components such as Hadoop and Spark have certain limitations in terms of functional integrity and operational stability, and some commercial big data platforms derived from Hadoop have deviations from the actual needs of power business scenarios. Therefore, in-depth analysis and research The business requirements of the electric power industry, the integration and integration of heterogeneous data sources is a practical problem often encountered in the process of enterprise information construction. With the rapid increase of data volume, especially the increase of unstructured data, traditional data warehouse technology and The performance of data extraction tools in data quality evaluation is stretched and cannot meet the data processing performance requirements of massive heterogeneous data and messy low-quality data. Building a graphical data quality evaluation device and method based on big data technology has far-reaching Significance and strong use value

Images