Mass data quality verification method based on Hadoop

Abstract

The invention discloses a mass data quality verification method based on Hadoop. The method comprises the following steps: formulating a data quality standard; for the DDL instruction, writing metadata information of the creation table into Hive; for the DQL statement, converting the SQL character string into an abstract syntax tree, performing syntax analysis on the abstract syntax tree, analyzing whether latest generated SQL semantics are wrong or not according to a data quality standard, and adding extension information; compiling the abstract syntax tree to generate a corresponding logic execution plan, optimizing the logic execution plan, converting the optimized logic execution plan into a physical plan, generating a MapReduce job, submitting the MapReduce job to the Yarn for execution, and finally returning an execution result; storing a returned execution result into the HDFS, and carrying out data visualization and abnormal data exporting, tracking and tracing. Therefore, thedata quality verification effects of abnormal data display, traceability, easy configuration and easy classification are achieved.

Description

Technical field [0001] The invention relates to the technical field of big data, and in particular to a method for checking data quality. Background technique [0002] With the current rapid development of information technology and Internet technology, the amount of data is growing in a blowout pattern, and the types of data are gradually increasing, and the complexity is getting higher and higher. Modern society has entered the era of big data. In this context, in order to give full play to the application value of big data, data quality management must be strengthened, and the security, accuracy, and stability of data transmission and use must be improved. [0003] In the past decades of development, large-scale relational databases such as Oracle have been the mainstay. In recent years, various open source databases have emerged in an endless stream, such as relational databases such as MySQL and PGSQL, and many semi-structured databases, such as ElasticSearch. , Mongodb, etc....

AI Technical Summary

Problems solved by technology

Data quality in data governance is particularly important in data warehouse-based data center construction; if non-compliant data is not standardized and screened; not only will it face the problem of data storage, but it will also cause a lot of valuable information to become very Difficult to obtain, a large amount of invalid data pollution, increase the investment of unnecessary human and financial resources

[0004] In the traditional data quality process, many people ignore the importance of data quality, resulting in a lot of normal data being polluted by abnormal data, usually passively discovered by downstream users or application teams, and then telling the big data data analysis team to go Find the cause of abnormal data, and then go upstream to find the root cause

This will lead to slow investigation cycle, complex process, time-consuming and laborious, only specialized personnel can understand, increased learning costs, data accumulation and many other problems

Images