Method and device for detecting hive data tables

Abstract

Embodiments of the present invention provide a method and device for detecting hive data tables, which relate to the field of computer technology and can quickly and accurately detect data uniqueness of hive data tables. The method for detecting the hive data table of the embodiment of the present invention comprises: setting up a configuration file for the hive data table to be tested, the configuration information of the configuration file includes a table name and a primary key; constructing a data file for counting the number of occurrences of the primary key in the hive data table, the data file It is a key-value pair type, with the primary key of the data record in the hive data table as the key, and the number of occurrences of the primary key in the hive data table as the value; when it is detected that a new data record is inserted into the hive data table, update the data file; In the case that the data file has a value greater than 1, a first alarm message for reminding data duplication is issued.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for detecting a hive data table. Background technique [0002] In the era of big data, data analysis and data application are already a very common thing in society. Data analysis and data application are inseparable from big data development. Now big data development uses Hadoop architecture, and data is stored on the distributed file system HDFS (Hadoop Distributed File System). Daily data development is to convert SQL into MapReduce through hive Or directly use MapReduce for data processing, which is very different from developing on a relational database. Hive uses sql query statement HQL. The biggest difference between HQL and database sql is that database sql supports data update, but HQL does not support it. That is, HQL cannot update (update) or delete (delete) data, and can only use insert ( Insert) realizes update and delete in disguise. This di...

AI Technical Summary

Problems solved by technology

However, data accuracy detection is actually a simple logical SQL processing. In MapReduce processing, it is still necessary to calculate how many map numbers will be generated by the SQL, and then perform real calculations. It takes more time to calculate the map number and the efficiency is very low. up

[0013] (2) There is hysteresis

[0014] Now this way of judging the accuracy of data through hive sql is to wait until the execution of the data program is completed, and the judgment can only be made after the result has been inserted into the target table. Real-time monitoring of feedback information during execution

[0015] (3) The error caused by the data file cannot be located

[0016] Sometimes it is found that the inserted data is correct, but the result of checking the target table is wrong

This is often because some wrong data files are stored, resulting in wrong final table results

However, it is impossible to check the problem simply through hive sql.

Images