Intelligent data blood relationship tracing method and device based on clustering analysis

Abstract

An intelligent data blood relationship tracing method based on clustering analysis comprises the steps that 1, a table structure and data are read, and forming data characteristics of all fields through a data engineering means; step 2, learning the data sample by using a clustering analysis algorithm in machine learning by taking a field as a unit and a field data feature set as a feature; 3, repeatedly executing the clustering analysis in the step 2 until the optimal classification number and the optimal classification number are found; step 4, under the optimal classification, automaticallyjudging the data fields in the same classification as fields which may have a blood relationship; 5, for each blood relationship, inferring the direction of the blood relationship according to the sequence of table creation time pointed by the relationship, namely inferring which field is a source and which field is a target, and if the field of the blood relationship comes from the same table, marking the blood relationship as an invalid blood relationship; and 6, calculating the table blood relationship according to the effective field blood relationship.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a method and device for intelligent traceability of blood relationship of data based on cluster analysis. Background technique [0002] With the development and popularization of big data and machine learning technology, the amount of data used, managed and generated by data analysis software is increasing, and the dependence on the format, content and quantity of data is also increasing. Before the data analysis system runs, it needs to perform various extraction, cleaning, conversion and desensitization operations on the data. The complexity of these businesses determines that there are many procedures, long processes and complicated methods in the data processing process. It is necessary to trace back the lineage of the data to judge the credibility of the data, analyze the influence of the data, and analyze and process the source of the wrong data. Therefore, e...

AI Technical Summary

Problems solved by technology

Its principle determines that only the blood relationship of data within the same database can be recorded, and blood relationship tracking across databases or even across database types (such as data conversion from mysql to oracle) cannot be completed

Moreover, the method based on the database plug-in requires the data blood relationship to be recorded at the same time as the data is generated. If the record fails when the data is generated (for example, the temporary network is unreachable), the blood relationship cannot be obtained again

At the same time, the use of plug-ins will affect the performance of software access to data, and some software with high real-time requirements cannot use this type of plug-in

Not only that, this method cannot handle the data lineage at the field level, and the granularity of data lineage tracing is not enough

Images