Field-level data blood relationship determination method and device based on knowledge graph

Abstract

The invention relates to a field-level data blood relationship determination method and device based on a knowledge graph. The method comprises the steps of obtaining table names and table structure information of two data tables in a first data table pair, wherein the first data table pair comprises any two data tables in a database; calculating a first table name similarity between table names of two data tables in the first data table pair, and calculating a similarity between fields in the table structure information to obtain a first field similarity matrix; according to the first table name similarity, the first field similarity matrix and a pre-acquired target weight, determining whether a blood relationship exists between two data tables in the first data table pair or not; acquiring a plurality of first data table pairs with a blood relationship as target data table pairs; and generating a field-level data relationship graph according to the blood relationship between the knowledge graph and the data table in the target data table pair. The method and device can be used to reduce the difficulty in determining the blood relationship of the data table, and improves the determination accuracy of the blood relationship.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular to a field-level data lineage determination method and device based on a knowledge graph. Background technique [0002] Data Lineage refers to the link of data generation, which is used to describe which other tables a table depends on, and how the generation of fields in the table depends on the fields of other tables. Through the data lineage, the upstream and downstream dependencies of data production can be clearly known. When an enterprise has a wide variety of businesses and a large business volume, the entire database system supporting the business will involve hundreds or thousands of tables, and there will be very complicated dependencies between tables. [0003] If the business changes are frequent, the development cycle is short, and the turnover of personnel is high, the table structure and table dependencies may change frequently as time goes by, e...

AI Technical Summary

Problems solved by technology

[0003] If the business changes are frequent, the development cycle is short, and the turnover of personnel is high, the table structure and table dependencies may change frequently as time goes by, eventually making the relationship between tables complicated and difficult to trace

The common dependencies are the dependencies between the data table and the production task. Which fields in the upstream data table are used for the production task can only be reflected in the coding logic. For example, database A or department A saves the updated data file Go to the specified directory, send it to database B through FTP (File Transfer Protocol, file transfer protocol) or other synchronous methods, database B calls the code to analyze the data file, and inserts the final calculation result into the database after a series of calculations on several fields In B, it is used to update an existing field or save it as a new field. The specific calculation process cannot be exposed to the data user

When enterprises with data ask external software service providers to perform data governance tasks, they often only provide information such as table headers and data dictionaries to external partners. It is difficult for external partners to obtain intermediate process records of table generation and change history, while database audit logs It is also difficult to reflect all the intermediate processing processes

Images