Construction method of deep learning-based automatic association matching of data

Abstract

The invention provides a construction method of deep learning-based automatic association matching of data. A process of manually establishing matching association among the wide range of data of multiple domains is simulated, matching ability and accuracy are enabled to be continuously improved with optimization of deep learning. The method is specifically to establish feature observation valuesfor abstraction on data entities, and an abstraction process includes feature extraction, automatic execution and automatic learning and evolution. The method is characterized by including the steps of: using a plurality of manual association rules as feature observation values to import the same to a deep-learning model, and setting available ranges of the data and judgment criteria of successfulassociation; and using multiple core object tables as a basis, indexing other tables through the core object tables to establish strong association or soft association, and then outputting a result set of successful relationships, wherein the result set includes an association table, association rules and association matching degrees, and is used as a task to be executed on a corresponding platform or is embedded in a mining analysis task to be used as data processing steps to be executed.

Description

technical field [0001] The invention relates to a construction method for automatic association and matching of data based on deep learning. Background technique [0002] Data association matching is an important processing process for multi-source data of different sources, different objects and different formats. Different sources are because the data is generated from different business systems, and different objects are because of the data stored in different databases and data tables. The entity objects are different (for example, some are related to people, and some are related to enterprises). The different formats are because the actual data requirements of the system and business management are different. In the era of big data, the data in many scenarios has the above characteristics, which brings great obstacles to the combined mining analysis. Data mining analysis depends on the correlation between data, whether it is weak correlation or strong correlation. There...

AI Technical Summary

Problems solved by technology

[0002] Data association matching is an important processing process for multi-source data of different sources, different objects and different formats. Different sources are because the data is generated from different business systems, and different objects are because of the data stored in different databases and data tables. The entity objects are different (for example, some are related to people, and some are related to enterprises). The different formats are because the actual data requirements of the system and business management are different. In the era of big data, the data in many scenarios has the above characteristics, which brings great obstacles to the combined mining analysis. Data mining analysis depends on the correlation between data, whether it is weak correlation or strong correlation. Therefore, mining and analysis of large-scale data with the above characteristics Often, a lot of manpower and material resources are spent on the processing of data association matching

[0003] In the process of operating and maintaining city-level government data for many years, we found that the data differences and mismatches between various departments are very serious, and the use of data to create greater value (including building applications and mining analysis) is very important for many The integration and use of domain data is essential, so in the early stage we mainly established some association rules manually and then realized it through database operations. The specific steps are divided into three steps: first, we need to read and understand the data to find matching data, Second, it is necessary to check and establish matching rules one by one among thousands of data tables and hundreds of thousands of fields. Third, data sampling inspection is required when verifying the matching results. The entire matching process is time-consuming and labor-intensive, and Prone to errors and omissions, resulting in frequent iterations of work

Images