Text data comparison method and device

Abstract

The invention relates to a text data comparison method and device in the technical field of information processing. The method comprises the steps that text data item sets in two data dictionary tables are obtained, word segmentation processing is conducted on the two text data item sets, a Chinese word set of each element in the two text data item sets is obtained, similarity measurement between the elements of the two text data item sets is calculated, and the Chinese word set of each element in the two text data item sets is obtained. The method comprises the following steps of: obtaining similarity measurement, preprocessing the similarity measurement through a preset similarity ratio threshold value to obtain a similarity measurement matrix, converting a comparison analysis problem of two text data item sets into a problem of seeking an optimal matching scheme by a bipartite graph through abstraction and modeling of a dictionary table comparison analysis problem, and solving the problem by utilizing a KM algorithm. According to the method, automatic comparison and analysis of the dictionary table data based on semantics are realized, the working pressure of manual comparison in the data reorganization process is effectively relieved, and a new thought is provided for automatic processing of data comparison.

Description

technical field [0001] The present application relates to the technical field of information processing, and in particular, to a text data comparison method and device. Background technique [0002] With the reduction of data collection and storage costs, the amount of data shows explosive growth, but at the same time, more and more requirements are put forward for data association and integration, and data association and integration face more and more challenges. As a key bridge between original data and high-value data, data integration and reorganization plays an increasingly important role in data-based statistical analysis, and has also become an increasingly basic and heavy task in data processing. [0003] The data dictionary table is the basic data that defines the metadata such as data items in the current database system, and is the key information for the application and understanding of the entire database system. Therefore, the comparison, association and conne...

AI Technical Summary

Problems solved by technology

At present, the Extract-Transform-Load (ETL) technology of data warehouses is widely used in the industry to realize the extraction, transformation and fusion of heterogeneous data. The existing research results: use Python as the intermediate unit to realize the records of tables in heterogeneous databases Data comparison solves the problem of data access and table record level comparison in different storage databases, but still fails to solve the problem of automatic comparison of the same data ontology in different expressions, and lacks automatic processing methods based on semantics

Images