Automatic data processing method and system, computer equipment and readable storage medium

Abstract

The invention discloses an automatic data processing method, which comprises the following steps of: respectively scanning each data table in metadata, and extracting field information of each field in each data table to generate a data dictionary of an intermediate process; adjusting the data dictionary, and carrying out association between tables to extract a repetition relationship and a mapping relationship; and constructing a data view according to the repetitive relationship and the mapping relationship, updating the data dictionary according to translation information of each field in the data view, and performing anomaly detection on metadata. The invention further discloses an automatic data processing system, computer equipment and a computer readable storage medium. By the adoption of the method and device, automatic data processing is achieved, and standard unified management and effective execution are truly achieved by effectively carding the data.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data automatic processing method, a data automatic processing system, computer equipment and a computer-readable storage medium. Background technique [0002] With the continuous maturity and development of Internet technology and big data technology, the status of campus information sharing is getting higher and higher in today's era. In the context of the development of the times, it is particularly important to improve data quality. The improvement of data processing capabilities and the mining of data value are the requirements of the times, and the standards need to be gradually improved. At the same time, the actual level of data sharing is also related to the ability of data processing, and the security and privacy issues involved in the improvement of the level of data sharing are gradually being exposed. [0003] However, in the current early stage of ...

AI Technical Summary

Problems solved by technology

[0003] However, in the current early stage of data governance, the standardized mapping of data dictionaries of various types of data is mostly through national / industry standards, and the metadata is manually mapped to the corresponding standard dictionary one by one, so as to realize the link between metadata and standard data. conversion, which consumes a lot of manpower and has a high error rate

On the other hand, each implementation is based on the original data dictionary, table name, field name and other information to complete the data standardization conversion, which has a high error rate

[0004] Therefore, in view of the problems of inconsistent standards, multi-source and heterogeneous data, lack of manufacturers’ dictionaries, and excessive manpower and time spent on data processing, it is necessary to design an automatic data processing method to sort out data items and truly achieve standards Unified management and effective execution

Images