Data processing method and device in data modeling

Abstract

The invention discloses a data processing method and device in data modeling, which are used to solve the problems in the prior art that the preprocessing flow of original data has a large amount of calculation, a long calculation time, a waste of calculation resources, and a reduction in work efficiency. . The method is: according to the data conversion function corresponding to the preset data processing category identification, perform data conversion on the data column corresponding to each feature in the read original data to generate a corresponding extended feature column, and convert all the features in the original data to The corresponding extended feature columns are combined to generate an extended feature set; the correlation coefficient of each feature in the extended feature set is determined; the feature whose correlation coefficient meets the set conditions is selected as an important feature, and the data column corresponding to the important feature is selected in the extended feature set . In this way, the problems of long time consumption and large calculation amount caused by data modeling through exhaustive data preprocessing methods are avoided, the calculation efficiency is improved, and the flexibility and adaptability of automatic data modeling are improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data processing method and device in data modeling. Background technique [0002] Data mining is one of the steps in database knowledge discovery, which is to find hidden relationships from large amounts of data and extract valuable information. Usually, in data mining, methods and technologies in the fields of database technology, statistics, online analytical processing, and machine learning are combined to process data from different perspectives. [0003] The specific process of data mining includes the following steps: business understanding, data understanding, data preparation, model building, model evaluation, and model deployment. [0004] In the data preparation process, the acquired raw data needs to be preprocessed. The original data is the wide table data stored in the database or data warehouse, as shown in Table 1, the original data includes missing values ​​...

AI Technical Summary

Problems solved by technology

[0011] Embodiments of the present invention provide a data processing method and device in data modeling, which are used to solve the problem of long calculation time and large amount of calculation in the original data preprocessing flow in the prior art, which increases the operating load of the computer, wastes computing resources, and reduces computer productivity problems

Images