Data processing method, device, terminal and storage medium based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence, and provides a data processing method, device, terminal and storage medium based on artificial intelligence, including: dividing an important data set selected from an original data set using an XGBoost model into multiple sub-data sets ; Use each sub-data set to train and test the lightGBM model to get the test pass rate; pre-standardize the training data and test data corresponding to the first field in the sub-data set to obtain a new sub-data set; use the new sub-data set to train And test the lightGBM model to get the test pass rate; judge whether the data corresponding to the first field needs to be standardized according to the two test pass rates; repeat the above process until it is judged whether the data corresponding to the last field in the important data set needs to be standardized processing, and update the important data set according to all the judgment results to obtain the target data set. The present invention can select a data set with strong stability and great contribution to the prediction model.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence, in particular to an artificial intelligence-based data processing method, device, terminal and storage medium. Background technique [0002] In the agent retention / low performance prediction model, some characteristics are affected by seasonality (such as the number and quality of recruits in February) and business aspects (such as the forced promotion of APP, the quality of newcomers in assessment months and non-assessment months), there are The phenomenon of instability across time, that is, the distribution of features (mean, standard deviation, etc.) changes over time, which brings difficulties to model training and prediction across time. [0003] In addition, since the training of the agent retention / low performance prediction model is to train the data of the first 3 to 6 months of the forecast month as a whole training set (for example, the training set of the rete...

AI Technical Summary

Problems solved by technology

However, the quality of the recruited population in each month may be slightly different, which leads to the finding of the optimal split point may not be stable in multiple months of training; the quality and feature distribution of the recruited population in the forecast month and training month are also discrepancies, which creates a potential hazard for the stability of the model's predictions across time

Images