An automated machine learning system and method for streaming data analysis of concept migration

Abstract

The invention discloses an automatic machine learning system and method for streaming data analysis of concept migration, and the system comprises an obtaining module which is used for obtaining serialized streaming data sets of all batches; a generation module which is used for carrying out automatic feature processing, self-adaptive concept migration and hyper-parameter optimization on the serialized streaming data set of each batch to generate a prediction model; a feedback module which is used for inputting a preset batch of streaming data sets into the prediction model to generate a feedback result; and an online adjustment module which is used for carrying out online adjustment on the prediction model according to the feedback result so as to realize automatic machine learning. The system automatically selects a feature processing technology for different application scenes, self-adapts to concept migration, is good in performance in a real scene, automatically compresses an automatic learning search space through a resource control technology, and ensures that a model operation scale adapts to a data scale and features.

Description

Technical field [0001] The present invention relates to the field of automatic learning technology, and in particular to an automatic machine learning system and method for streaming data analysis of concept migration. Background technique [0002] Automatic machine learning technology has very important value in reducing the labor cost and time cost of building machine learning systems, and has received extensive attention from the industry. [0003] Automatic machine learning technology in related technologies builds systems based on the assumption that training data and test data have the same distribution. However, in the application scenarios of streaming data analysis, such as online advertising, recommendation systems, fraud detection, industrial manufacturing, etc., due to the existence of concept migration, data distribution will gradually change over time, leading to existing automatic machine learning Technology cannot be directly applied to these scenarios. [0004] In ...

AI Technical Summary

Problems solved by technology

Existing automatic machine learning systems only implement some primary feature transformations in automatic feature engineering, and there is still a big gap in performance in actual application scenarios compared with manually designed systems

Images