Machine learning and big data processing system

Abstract

A machine learning and big data processing system includes a data access unit, data storage unit, information processing unit, information classification unit, information reading unit, information collation unit, information optimization unit, information filtering unit, an intelligent audit unit and a manual audit unit, the database is bidirectionally connected with the data storage unit, An output end of the information optimization unit is connected to an input end of the information screening unit, The output end of the information screening unit is connected with the input end of the intelligent auditing unit, the output end of the intelligent auditing unit is connected with the input end of the manual auditing unit, the output end of the manual auditing unit is connected with the input end of the data storage unit, and the input end of the data storage unit is connected with the output end of the information processing unit. The invention has the advantages of ingenious design,reasonable design, high data processing efficiency, real-time updating database function, suitable for popularization, can be used for processing big data, and has high intelligence degree.

Description

technical field [0001] The invention relates to the technical field of machine learning and big data processing systems, in particular to a machine learning and big data processing system. Background technique [0002] In recent years, a large number of technologies and products have been developed in the field of big data, which have become effective means for big data acquisition, storage, processing analysis or visualization. However, there are high technical barriers to using these technologies and products. How to organize big data technologies in a standardized form and provide users with convenient and efficient usage methods has become a huge challenge. [0003] A series of big data analysis products based on cloud platforms, represented by many large Internet companies, have emerged as the times require. These products provide relatively efficient and relatively convenient means for the organization of big data analysis tasks by virtue of their abundant cloud compu...

AI Technical Summary

Problems solved by technology

However, the data analysts of enterprises are often proficient in domain knowledge and business background, but not very familiar with the technical details of data analysis, resulting in these big data analysis products still have a high technical threshold

Especially in today's rapid development of data analysis technology, most enterprises cannot bear the huge labor cost of mastering various data analysis technologies at the same time

In addition, in practical applications, it is often necessary to train batches of similar models according to different conditions, and existing platforms and products cannot efficiently support this scenario

[0004] After retrieval, the patent No. CN201711354629.6 proposes an interactive automated big data analysis application development system, which includes: a data receiving module, an operator management module, an analysis model generation module, a process analysis optimization module, and a task scheduling execution module; The data receiving module is used to receive the training data set of the big data to be analyzed inputted by the user through the data management interface; the operator management module is used to construct and store the corresponding data set operator according to the training data set, The operator management module is also used to store preset data processing operators, preset machine learning operators and preset process control operators; the analysis model generation module is used to call the The analysis process constructed by the data set operator, the preset data processing operator, the preset machine learning operator, and the preset process control operator generates the analysis model of the big data to be analyzed, and the comparison document Data processing does not support real-time update of data, and the screening of data is not strong, and it is difficult to remove garbage information. Therefore, the present invention proposes a machine learning and big data processing system

Images