A general machine learning data analysis platform

Abstract

The invention provides a universal machine learning data analysis platform. The universal machine learning data analysis platform comprises an interface module, a data storage module, a preprocessing module, a feature extraction module, a feature conversion module, an algorithm module and a selection optimization module, wherein the feature extraction module extracts feature parameters from to-be-analyzed data according to the feature parameters set by a user; the feature conversion module is used for converting features set by the user to be in a representation form required by the user; the algorithm module comprises a plurality of algorithm models for the user to select and build, and the user builds at least one group of models; the selection optimization module selects out an optimal model and an optimal parameter from the built models and then stores the optimal model; and data generated by the modules is stored in the data storage module. According to the platform, the user can freely combine and use the modules and the algorithm models, and also can build a complex model and quickly and iteratively develop a novel analytic model, so that the working efficiency is greatly improved.

Description

【Technical field】 [0001] The invention relates to the computer field, in particular to a general machine learning data analysis platform. 【Background technique】 [0002] Today is undoubtedly an era of data explosion, whether countries, organizations or individuals are continuously producing data. Simply put, the technology of data analysis is to collect, process, clean, and statistically calculate data for the purpose of discovering useful information, knowledge, and insights to support decision-making. For example, the United Nations will allocate humanitarian aid funds based on the GDP of each country, the Federal Reserve will decide whether to raise interest rates based on the employment index, Fujian Province will predict typhoons based on weather data, and so on. Smart people have obtained data through various means, and based on the processing, processing and analysis of these data, they can guide future decisions. It can be said that data analysis technology has bec...

AI Technical Summary

Problems solved by technology

However, commercial companies, enterprises, research institutes, etc. have complex data and require high performance, high flexibility, high reusability, and fast iterative data analysis tasks. Due to the overly simple automation system, these requirements are often not met.

[0010] As mentioned above, there are two main technical solutions for existing data analysis systems. When pursuing high performance and high accuracy, professionals usually need to spend a lot of manpower to build a specific system. The system is limited to the current use scenario and cannot be transplanted. Maintenance and upgrade costs are high; when pursuing simplicity and ease of use, automatic detection of data sets is usually used to automatically select the method of model training and analysis

This method is only suitable for the rough analysis purposes of ordinary users. It is not friendly to research and development personnel, and has a low degree of freedom of operation. It is not suitable for enterprises and scientific research units that require high accuracy and high performance.

[0011] In addition, for a data analysis platform with high performance and strong versatility, its machine learning training often requires a large amount of data input, and the capacity required for process data is also very large. How to transmit data efficiently is also a difficulty of this system

If you directly transfer data between modules, there are several problems: 1. Network bandwidth limitation. If multiple groups of online training are performed at the same time, large data transmission will greatly reduce operating efficiency without effective scheduling.

2. To transmit data directly, the modules need to know the communication address of each other, which is not conducive to distributed deployment

Images