Data structure prediction transmission and automatic data processing method based on flow chart

Abstract

The invention provides a data structure prediction transmission and automatic data processing method based on flow chart. The method comprises the steps of obtaining a modeling experience data table of a target business scene; after data source nodes are set, selecting and connecting operator nodes, setting parameters of the operator nodes, and generating a flow chart; predicting a data table structure operated and output by the current operator node according to the parameters of the operator node, transmitting the data table structure to all downstream operator nodes of the current operatornode, and configuring the parameters by the downstream operator nodes according to the transmitted predicted data table structure when the current node does not run or does not run; and executing operator calculation, the downstream operator node acquiring data flowing out of the upstream operator node, and after each operator node operates, generating and transmitting output data to the downstream operator node. According to the method, data transmission and data structure prediction transmission are conveniently and flexibly realized, and a simple and rapid building method is provided for data processing and model design by business personnel and technicians.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a flow chart-based data structure prediction transfer and automatic data processing method. Background technique [0002] With the advent of the era of big data, artificial intelligence technology is deeply applied to financial business scenarios, and machine learning is an inevitable product of artificial intelligence research to a certain stage. performance of the scene. "Experience" usually exists in the form of "data". Through machine learning algorithms, "models" can be generated from the data, and the data needs to be converted into machine learning samples including various characteristic variables. The characteristic variables in empirical data often need to be combined with statistical algorithms to process the data before machine learning and then model training. [0003] By using machine learning technology to mine value from data, traditional machine lea...

AI Technical Summary

Problems solved by technology

[0003] By using machine learning technology to mine value from data, traditional machine learning or data processing requires technicians to write codes to complete the overall process before building a model and generate valid data and variables for the model. This process requires human input. High cost, high technical threshold and low production efficiency

For business personnel, they have a clearer understanding of real business scenario requirements, while technical personnel often have a biased understanding of business scenario requirements, and the designed system often does not fully meet the needs of business scenarios

[0004] For example, traditional data processing methods need to write program codes for data processing of characteristic variables in data tables. There are a lot of dirty, messy, missing data in real data or data with many dimensions and large data volume. In related data processing software , cannot automatically filter and output field variables, lacks effective variable transmission and processing methods, and cannot effectively predict preprocessed field variables and pass them downstream

[0005] Existing related system platforms do not have table structure prediction and transmission. When calculations are performed to a certain operator node, new data structures and data are generated. At this time, downstream data needs to obtain other structures from rows and configure parameters. In complex data Very poor experience during processing and waste of time

Images