Heterogeneous system parallel random forest optimization method and system

Abstract

The invention discloses a heterogeneous system parallel random forest optimization system and method. The heterogeneous system parallel random forest optimization system is applied to a central processing unit and a coprocessor mixed heterogeneous cluster and comprises one master node and multiple slave nodes, wherein the main master node is used for dividing a data file to be calculated into multiple data fragments and transmitting the data fragments to all the slave nodes respectively; all the slave nodes are used for receiving the data fragments distributed by the master node to calculate the data fragments and transmitting a calculated optimal solution building decision-making tree to the master node to generate a random forest. As the data fragments are calculated by the slave nodes in parallel, the time for searching for the optimal solution is shortened, efficiency of the whole system is substantially improved, the system is unnecessarily limited by network bandwidth deficiency, small memory capacity and other conditions, and the requirement for processing large-scale data of high-performance application is met.

Description

technical field [0001] The invention relates to the field of machine learning, in particular to a heterogeneous system parallel random forest optimization system and method. Background technique [0002] In recent years, with the rapid development of today's social economy and technology, many application fields are rapidly accumulating a large amount of data, and analyzing these data to discover the information contained in the data has become a common demand in almost all fields. In practical applications, The role of machine learning in data mining analysis technology is becoming more and more important, and it has received extensive attention. [0003] In the prior art, the most commonly used classification method in machine learning is the random forest algorithm. Random forest is a supervised ensemble learning classification technology, which improves the prediction accuracy of the model by summarizing a large number of classification trees. It is precisely because of...

AI Technical Summary

Problems solved by technology

[0004] However, using the existing technology, it takes a long time to find the optimal solution in the process of data optimization. In the current era of exponential growth in the amount of data, it is obvious that the processing of very large data is powerless.

Images