Random forest parallelization machine studying method for big data in Spark cloud service environment

Abstract

The invention discloses a random forest parallelization machine studying method for big data in a Spark cloud service environment. The method comprises the steps that dimension reduction processing is performed on the high-dimensional big data through feature vector importance analysis, and prediction is performed by adopting a weighed voting mode; through a distributed memory management mechanism and a cloud computing platform, parallelization of random forest training process model building, single decision-making tree splitting process and prediction voting is improved. According to the method, dimension reduction processing is performed on the high-dimensional big data through feature vector importance analysis, prediction is performed by adopting the weighed voting mode, therefore, optimization of the random forest method is achieved, and the mining effect of the random forest machine studying method on the complex big data is improved; the random forest parallelization method based on the Spark cloud platform is performed on the basis, so that the operation efficiency of the random forest machine studying method is improved.

Description

technical field [0001] The invention belongs to the field of computers, and in particular relates to a large data-oriented random forest parallel machine learning method under a Spark cloud service environment. Background technique [0002] Explanation of terms: [0003] Feature dimensionality reduction: In the process of image or data feature extraction, too many feature dimensions extracted often lead to too complex feature matching and consume system resources. At this time, a low-latitude feature is used to represent a high-latitude feature. Dimensionality reduction. [0004] With the continuous emergence of various new information publishing methods, the rise of technologies such as cloud computing and the Internet of Things, and various sensors all over the world, data is growing and accumulating at an unprecedented rate. The data age has arrived. With the deepening of network applications, the value of big data applications is becoming more and more obvious. Massi...

AI Technical Summary

Problems solved by technology

[0008] Traditional classification methods can achieve ideal results on low-dimensional small data sets, but when the structure of the data becomes complex, the dimension of the data becomes higher, and the size of the data increases, the performance of the traditional classification method will be obvious. drop

In the face of massive big data, traditional classification methods take a lot of time in the process of modeling and forecasting

Images