Parallel random forest algorithm for processing unbalanced big data

Abstract

The invention discloses a parallel random forest algorithm for processing unbalanced big data, belongs to the field of unbalanced big data classification, and aims at solving the problem that a parallel random forest algorithm based on a MapReduce mechanism is low in the unbalanced big data classifying efficiency. The parallel random forest algorithm (SBWMR_RF) in which layered self-service sampling and sensitive cost ideas are combined is provided, self-service sampling is carried out on positive and negative classes of each partitioned data set, the information content of the position classis increased while the randomness is guaranteed, a cost sensitive matrix of each partition is calculated dynamically, and the sensitive cost is used for establishment of a base classifier. According to the invention, the unbalanced data is processed in the aspects of data distribution adjustment and algorithm adjustment, influence of data bias is weakened, the classification accuracy of the unbalanced big data is improved, and the algorithm has obvious advantages in the extremely unbalanced environment.

Description

technical field [0001] The invention relates to a parallel random forest algorithm for processing unbalanced large data. It belongs to the classification technology in machine learning and involves the classification method of unbalanced big data. Background technique [0002] The technical focus of processing big data is to achieve fast and scalable parallel analytical processing. As the mainstream framework, Hardtop uses the simple idea of ​​"divide and conquer" to integrate distributed file system HDFS, distributed computing framework Map Reduce, non-relational database HBASE, etc. for big data processing, and can store, manage and mine "big data" task. Big data and big data analysis and processing technologies promote each other. Map Reduce Random Forest algorithm based on Map Reduce, hereinafter referred to as MR_RF, as a kind of integrated classification, because it is simple and efficient in processing classification and regression problems, and has good parallelis...

AI Technical Summary

Problems solved by technology

[0005] When MR_RF deals with the problem of unbalanced data classification, because the positive sample density is too low and the base classifier is built on the sampled data of each block, the influence of data bias is deepened; secondly, MR_RF uses the global minimum gini when constructing the tree. The value is used as the criterion for selecting the split point, it is easy to ignore the positive class, there is a tendency to divide the positive class into negative class, and the classification efficiency is reduced

Images