A Text Incremental Dimensionality Reduction Method Based on Random Forest

Abstract

The present invention relates to a text incremental dimension reduction method based on random forest, comprising the following steps: 1) dividing original text data into multiple subsets to construct original text feature map clusters; 2) dividing the text feature map clusters of each subset into Expressed in the form of a data table; 3) Sampling the records in the data table with replacement to establish a random forest data training set, and obtain a random forest by constructing a classification tree; 4) Convert the text feature map clusters of the new text subset into In the form of a data table, input all the records in the data table into the trained random forest, summarize the voting results of each classification tree to obtain a new category for each record, and complete the merging of the two text feature map clusters according to the classification results, Thus, incremental dimensionality reduction of existing text features is achieved. Compared with the prior art, the present invention has the advantages of high precision, strong expansibility, no need for feature variable selection, and no overfitting.

Description

technical field [0001] The invention relates to the field of data dimensionality reduction in machine learning and natural language processing, in particular to a text incremental dimensionality reduction method based on random forests. Background technique [0002] With the rapid development of technologies such as the Internet of Things, cloud computing, and big data, data has been more comprehensively acquired and rationally utilized, but problems such as data diversity, massiveness, and high-dimensionality have followed. Therefore, in order to use the data better, the data needs to be preprocessed. Data dimensionality reduction is to map data from high-dimensional space to low-dimensional space, remove irrelevant or redundant data, and retain data that reflects the essence of the original data. Using the dimensionally reduced data to perform tasks such as data search, data processing, and data mining can increase the accuracy of data search, reduce the amount of data ca...

AI Technical Summary

Problems solved by technology

Text data has the characteristics of high dimensionality, large amount of data, and rich semantics. However, it is difficult for existing data dimensionality reduction methods to achieve good results in terms of efficiency and accuracy.

Statistics-based dimensionality reduction methods often ignore the semantic content of text data, and semantic-based dimensionality reduction methods are often inefficient

Images