Method for extracting features based on distributed mutual information documents

Abstract

The invention provides a method for extracting features based on distributed mutual information documents to solve the bottleneck problems of data processing scale limit and poor performance in the process of processing a great number of documents. The method comprises a first step of collecting and initializing the documents, a second step of calculating the frequency of the appearance of participles in the documents and the mutual information value of the participles in different classification and selecting feature word collection accordingly, and a third step of calculating the weight of all feature words to form the final document vector quantity collection. The feature words of the document classification and the weight of the feature words are extracted by utilizing the MapReduce distributed calculating frame, and the process of classifying the documents and the expandability of the document classification can be accelerated. By means of the key assignment pair design, the weight, in the documents, of the feature words can be calculated while the feature words are extracted, and the efficiency of classifying the documents is improved.

Description

technical field [0001] The invention belongs to the technical field of distributed computing and data mining, and in particular relates to a document feature extraction method based on distributed mutual information. Background technique [0002] With the rapid development of the Internet, it has also brought us a very spectacular information explosion. How to process the huge amount of data on the Internet is a severe test that Internet companies must face. To solve the problem of "rich data and poor information", It is necessary to analyze and mine massive data. A common and practical method for processing massive data is to classify documents, that is, document classification. [0003] The task of document classification is to classify a document with an unknown category label according to its content under a given classification system. It can be classified into multiple categories or not belong to any category (for a given set of categories ). [0004] Common document...

AI Technical Summary

Problems solved by technology

[0009] Aiming at the bottleneck problems of data processing scale limitation and insufficient performance in the process of massive document processing, the present invention provides a document feature extraction method based on distributed mutual information, using the MapReduce distributed computing framework to perform document classification feature words and its The extraction of weights can speed up the speed and scalability of document classification. Through the design of key-value pairs, the weights of feature words in documents can be calculated in parallel while extracting feature words, and the efficiency of document classification can be accelerated.

Images