Big data text classifying method based on cloud computing

Abstract

The invention discloses a big data text classifying method based on cloud computing. The method comprises the following steps: respectively pre-processing training texts with class labels and without class labels to obtain corresponding training data sets; respectively carrying out feature selection on the training data sets to obtain corresponding dimensionally reduced training data sets; respectively calculating the dimensionally reduced training data sets according to a TFIDF weighted model, and respectively converting the training data sets to corresponding one-dimensional vectors; calculating the one-dimensional vectors with class labels according to a bayesian algorithm to obtain the prior probability of each class and the prior probability that each entry belongs to each class, and initializing the parameters of a bayesian classifier; utilizing an EM algorithm to optimize the parameters of the bayesian classifier so as to obtain a classifying model; carrying out text classification on the to-be-classified texts through the classifying model. Through combining a traditional naive bayesian classifying technology and Hadoop and EM algorithms, calculating speed limitation and training data limitation problems in actual application are improved, and the efficiency and the accuracy of the classifier are improved.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a cloud computing-based big data text classification method. Background technique [0002] The Naive Bayesian classification algorithm has become the focus of simplicity and efficiency. Compared with other classification algorithms, it has lower time complexity and higher efficiency. It is widely used in various classification tasks and is currently one of the most widely used text classification methods. [0003] In recent years, with the development of information technology, the task of document classification has shown new characteristics, which are mainly reflected in the following two aspects: First, the data is growing explosively, and a large amount of information is represented by unstructured or semi-structured files such as text. Second, (2) A good classifier needs a large number of labeled samples for training, but the information provided by the labeled samples may be subj...

AI Technical Summary

Problems solved by technology

[0008] The present invention provides a large data text classification method based on cloud computing to solve the problems of calculation speed limitation and training data limitation in practical applications

Images