Method for text emotion classification through sparse multinomial logistic regression model under Spark framework

Abstract

The invention provides a method for text emotion classification through a sparse multinomial logistic regression model under a Spark framework. The method comprises the steps that a training sample dataset is stored in an HDFS (Hadoop Distributed File System); a Spark platform reads data from the HDFS to generate an RDD (Resilient Distributed Dataset); the Spark platform divides a preprocessing task of the data into multiple task groups, the RDD storing the read data in each task group is preprocessed, and the preprocessing result is stored into the HDFS; the sparse multinomial logistic regression model is trained, and a sparse multinomial logistic regression classifier is obtained through solving; the sparse multinomial logistic regression classifier is output into the HDFS; the preprocessed data of a to-be-predicted text and the sparse multinomial logistic regression classifier obtained through training are read from the HDFS; and the emotion classification of the to-be-predicted text is acquired. According to the method, an ADMM (Alternating Direction Method of Multipliers) parallel method is used to solve an optimization problem under the Spark computing framework, so that model training is faster, and the method is more suitable for text emotion classification under a big data scene; and classification efficiency and precision are effectively improved.

Description

technical field [0001] The invention relates to the field of distributed machine learning, in particular to a text sentiment classification method based on a sparse multiple logistic regression model under the Spark framework. Background technique [0002] As a key part of machine learning and data mining, classification has a wide range of applications in image recognition, drug development, speech recognition, handwriting recognition, etc. It is a supervised learning problem of identifying which category a new instance belongs to based on a known training set. [0003] With the continuous expansion of data scale, the serial solution method of Sparse Multinomial Logistic Regression (SMLR) problem has been difficult to meet the time and storage space constraints in big data applications. Among many distributed algorithms, the Alternating Direction Method of Multipliers (ADMM) is widely used in the field of distributed machine learning because of its high decomposition and c...

AI Technical Summary

Problems solved by technology

However, the data that people need to deal with in these fields often have a large sample size or a large feature scale.

There are two problems in solving the original SMLR algorithm, the first one is that the computational complexity of the original SMLR algorithm is too high

The second is that the current distributed machine learning field has not done too much parallelization of the SMLR problem, and cannot effectively handle large-scale samples or feature data.

Images