Multi-modal sentiment classification method based on heterogeneous fusion network

Abstract

The invention discloses a multi-modal sentiment classification method based on a heterogeneous fusion network, and belongs to the technical field of opinion mining and sentiment analysis. The method comprises the steps of 1) preprocessing video data; 2) constructing a text feature vector and identifying a text emotion category; 3) constructing a picture feature vector and identifying a picture emotion category; 4) constructing an audio feature vector and identifying an audio emotion category; 5) constructing a multi-modal global feature vector and identifying a multi-modal global emotion category; 6) constructing a multi-modal local feature vector and identifying a multi-modal local emotion category; and 7) adopting a voting strategy to obtain a final sentiment classification result. The heterogeneous fusion network adopts two fusion forms of intra-modal fusion and inter-modal fusion, two fusion angles of macroscopic fusion and microscopic fusion, and two fusion strategies of feature layer fusion and decision-making layer fusion. According to the method, the implied associated information among the multi-modal data can be deeply mined, and mutual complementation and fusion among the multi-modal data are realized, so that the accuracy of multi-modal sentiment classification is improved.

Description

technical field [0001] The invention relates to a multimodal emotion classification method based on a heterogeneous fusion network, and belongs to the technical field of opinion mining and emotion analysis. Background technique [0002] Multimodal sentiment classification is an important research topic in the fields of social computing and big data mining. Multimodal sentiment classification refers to identifying the emotional polarity of network users based on various modal data such as text, pictures, and videos of network user comments. Emotional polarity includes both negative and non-negative categories. [0003] Multimodal sentiment classification methods include multimodal sentiment classification methods based on feature layer fusion and multimodal sentiment classification methods based on decision layer fusion. [0004] The multi-modal emotion classification method based on feature layer fusion first constructs the feature vectors of various modal data, and then f...

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to solve the problem that the existing multimodal emotion classification method has a single fusion method and it is difficult to mine the hidden correlation features of multimodal data, resulting in low accuracy of multimodal emotion classification. A multi-modal emotion classification method based on quality fusion network. This method extracts three modal data of text, picture and audio from videos posted by network users, and uses a heterogeneous fusion network model based on deep learning to recognize text, picture and audio respectively. and the sentiment category of the overall video

Images