Multi-modal emotion analysis method based on multi-dimensional attention fusion network

Abstract

The invention discloses a multi-modal emotion analysis method based on a multi-dimensional attention fusion network. The multi-modal emotion analysis method comprises the steps: extracting voice preprocessing features, video preprocessing features and text preprocessing features from sample data containing multiple modals such as voice, video and text; then, constructing the multi-dimensional attention fusion network for each mode; extracting first-level autocorrelation features and second-level autocorrelation features by using an autocorrelation feature extraction module in the network, thencombining the autocorrelation information of the three modes, and obtaining cross-modal fusion features of the three modes by using a cross-modal fusion module in the network; combining the secondaryautocorrelation features and the cross-modal fusion features to obtain modal multi-dimensional features; and finally, splicing the modal multi-dimensional features, determining emotion scores, and performing emotion analysis. According to the method, feature fusion can be effectively carried out in a non-aligned multi-modal data scene, and multi-modal associated information is fully utilized to carry out emotion analysis.

Description

technical field [0001] The invention belongs to the field of multi-modal emotion computing, and more specifically relates to a multi-modal emotion analysis method based on a multi-dimensional attention fusion network. Background technique [0002] Sentiment analysis has many applications in everyday life. With the development of big data and multimedia technology, the use of multi-modal sentiment analysis technology to analyze the different modes of data voice, video, and text is more conducive to mining the shallow meaning behind the data. For example, in the return visit survey, through the comprehensive analysis of the user's voice, face and speech content, we can know the user's satisfaction with the service or product. [0003] At present, the difficulty of multimodal sentiment analysis lies in how to effectively integrate multimodal information, because the acquisition methods of voice, video and text features are completely different. When describing the same conten...

AI Technical Summary

Problems solved by technology

This method is only a simple result integration of the three modalities, and does not take into account the correlation information between the modalities, and it is easy to cause model overfitting due to information redundancy.

Another approach is based on modal labeling alignment, that is, when performing data labeling, the three modalities are forced to align in the time dimension based on text or phonemes, thus ensuring the corresponding relationship of the three modalities in time. Then use the recurrent neural network, convolutional neural network, attention mechanism, and Seq2Seq framework for modal fusion. This method is expensive to label and is not conducive to the actual production and living environment.

Images