Feature fusion and decision fusion mixed multi-modal emotion recognition method

Abstract

The invention discloses a feature fusion and decision fusion mixed multi-mode emotion recognition method, and belongs to the field of mode recognition and emotion recognition. The implementation method comprises the following steps: 1, constructing an image emotion recognition network by using a convolutional neural network framework, and obtaining image features and image emotion states; 2, constructing a text emotion recognition network by using a recurrent neural network framework, and obtaining text features and a text emotion state; and 3, constructing a multi-modal information fusion emotion recognition network, constructing a main classifier for fusing the image emotion state and the text emotion state and obtaining main emotion classification, constructing an auxiliary classifier for fusing the image features and the text features and obtaining auxiliary emotion classification, and fusing the main emotion classification and the auxiliary emotion classification to obtain final emotion classification. According to the method, information complementation among multi-modal information is utilized, the problem of low emotion recognition accuracy caused by factors such as information fuzziness or missing of single-modal information is avoided, and a new thought is provided for multi-modal data fusion and emotion recognition.

Description

technical field [0001] The present invention relates to the fields of data fusion, neural network, emotion recognition, etc., and more specifically, relates to a multi-modal information fusion emotion recognition method based on hybrid fusion. Background technique [0002] Human beings express emotional information through multiple modalities such as facial expressions, gestures, voices, and language. Emotional behavior is an important indicator reflecting human satisfaction. With the development of artificial intelligence technology, emotion recognition is an important means to achieve good human-computer interaction. Emotion recognition is to obtain the mapping relationship between the external appearance characteristics of emotion and the internal emotional state by extracting the characteristics of emotional signals, so as to identify the internal emotional type of the recognized object. Emotion recognition has very broad application prospects in machine services, healt...

AI Technical Summary

Problems solved by technology

Data-level fusion is only applicable to signals of similar types, and cannot handle signals with large differences, such as image and sound signals, etc.

Images