Bimodal fusion emotion recognition method based on video image and voice

Abstract

The invention discloses a bimodal fusion emotion recognition method based on video images and voice. The method is composed of a camera, a microphone and an emotion recognition unit, wherein the emotion recognition unit is composed of a video image mode and a voice mode. The training process of the bimodal model is as follows: inputting an image training data set into a convolutional neural network model for training to obtain a video image modal model; and inputting the voice training data set into the long-short-term memory neural network model for training to obtain a voice modal model. The camera collects video images and sends the video images to the emotion recognition unit, and the facial expression features are analyzed to obtain recognition results; the microphone collects voice data and sends the voice data to the emotion recognition unit, and analyzes voice emotion features to obtain a recognition result; and fusing the identification results of the two modals at a decision-making layer according to a weight criterion and outputting the fused identification results. The recognition method can improve the accuracy of emotion recognition and realize real-time detection.

Description

technical field [0001] The invention relates to the field of emotion recognition, in particular to a dual-modal fusion emotion recognition method based on video image and voice. Background technique [0002] With the rapid development of artificial intelligence technology, people hope that there will be a more vivid interaction between AI and users, so as to bring users a better user experience. In terms of engineering application value, emotion recognition is a research topic involving machine vision, medicine, psychology and many other fields. The research can not only promote the progress of other interdisciplinary subjects, but also bring huge commercial value and practical significance to the society. . According to different analysis information, emotion recognition technology can be divided into two categories at present; one is based on physiological signals, such as EEG, ECG, etc.; the other is based on the analysis of emotional behavior, such as facial expressions...

AI Technical Summary

Problems solved by technology

Due to the difficulty of collecting physiological parameters, this analysis method is rarely used; the accuracy of body movement recognition is low, and it usually appears as an auxiliary recognition method; the difficulty of collecting voice and facial expressions is not high, but the recognition effect is good, which is practical application The most extensive emotion recognition method

[0003] The currently used emotion recognition methods mainly have the following shortcomings: many single-modal recognition methods are used, and the accuracy of single-modal recognition is difficult to continue to improve; manual feature extraction is required, and real-time processing cannot be achieved; fusion of different modalities mostly uses feature fusion technology , resulting in higher feature dimensions and unable to achieve real-time processing

These shortcomings make it difficult to improve the recognition accuracy to a higher level, and it is also difficult to realize real-time emotion recognition, so it is necessary to improve

Images