Bimodal emotion recognition method and system based on multi-source signal and neural network

Abstract

The invention discloses a bimodal emotion recognition method and system based on a multi-source signal and a neural network. The method comprises the following steps: firstly, extracting a respiratory signal from a radar echo signal, extracting a PPG signal from a video face cheek region, extracting a heartbeat signal from the PPG signal, extracting the characteristics of a physiological signal by using a one-dimensional convolutional neural network, secondly, extracting continuous picture frames of eye and mouth regions from a video, and finally extracting the characteristics of the physiological signal by using the continuous picture frames. The emotion recognition method comprises the following steps: extracting features by using a two-dimensional convolutional neural network and a long-short term memory network, then carrying out feature fusion based on a multi-modal compact bilinear pooling algorithm, assigning different weights to each dimension of fused features by using an attention mechanism, and finally carrying out emotion recognition through a classification layer. According to the method, the bimodal sensor is combined with the compact bilinear pooling feature fusion algorithm to realize emotion recognition, and compared with a traditional single-modal sensor and feature splicing type feature fusion, the feature dimension is effectively reduced, dimension explosion is avoided, and meanwhile, the emotion recognition accuracy is improved.

Description

technical field [0001] The invention belongs to the field of radar and multi-sensor fusion, and in particular relates to a dual-modal emotion recognition method and system based on multi-source signals and neural networks. Background technique [0002] Emotion recognition is an important research content in the fields of psychology, cognitive science and computer science. [0003] Emotion recognition first started with facial expressions. Due to its straightforwardness and individual differences, facial expressions have small changes in expressions. In addition, they are affected by factors such as ambient light intensity in actual application scenarios. The degree of discrimination of facial expressions is insufficient, and in some specific scenes, the facial expressions of people are easy to be disguised, which brings a subjective impact on emotion recognition to a certain extent, so that the actual emotion recognition accuracy is often lower than laboratory tests. obtain...

AI Technical Summary

Problems solved by technology

However, most of the existing physiological signal acquisitions are contact-type. This kind of contact-type acquisition method is easy to cause discomfort to the human body and has a certain impact on the stimulation of human emotions.

Moreover, in actual application scenarios, the acquisition of physiological signals is easily affected by various environmental factors, resulting in a large degree of distortion and poor robustness, which makes the emotion recognition based on physiological signals also have certain limitations.

Images