Multi-modal emotion recognition method and system

Abstract

The invention provides a multi-modal emotion recognition method and system, and the method comprises the steps: employing a novel and robust endpoint detection algorithm for voice components in an emotion video sample, and employing a prediction residual conditional entropy parameter generated in a sample reconstruction process under a compressed sensing theory, calculating a residual conditional entropy difference value in the algorithm iteration process of an orthogonal matching pursuit (OMP) algorithm, completing endpoint detection according to an empirical threshold value, and completing feature learning of emotional speech of a voiced segment based on a reconstructed sample; and screening the facial expression images according to the endpoint detection result of the emotional voice, and only reserving the facial expression images with the active emotional voice in the same time period, so that the purposes of enhancing the emotional separability of the facial expression data set and reducing the redundancy are achieved; the emotion speech features and the facial expression features are fused, an effective multi-modal emotion recognition model is trained, and the purpose of effective multi-modal emotion recognition is achieved.

Description

technical field [0001] The disclosure belongs to the technical field of emotion recognition, and in particular relates to a multi-modal emotion recognition method and system. Background technique [0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art. [0003] Emotion recognition is a research hotspot in the field of affective computing. The emotional signals of the two modalities of emotional speech and facial expression images have the characteristics of convenient collection and large amount of emotional information. They are important and highly relevant in the research of emotion recognition. Two types of data sources. [0004] The inventors found that the emotional voice data and facial expression images currently used in the field of multi-emotion recognition are usually obtained by storing the emotional voice components and image components in emotional video samples...

AI Technical Summary

Problems solved by technology

Under the slow-changing characteristics of facial expressions, this non-discrimination and non-filtering method of facial expression image collection ignores the connection between different modal emotional expressions, and does not consider the emotional differentiation of facial expression images, resulting in the collected The emotional recognizability of facial expression images is low and the redundancy is large, which leads to the poor performance of the model trained and learned in the follow-up emotion recognition research.

Images