Multi-modal emotion recognition method based on acoustic and text features

Abstract

The invention provides a multi-modal emotion recognition method based on acoustic and text features, which is suitable for extracting speech and text emotion features. The method comprises the following steps: extracting emotional shallow-layer features of input voice by using OpenSMILE, and fusing the emotional shallow-layer features with deep-layer features obtained by learning the shallow-layer features through a Transform network to generate multi-level acoustic features; performing forced alignment on the voice and the transcriptional text to obtain pause information, encoding the speaking pause information in the voice, adding the encoded speaking pause information to the transcriptional text, sending the encoded speaking pause information to a hierarchical dense connection DC-BERT model to obtain text features, and fusing the text features with acoustic features; according to the method, effective context information is acquired by utilizing priori knowledge through a BiLSTM network, a part which highlights emotion information in features is extracted through an attention mechanism to avoid information redundancy, a global average pooling layer is added behind the attention mechanism to replace a traditionally used full connection layer, and finally the information is sent to a softmax layer for emotion classification. The method has the advantages of simple steps, accurate identification and wide practical value.

Description

technical field [0001] The invention relates to a multimodal emotion recognition method based on acoustic and text features, which is suitable for the extraction of voice and text emotion features, and belongs to the technical fields of artificial intelligence and voice emotion recognition. Background technique [0002] With the development of technology, speech emotion recognition and natural language processing have made great progress, but humans still cannot communicate with machines naturally. Therefore, it is crucial to build a system capable of detecting emotion in human-computer interaction. But it is still a challenging task due to the variability and complexity of human emotions. [0003] Traditional emotion recognition is mainly aimed at a single modality, such as text, voice, image, etc., and there are certain limitations in recognition performance. For example, in the early speech emotion recognition tasks, researchers mainly used the acoustic features and som...

AI Technical Summary

Problems solved by technology

Although BERT can be used to obtain contextual word embeddings to represent the information contained in the transcribed text, it does not take into account the mismatch between the complex network structure of BERT and the insufficient amount of data in the emotional corpus

Although BERT can be used to generate representations of text information, it cannot make up for the lack of potential emotional information that the transcribed text itself ignores.

Images