Self-learning emotion interaction method based on multi-modal recognition

Abstract

The invention discloses a self-learning emotion interaction method based on multi-modal recognition. The method comprises the following steps: respectively collecting voice, face and gesture signals by a non-contact channel; performing feature extraction on the signals to obtain preliminary features of the signals; inputting the features into a bidirectional LSTM layer to obtain single-mode private information and multi-mode interaction information, and obtaining fusion features according to the information; predicting the emotion of a user based on a classification learning algorithm in combination with the multi-modal fusion features and a historical emotional state curve, and selecting an interaction mode; in the interaction mode, giving an interaction response according to the dialoguememory network; and finally, feeding back and optimizing the emotional state curve and the dialogue memory network according to the interaction effect. According to the method, an operator is allowedto input information through multiple channels of a non-contact man-machine interaction interface, multi-modal fusion features are comprehensively considered, and an interaction task is completed incombination with the historical emotional state and the dialogue memory network.

Description

technical field [0001] The invention relates to the technical field of human-computer interaction, in particular to a self-learning emotional interaction method based on multi-modal recognition. Background technique [0002] Intelligent human-computer interaction is an important direction for the development of artificial intelligence. With the development of mobile Internet, higher requirements are put forward for the human-like and natural nature of human-computer interaction. [0003] The current interaction technology is relatively simple, mostly pure text or voice interaction. Some so-called multi-modal interaction methods are only simple addition operations on multi-modal features, which are separate processing of multiple single-modal information. The interaction information between multimodal information causes ambiguity between multiple modalities, and it is impossible to achieve a complete and unambiguous interaction task. At the same time, most of the interaction...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to solve the defect of insufficient interaction ability, and provide a self-learning emotional interaction method based on multi-modal recognition. Comprehensive consideration of fusion features, combined with emotional history state and dialogue memory network, to complete interactive tasks

Images