A Self-learning Emotional Interaction Method Based on Multimodal Recognition

Abstract

The invention discloses a self-learning emotion interaction method based on multi-modal recognition, the steps are as follows: the non-contact channel collects voice, face and gesture signals respectively; the feature extraction is performed on the signal to obtain the preliminary feature of the signal; the feature is input to the two-way The LSTM layer obtains single-modal private information and multi-modal interactive information, and obtains fusion features based on these information; based on the classification learning algorithm, combines multi-modal fusion features and historical emotional state curves to predict user emotions and select interactive modes; Next, an interactive response is given according to the dialogue memory network; finally, the emotional state curve and the dialogue memory network are fed back and optimized according to the interaction effect. The invention allows the operator to input information through multiple channels through the non-contact human-computer interaction interface, comprehensively considers the fusion characteristics of multiple modes, combines the emotional history state and the dialogue memory network, and completes the interactive task.

Description

technical field [0001] The invention relates to the technical field of human-computer interaction, in particular to a self-learning emotional interaction method based on multimodal recognition. Background technique [0002] Intelligent human-computer interaction is an important direction for the development of artificial intelligence. With the development of mobile Internet, higher requirements are put forward for the human-like and natural nature of human-computer interaction. [0003] The current interaction technology is relatively simple, mostly pure text or voice interaction. Some so-called multi-modal interaction methods are only simple addition operations on multi-modal features, which are separate processing of multiple single-modal information. The interaction information between multimodal information causes ambiguity between multiple modalities, and it is impossible to achieve a complete and unambiguous interaction task. At the same time, most of the interaction ...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to solve the defect of insufficient interaction ability, and provide a self-learning emotional interaction method based on multi-modal recognition. Comprehensive consideration of fusion features, combined with emotional history state and dialogue memory network, to complete interactive tasks

Images