Virtual learning environment natural interaction method based on multimode emotion recognition

Abstract

The invention provides a virtual learning environment natural interaction method based on multimode emotion recognition. The method comprises the steps that expression information, posture information and voice information representing the learning state of a student are acquired, and multimode emotion features based on a color image, deep information, a voice signal and skeleton information are constructed; facial detection, preprocessing and feature extraction are performed on the color image and a depth image, and a support vector machine (SVM) and an AdaBoost method are combined to perform facial expression classification; preprocessing and emotion feature extraction are performed on voice emotion information, and a hidden Markov model is utilized to recognize a voice emotion; regularization processing is performed on the skeleton information to obtain human body posture representation vectors, and a multi-class support vector machine (SVM) is used for performing posture emotion classification; and a quadrature rule fusion algorithm is constructed for recognition results of the three emotions to perform fusion on a decision-making layer, and emotion performance such as the expression, voice and posture of a virtual intelligent body is generated according to the fusion result.

Description

technical field [0001] The invention relates to the fields of emotion recognition, multimodality, human-computer interaction technology, virtual reality, education, etc., and in particular to a natural interaction method for a virtual learning environment based on multimodal emotion recognition. Background technique [0002] The virtual learning environment is an organic combination of virtual reality technology and classroom teaching. By constructing classroom teaching scenes, teaching strategies, teaching content, etc. in a virtual environment, it can break the limitations of time, space, and teaching resources, allowing students to "immerse themselves in the scene" Experience the process of various teaching experiments, strengthen the understanding of various principles, concepts, and methods, and enhance students' learning interest and effect. [0003] The establishment of a virtual learning environment is an integrated and comprehensive technology, involving virtual rea...

AI Technical Summary

Problems solved by technology

In a virtual learning environment, it is difficult to accurately convey the true emotions of students only by using single-modal emotion recognition information such as human expressions, voice, or gestures.

However, there is still a lack of effective research at home and abroad on how to build a multi-modal emotion recognition method based on facial expressions, voice, and gestures and its natural interaction in a virtual learning environment, and there is no patent application for this aspect.

Images