Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism and experimental evaluation method

A multi-task learning and emotion recognition technology, applied in the field of human-computer interaction, can solve the problems of multi-modal emotion recognition process efficiency and low accuracy, and achieve the effect of improving calculation efficiency and accurate recognition

Inactive Publication Date: 2021-09-21
HARBIN UNIV OF SCI & TECH
View PDF3 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem of low efficiency and low accuracy of the multi-modal emotion recognition process that introduces the multi-modal fusion mechanism in the prior art, the present invention proposes a multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism And experimental evaluation method; Technical scheme of the present invention is as follows:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism and experimental evaluation method
  • Multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism and experimental evaluation method
  • Multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism and experimental evaluation method

Examples

Experimental program
Comparison scheme
Effect test

specific Embodiment approach 1

[0045] Specific implementation mode 1: In this embodiment, by introducing multi-task learning and multi-modal emotion analysis, the multi-modal fusion emotion recognition process based on multi-task learning and attention mechanism is completed. The specific process is as follows:

[0046] First of all, multi-task learning (MTL) is a machine learning method that combines multiple tasks to learn at the same time. During the learning process, it helps each task to learn by using useful information between multiple related tasks. Obtain more accurate learning performance and enhance the representation and generalization capabilities of the model. The core of multi-task learning lies in the sharing of knowledge between tasks, so the challenge is the sharing mechanism between tasks. In deep learning, there are two sharing strategies, namely the hard sharing mechanism and the soft sharing mechanism of parameters, such as figure 2 As shown, the hard sharing mechanism shares hidden l...

specific Embodiment approach 2

[0096] Embodiment 2: According to the multi-modal fusion emotion recognition system and method based on multi-task learning and attention mechanism proposed in Embodiment 1, this embodiment proposes a multi-modal fusion emotion based on multi-task learning and attention mechanism Identify experimental analysis, process:

[0097] First, use the CMU-MOSI and CMU_MOSEI data sets for experimental simulation;

[0098] The content also comes from a certain website, including 22856 video clips and corresponding emotional tags. The range of emotional tags is [-3,+3]. for negative. The statistical information of the experimental data set is shown in Table 1:

[0099] Table 1 MOSI and MOSEI dataset information

[0100]

[0101] Secondly, regarding the experimental settings, it is written in python3.7.8 language and uses the deep learning framework pytorch1.4.0 to implement the neural network structure. The experimental environment is Windows10 system, and the experimental hardwar...

specific Embodiment approach 3

[0144] Specific Embodiment Three: According to the system and method provided in Embodiment 1 or 2, this embodiment divides the functional modules according to the block diagram shown in the accompanying drawings. For example, each functional module can be divided corresponding to each function, or two or More than two functions are integrated into one processing module; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present invention is schematic, and is only a logical function division, and there may be another division manner in actual implementation.

[0145] Specifically, the system includes a processor, a memory, a bus, and a communication device; the memory is used to store computer-executable instructions, the processor is connected to the memory through the bus, and the processor executes the instructions stored in the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a multi-modal fusion emotion recognition system and method based on multi-task learning and an attention mechanism and an experimental evaluation method, and aims to solve the problems that in the prior art, a multi-modal emotion recognition process without introducing a multi-modal fusion mechanism is low in efficiency and accuracy. The invention belongs to the field of human-computer interaction, and provides a multi-modal fusion emotion recognition method based on combination of multi-task learning and an attention mechanism, and compared with single-modal emotion recognition work, the multi-modal fusion emotion recognition method based on combination of multi-task learning and the attention mechanism is wider in application. Multi-task learning is utilized to introduce an auxiliary task, so that the emotion representation of each mode can be more efficiently learned, and an interactive attention mechanism can enable the emotion representations among the modes to mutually learn and complement each other, so that the recognition accuracy of the multi-mode emotion is improved; experiments are carried out on the multi-modal data sets CMU-MOSI and CMU-MOSEI, the accuracy and the F1 value are both improved, and meanwhile the accuracy and efficiency of emotion information recognition are improved.

Description

technical field [0001] The present invention is a multimodal fusion emotion recognition system and method, in particular to a multimodal fusion emotion recognition system, method and experimental evaluation method based on multi-task learning and attention mechanism, belonging to the field of human-computer interaction, Background technique [0002] Emotion recognition is a basic task in natural language processing, which aims to use computers to analyze and process signals collected from sensors, so as to obtain the emotional state of the other party. For a long time, research on emotion recognition tasks has been focused on the research and processing of text. With the popularity of emerging social media, more and more people like to express their views and views on some things or hot events by sharing videos on the platform. Comments, the data form of social media is no longer limited to a single text form, but more data including text, sound, and pictures. These multi-mo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06K9/62G06N20/20
CPCG06N20/20G06F18/2415G06F18/25
Inventor 王庆岩王吉予殷楠楠谢金宝梁欣涛
Owner HARBIN UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com