Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Lightweight multi-modal sentiment analysis method based on multi-element hierarchical deep fusion

A sentiment analysis, multi-element technology, applied in neural learning methods, character and pattern recognition, biological neural network models, etc., can solve problems such as lack of multimodal characteristics

Pending Publication Date: 2021-03-23
HANGZHOU DIANZI UNIV
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this "translation method" is mainly implemented between two modalities and must include "translation" in both directions, making the joint representation have strong local characteristics and lack important global multimodal properties.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Lightweight multi-modal sentiment analysis method based on multi-element hierarchical deep fusion
  • Lightweight multi-modal sentiment analysis method based on multi-element hierarchical deep fusion
  • Lightweight multi-modal sentiment analysis method based on multi-element hierarchical deep fusion

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] like figure 1 As shown, the lightweight multi-modal sentiment analysis method based on multi-element layered deep fusion, the specific steps are as follows:

[0044] Step 1. Data matrix representation of a single modality

[0045] The data of multiple modalities involved in the present invention are represented in the form of X, Y and Z respectively. In the actual use of the framework, three types of modal data are used: text modal, visual modal and acoustic modal to perceive the emotional state of the subjects. We refer to the text mode with a capital letter Y, the visual mode with a capital letter X, and the acoustic mode with a capital letter Z. The data of each modality can be organized in the form of a two-dimensional matrix, namely Among them, t x , t y , t z Respectively represent the number of three modal elements, d x 、d y 、d z represent the characteristic lengths of the corresponding elements, respectively. Taking the text mode as an example, the su...

Embodiment 2

[0097] The difference between this embodiment and Embodiment 1 is that the lightweight multi-modal sentiment analysis method based on multi-element layered deep fusion is applied to four modal data (for example: text modal, visual modal, acoustic modal and EEG data), we can also calculate the matching matrix between the two modalities first, and then use our proposed Ex-KR addition to integrate the matching matrix representing the local fusion of the two modalities to obtain a multimodal global tensor express. In order to illustrate the calculation process, the four modalities participating in the fusion can be denoted by m 1 , m 2 , m 3 , m 4 To represent, the corresponding modal data is organized into a two-dimensional matrix, that is, and In the same way as in equations (1) and (2), the context within a single modality can be captured using the bidirectional long-short-term memory network layer transformation LSTM( ) and the nonlinear feed-forward fully-connected lay...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a lightweight multi-modal sentiment analysis method based on multi-element hierarchical deep fusion. According to the method, the direct correlation among the multi-modal elements is established in a layered manner, and the short-time and long-time dependence among different modals can be captured. In order to avoid reduction of resolution and retain original spatial structure information corresponding to each modal, corresponding attention weights are applied in a broadcast form when multi-modal information interaction is selected and emphasized. Moreover, the invention also provides a new tensor operator, called Ex-KR addition, so that the shared information is utilized to fuse the multi-modal element information to obtain global tensor representation. The methodis an effective supplement for solving the problems that in the current multi-modal emotion recognition field, most methods only pay attention to modeling in a local time sequence multi-modal space, and all complete representation forms participating in modal fusion cannot be clearly learned.

Description

technical field [0001] The invention belongs to the field of multi-modal emotion analysis of cross-fusion of natural language processing, computer vision, and voice signal processing, and specifically relates to a multi-element layered deep fusion technology based on cross-modality. The use of high-level multi-modal element fusion, and then the emotional state of the subject can be analyzed according to the tensor representation of cross-modal and cross-time series. Background technique [0002] With recent advances in machine learning research, the analysis of multimodal time-series data has become an increasingly important research area. Multimodal learning aims to build neural networks that can process and integrate information from multiple modalities, and multimodal sentiment analysis is a research subfield of multimodal learning. When people express their emotions (negative or positive) in real life, various types of information are involved in this communication, inc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/62G06F17/16G06F40/253G06N3/04G06N3/08
CPCG06F17/16G06F40/253G06N3/08G06N3/045G06F18/25
Inventor 李康孔万增金宣妤唐佳佳
Owner HANGZHOU DIANZI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com