Cross-modal image generation method and device based on audio-tactile signal fusion

Abstract

The invention discloses a cross-modal image generation method based on audio-tactile signal fusion. The method comprises the following steps: 1), selecting a multi-modal data set containing audio data, image data and tactile signals, and dividing the data set into a training set and a test set; 2) designing an audio-tactile signal fused cross-modal image generation model, wherein the model comprises a deep semantic fusion module, a potential space learning module and a cross-modal image generation module; 3) training the model by using a training set to obtain an optimal parameter; and 4) generating a corresponding image in a cross-modal manner based on the trained model by using the tactile signal and the audio data in the test set. The invention also discloses a cross-modal image generation device based on audio-tactile signal fusion, which introduces a strong generative adversarial mechanism, utilizes label information and effectively improves the accuracy and robustness of image generation.

Description

technical field [0001] The invention relates to the technical field of image generation, in particular to a cross-modal image generation method and device based on audio-tactile signal fusion. Background technique [0002] With the rapid development of wireless communication and multimedia technology, people began to pursue a more realistic immersive experience. Touch, as a new sensory dimension, gradually began to integrate and sublimate traditional audio-visual services, forming a cross-modal business. The cross-modal communication of collaborative audio-visual-touch transmission is considered to be a reasonable and efficient communication method to support cross-modal business. However, due to the unreliability of transmission and the difference in communication quality of signals of different modalities, visual signals often face severe loss, and restoration and reconstruction are urgently needed. [0003] Existing image generation work mainly utilizes the inherent inf...

AI Technical Summary

Problems solved by technology

[0006] In addition, the existing generative models are mainly represented by generative confrontation network (GAN) and variational autoencoder (VAE), but GAN has the defects of gradient disappearance and training instability, while the image generated by autoencoder is relatively blurred

At the same time, cross-modal image generation research is mainly based on text-based image generation, and these models can only handle cross-modal generation of a single modality, even if they can be extended, considering that the word-level features of text are relatively tactile and audio-usually in the form of Time-domain sequence representation is more complex, and text-based cross-modal models are not suitable for multi-modal scenarios

Images