Multi-modal joint representation learning method and system based on variational distillation

Abstract

The invention relates to a variational distillation-based multi-modal joint representation learning method, which comprises the steps of deploying a student model, a text teacher model and an image teacher model, arranging multi-modal data including original text modal data and image modal data to obtain text modal input and image modal input with the same input; inputting the original text modal data and the original image modal data into a modal joint representation module to obtain text output and image output, and inputting the original text modal data and the original image modal data into a text teacher model and an image teacher model to obtain text output and image output; and representing the correlation between the text output and the image output corresponding to the student model and the teacher model by using variation mutual information, and carrying out combined distillation training on the text output and the image output by using a distillation loss function, so that the student model obtains the ability of matching the teacher model. The invention provides a variational distillation-based multi-modal joint representation learning method and system. The variational distillation-based multi-modal joint representation learning method and system surpass the existing reference model on different modal data sets.

Description

technical field [0001] The invention relates to the technical field of multimodal distillation, in particular to a multimodal joint representation learning method and system based on variational distillation. Background technique [0002] Large-scale pre-trained models, such as BERT, GPT, and RoBERTa in text modalities, or ResNet, BiT, ViT, etc. in image modalities, have brought revolutionary advances in different modalities. However, as pretrained models get larger, it becomes increasingly challenging to deploy them in resource-poor environments. Therefore, these model compression methods that reduce the size of pre-trained models and preserve most of the performance are also gaining attention. [0003] In the field of text modality, PKD is an earlier exploration, which is very simple and effective, mainly compressing the BERT model in the fine-tuning stage. Subsequently, DistillBERT, TinyBERT, and MobileBERT used KL divergence or L2 loss function to perform task-independ...

AI Technical Summary

Problems solved by technology

[0006] For this reason, the technical problem to be solved by the present invention is to overcome the problems existing in the prior art, and propose a multimodal joint representation learning method and system based on variational distillation, which solves the problem that the prior art lacks a unified modal distillation method , which outperforms existing baseline models on different modality datasets

Images