Multi-modal small sample learning method based on significance

Abstract

The invention discloses a multi-modal small sample learning method based on significance, which comprises a multi-modal combination part and a label propagation part, and specifically comprises the following steps: in a multi-modal combination process, firstly, carrying out significance map extraction on sample images of a support set through a pre-trained significance detection network, and separating out foregrounds and backgrounds of the sample images; secondly, acquiring word embedding of support set sample image foreground and background region semantics through a GloVe model as semanticinformation to assist visual information classification; and finally, performing adaptive combination on foreground, background and semantic information of the acquired support set sample image through a modal mixing mechanism to obtain sample feature representation with multi-modal information; in a label propagation process, firstly, subjecting a support set sample and a query set sample after modal combination to graph construction according to a K-nearest neighbor method; and finally, predicting the category of the query set sample without the label through the support set sample with thelabel.

Description

technical field [0001] The invention relates to a small-sample image classification method, in particular to a small-sample learning method based on salient multimodal data processing. Background technique [0002] With the introduction of convolutional neural networks, deep learning has made breakthroughs in image classification, speech recognition, and object detection. However, these studies usually require a large amount of labeled data for training, such as ImageNet, etc., but in reality In daily life, it is very difficult to obtain a large amount of data, such as photos of endangered species, medical images, etc., which seriously limits the applicability of the model in the real world, and the labeling of images will also consume a lot of manpower and financial resources. It can identify a new object with a very small number of samples, use previously learned knowledge to help quickly learn new content, and integrate new concepts into existing concept networks to learn...

AI Technical Summary

Problems solved by technology

[0002] With the introduction of convolutional neural networks, deep learning has made breakthroughs in image classification, speech recognition, and object detection. However, these studies usually require a large amount of labeled data for training, such as ImageNet, etc., but in reality In daily life, it is very difficult to obtain a large amount of data, such as photos of endangered species, medical images, etc., which seriously limits the applicability of the model in the real world, and the labeling of images will also consume a lot of manpower and financial resources. Able to recognize a new object through a very small number of samples, help to quickly learn new content through previously learned knowledge, and integrate new concepts into existing concept networks to learn new concepts

Images