A multi-attention mechanism video description method based on spatio-temporal and channel

Abstract

The invention discloses a multi-attention mechanism video description method based on space-time and channels, which extracts video features from the video through a CNN network, and then calculates the video features and the output of the encoding at the previous moment based on the multi-attention network, thereby obtaining The attention weights of video features in the time domain, space domain, and channel, and then calculate the three sets of weights with the video features again to obtain the fused features, so that we can get more effective video features, and finally encode the fused feature lines output, resulting in a description that is more consistent with the video content.

Description

technical field [0001] The invention belongs to the technical field of optical communication, and more specifically relates to a multi-attention mechanism video description method based on time, space and channels. Background technique [0002] Video description is a research in two fields of computer vision and natural language processing, which has received great attention in recent years. Venugopalan released a video description model based on the "encoding-decoding" framework in 2014. The encoding model in the paper first uses CNN to extract features for a single video frame, and then adopts two encoding models of mean pooling and time sequence encoding. Although the model has been successfully applied to video description, there are still some problems in the video description model: [0003] The first problem is that video features are not effectively utilized. In the paper, the video features are only used in the first decoding, and the video features are not used ...

AI Technical Summary

Problems solved by technology

[0003] The first problem is that there is no effective use of video features

In the paper, the video features are only used in the first decoding, and the video features are not used in the subsequent moments, which leads to the weakening of the impact of video features on word prediction when the time sequence increases, which will reduce the ability of the model to generate sentences

[0004] A direct solution to this problem is to add video features every time, but since the video features are continuous multiple images, if the mean pooling method is still used to send the decoding model every moment, obviously this is still not effective Take advantage of video features

[0006] The second problem is the consistency of visual content features and sentence descriptions

The first problem is that although the method based on temporal attention improves the utilization of video features, but in a deeper way, this method still does not fully model the relationship between video features and sentence descriptions, which leads to The second problem that comes is how to ensure the consistency of visual content feature sentence description

Images