Bidirectional reconstruction network video description method based on hierarchical attention mechanism

Abstract

The invention discloses a bidirectional reconstruction network video description method based on a hierarchical attention mechanism. The method comprises the following steps: firstly, extracting multi-scale region features of a video frame by using a convolutional neural network as an encoder, and processing the video features by using a hierarchical attention mechanism to obtain video feature dynamic representation; secondly, obtaining probability distribution of vocabulary words by minimizing a cross entropy loss function by taking a long-short-term memory neural network as a decoder and taking video feature dynamic representation and text description thereof as input, and obtaining a generated statement according to the probability distribution; and thirdly, constructing a bidirectionalreconstruction network taking the hidden vector of the decoder as input, minimizing the reconstruction loss, and outputting reconstructed video features, so as to enable the generated text description and video content to have very high semantic similarity. According to the method, multi-scale video features can be effectively extracted to reflect a video space-time structure, irrelevant information interference is reduced, potential video semantic information is mined, and more accurate, more natural and smoother video content description is generated.

Description

technical field [0001] The invention belongs to the technical field of computers, in particular to the technical field of video description in visual computing, and relates to a two-way reconstruction network video description method based on a hierarchical attention mechanism. Background technique [0002] In today's Internet era, smart devices such as live broadcast network platforms, video surveillance, and mobile phones generate a large amount of video data every day, and these data are showing explosive growth. It is time-consuming and labor-intensive to manually describe the content of these videos, and the field of video description was born. The video description method can be mainly used in practical application scenarios such as video title generation, video retrieval, and visually impaired people watching videos. [0003] The video description task is to describe the content of the video with a piece of text. Its goal is not only to capture the complex high-dimen...

AI Technical Summary

Problems solved by technology

[0005] The shortcomings of the above methods are mainly manifested in the following aspects: first, the scale of extracting video frame features is single, and it is difficult to fully represent the rich video information; second, only consider the forward information propagation from video content to text description, there is no Consider the reverse information propagation from text description to video content, so that the semantic similarity between the generated text description and video content is not high; third, the correlation between the video frame region features and the generated text description is not considered, when the described object When it is small, it is easy to introduce irrelevant background information and interfere with the generation of text descriptions

Therefore, these methods are difficult to accurately describe video content and cannot fully capture video semantic information

Images