Video Description Method for Semantic Reconstruction Based on Temporal Gaussian Mixture Atrous Convolution

Abstract

The invention discloses a video description method for semantic reconstruction based on temporal Gaussian mixture hole convolution. The method of the present invention firstly extracts appearance features and action features from sampled video frames containing text descriptions, and then splicing them and inputting them into a time series Gaussian mixed hole convolution encoder to obtain time series Gaussian features; , get the probability distribution of generated sentences and hidden vectors; then build a semantic reconstruction network and calculate the semantic reconstruction loss; use the stochastic gradient descent algorithm to optimize the model, and obtain the probability distribution of generated sentences through the above steps for new videos, and use the greedy search algorithm to obtain video Descriptive statement. The method of the present invention uses time-series Gaussian mixed hole convolution to model the long-term time-series relationship of videos, and obtains the sentence-level probability distribution difference through the semantic reconstruction network, which can narrow the semantic gap between generated sentences and video content, thereby generating a more accurate description of the video A natural sentence for the content.

Description

technical field [0001] The invention belongs to the field of computer technology, in particular to the field of video description in computer vision, and relates to a semantic reconstruction video description method based on temporal Gaussian mixture hole convolution. Background technique [0002] The rapid development of the Internet has produced a variety of multimedia data resources, such as video, image, audio and text. In recent years, with the popularity of smart terminals such as mobile phones and cameras and the substantial increase in Internet bandwidth, video platforms such as Douyin and Kuaishou have become popular among users. The webcast and self-media industries have risen rapidly, and tens of thousands of videos are uploaded every day. Generation and dissemination, the number of videos has shown explosive growth, which has had a great impact on people's daily life. In the era of big data, how to effectively use massive video is very important. Compared with d...

AI Technical Summary

Problems solved by technology

[0005] The shortcomings of the above methods are mainly manifested in the following aspects: (1) Since LSTM still has the problem of gradient disappearance or gradient explosion, it is difficult to effectively capture long-term video sequence information, which is not conducive to learning the feature representation of video context; (2) natural sentences and Videos belong to two data modalities with different structures. It is difficult to accurately convert the semantics of video content into natural sentences. There is a semantic gap between generated sentences and video content. Existing methods often use the cross-entropy loss function to reduce the gap between generated sentences and videos from the word level. Semantic differences, while ignoring semantic differences at the statement level

Images