Multi-event natural language description algorithm in video with event relation coding orientation

Abstract

The invention discloses a multi-event natural language description algorithm in a video with event relation coding orientation. The multi-event natural language description algorithm is implemented asfollows: S1, depth features are extracted from a given video sequence by using a three-dimensional convolutional neural network to obtain a plurality of depth feature vectors and form a depth featuresequence; S2, on the basis of the depth feature sequence, a proposed start-stop interval of an event in the video sequence is calculated by using a recurrent neural network as a time sequence analysis method; S3, a to-be-described event needing description is selected in the video sequence, and a corresponding sub-sequence of the to-be-described event in the depth feature sequence is coded againbased on the start-stop interval of the event to obtain a descriptor of the to-be-described event; and S4, the descriptor is decoded by using an attention-model-based LSTM adaptive decoder to obtain anatural language for describing the to-be-described event.

Description

technical field [0001] The invention relates to the technical field of natural language description, in particular to an algorithm for detecting events from videos and describing the events with natural language. Background technique [0002] Visual natural language description (Captioning) is the task of converting visual information into natural language. Usually this task uses codec architecture as a key technology. As the main steps in this process, the quality of the features output by the encoder and the generative model of the decoder have a significant impact on the final natural language results. Visual natural language description has been explored in both video and image. In general, state-of-the-art prevalents use neural network-based computational models to address the actual modeling of architectures. On the image, the convolutional neural network has good results in many visual understanding tasks, and the work in the image description task often uses this ...

AI Technical Summary

Problems solved by technology

However, in the description task of multi-event video, the existing methods have different shortcomings.

The main problems are: 1) For the description of multi-event videos, these methods cannot obtain the relationship between events; 2) For video clips with large differences in length, the effect of using a unified encoder-decoder architecture is not good. ideal

The shortcomings of these two aspects lead to a decline in the accuracy and naturalness of the description language

Images