A Method for Multi-Event Natural Language Description in Video Based on Event Relation Coding

Abstract

The present invention discloses a multi-event natural language description algorithm in video oriented to event relationship encoding, which includes the following steps: S1. Using a three-dimensional convolutional neural network to extract depth features from a given video sequence to obtain several depth feature vectors , forming a deep feature sequence; S2, based on the deep feature sequence, using a recurrent neural network as a time series analysis method to calculate the proposed start and end intervals of events in the video sequence; S3, selecting the events to be described in the video sequence Describe the event, and re-encode the corresponding subsequence of the event to be described in the deep feature sequence according to the proposed start and end interval of the event to obtain the descriptor of the event to be described; S4. Using attention-based The LSTM adaptive decoder of the model decodes the descriptor to obtain a natural language used to describe the event to be described.

Description

technical field [0001] The invention relates to the technical field of natural language description, in particular to an algorithm for detecting events from videos and describing the events with natural language. Background technique [0002] Visual natural language description (Captioning) is the task of converting visual information into natural language. Usually this task uses codec architecture as a key technology. As the main steps in this process, the quality of the features output by the encoder and the generative model of the decoder have a significant impact on the final natural language results. Visual natural language description has been explored in both video and image. In general, state-of-the-art prevalents use neural network-based computational models to address the actual modeling of architectures. On the image, the convolutional neural network has good results in many visual understanding tasks, and the work in the image description task often uses this ...

AI Technical Summary

Problems solved by technology

However, in the description task of multi-event video, the existing methods have different shortcomings.

The main problems are: 1) For the description of multi-event videos, these methods cannot obtain the relationship between events; 2) For video clips with large differences in length, the effect of using a unified encoder-decoder architecture is not good. ideal

The shortcomings of these two aspects lead to a decline in the accuracy and naturalness of the description language

Images