Video abstraction method based on attention expansion coding and decoding network

Abstract

A video abstraction method based on attention expansion coding and decoding network comprises: regarding a video abstract as a sequence-to-sequence learning process; using time domain correlation between video frames is utilized to obtain a video frame feature sequence from an original video in the SumMe or the TVSum through a pre-training network; taking the video frame feature sequence as inputof an encoder network in an attention expansion coding and decoding network to obtain a semantic information sequence of the video frames, and then obtaining a score corresponding to each video framethrough a multiplicative attention decoding network; the scores of all the video frames forming an abstract sequence; obtaining a semantic information sequence of an abstract sequence through a retrospective encoder, constructing global semantic discrimination loss, introducing a moving average model, learning semantic correlation between the abstract sequence and a video frame feature sequence, obtaining a new abstract sequence retaining important information of an original video, and finally selecting a set final abstract through the new abstract sequence. The robustness of the model is enhanced.

Description

technical field [0001] The invention relates to a video summary. In particular, it relates to a method for video summarization based on attention-expanding codec networks for video processing and indexing. Background technique [0002] With the rapid development of information technology, the explosive growth of video data, redundant and repeated information exists in a large amount of video data, which makes it more difficult for each user to quickly obtain the required information. In this case, video summarization technology came into being. Its goal is to generate a compact and comprehensive summary to provide users with the maximum information of the target video in the shortest time to satisfy people who want to browse videos quickly and accurately. The need for important information and the ability to improve people's access to information. [0003] Research on video summarization is generally divided into two categories: supervised learning and unsupervised learnin...

AI Technical Summary

Problems solved by technology

[0006] The above methods only focus on the local correspondence between the generated summaries and the real annotations, but do not comprehensively consider the global constraints on the generated summaries, and fail to make full use of the semantic information of the video

And there is no clear solution to the outlier processing of the parameter update process, which will also affect the final summary performance

Images