Video prediction method based on space-time propagation level codec

Abstract

The invention discloses a video prediction method based on a space-time propagation level codec. The method comprises the following steps: firstly, sampling a given original video to obtain a frame sequence, and inputting the frame sequence into a low-layer visual memory coder to obtain low-layer visual coding features and low-layer memory state features; then, extracting space-time coding features from low-layer visual coding representation through a space-time propagation module, and extracting high-layer semantic features through a high-layer semantic coder; and carrying out information fusion on the obtained low-layer visual coding features, low-layer memory state features and high-layer semantic coding features through a hierarchical feature decoder, and outputting a predicted video frame. According to the method, the low-level visual memory features and the high-level semantic features can be fused, the low-level visual information is propagated in the time sequence direction through the space-time propagation module, the problem of video frame blurring can be solved to a certain extent by utilizing the priori knowledge of the first frame of the video, and the definition and visual quality of the predicted video frame are improved as a whole.

Description

technical field [0001] The invention belongs to the field of computer vision, in particular to the technical field of video prediction in video perception, and relates to a video prediction method based on a time-space propagation layer codec. Background technique [0002] In the era of Internet +, thousands of videos are generated on various terminal devices every day. Video perception has attracted extensive attention from academia and industry, and video prediction is one of the challenges and has high application value. visual task. This task aims to generate video segments at subsequent moments given a partial video segment. Video prediction has a wide range of applications in practical scenarios such as radar weather map prediction, traffic flow prediction, robot object interaction prediction, and driverless driving. For example, in an unmanned driving environment, traditional radar ranging can only judge close-range vehicle interaction, while video prediction can us...

AI Technical Summary

Problems solved by technology

[0004] The shortcomings of the above methods are mainly manifested in the following aspects: First, the method of stacking multiple ConvLSTMs does not integrate the learned low-level visual features and high-level semantic features well, and does not make full use of the learned features, resulting in prediction results There are still ambiguities; second, the video prediction method based on the dual-stream architecture does not effectively propagate the low-level visual features of the video in time sequence, resulting in inaccurate prediction of the position of the object outline in the video; third, the self-recursive method passes the The video frames generated by the network are sent to the network again to realize the video prediction of multiple video frames, but there are errors in the video frames generated by the network, and they will continue to accumulate due to this form, resulting in blurring in the late stage of multi-video frame prediction

Therefore, in order to alleviate the problems of insufficient fusion of different levels of features, inaccurate contour positions of video objects, and blurring in the later stage of prediction, there is an urgent need for a method that can fuse different levels of features, provide more accurate contour position information, and alleviate error accumulation. thereby improving the accuracy of video predictions

Images