Video frame rate up-conversion system and method based on scene depth estimation

Abstract

The invention provides a video frame rate up-conversion system and method based on scene depth estimation. The system comprises a module 1: an optical flow estimation network; a module 2: a scene depth estimation network; a module 3: a context extraction network; a module 4: an nterpolating kernel estimation network; a module 5: an optical flow field mapping of depth perception; a module 6: a self-adaptive interpolation mapping; a module 7: an intermediate frame generation network. According to the method, scene depth estimation is utilized, and a solution for the exposure shielding problem in video frame rate up-conversion is provided; a neural network system capable of being trained end to end by utilizing a deep learning technology, so that the system can be trained and optimized on alarge amount of label-free video data.

Description

technical field [0001] The present invention relates to the technical field of video frame rate conversion, in particular to a video frame rate up-conversion system and method based on scene depth estimation, especially to a video frame rate up-conversion system and method based on deep learning technology. Background technique [0002] Video frame rate up-conversion is a technology for up-converting low frame rate videos such as 24fps and 30fps to high frame rate videos such as 48fps and 60fps. The technology is implemented by inserting a visually reasonable new image between every two frames of the original low frame rate video, thereby improving the viewing experience of the video. [0003] For example, a video frame interpolation processing method disclosed in patent document CN109640117A uses a Lagrange interpolation method to perform frame interpolation processing on a video. Read digital video, obtain video information such as duration, rate, total frame number, etc....

AI Technical Summary

Problems solved by technology

However, a serious defect of such video interpolation methods based on motion estimation is that they are not capable of dealing with exposure and occlusion problems between different moving objects.

Since these methods do not obtain the occlusion relationship between different objects, when interpolating the intermediate frame, when two or more different objects move to the same position, it is impossible to know which object's pixel brightness should be presented

Images