High-frame-rate video generation method and device based on data fusion

Abstract

The invention discloses a high-frame-rate video generation method and device based on data fusion, and the method comprises the steps: obtaining a low-frame-rate video and event data from an event camera; every two adjacent image frames in a video are combined, a timestamp set of all expected intermediate frames is calculated, a first event stream and a second event stream from two boundary frames to the expected intermediate frames are intercepted, and the first event stream and the second event stream are input into a preset pulse neural network for forward propagation. Obtaining a first event stream data feature vector and a second event stream data feature vector, splicing the first event stream data feature vector and the second event stream data feature vector with adjacent image frames, and inputting the spliced image frames into a preset multi-modal fusion network for forward propagation to obtain all intermediate frames; a high frame rate video is generated based on all the intermediate frames. Therefore, the technical problem that the quality of the generated image is low due to the fact that only the event stream is used as data input, the initial brightness value of each pixel point is lacked and the stability of brightness estimation only depending on the brightness change record is poor in the prior art is solved.

Description

technical field [0001] The present application relates to the technical field of computer vision and neuromorphic computing, and in particular, to a method and device for generating high frame rate video based on data fusion. Background technique [0002] The event camera is a bio-inspired sensor, and its working principle is very different from that of a traditional camera. Unlike a traditional camera, which captures the absolute light intensity of the scene at a fixed frame rate, the event camera outputs data if and only when the scene light intensity changes. , this output data is called event stream. Compared with traditional cameras, event cameras have the advantages of high dynamic range, high temporal resolution, and no motion blur. [0003] In the related art, the video generation method uses an event camera as a sensor, uses a pure event stream to generate video, converts the event stream into a grid-like tensor representation by stacking, and then uses a deep lear...

AI Technical Summary

Problems solved by technology

[0005] This application provides a high frame rate video generation method and device based on data fusion to solve the problem of only using event streams as data input and lacking the initial brightness value of each pixel in the related art, only relying on brightness change records to estimate brightness Poor stability, resulting in technical issues with lower-quality generated images

Images