Monocular depth estimation method fusing multi-modal information

Abstract

The invention relates to a monocular depth estimation method fusing multi-modal information. Firstly, a single RGB image is input, features are extracted through a conventional backbone network (such as ResNet), and then the RGB image passes through parallel global pooling operators in the horizontal and vertical directions, cross-channel 1 * 1 convolution, multi-scale cavity convolution and a semantic segmentation prediction module. In this way, two feature maps D and S (depth and semantic modals) with different modalities are obtained. And multiplying the probability distribution vector, belonging to the specific semantic category, of each pixel in the S by the depth representation vector in the D to obtain a multi-modal fusion similarity matrix, and finally performing residual connection with the feature map D to obtain a final output feature map F. In addition, a loss function is redesigned to adapt to and train the deep neural network model. Compared with an existing method, the method can better reflect contours of different object types in a scene and has higher depth estimation precision.

Description

technical field [0001] The invention belongs to the intersection fields of computer vision, robot visual perception and deep learning, and relates to a monocular depth estimation method integrating multimodal information. Background technique [0002] Ordinary cameras can only record the color information of the scene during the imaging process, but cannot record the distance information between the actual object and the camera, that is, in the process of projecting the three-dimensional space to the two-dimensional plane, the depth information is lost. Compared with obtaining the depth of a certain number of points on the surface of the object through various hardware devices such as laser rangefinders, the depth estimation method based on a single image has a wider range of applications because it does not require expensive equipment, instruments and professionals. Methods for introducing semantic cues to improve the performance of depth estimation can be roughly divided i...

AI Technical Summary

Problems solved by technology

[0004] In the present invention, the problem of predicting a depth map from a single RGB image under a supervised learning framework is addressed

Images