Real-time large-scene three-dimensional semantic modeling method

Abstract

The invention provides a real-time large-scene three-dimensional semantic modeling method. The method comprises the following steps of S1, constructing a three-dimensional geometric model through an RGB image and a depth image obtained by scanning a scene through a sensor; s2, inputting the three-dimensional geometric model into a three-dimensional convolutional neural network to complete semanticsegmentation; s3, integrating semantic tags output by the three-dimensional convolutional neural network into the three-dimensional geometric model to complete semantic modeling; wherein the step ofconstructing the three-dimensional geometric model and the step of semantic segmentation are combined in a multi-thread mode and are carried out at the same time. Joint real-time three-dimensional geometric reconstruction and semantic reconstruction are realized; by adopting the sparse convolutional neural network and accelerating the calculation of the convolutional network, the performance of real-time operation can be achieved; according to the method, two-dimensional convolution in UNet is replaced by three-dimensional convolution, and meanwhile, the depth of the UNet network is increased,so that the capacity of the convolutional network is larger, and the semantic segmentation accuracy is improved.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to a method for real-time three-dimensional semantic modeling of a large scene. Background technique [0002] 3D reconstruction of large scenes is an important problem in the field of computer vision. Technologies such as autonomous driving, indoor robot navigation, AR, and VR all rely on large-scene 3D reconstruction technology. Its goal is to use portable devices to dynamically scan the scene, and through algorithm processing, it can easily generate a 3D model of the entire scene. [0003] In addition to obtaining the geometric information of the scene, in practical applications, another information we are interested in is semantic information, that is, we also want to know which parts the scene can be divided into and what objects they are. This is the goal of 3D semantic segmentation. Good semantic information is very important to realize the complex interaction of in...

AI Technical Summary

Problems solved by technology

Due to the limitations of 2D semantic segmentation on 3D space understanding, semantic SLAM systems based on 2D semantic segmentation are often less effective

[0013] The existing technology does not solve the problem of 3D semantic reconstruction

Images