Multi-sensor real-time fusion method for mobile robot remote takeover scene

Abstract

The invention discloses a multi-sensor real-time fusion method for a mobile robot remote takeover scene. According to the multi-sensor real-time fusion method, the laser radar and the multiple camerasare used for collecting data, the defects of a single sensor are overcome, and the laser radar and the multiple cameras are fused to generate a video stream with depth information, so that an operator knows distance information of surrounding objects. Aiming at the problems of splicing seams and ghosting in a linear fusion method, the invention provides a linear fusion method with a threshold value, so that the splicing seams and the ghosting are eliminated while the real-time performance is ensured, and the splicing effect is improved. In combination with the actual situation of an intelligent mobile robot, real-time fusion of multiple paths of high-definition video streams with depth information is completed through offline parameter updating and adoption of a specific parallelization method, the time delay of data transmission is reduced while redundant information of multiple sensors is removed, and the real-time fusion method has actual application value.

Description

technical field [0001] The invention belongs to the technical field of multi-sensor fusion, and in particular relates to a multi-sensor real-time fusion method for remote takeover scenarios of mobile robots. Background technique [0002] With the development of science and technology in recent years, intelligent mobile robots have become more and more widely used in people's lives. They can complete corresponding tasks in specific scenarios and partially replace people's labor, such as AGV sorting robots, sweeping robots and automatic robots. Driving a van and more. However, at this stage, the intelligent mobile robot has not yet reached the level of completely autonomous problem solving. When the intelligent mobile robot is in a complex scene or an unknown scene occurs, its own task decision-making system is not enough to deal with new problems. Without remote assistance, it will be uncontrollable situation, and even cause accidents and losses. The abnormality of the func...

AI Technical Summary

Problems solved by technology

[0003] At this stage, the remote takeover system usually transmits multiple video stream data directly to the client for display, but this method has the following problems: 1. The sensor uses a single, only the camera is used to collect environmental information, and important distance information is lost; 2. .The amount of transmitted data is large, and there is a lot of redundant data, which has high requirements for network bandwidth and transmission speed; 3. The multi-channel video data display is not synchronized; 4. The multi-channel video stream is directly displayed, which is inconvenient for the operator to make decisions

[0004] The Chinese patent with the publication number CN108495060A proposes a real-time splicing method for high-definition video. After the method is initialized, each step of the video splicing stage is accelerated in a parallel manner, which can make the splicing frame rate of the two-way video stream It is greater than or equal to 30fps, and solves problems such as ghosting and blurring at the same time. However, this method only splices two-way video streams, and cannot handle the splicing of three or more video streams. It still cannot meet the needs of modern intelligent mobile robot application scenarios.

[0005] The Chinese patent with the publication number CN103856727B proposes a real-time splicing processing method for multi-channel video streams. This method speeds up the execution time of the system by updating offline parameters and achieves the effect of real-time splicing of multi-camera video streams. However, this method has Many problems: 1. The image resolution used is 704×576. For the remote takeover system, the resolution is low, which is not conducive to recognition and decision-making; 2. The multi-image fusion of this method uses pairwise fusion 3. This method is only suitable for relatively static supervision scenarios, and no corresponding processing is performed for dynamic objects with seams

[0006] The Chinese patent with the publication number CN106526605B proposes a data fusion method of lidar and depth camera, which uses the first pixel point calculation of the depth image and the angle and distance of the lidar point cloud to obtain two sets of different polar coordinates series, and sequence fusion according to the angle, this method can expand the detection range and improve the information accuracy, but there are several problems in this method: 1. This method uses the fusion of a single binocular camera and lidar, binocular camera The cost is high, and at the same time, the visual information obtained by a single binocular camera is less; 2. This fusion method cannot meet the real-time requirements, and is not suitable for remote takeover scenarios with intelligent mobile robots

Images