Hand posture estimation system and method based on RGBD fusion network

Abstract

The invention provides a hand posture estimation system and method based on an RGBD fusion network. The system comprises a global depth feature extraction module, a residual module, a multi-mode feature fusion module and a branch parallel interference elimination module. The global depth feature extraction module adopts two parallel paths of cross-fused residual networks, wherein the upper path isa high-resolution feature map, the lower path is a low-resolution feature map, carries out the multi-scale feature fusion by the cross-fusing multi-resolution information, and finally predicts the network output in a high-resolution feature map. An input part of the system is divided into a depth image processing branch and an RGB color image processing branch, the features extracted by the two branches are subjected to multi-mode fusion to form the global features, the global features are sent to a branch parallel interference elimination module to perform feature extraction of hand branches, and the reinforced hand branch features are obtained and used for final joint position prediction. According to the method, the hand posture estimation with higher accuracy is achieved mainly through the information synthesis of the color images and the depth images.

Description

technical field [0001] The present invention relates to the technical fields of computer vision and deep learning, in particular, to a hand posture estimation system and method based on RGBD fusion network. Background technique [0002] Vision-based 3D hand pose estimation is a hot research issue in computer vision, virtual reality, robotics and other fields, and there have been many research results. However, so far, the estimation of human hand pose based on visual information is still a problem that has not been perfectly solved. The highly flexible changes of finger joints, the high similarity between different fingers, the occlusion of each other when fingers move, and the objects The occlusion of fingers brings great challenges to hand pose estimation and hinders the development of intelligence and full automation in corresponding application fields. Therefore, it is of great significance to develop a hand pose estimation method with better performance. [0003] Rese...

AI Technical Summary

Problems solved by technology

Moon et al. converted depth data into 3D voxel representation, and used a more complex 3D CNN for learning, achieving higher prediction accuracy. However, the conversion process for constructing 3D data is complex and time-consuming.

[0005] The above research on hand posture detection limits the environment to empty-handed movements and a third perspective facing the camera, reducing occlusion as much as possible, which brings great benefits to joint detection. Convenience, but when the scene is not a third perspective or when interacting with objects, there is strong occlusion. To solve this problem, Goudie uses real depth images, and realizes hand object segmentation network and joint estimation network through the two-stage operation of the hand object segmentation network. Hand joint estimation when operating a small ball, however this method is limited to known objects, requires additional segmentation annotations, and the generalization ability to other objects is unknown

Images