Grabbed object recognition method based on tactile vibration signal and visual image fusion

Abstract

The invention relates to the field of robot grabbing cognition, in particular to a grabbed object recognition method based on tactile vibration signal and visual image fusion. The recognition method comprises the following steps of firstly, obtaining a visual color image of a grabbed object through a camera, and then converting the tactile vibration signal obtained during grabbing of the object into the tactile color image according to the numerical value; finally, converting the sizes of the visual color image and the tactile color image to obtain a unified size, and combining the RGB three channels of the visual color image and the RGB three channels of the tactile color image to form six-channel input; and at last, inputting the six-channel image into a convolutional neural network forobject type recognition. According to the recognition method, the types and physical properties of the object can be recognized properly, and the cognitive ability of the robot can be improved.

Description

technical field [0001] The invention relates to the technical fields of robot grasping cognition and multi-modal data fusion, in particular to a grasping object recognition method based on the fusion of tactile vibration signals and visual images. Background technique [0002] At present, the vast majority of robots' cognition process of the outside world is completed through the image information captured by the camera, and more than 70% of the information in the process of human interaction with the outside world also comes from vision. But the sense of touch also plays an important role in human grasping behavior, especially in the identification of some real and fake objects, the sense of touch has an irreplaceable advantage of vision. For example, it is difficult to distinguish between simulated plastic fruits and real fruits at the visual level, but it is easy to draw conclusions through touch. However, the current tactile technology is not yet mature enough. It can o...

AI Technical Summary

Problems solved by technology

[0005] 1. The information of different modalities uses different neural networks to extract features, and finally obtains independent features of different modalities. Since the process of feature extraction is independent of each other, some correlation characteristics between different modal data are ignored. For example, there is a certain relationship between the rough surface of the image and the micro-vibration on the touch. When using the previous method to extract features independently of vision and touch, the surface roughness may not be extracted in the respective feature vectors because the features are not prominent. information or small vibrations, which will lead to the loss of some key information

[0006] 2. Different modalities use different networks to extract features, which leads to a huge amount of network parameters in the whole method

Images