Robot sequence task learning method based on visual simulation

Abstract

The invention provides a robot sequence task learning method based on visual simulation. The robot sequence task learning method based on the visual simulation is used for guiding a robot to simulateand execute human actions from a video containing the human actions. The robot sequence task learning method comprises the following steps of (1) identifying object types and masks by using a region-based mask convolutional neural network according to an input image; (2) calculating actual plane physical coordinates (x, y) of objects according to the masks; (3) identifying atomic actions in a target video; (4) converting an atomic action sequence and the identified object types into a one-dimensional vector; (5) inputting the one-dimensional vector into a task planner, and outputting a task description vector capable of guiding the robot; and (6) controlling the robot to simulate a sequence task in the target video by combining the task description vector and the object coordinates. According to the robot sequence task learning method based on the visual simulation, the video and the image serve as input, the objects are recognized, a task sequence is deduced, the robot is guided to simulate the target video, and meanwhile, the generalization performance is high, so that the simulation of the task can still be completed under different environments or object types.

Description

Technical field [0001] The invention relates to a method for robots to imitate humans to complete various tasks based on visual sensors and video input, belonging to the field of robot control, deep learning and imitation learning, and is mainly used to teach robots to imitate humans to complete handling, cleaning, classification or Place objects and other application scenarios. Background technique [0002] In recent years, with the rapid development of the field of artificial intelligence and intelligent robots, intelligent products such as robots occupy an increasingly important role in human life, and behind the intelligence are complex algorithms and control methods. In the era of "Industrial Revolution 4.0" and "Made in China 2025", research in the field of robots, robotic arms, and artificial intelligence has increasingly become the mainstream of research and innovation in universities, companies, and major laboratories in various countries. Using artificial intelligence ...

AI Technical Summary

Problems solved by technology

However, the model trained by this method is difficult to apply to new unfamiliar scenes, and its generalization ability is limited.

Images