Robot learning control method based on policy gradient

Abstract

The invention discloses a strategy gradient method suitable for robot learning control, which relates to robot learning control technology, including a data acquisition module, which acquires information data during the operation of the robot; a value function approximation module, which uses observed state information and obtained The timely reward is input to obtain an approximate estimation model of the value function; the strategy gradient optimization module parameterizes the robot learning control strategy, and adjusts and optimizes the parameters to make the robot reach an ideal operating state. The action execution module maps the action output by the controller to the action command actually executed by the robot. The method proposed by the present invention can be used for different types of robots, especially multi-degree-of-freedom robots, which have the ability to learn complex actions and solve random strategies, thereby improving the intelligence of the robot, reducing the danger in the learning process, and shortening the robot life. Learning time simplifies controller design difficulty.

Description

technical field [0001] The invention relates to robot learning control technology, in particular to a robot learning control method capable of parameterizing control strategies. Background technique [0002] There are already some technical methods in the field of robot learning control, the most commonly used method is based on value function approximation. In order to obtain the value of the state-action pair, TD (Temporal Difference) learning algorithm and Q-learning algorithm are usually used. While these methods are effective for discrete state-action spaces, many problems arise when solving continuous state-action problems. In a continuous space, in order to achieve a reliable estimation of the value function, it is often necessary to collect a large amount of data in the corresponding space, which is difficult to achieve in the actual complex robot system. And as the robot's degrees of freedom continue to increase, the problem of "dimensional disaster" will appear. ...

AI Technical Summary

Problems solved by technology

However, these methods are more effective for discrete state-action spaces. When solving continuous state-action problems, many problems will arise.

In a continuous space, in order to achieve a reliable estimation of the value function, it is often necessary to collect a large amount of data in the corresponding space, which is difficult to achieve in the actual complex robot system

And as the robot's degrees of freedom continue to increase, there will be a "curse of dimensionality" problem

[0003] In addition, the method based on value function approximation also faces other problems: 1. This method is often used to solve deterministic strategies, and it is very difficult to deal with random strategies, but the best strategy is often random; 2. A random small change in the estimated value of an action will cause the action to not be executed, and this discontinuous change has been identified as a key obstacle to ensure algorithm convergence; 3. This method cannot guarantee that in robot learning The instructions sent to the robot during the process are safe and reliable

Therefore, most of the methods first search for strategies in the simulation environment, which cannot be directly applied in the actual physical environment.

Images