Continuous reinforcement learning system and method based on stochastic differential equation

Abstract

The invention discloses a continuous reinforcement learning system and method based on a stochastic differential equation. The system comprises an action strategy generator APG, an environment state estimator ESE, a value estimator VE, a memory storage module MS and an external environment EE. The method comprises the following specific steps: an action strategy generator APG, an environment stateestimator ESE and a value estimator VE are initialized; an action strategy generator APG calculates an output action value increment delta ak; the external environment EE outputs a next action valueak + 1, a next environment state value sk + 1 and a current reward value Rk and stores the values in a memory storage module MS; the environment state estimator ESE updates the environment state parameter set theta p and predicts a future environment state estimated value s'k; the VE optimizer updates the Q function network and predicts a future reward estimation value R'k; and the APG optimizer updates the action value parameter set theta v. The method is based on a stochastic differential equation as a basic model, continuity of action control can be achieved, variance of the training process can be controlled, and actions can be selected by predicting changes of the environment so as to achieve better environment interaction.

Description

technical field [0001] The invention relates to the fields of reinforcement learning and stochastic process, and in particular relates to a reinforcement learning method for continuous systems. Background technique [0002] Deep reinforcement learning is an end-to-end learning system that combines the perception ability of deep learning with the decision-making ability of reinforcement learning, has strong versatility, and realizes direct control from raw input to output. Reinforcement learning has become a very important unsupervised learning method, which enables the agent to judge the current environment state through the value function in the interaction with the environment, and thus make corresponding actions to obtain better rewards. At present, reinforcement learning algorithms mainly focus on discrete action policy sets, while classic continuous reinforcement learning methods such as DDPG and A3C can be used for continuous motion control in applications such as robo...

AI Technical Summary

Problems solved by technology

[0003] However, most of the current continuous reinforcement learning methods have their shortcomings in theory. For example, although the noise introduced by DDPG can guarantee the continuity of the action, it cannot control the variance; and A3C under the Gaussian strategy, although it can control the variance, it cannot control the variance. Does not satisfy the continuity condition in theory

Images