A continuous reinforcement learning system and method based on stochastic differential equations

Abstract

The invention discloses a continuous reinforcement learning system and method based on stochastic differential equations. The system includes an action policy generator APG, an environment state estimator ESE, a value estimator VE, a memory storage module MS and an external environment EE; the specific steps are as follows : Initialize the action policy generator APG, the environment state estimator ESE and the value estimator VE; the action policy generator APG calculates the output action value increment Δa k ;The external environment EE outputs the next action value a k+1 , next step environment state value s k+1 and the current step reward value R k , and stored in the memory storage module MS; the environmental state estimator ESE updates the environmental state parameter set θ p and predicted future environmental state estimates s′ k ; The VE optimizer updates the Q-function network and predicts the future reward estimate R′ k ; The APG optimizer updates the action value parameter set θ v . This method is based on stochastic differential equations as the basic model, which can realize the continuity of action control and control the variance of the training process, and can select actions by predicting changes in the environment to achieve better environmental interaction.

Description

technical field [0001] The invention relates to the fields of reinforcement learning and stochastic process, and in particular relates to a reinforcement learning method for continuous systems. Background technique [0002] Deep reinforcement learning is an end-to-end learning system that combines the perception ability of deep learning with the decision-making ability of reinforcement learning, has strong versatility, and realizes direct control from raw input to output. Reinforcement learning has become a very important unsupervised learning method, which enables the agent to judge the current environment state through the value function in the interaction with the environment, and thus make corresponding actions to obtain better rewards. At present, reinforcement learning algorithms mainly focus on discrete action policy sets, while classic continuous reinforcement learning methods such as DDPG and A3C can be used for continuous motion control in applications such as robo...

AI Technical Summary

Problems solved by technology

[0003] However, most of the current continuous reinforcement learning methods have their shortcomings in theory. For example, although the noise introduced by DDPG can guarantee the continuity of the action, it cannot control the variance; and A3C under the Gaussian strategy, although it can control the variance, it cannot control the variance. Does not satisfy the continuity condition in theory

Images