A Floating Control Method for Target Area of ​​Underwater Vehicle Based on Double Critic Reinforcement Learning Technology

Abstract

The invention relates to a method for controlling the floating of the target area of ​​an underwater vehicle based on double-critician reinforcement learning technology, which belongs to the technical field of marine control experiments, and is based on the DDPG algorithm framework in deep reinforcement learning. Both the previously obtained expert data and the interaction data obtained from the interaction between the agent and the task environment are used, and the mixed collection of the two greatly improves the convergence speed of the algorithm. At the same time, the present invention utilizes two sets of critic networks independent of each other, and obtains the loss function of the actor network by taking the minimum value of Q(s, a) respectively output by the two groups, which effectively reduces the overestimation existing in the reinforcement learning algorithm.

Description

technical field [0001] The invention relates to a method for controlling the floating of a target area of ​​an underwater vehicle based on double-critician reinforcement learning technology, and belongs to the technical field of ocean control experiments. Background technique [0002] As a key marine equipment, underwater vehicles are widely used in many scientific research and engineering fields such as ocean topographic mapping, resource exploration, archaeological investigation, pipeline maintenance, biological monitoring, etc., and are an important means for human beings to explore the ocean. However, the seabed environment is complex and changeable. Underwater vehicles working in such an environment will inevitably lead to economic losses and loss of important data if they fail to float up to the area where the mother ship is located in a timely, safe and intelligent manner when encountering a fault or strong interference. . Therefore, in order to enhance the adaptabil...

AI Technical Summary

Problems solved by technology

[0005] However, existing research and inventions based on traditional RL for underwater vehicle control have some significant defects: First, based on traditional reinforcement learning algorithms such as Q-learning, it is necessary to construct a huge Q-value table to store high The Q(s, a) value exists in the latitude action space and state space, and with the continuous training of the agent in the high-dimensional action and state space, the number of Q values ​​in the Q value table will explode. makes this method very limited

Then, with the combination of deep learning and traditional Q-learning technology proposed by the Google Deepmind team, the deep reinforcement learning algorithm DRL (Deep reinforcement learning) algorithm was born. In this algorithm, the Q value table is replaced by the neural network and becomes DQN (Deep Q Net) (V.Mnih et al., "Human-level control through deep reinforcement learning," Nature, vol.518, pp.529-533, 2015.), but the DQN algorithm is only suitable for discrete action spaces, which restricts its application to Intelligent control of underwater vehicles; DDPG (Deep Deterministic Policy Gradient) (Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep inforcement learning [J]. Computer ence, 2015, 8(6): A187.) It is a control algorithm suitable for continuous action space, but the Q(s, a) output by the critic network comes from the expectation of the action-value function, which leads to the disadvantage of overestimation

Moreover, the above RL method does not consider expert data that is easy to obtain and has reliable performance, which leads to the slow convergence speed of the algorithm in training, and there is a lot of randomness in the early stage of training

Images