摘要
为满足智能船舶自主航行的发展需求,解决基于强化学习的船舶避碰决策方法存在的学习效率低、泛化能力弱以及复杂会遇场景下鲁棒性差等问题,针对船舶避碰决策信息的高维性和动作的连续性等特点,考虑决策的合理性和实时性,研究了基于双延迟深度确定性策略梯度(TD3)的船舶自主避碰方法。根据船舶间相对运动信息与碰撞危险信息,从全局角度构建具有连续多时刻目标船信息的状态空间;依据船舶操纵性设计连续决策动作空间;综合考虑目标导向、航向保持、碰撞危险、《1972年国际海上避碰规则》(COLREGs)和良好船艺等因素,设计船舶运动的奖励函数;基于TD3算法,根据状态空间结构,结合长短期记忆(LSTM)网络和一维卷积网络,利用Actor-Critic结构设计船舶自主避碰网络模型,利用双价值网络学习、目标策略平滑以及策略网络延迟更新等方式稳定网络训练,利用跳帧以及批量大小和迭代更新次数动态增大等方式加速网络训练;为解决模型泛化能力弱的问题,提出基于TD3的船舶随机会遇场景训练流程,实现自主避碰模型应用的多场景迁移。运用训练得到的船舶自主避碰模型进行仿真验证,并与改进人工势场(APF)算法进行比较,结果表明:所提方法学习效率高,收敛快速平稳;训练得到的自主避碰模型在2船和多船会遇场景下均能使船舶在安全距离上驶过,并且在复杂会遇场景中比改进APF算法避碰成功率高,避让2~4艘目标船时成功率高达99.233%,5~7艘目标船时成功率97.600%,8~10艘目标船时成功率94.166%;所提方法能有效应对来船的不协调行动,避碰实时性高,决策安全合理,航向变化快速平稳、震荡少、避碰路径光滑,比改进APF方法性能更强。
In order to meet the requirements of developingautonomous navigation of intelligent ships and solve the problems of low learning efficiency,weak generalization ability and poor robustness ofdecision-making methods for collision avoidance based on reinforcement learning,an autonomous collision avoidance method based on Twin Delayed Deep Deterministic Policy Gradient(TD3)algorithmis proposed based on the high-dimensional character-istics of the information processed in the process of collision avoidanceand continuity nature of ship maneuvers,al-so considering the rationality and real-time progress of decision-making.The collision risk of a given ship is calcu-lated by considering geographical location of the ship and the other ships nearby.The state space of intelligent colli-sion avoidance model for autonomous ships is developed from the perspective of the global point of view.The con-tinuous decision-making and action space of the ship is designed according to the maneuvering characteristics of en-countered ships.An intelligent collision avoidance model is developed considering factors such as orientation of tar-get ship,course keeping,collision risk,the COLREGs and good seamanship.Based on the TD3 algorithm,the ship autonomous collision avoidance network model is designed based on the state space structure,combining Long Short Term Memory(LSTM)networks and 1D-convolutional networks,and a network model is designed by using Actor-Critic structure.The network training is stabilized by means of clipped double q-learning,target strategy smoothing,and delayed policy updates.The developed collision avoidance model is trained and updated with ran-dom scenarios by usingframe skipping,dynamic increase of batch size,and iterative update times.In order to solve the problem of weak generalization ability of the model,a training process of random shipencounter scenariosbased on TD3 is proposed to achievemulti-scenario migration for theapplications of the model.A simulationis carried out to verify the model,then compared with the modified Artificial Potential Field(APF)method.The results show that the proposed method has high learning efficiency,fast and stable convergence.The trained model is applicable for the ships to passa safe distance in both two-ship and multi-ship encounter scenarios.In a complex encounter sce-nario,the success rate of collision avoidance reaches 99.233%when avoiding 2~4 target ships,97.600%when 5~7 target ships,94.166%when 8~10 target ships,is higher than that of the modified APF algorithm.The proposed method can effectively respond to the uncoordinated actions of incoming ships,with real-time performance,as well as safe and reasonable decision-making.The course change is fast,stable,and the vibration is small,also the path for avoiding collisions is smooth,which has better performance than the modified APF method.
作者
刘钊
周壮壮
张明阳
刘敬贤
LIU Zhao;ZHOU Zhuangzhuang;ZHANG Mingyang;LIU Jingxian(School of Navigation,Wuhan University of Technology,Wuhan 430063,China;Hubei Key Laboratory of Inland Shipping Technology,Wuhan University of Technology,Wuhan 430063,China;National Engineering Research Center for Water Transport Safety,Wuhan University of Technology,Wuhan 430063,China;School of Engineering,Department of Mechanical Engineering,Aalto University,Espoo 20110,Finland)
出处
《交通信息与安全》
CSCD
北大核心
2022年第3期60-74,共15页
Journal of Transport Information and Safety
基金
国家自然科学基金项目(52171351)资助。
关键词
交通信息工程
船舶避碰
智能决策
深度强化学习
双延迟深度确定性策略梯度
traffic information engineering
ship collision avoidance
intelligent decision-making
deep reinforce-ment learning
twin delayed deep deterministic policy gradient
