By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning...By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning of mobile robots.However,the traditional DDQN algorithm suffers from sparse rewards and inefficient utilization of high-quality data.Targeting those problems,an improved DDQN algorithm based on average Q-value estimation and reward redistribution was proposed.First,to enhance the precision of the target Q-value,the average of multiple previously learned Q-values from the target Q network is used to replace the single Q-value from the current target Q network.Next,a reward redistribution mechanism is designed to overcome the sparse reward problem by adjusting the final reward of each action using the round reward from trajectory information.Additionally,a reward-prioritized experience selection method is introduced,which ranks experience samples according to reward values to ensure frequent utilization of high-quality data.Finally,simulation experiments are conducted to verify the effectiveness of the proposed algorithm in fixed-position scenario and random environments.The experimental results show that compared to the traditional DDQN algorithm,the proposed algorithm achieves shorter average running time,higher average return and fewer average steps.The performance of the proposed algorithm is improved by 11.43%in the fixed scenario and 8.33%in random environments.It not only plans economic and safe paths but also significantly improves efficiency and generalization in path planning,making it suitable for widespread application in autonomous navigation and industrial automation.展开更多
船舶在自动靠泊过程中会受到风、浪、流和岸壁效应等因素的影响,故需要精确的路径规划方法防止靠泊失败。针对全驱动船舶靠泊过程的基于双深度Q网络(double deep Q network,DDQN)算法,设计了一种船舶自动靠泊路径规划方法。首先建立船...船舶在自动靠泊过程中会受到风、浪、流和岸壁效应等因素的影响,故需要精确的路径规划方法防止靠泊失败。针对全驱动船舶靠泊过程的基于双深度Q网络(double deep Q network,DDQN)算法,设计了一种船舶自动靠泊路径规划方法。首先建立船舶三自由度模型,然后通过将距离、航向、推力、时间和碰撞作为奖励或惩罚,改进奖励函数。随后引入DDQN来学习动作奖励模型,并使用学习结果来操纵船舶运动。通过追求更高的奖励值,船舶可以自行找到最优的靠泊路径。实验结果表明,在不同水流速度下,船舶都可以在完成靠泊的同时减小时间和推力,并且在相同水流速度下,DDQN算法与Q-learning、SARSA(state action reward state action)、深度Q网络(deep Q network,DQN)等算法相比,靠泊过程推力分别减小了241.940、234.614、80.202 N,且时间仅为252.485 s。展开更多
针对无人机(UAV)空战环境信息复杂、对抗性强所导致的敌机机动策略难以预测,以及作战胜率不高的问题,设计了一种引导Minimax-DDQN(Minimax-Double Deep Q-Network)算法。首先,在Minimax决策方法的基础上提出了一种引导式策略探索机制;然...针对无人机(UAV)空战环境信息复杂、对抗性强所导致的敌机机动策略难以预测,以及作战胜率不高的问题,设计了一种引导Minimax-DDQN(Minimax-Double Deep Q-Network)算法。首先,在Minimax决策方法的基础上提出了一种引导式策略探索机制;然后,结合引导Minimax策略,以提升Q网络更新效率为出发点设计了一种DDQN(Double Deep Q-Network)算法;最后,提出进阶式三阶段的网络训练方法,通过不同决策模型间的对抗训练,获取更为优化的决策模型。实验结果表明,相较于Minimax-DQN(Minimax-DQN)、Minimax-DDQN等算法,所提算法追击直线目标的成功率提升了14%~60%,并且与DDQN算法的对抗胜率不低于60%。可见,与DDQN、Minimax-DDQN等算法相比,所提算法在高对抗的作战环境中具有更强的决策能力,适应性更好。展开更多
电力传感网可以用于对电力网络的设备工作状态和工作环境等信息实时采集和获取,对于电力网络设施的实时监控与快速响应具有重要作用。针对系统在数据排队时延和丢包率上的特殊要求,提出了一种基于强化学习的电力传感网资源分配方案。在...电力传感网可以用于对电力网络的设备工作状态和工作环境等信息实时采集和获取,对于电力网络设施的实时监控与快速响应具有重要作用。针对系统在数据排队时延和丢包率上的特殊要求,提出了一种基于强化学习的电力传感网资源分配方案。在资源受限的情况下,通过资源分配算法来优化传感器节点的排队时延和丢包率,并将该优化问题建模为马尔可夫决策过程(Markov decision process,MDP),通过双深度Q网络(double deep Q-learning,DDQN)来对优化目标函数求解。仿真结果与数值分析表明,所提方案在收敛性、排队时延和丢包率等方面的性能均优于基准方案。展开更多
Edge computing nodes undertake an increasing number of tasks with the rise of business density.Therefore,how to efficiently allocate large-scale and dynamic workloads to edge computing resources has become a critical ...Edge computing nodes undertake an increasing number of tasks with the rise of business density.Therefore,how to efficiently allocate large-scale and dynamic workloads to edge computing resources has become a critical challenge.This study proposes an edge task scheduling approach based on an improved Double Deep Q Network(DQN),which is adopted to separate the calculations of target Q values and the selection of the action in two networks.A new reward function is designed,and a control unit is added to the experience replay unit of the agent.The management of experience data are also modified to fully utilize its value and improve learning efficiency.Reinforcement learning agents usually learn from an ignorant state,which is inefficient.As such,this study proposes a novel particle swarm optimization algorithm with an improved fitness function,which can generate optimal solutions for task scheduling.These optimized solutions are provided for the agent to pre-train network parameters to obtain a better cognition level.The proposed algorithm is compared with six other methods in simulation experiments.Results show that the proposed algorithm outperforms other benchmark methods regarding makespan.展开更多
基金funded by National Natural Science Foundation of China(No.62063006)Guangxi Science and Technology Major Program(No.2022AA05002)+1 种基金Key Laboratory of AI and Information Processing(Hechi University),Education Department of Guangxi Zhuang Autonomous Region(No.2022GXZDSY003)Central Leading Local Science and Technology Development Fund Project of Wuzhou(No.202201001).
文摘By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning of mobile robots.However,the traditional DDQN algorithm suffers from sparse rewards and inefficient utilization of high-quality data.Targeting those problems,an improved DDQN algorithm based on average Q-value estimation and reward redistribution was proposed.First,to enhance the precision of the target Q-value,the average of multiple previously learned Q-values from the target Q network is used to replace the single Q-value from the current target Q network.Next,a reward redistribution mechanism is designed to overcome the sparse reward problem by adjusting the final reward of each action using the round reward from trajectory information.Additionally,a reward-prioritized experience selection method is introduced,which ranks experience samples according to reward values to ensure frequent utilization of high-quality data.Finally,simulation experiments are conducted to verify the effectiveness of the proposed algorithm in fixed-position scenario and random environments.The experimental results show that compared to the traditional DDQN algorithm,the proposed algorithm achieves shorter average running time,higher average return and fewer average steps.The performance of the proposed algorithm is improved by 11.43%in the fixed scenario and 8.33%in random environments.It not only plans economic and safe paths but also significantly improves efficiency and generalization in path planning,making it suitable for widespread application in autonomous navigation and industrial automation.
文摘针对无人机(UAV)空战环境信息复杂、对抗性强所导致的敌机机动策略难以预测,以及作战胜率不高的问题,设计了一种引导Minimax-DDQN(Minimax-Double Deep Q-Network)算法。首先,在Minimax决策方法的基础上提出了一种引导式策略探索机制;然后,结合引导Minimax策略,以提升Q网络更新效率为出发点设计了一种DDQN(Double Deep Q-Network)算法;最后,提出进阶式三阶段的网络训练方法,通过不同决策模型间的对抗训练,获取更为优化的决策模型。实验结果表明,相较于Minimax-DQN(Minimax-DQN)、Minimax-DDQN等算法,所提算法追击直线目标的成功率提升了14%~60%,并且与DDQN算法的对抗胜率不低于60%。可见,与DDQN、Minimax-DDQN等算法相比,所提算法在高对抗的作战环境中具有更强的决策能力,适应性更好。
文摘电力传感网可以用于对电力网络的设备工作状态和工作环境等信息实时采集和获取,对于电力网络设施的实时监控与快速响应具有重要作用。针对系统在数据排队时延和丢包率上的特殊要求,提出了一种基于强化学习的电力传感网资源分配方案。在资源受限的情况下,通过资源分配算法来优化传感器节点的排队时延和丢包率,并将该优化问题建模为马尔可夫决策过程(Markov decision process,MDP),通过双深度Q网络(double deep Q-learning,DDQN)来对优化目标函数求解。仿真结果与数值分析表明,所提方案在收敛性、排队时延和丢包率等方面的性能均优于基准方案。
基金supported by the National Key Research and Development Program of China(No.2021YFE0116900)National Natural Science Foundation of China(Nos.42275157,62002276,and 41975142)Major Program of the National Social Science Fund of China(No.17ZDA092).
文摘Edge computing nodes undertake an increasing number of tasks with the rise of business density.Therefore,how to efficiently allocate large-scale and dynamic workloads to edge computing resources has become a critical challenge.This study proposes an edge task scheduling approach based on an improved Double Deep Q Network(DQN),which is adopted to separate the calculations of target Q values and the selection of the action in two networks.A new reward function is designed,and a control unit is added to the experience replay unit of the agent.The management of experience data are also modified to fully utilize its value and improve learning efficiency.Reinforcement learning agents usually learn from an ignorant state,which is inefficient.As such,this study proposes a novel particle swarm optimization algorithm with an improved fitness function,which can generate optimal solutions for task scheduling.These optimized solutions are provided for the agent to pre-train network parameters to obtain a better cognition level.The proposed algorithm is compared with six other methods in simulation experiments.Results show that the proposed algorithm outperforms other benchmark methods regarding makespan.
文摘针对海上船舶自主避碰决策中深度Q网络(deep Q-network,DQN)算法的高估和收敛性差的问题,提出一种融合噪声网络的裁剪双DQN(double DQN,DDQN)算法,记为NoisyNet-CDDQN算法。该算法采用裁剪双Q值的方式减小DQN算法的高估问题,并通过引入噪声网络来增强算法的稳定性以解决DQN算法收敛性差的问题。充分考虑船舶运动数学模型和船舶领域模型,并在奖励函数设计中考虑到偏航、《国际海上避碰规则》(International Regulations for Preventing Collisions at Sea,COLREGs)等要素。多会遇场景仿真实验证明,本文所提出的NoisyNet-CDDQN算法相较于融合噪声网络的DQN算法在收敛速度上提升了27.27%,相较于DDQN算法提升了54.55%,相较于DQN算法提升了87.27%,并且船舶自主避碰决策行为符合COLREGs,可为船舶的自主避碰提供参考。
