基于自注意力机制的深度强化学习交通信号控制

doi:10.16097/j.cnki.1009-6744.2024.02.010

交通运输系统工程与信息 ›› 2024, Vol. 24 ›› Issue (2): 96-104.DOI: 10.16097/j.cnki.1009-6744.2024.02.010

• 智能交通系统与信息技术 • 上一篇下一篇

基于自注意力机制的深度强化学习交通信号控制

张玺君^*，聂生元，李喆，张红

兰州理工大学，计算机与通信学院，兰州730050

收稿日期:2023-05-22 修回日期:2023-12-08 接受日期:2024-01-29 出版日期:2024-04-25 发布日期:2024-04-25
作者简介:张玺君(1980- )，男，甘肃临洮人，副教授。
基金资助:
国家自然科学基金 (62162040)；甘肃省自然科学基金重点项目 (22JR5RA226)；甘肃省高等学校创新基金项目(2021A-028)。

Traffic Signal Control with Deep Reinforcement Learning and Self-attention Mechanism

ZHANGXijun^*,NIE Shengyuan,LI Zhe,ZHANG Hong

School of Computer and Communication, Lanzhou University of Technology, Lanzhou 730050, China

Received:2023-05-22 Revised:2023-12-08 Accepted:2024-01-29 Online:2024-04-25 Published:2024-04-25
Supported by:
NationalNaturalScienceFoundation of China (62162040)；Key Program of the Natural Science Foundation of Gansu Province, China (22JR5RA226)；Gansu Province Higher Education Innovation Fund-funded Project (2021A-028)。

摘要/Abstract

摘要： 交通信号控制(TrafficSignal Control, TSC)仍然是交通领域中最重要的研究课题之一。针对现有基于深度强化学习(DeepReinforcementLearning, DRL)的交通信号控制方法的状态需要人为设计，导致提取交通状态信息难度大以及交通状态信息无法全面表达的问题，为了从有限特征中挖掘潜在交通状态信息，从而降低交通状态设计难度，提出一种引入自注意力网络的DRL算法。首先，仅获取交叉口各进入口车道车辆位置，使用非均匀量化和独热编码方法预处理得到车辆位置分布矩阵；其次，使用自注意力网络挖掘车辆位置分布矩阵的空间相关性和潜在信息，作为DRL算法的输入；最后，在单交叉口学习交通信号自适应控制策略，在多交叉口路网中验证所提算法的适应性和鲁棒性。仿真结果表明，在单交叉口环境下，与3种基准算法相比，所提算法在车辆平均等待时间等指标上具有更好的性能；在多交叉口路网中，所提算法仍然具有良好的适应性。

关键词: 智能交通, 自适应控制, 深度强化学习, 自注意力网络, 近端策略优化

Abstract: Traffic signal control (TSC) is still one of the most important research topics in the transportation field. The existing traffic signal control method based on deep reinforcement learning (DRL) needs to be designed manually, and it is often difficult to extract the complete traffic state information in the real operations. This paper proposes a DRL algorithm based on the self-attention network for the traffic signal control to analyze the potential traffic from limited traffic state information and reduce the difficulty of traffic state design. The vehicle position of each entry lane at the intersection is obtained, and the vehicle position distribution matrix is established through the non-uniform quantization and one-hot encoding method. The self-attention network is then used to analyze the spatial correlation and latent information of the vehicle location distribution matrix which is an input of the DRL algorithm. The traffic signal adaptive control strategy is trained at a single intersection and the adaptability and robustness of the proposed algorithm are verified in a multi-intersection road network. The simulation results show that in a single intersection environment, the proposed algorithm has better performance on the average vehicle delay and other indicators compared with three benchmark algorithms. The proposed algorithm also has good adaptability in the multi-intersection road network.

Key words: intelligent transportation, adaptive control, deep reinforcement learning, self-attention network, proximal policy optimization

中图分类号:

U491

张玺君, 聂生元, 李喆, 张红. 基于自注意力机制的深度强化学习交通信号控制[J]. 交通运输系统工程与信息, 2024, 24(2): 96-104.

ZHANGXijun, NIE Shengyuan, LI Zhe, ZHANG Hong. Traffic Signal Control with Deep Reinforcement Learning and Self-attention Mechanism[J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(2): 96-104.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.tseit.org.cn/CN/10.16097/j.cnki.1009-6744.2024.02.010

http://www.tseit.org.cn/CN/Y2024/V24/I2/96

参考文献

[1]陈喜群,朱奕璋,吕朝锋.基于混合近端策略优化的交叉口信号相位与配时优化方法[J].交通运输系统工程与信息,2023,23(1):106-113. [CHENXQ, ZHUYZ, LVCF.Signalphaseandtimingoptimizationmethodfor intersection based on hybrid proximal policy optimization [J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(1): 106-113.]

[2]WU Q, ZHANG L, SHEN J, et al. Efficient pressure: Improving efficiency for signalized intersections[J]. ArXiv, 2021. DOI: 10.48550/arXiv.2112.02336.

[3]ZHANG L, WU Q, SHEN J, et al. Expression might be enough: Representing pressure and demand for reinforcement learning based traffic signal control[C]// International Conference on Machine Learning. PMLR, 2022: 26645-26654.

[4]OROOJLOOY A, NAZARI M, HAJINEZHAD D, et al. Attendlight: Universal attention-based reinforcement learning model for traffic signal control[J]. Advances in Neural Information Processing Systems, 2020, 33: 4079 4090.

[5]唐慕尧,周大可,李涛.结合状态预测的深度强化学习交通信号控制[J].计算机应用研究,2022,39(8):2311 2315. [TANG M Y, ZHOU D K, LI T. Traffic signal control with deep reinforcement learning combined with state prediction[J]. Application Research of Computers， 2022, 39(8): 2311-2315.]

[6]马东方,陈曦,吴晓东,等.基于强化学习的干线信号混合协同优化方法[J]. 交通运输系统工程与信息, 2022, 22(2): 145-153. [MA D F, CHEN X, WU X D, et al. Mixed-coordinated decision-making method for arterial signals based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2022, 22(2): 145-153.]

[7]刘利军,王州,余臻.一种改进的深度确定性策略梯度网络交通信号控制系统[J].四川大学学报(自然科学版), 2021, 58(4): 93-99. [LIU L J, WANG Z, YU Z. An improved deep deterministic policy gradient network traffic signal control system[J]. Journal of Sichuan University(Natural Science Edition), 2021, 58(4): 93-99.]

[8]YU P, LUO J. Minimize pressure difference traffic signal control based on deep reinforcement learning[C]//2022 41st Chinese Control Conference (CCC), IEEE, 2022: 5493-5498.

[9] MA D, ZHOU B, SONG X, et al. A deep reinforcement learning approach to traffic signal control with temporal traffic pattern mining[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(8): 11789 11800.

[10] 任安妮, 周大可,冯锦浩,等.基于注意力机制的深度强化学习交通信号控制[J].计算机应用研究,2023,40 (2): 430-434. [REN A N, ZHOU D K, FENG J H, et al. Attention mechanism-based deep reinforcement learning traffic signal control[J]. Application Research of Computers, 2023, 40(2): 430-434.]

[11] GE Z. Reinforcement learning-based signal control strategies to improve travel efficiency at urban intersection[C]//2020 International Conference on Urban Engineering and Management Science (ICUEMS), IEEE, 2020: 347-351.

[12] ZHU Y, CAI M, SCHWARZ C, et al. Intelligent traffic light via policy-based deep reinforcement learning [J]. International Journal of Intelligent Transportation Systems Research, 2022. DOI: 10.1007/S13177-022 00321-5.

[13] LI Y, HE J, GAO Y. Intelligent traffic signal control with deep reinforcement learning at single intersection[C]// 2021 7th International Conference on Computing and Artificial Intelligence, 2021: 399-406.

[14] 刘智敏, 叶宝林,朱耀东,等.基于深度强化学习的交通信号控制方法[J].浙江大学学报(工学版),2022,56 (6): 1249-1256. [LIU Z M, YE B L, ZHU Y D, et al. Traffic signal control method based on deep reinforcement learning[J]. Journal of Zhejiang University (Engineering Science), 2022, 56(6):1249-1256.]

[15] WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7794-7803.

[16] JIE H, LI S, GANG S. Squeeze-and-excitation networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2018.

[17] CAO Y, XU J, LIN S, et al. GCNet: Non-local networks meet squeeze-excitation networks and beyond[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE, 2020.

[18] ZHENG G, XIONG Y, ZANG X, et al. Learning phase competition for traffic signal control[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019: 1963-1972.

[19] WEI H, XU N, ZHANG H, et al. Colight: Learning network-level cooperation for traffic signal control[C]// Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019: 1913-1922.

[20] ZHAO C, HU X，WANG G. PRGLight: A novel traffic light control framework with pressure-based reinforcement learning and graph neural network[C]. In IJCAI 2021 Reinforcement Learning for Intelligent Transportation Systems (RL4ITS) Workshop, 2021.

[21] FUJIMOTO S, GU S S. A minimalist approach to offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 20132-20145.

基于自注意力机制的深度强化学习交通信号控制

Traffic Signal Control with Deep Reinforcement Learning and Self-attention Mechanism

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	董红召, 杨嘉炜, 全程. 车联网下公交专用道复用的动态清空控制方法[J]. 交通运输系统工程与信息, 2024, 24(3): 12-20.
[2]	刘江, 张楚, 蔡伯根, 王剑, 陆德彪. 基于样本增强的列车卫星定位伪距欺骗检测方法[J]. 交通运输系统工程与信息, 2024, 24(3): 32-42.
[3]	李建市, 徐友春, 齐尧, 谢德胜, 李华. 起伏道路条件下无人车纵向速度控制方法研究[J]. 交通运输系统工程与信息, 2024, 24(3): 43-52.
[4]	张金雷, 杨健, 刘晓冰, 陈瑶, 杨立兴, 高自友. 基于计算机视觉的轨道交通站内火灾检测与定位[J]. 交通运输系统工程与信息, 2024, 24(3): 53-63.
[5]	陈喜群, 朱奕璋, 谢宁珂, 耿茂思, 吕朝锋. 基于异构多智能体自注意力网络的路网信号协调顺序优化方法[J]. 交通运输系统工程与信息, 2024, 24(3): 114-126.
[6]	辛琪, 王嘉琪, 杨文科, 徐猛, 袁伟. 信控路段混行交通生态驾驶深度强化学习模型[J]. 交通运输系统工程与信息, 2024, 24(3): 127-139.
[7]	倪浩原, 余贵珍, 李涵, 陈鹏, 刘喜, 王文达. 露天矿作业区无人矿车协同通行决策方法研究[J]. 交通运输系统工程与信息, 2024, 24(3): 277-289.
[8]	张磊, 彭金栓, 陈晓利. 接管场景驱动下非驾驶任务对驾驶人视觉与生理特征的影响[J]. 交通运输系统工程与信息, 2024, 24(3): 290-298.
[9]	张河山, 谭鑫, 范梦伟, 潘存书, 徐进, 张羽. 无人机高空航拍视角下小尺度车辆精确检测方法[J]. 交通运输系统工程与信息, 2024, 24(3): 299-309.
[10]	魏丽英, 吴润泽. 基于鱼群效应的智能网联车队形成与演化机理研究[J]. 交通运输系统工程与信息, 2024, 24(2): 76-85.
[11]	黄玮, 李世昌, 曾海鹏. 基于非冲突合流策略的交叉口灵活信号控制方法[J]. 交通运输系统工程与信息, 2024, 24(2): 86-95.
[12]	唐斌, 刘光耀, 江浩斌, 田宁, 米伟, 王春宏. 基于柔性演员-评论家算法的决策规划协同研究[J]. 交通运输系统工程与信息, 2024, 24(2): 105-113.
[13]	王婉琦, 程国柱, 徐亮. 基于无监督聚类分析的激进换道行为识别方法[J]. 交通运输系统工程与信息, 2024, 24(2): 166-178.
[14]	陈峥, 赵文龙, 郭凤香, 赵志刚, 刘昱, 刘永刚. 计及安全性与舒适性的智能车辆换道轨迹规划研究[J]. 交通运输系统工程与信息, 2024, 24(1): 55-65.
[15]	裴玉龙, 傅博涵, 王子奇, 张杰. 引力理论框架下基于综合竞争力的自动驾驶拟人换道决策模型[J]. 交通运输系统工程与信息, 2024, 24(1): 66-80.