基于深度强化学习的城市交通信号分层协同控制方法

doi:10.16097/j.cnki.1009-6744.2025.04.007

交通运输系统工程与信息 ›› 2025, Vol. 25 ›› Issue (4): 63-72.DOI: 10.16097/j.cnki.1009-6744.2025.04.007

• 智能交通系统与信息技术 • 上一篇下一篇

基于深度强化学习的城市交通信号分层协同控制方法

代亮^*，杜鹏飞，黄自彬，杨朋博

长安大学，电子与控制工程学院，西安710064

收稿日期:2025-02-28 修回日期:2025-03-28 接受日期:2025-04-09 出版日期:2025-08-25 发布日期:2025-08-25
作者简介:代亮(1981—)，男，陕西安康人，教授，博士。
基金资助:
陕西省交通运输厅交通科研项目 (24-15R)；长安大学中央高校基本科研业务费专项资金(300102323201)。

Hierarchical Collaborative Control of Urban Traffic Signals Based on Deep Reinforcement Learning

DAI Liang^*, DU Pengfei, HUANG Zibin,YANG Pengbo

School of Electronics and Control Engineering, Chang'an University, Xi'an 710064, China

Received:2025-02-28 Revised:2025-03-28 Accepted:2025-04-09 Online:2025-08-25 Published:2025-08-25
Supported by:
Transportation Research Project of Shaanxi Provincial Department of Transport (24-15R)；The Fundamental Research Funds for the Central Universities, CHD (300102323201)。

摘要/Abstract

摘要： 强化学习具有强大的自适应性和学习能力，能够根据环境变化和反馈信号不断调整策略和行为，实现持续优化，为城市交通信号控制提供新的技术手段。针对现有强化学习方法在交通信号协同控制中存在的智能体协作效率低下与控制区域划分机制缺失问题，本文提出一种交通信号分层协同控制架构，通过构建交叉口智能体，进行状态空间与回报函数的关联性协同设计，并建立基于拥堵扩散的交通控制子区划分模型，实现动态划分交通控制子区。最后，构建子区智能体协调子区内部交叉口智能体，交叉口智能体根据子区智能体提供的全局性建议以及所在交叉口情况完成信号控制方案的优化，实现区域交通信号分层协同控制。仿真结果表明，与现有定时控制与强化学习方法相比，本文方法平均行程时间分别降低56.78%和29.23%。相比MPLight(Max Pressure Light)方法，平均速度提升7.21%，平均行程时间与停车次数分别减少22.62%和3.98%。此外，通过对比在不同规模以及拓扑结构路网的性能表现，验证本文方法在同质交叉口路网中具有一定可移植性。

关键词: 智能交通, 交通信号控制, 深度强化学习, 多智能体, 子区划分

Abstract: Reinforcement learning has strong adaptability and learning ability, which can continuously adjust strategies and behaviors based on changes in the environment and feedback signals, thereby achieving continuous optimization and providing new technological means for urban traffic signal control. In response to the low efficiency of intelligent agent collaboration and the lack of control area partitioning mechanism in existing reinforcement learning methods for traffic signal collaborative control, this paper proposes a traffic signal hierarchical collaborative control architecture. By constructing intersection intelligent agents, this paper performs the correlation and collaborative design of state space and reward function. A traffic control sub zone partitioning model based on congestion diffusion is established to dynamically partition traffic control sub zones. The regional coordinated traffic signal control framework is established by deep reinforcement learning, and the information interaction mechanism is proposed with multi-level agents. The dynamic division method of traffic control sub-area is developed based on the congestion diffusion relationship of urban intersections. The results show that, compared with the existing timing control and reinforcement learning methods, the average travel time of the proposed method is reduced by 56.78% and 29.23%, respectively. In addition, the proposed method has certain portability in the road network of homogeneous intersections.

Key words: intelligent transportation, traffic signal control, deep reinforcement learning, multi-agent, sub-area division

中图分类号:

U491.5

代亮, 杜鹏飞, 黄自彬, 杨朋博. 基于深度强化学习的城市交通信号分层协同控制方法[J]. 交通运输系统工程与信息, 2025, 25(4): 63-72.

DAI Liang, DU Pengfei, HUANG Zibin, YANG Pengbo. Hierarchical Collaborative Control of Urban Traffic Signals Based on Deep Reinforcement Learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2025, 25(4): 63-72.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.tseit.org.cn/CN/10.16097/j.cnki.1009-6744.2025.04.007

http://www.tseit.org.cn/CN/Y2025/V25/I4/63

参考文献

[1] SHAIKH P W, EL-ABD M, KHANANFER M, et al. A Review on swarm intelligence and evolutionary algorithms for solving the traffic signal control problem [J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(1): 48-63.

[2] MOHAMMAD N, ATHARVA N, LIANA G, et al. Reinforcement learning in urban network traffic signal control: A systematic literature review [J]. Expert Systems with Applications, 2022, 199: 116830.1 116830.32.

[3] GENDERS W, RAZAVI S. Asynchronous n-step q learning adaptive traffic signal control [J]. Journal of Intelligent Transportation Systems, 2019, 23(4): 319 331.

[4]张玺君,聂生元,李喆,等.基于自注意力机制的深度强化学习交通信号控制[J].交通运输系统工程与信息, 2024, 24(2): 96-104. [ZHANG X J, NIE S Y, LI Z, et al. Traffic signal control with deep reinforcement learning and self-attention mechanism[J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(2): 96-104.]

[5] XIAO N, YU L, YU J, et al. A cold-start-free reinforcement learning approach for traffic signal control [J]. Journal of Intelligent Transportation Systems, 2021, 26(4): 476-85.

[6] CHU T, WANG J, CODCA L, et al. Multi-agent deep reinforcement learning for large-scale traffic signal control[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(3): 1086-1095.

[7] ZHU H , FENG J, SUN F, et al. Sharing control knowledge among heterogeneous intersections: A distributed arterial traffic signal coordination method using multi-agent reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(2): 2760-2776.

[8]陈喜群,朱奕璋,谢宁珂,等.基于异构多智能体自注意力网络的路网信号协调顺序优化方法[J].交通运输系统工程与信息,2024,24(3): 114-126. [CHEN X Q, ZHU Y Z, XIE N K, et al. Coordinated sequential optimization for network-wide traffic signal control based on heterogeneous multi-agent transformer[J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 114-126.]

[9] 马东方,陈曦,吴晓东,等.基于强化学习的干线信号混合协同优化方法[J]. 交通运输系统工程与信息, 2022, 22(2): 145-153. [MA D F, CHEN X, WU X D, et al. Mixed-coordinated decision-making method for arterial signals based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2022, 22(2): 145-153.]

[10] 郑晔, 郭仁忠,马丁,等.面向地理路网的交通信号智能协同控制方法[J]. 测绘学报, 2021, 50(9): 1203 1210. [ZHENG X, GUO R Z, MA D, et al. Multi-agent cooperative control for traffic signal on geographic road network[J] Acta Geodaetica et Cartographica Sinica, 2021, 50(9): 1203-1210.]

[11] LI Y, ZHANG Y, LI X, et al. Regional multi-agent cooperative reinforcement learning for city-level traffic grid signal control[J]. IEEE/CAA Journal of Automatica Sinica, 2024, 11(9): 1987-1998.

[12] TAN T, BAO F, DENG Y, et al. Cooperative deep reinforcement learning for large-scale traffic grid signal control [J]. IEEE Transactions on Cybernetics, 2020, 50 (6): 2687-2700.

[13] 胡立伟, 赵雪亭,杨锦青,等.基于双层规划的城市交通拥塞疏导优化研究[J].交通运输系统工程与信息, 2021, 21(5): 222-227, 234. [HU L W, ZHAO X T, YANG J Q, et al. Urban traffic congestion mitigation and optimization study based on two-level programming[J]. Journal of Transportation Systems Engineering and Information Technology, 2021, 21(5): 222-227, 234.]

[14] NG A Y, HARADA D, RUSSELL S. Policy invariance under reward transformations: Theory and application to reward shaping[C]. Bled: International Conference on Machine Learning, 1999.

[15] Transportation Research Board. The highway capacity manual 7th Edition: A guide for multi-modal mobility analysis[M]. USA: National Academies Press, 2016.

[16] ZHANG G, WANG Y. Optimizing minimum and maximum green time settings for traffic actuated control at isolated intersections[J]. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(1): 164 173.

[17] WEI H, XU N, ZHANG H, et al. Colight: Learning network-level cooperation for traffic signal control[C]. Beijing: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019.

[18] CHEN C, WEI H, XU N, et al. Toward a thousand lights: Decentralized deep reinforcement learning for large scale traffic signal control[C]. New York: Proceedings of the AAAI Conference on Artificial Intelligence, 2020.

基于深度强化学习的城市交通信号分层协同控制方法

Hierarchical Collaborative Control of Urban Traffic Signals Based on Deep Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	姚振兴, 刘贤, 赵一飞, 王亮, 王彦琛. 手机信令不均匀定位下出行端点自适应识别方法[J]. 交通运输系统工程与信息, 2025, 25(4): 44-52.
[2]	张鹏, 李兴旺, 姬炳豪, 孙超, 李文权. 路口重复放行的公交与社会车辆协同绿波优化模型[J]. 交通运输系统工程与信息, 2025, 25(4): 53-62.
[3]	王庞伟, 王思淼, 雷方舒, 徐京辉, 王子鹏, 王力. 混合动作表示强化学习下的城市交叉口智慧信控方法[J]. 交通运输系统工程与信息, 2025, 25(4): 73-83.
[4]	王连震, 沈超文, 王宇萍, 薛淑祺. 网联高速公路合流区基于间隙优化的车辆协同控制方法[J]. 交通运输系统工程与信息, 2025, 25(4): 84-95.
[5]	王维锋, 黄建鑫, 王晓全, 吴昕韩, 卞子馨. 基于无锚旋转框的航拍图像车辆全向检测方法[J]. 交通运输系统工程与信息, 2025, 25(4): 104-115.
[6]	陈峥, 张景, 陈博闻, 李春宇, 郭凤香, 魏福星. 基于异构多图时空融合的长时域车辆轨迹预测[J]. 交通运输系统工程与信息, 2025, 25(4): 126-136.
[7]	王祥, 任浩, 谭国真, 李健平, 王珏, 王妍力. 大语言模型协同强化学习的自动驾驶决策方法[J]. 交通运输系统工程与信息, 2025, 25(4): 137-146.
[8]	郑展骥, 冯昌奎, 赵杨洋, 凃强, 张河山, 徐进. 无人机航拍视角下密集场景非机动车小目标检测方法[J]. 交通运输系统工程与信息, 2025, 25(4): 147-161.
[9]	吴剑凡, 谢征宇, 秦勇, 王力, 王佳丽. 基于计算机视觉的地铁车站内乘客异常行为检测模型[J]. 交通运输系统工程与信息, 2025, 25(4): 162-174.
[10]	宋翠颖, 丁杰, 张春波. 模块化公交车辆调度研究综述[J]. 交通运输系统工程与信息, 2025, 25(4): 175-192.
[11]	谢秉磊, 冯健茜, 秦筱然. 多特征融合的网约车拼车起讫点需求时空预测[J]. 交通运输系统工程与信息, 2025, 25(4): 193-205.
[12]	陈喜群, 祝文琪, 吕朝锋. 融合轨迹时序与行为修正的车辆冲突风险预测[J]. 交通运输系统工程与信息, 2025, 25(4): 219-229.
[13]	朱琴跃, 李纪元, 李泓羿, 钱舒杨, 赵亚辉. 考虑负载不确定性的城轨列车目标速度曲线实时鲁棒优化[J]. 交通运输系统工程与信息, 2025, 25(4): 254-264.
[14]	高远, 付金龙, 冯文文. 考虑时空特征动态耦合的车辆轨迹预测方法[J]. 交通运输系统工程与信息, 2025, 25(3): 107-116.
[15]	赵霞, 李之红, 刘剑锋, 杨静, 吴梦琳, 秦伊萌. 行为模式时空动态超图聚类的公共交通异常团体检测[J]. 交通运输系统工程与信息, 2025, 25(3): 132-141.