Coordinated Sequential Optimization for Network-wide Traffic Signal
Control Based on Heterogeneous Multi-agent Transformer

doi:10.16097/j.cnki.1009-6744.2024.03.012

Abstract

Abstract: Focusing on the complex traffic signal control task in an urban network, this study proposes a coordinated sequential optimization method based on a Heterogeneous Multi-Agent Transformer (HMATLight) to optimize network-wide traffic signals and improve the performance of signal control policy at intersections within the urban network. Specifically, considering the spatial correlation of multi-intersection traffic flow, a value encoder based on a self-attention mechanism is first designed to learn traffic observation representations and realize network-level communication. Secondly, in response to the non-stationary environment for multi-agent policy updates, a policy decoder based on the multi-agent advantage decomposition is constructed, which can sequentially output the optimal responsive action on the basis of the joint actions of preceding agents. Besides, an action-masking mechanism based on effective driving vehicles, adapting the decision frequency within the time-adequate interval, and a spatio-temporal pressure reward function considering the waiting fairness are constructed, which further enhance policy performance and practicality. A series of experiments are carried out on Hangzhou network datasets to validate the effectiveness of the proposed method. Experimental results show that the proposed HMATLight outperforms all baselines on two datasets with five metrics. Compared with the best-performed baseline, HMATLight decreases the average travel time by 10.89%, the average queue length by 18.84% and the average waiting time by 22.21%. Furthermore, HMATLight is dramatically higher in generalization and significantly reduces instances of long vehicle waiting times.

Key words: intelligent transportation, deep reinforcement learning, network-wide traffic signal control, heterogeneous multi-agent, spatio-temporal pressure reward

摘要： 针对路网交通信号控制的复杂性，本文提出基于异构多智能体自注意力网络的路网信号协调顺序优化方法，提升路网范围内多交叉口信号控制策略性能。首先，模型考虑多交叉口交通流的空间相关性，采用基于自注意力机制的价值编码器学习交通观测表征，实现路网级通信；其次，面向多智能体策略更新的非稳态环境，模型在前序智能体的联合动作基础上，基于多智能体优势分解的策略解码器，顺序决策最优反应动作；最后，设计基于有效行驶车辆的动作掩码机制，在时效完备区间自适应调节决策频率，并提出考虑等待公平性的时空压力奖励函数，进一步提高策略性能与实用性。在杭州路网数据集上验证模型有效性，结果表明：所提模型在2个数据集和5个性能指标上均优于基准模型；相比最优基准模型，所提模型平均行程时间降低10.89%，平均排队长度降低18.84%，平均等待时间降低22.21%。此外，所提模型的泛化能力更强，且显著减少车辆等待时间过长的情形。

关键词: 智能交通, 深度强化学习, 路网信号控制, 异构多智能体, 时空压力奖励

CLC Number:

U491

CHEN Xiqun, ZHU Yizhang, XIE Ningke, GENG Maosi, LV Chaofeng. Coordinated Sequential Optimization for Network-wide Traffic Signal Control Based on Heterogeneous Multi-agent Transformer[J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 114-126.

陈喜群, 朱奕璋, 谢宁珂, 耿茂思, 吕朝锋. 基于异构多智能体自注意力网络的路网信号协调顺序优化方法[J]. 交通运输系统工程与信息, 2024, 24(3): 114-126.

Add to citation manager EndNote|Ris|BibTeX

URL: http://www.tseit.org.cn/EN/10.16097/j.cnki.1009-6744.2024.03.012

http://www.tseit.org.cn/EN/Y2024/V24/I3/114

References

[1] ROBERTSON D I. "TRANSYT" method for area traffic control[J]. Traffic Engineering & Control, 1969, 10(6): 181-182.

[2] ROBERTSON D I, BRETHERTON R D. Optimizing networks of traffic signals in real time-the SCOOT method [J]. IEEE Transactions on Vehicular Technology, 1991, 40(1): 11-15.

[3] SIMS A G, DOBINSON K W. The sydney coordinated adaptive traffic (SCAT) system philosophy and benefits [J]. IEEE Transactions on Vehicular Technology, 1980, 29(2): 130-137.

[4] VARAIYA P. Max pressure control of a network of signalized intersections[J]. Transportation Research Part C: Emerging Technologies, 2013, 36: 177-195.

[5] 马东方, 陈曦, 吴晓东, 等. 基于强化学习的干线信号混合协同优化方法[J]. 交通运输系统工程与信息, 2022, 22(2): 145-153. [MA D F, CHEN X, WU X D, et al. Mixed-coordinated decision-making method for arterial signals based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2022, 22(2): 145-153.]

[6] WEI H, CHEN C, ZHENG G, et al. Presslight: Learning max pressure control to coordinate traffic signals in arterial network[C]. Anchorage: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.

[7] ZHENG G J, XIONG Y H, ZANG X S, et al. Learning phase competition for traffic signal control[C]. Beijing: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019.

[8] CHEN C C, WEI H, XU N, et al. Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control[C]. New York: Proceedings of the AAAI Conference on Artificial Intelligence, 2020.

[9] PAPOUDAKIS G, CHRISTIANOS F, RAHMAN A, et al. Dealing with non-stationarity in multi-agent deep reinforcement learning[J]. ArXiv Preprint ArXiv, 2019, 1906: 04737.

[10] WEI H, XU N, ZHANG H C, et al. CoLight: Learning network-level cooperation for traffic signal control[C]. Beijing: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019.

[11] ZHANG L, WU Q, SHEN J, et al. Expression might be enough: Representing pressure and demand for reinforcement learning based traffic signal control[C]. Maryland: Proceedings of the 39th International Conference on Machine Learning, 2022.

[12] MAO F, LI Z H, LI L. A comparison of deep reinforcement learning models for isolated traffic signal control[J]. IEEE Intelligent Transportation Systems Magazine, 2022, 15(1): 160-180.

[13] XU M, WU J P, HUANG L, et al. Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning[J]. Journal of Intelligent Transportation Systems, 2020, 24(1): 1-10.

[14] WEN M N, KUBA J G, LIN R J, et al. Multi-agent reinforcement learning is a sequence modeling problem [J]. Advances in Neural Information Processing Systems, 2022, 35: 16509-16521.

[15] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. ArXiv Preprint ArXiv, 2017, 1707: 06347.

[16] KUBA J G, WEN M N, MENG L H, et al. Settling the variance of multi-agent policy gradients[J]. Advances in Neural Information Processing Systems, 2021, 34: 13458-13470.

[17] SCHULMAN J, MORITZ P, LEVINE S, et al. High-dimensional continuous control using generalized advantage estimation[J]. ArXiv Preprint ArXiv, 2015, 1506: 02438.

[18] ZHANG H C, FENG S Y, LIU C, et al. CityFlow: A multi-agent reinforcement learning environment for large scale city traffic scenario[C]. San Francisco: Proceedings of the World Wide Web Conference (WWW 2019), 2019.

[19] KOONCE P, RODEGERDTS L, LEE K, et al. Traffic signal timing manual[R]. Washington: Federal Highway Administration, 2008.

[20] COOLS S B, GERSHENSON C, D'HOOGHE B. Self-organizing traffic lights: A realistic simulation[M]. London: Advances in Applied Self-Organizing Systems, 2013.

[21] OROOJLOOY A, NAZARI M, HAJINEZHAD D, et al. Attendlight: Universal attention-based reinforcement learning model for traffic signal control[J]. Advances in Neural Information Processing Systems, 2020, 33: 4079- 4090.

[22] ZHANG G H, WANG Y H. Optimizing minimum and maximum green time settings for traffic actuated control at isolated intersections[J]. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(1): 164- 173.

Coordinated Sequential Optimization for Network-wide Traffic Signal Control Based on Heterogeneous Multi-agent Transformer

基于异构多智能体自注意力网络的路网信号协调顺序优化方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	LIU Meiqi, JIN Kairan, LI Yalan, GUO Ge. Robust Model Predictive Control of Connected and Automated Vehicle Trajectories on Urban Roads [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(4): 31-40.
[2]	WENHuiying, ZHANG Xinyi, HUANG Junda, XU Pengpeng. Intelligent Vehicle Trajectory Prediction Considering Dynamic Interactions [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(4): 60-68.
[3]	ZHAOShuo, QI Geqi, LI Peihao, GUAN Wei. Driving Behavior Recognition Based on EEG Channel Attention Mechanism [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(4): 283-291.
[4]	MAQianli, WANG Tong, SHAO Shuai, JIAPeng. Evaluation of Driverless Minibus Operation in Complex Scenarios Under Road Test Environment [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(4): 300-310.
[5]	DONG Hongzhao, YANG Jiawei, QUAN Cheng. Dynamic Clearance Control Method for Reusing Bus Lanes Under Vehicular Networking [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 12-20.
[6]	LIU Jiang, ZHANG Chu, CAI Baigen, WANG Jian, LU Debiao. Pseudo-range Spoofing Detection Method for Satellite-based Train Positioning Using Sample Augmentation [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 32-42.
[7]	LI Jianshi, XU Youchun, QI Yao, XIE Desheng, LI Hua. Longitudinal Speed Control Method of Unmanned Vehicle Under Undulating Road Conditions [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 43-52.
[8]	ZHANG Jinlei, YANG Jian, LIU Xiaobing, CHEN Yao, YANG Lixing, GAO Ziyou. Computer Vision-based Fire Detection and Localization Inside Urban Rail Transit Stations [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 53-63.
[9]	XIN Qi, WANG Jiaqi, YANG Wenke , XU Meng, YUAN Wei. Eco-driving Under Mixed Autonomy at Signalized Intersection: A Deep Reinforcement Learning Model [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 127-139.
[10]	NI Haoyuan, YU Guizhen, LI Han, CHEN Peng, LIU Xi, WANG Wenda. Collaborative Driving Decision-making Method of Unmanned Mining Trucks in Open-pit Mine Operation Areas [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 277-289.
[11]	ZHANG Lei , PENG Jinshuan , CHEN Xiaoli. Impact of Non-driving Tasks on Drivers' Visual and Physiological Characteristics Under Take-over Scenario [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 290-298.
[12]	ZHANG Heshan, TAN Xin , FAN Mengwei, PAN Cunshu, XU Jin, ZHANG Yu. Accurate Detection Method of Small-scale Vehicles from Perspective of Unmanned Aerial Vehicle High-altitude Aerial Photography [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(3): 299-309.
[13]	HUANGWei, LI Shichang, ZENG Haipeng. Flexible Traffic Signal Control Method Based on Non-conflict Merging Strategy [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(2): 86-95.
[14]	ZHANGXijun, NIE Shengyuan, LI Zhe, ZHANG Hong. Traffic Signal Control with Deep Reinforcement Learning and Self-attention Mechanism [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(2): 96-104.
[15]	TANGBin, LIU Guangyao, JIANG Haobin, TIAN Ning, MI Wei, WANG Chunhong. Collaborative Study of Decision-making and Trajectory Planning for Autonomous Driving Based on Soft Acto-Critic Algorithm [J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(2): 105-113.