混合交通流环境下基于改进强化学习的可变限速控制策略

doi:10.16097/j.cnki.1009-6744.2023.03.013

摘要/Abstract

摘要： 现有的可变限速(VSL)控制策略灵活性较差，响应速度较慢，对驾驶人遵从度和交通流状态预测模型的依赖性较高，且单纯依靠可变限速标志(VMS)向驾驶人发布限速值，难以在智能网联车辆(CAVS)与人工驾驶车辆(HDVS)混行的交通环境中实现较好的控制效果。对此，结合深度强化学习无需建立交通流预测模型，能自动适应复杂环境，以及CAVs可控性的优势，提出一种混合交通流环境下基于改进竞争双深度Q网络(PD3QN)的VSL控制策略，即IPD3QN-VSL。首先，将优先经验回放机制引入深度强化学习的竞争双深度Q网络(D3Q)框架中，提升网络的收敛速度和参数更新效率；并提出一种新的自适应ε-贪婪算法克服深度强化学习过程中探索与利用难以平衡的问题，实现探索效率和稳定性的提高。其次，以最小化路段内车辆总出行时间(TTS)为控制目标，将实时交通数据和上个控制周期内的限速值作为PD3QN算法的输入，构造奖励函数引导算法输出VSL控制区域内执行的动态限速值。该策略通过基础设施到车辆通信(I2V)向CAVs发布限速信息，HDVs则根据VMS上公布的限速值以及周围CAVs的行为变化做出决策。最后，在不同条件下验证IPD3QN-VSL控制策略的有效性，并与无控制情况、反馈式VSL控制和D3QN-VSL控制进行控制效果上的优劣对比。结果表明：在30%渗透率下，所提策略即可发挥显著控制性能，在稳定和波动交通需求情境中均能有效提升瓶颈区域的通行效率，缩小交通拥堵时空范围，与次优的D3QN-VSL控制相比，两种情境中的TTS分别改善了14.46%和10.36%。

关键词: 智能交通, 可变限速控制, 改进竞争双深度Q网络, 混合交通流, 智能网联车辆, 深度强化学习

Abstract: Existing variable speed limit (VSL) control strategies suffer from poor flexibility, slow response time, and a heavy reliance on the compliance rate and traffic flow prediction models. Additionally, it is difficult to achieve effective control by relying solely on variable message signs (VMS) to post speed limits to drivers in the mixed traffic environment where connected and automated vehicles (CAVs) and human-driven vehicles (HDVs) coexist. To this end, this paper proposes a VSL control strategy based on the improved dueling double deep Q network (IPD3QN) under the mixed traffic flow environment, i.e., IPD3QN-VSL. This strategy integrates the ability of deep reinforcement learning to automatically adapt to complex environments without establishing traffic flow prediction models, and the advantages of controllability of CAVs. Firstly, the prioritized experience replay mechanism is introduced into the dueling double deep Q network (D3QN) framework of deep reinforcement learning to enhance the convergence speed and parameter update efficiency of the network. Meanwhile, a novel adaptive 𝜀-greedy algorithm is proposed to solve the problem of balance between exploration and utilization in D3QN’s learning process. The proposed VSL control strategy aims to minimize the total time spent (TTS) of vehicles on the freeway section. Real-time traffic data and speed limits within the previous control cycle are used as inputs to the IPD3QN algorithm. Then, a reward function is constructed to guide the algorithm to generate the dynamic speed limit value executed in the VSL control area. Finally, the effectiveness of the IPD3QN-VSL control strategy is verified under different conditions and compared to no control, feedback control, and D3QN-VSL control in terms of control performance. Analysis results indicate that the proposed strategy can achieve remarkable control performance at a 30% penetration rate and effectively improve bottleneck traffic efficiency and reduce the spatiotemporal range of traffic congestion in both stable and fluctuating demand scenarios. Compared to the suboptimal D3QN-VSL control, the proposed strategy can achieve improvements of 14.46% and 10.36% on TTS in stable and fluctuating traffic demand scenarios, respectively.

Key words: intelligent transportation, variable speed limit control, improved dueling double deep Q network, mixed traffic flow, connected and automated vehicles, deep reinforcement learning

中图分类号:

U491.4

韩磊, 张轮, 郭为安. 混合交通流环境下基于改进强化学习的可变限速控制策略[J]. 交通运输系统工程与信息, 2023, 23(3): 110-122.

HAN Lei, ZHANG Lun, GUO Wei-an. Variable Speed Limit Control Based on Improved Dueling Double Deep Q Network Under Mixed Traffic Environment[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(3): 110-122.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.tseit.org.cn/CN/10.16097/j.cnki.1009-6744.2023.03.013

http://www.tseit.org.cn/CN/Y2023/V23/I3/110

参考文献

[1] CHEN D J, AHN S Y. Variable speed limit control for severe non-recurrent freeway bottlenecks[J]. Transportation Research Part C: Emerging Technologies, 2006, 14(3): 213-228.

[2] 李志斌. 快速道路可变限速控制技术[D]. 南京：东南大学, 2014. [LI Z B. Variable speed limit control technique on expressway[D]. Nanjing: Southeast University, 2014.]

[3] SORIGUERA F, TORNE J M, Rosas D. Assessment of dynamic speed limit management on metropolitan freeways[J]. Journal of Transportation Engineering, 2013, 17(1), 78-90.

[4] GHODS H, FU L, KIAN A R. An efﬁcient optimization approach to real-time coordinated and integrated freeway trafﬁc control[J]. IEEE Transactions on Intelligent Transportation Systems, 2010, 11(4): 873-884.

[5] CARLSON R C, PAPAMICHAIL I, PAPAGEORGIOU M. Local feedback-based mainstream traffic flow control on motorways using variable speed limits [J]. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(4): 1261-1276.

[6] LI Z B, LIU P, XU C C, et al. Reinforcement learning-based variable speed limit control strategy to reduce traffic congestion at freeway recurrent bottlenecks [J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(11): 3204-3217.

[7] ZHU F, UKKUSURI S V. Accounting for dynamic speed limit control in a stochastic traffic environment: a reinforcement learning approach[J]. Transportation Research Part C: Emerging Technologies, 2014, 41: 30-47.

[8] 段荟, 刘攀, 李志斌, 等. 基于强化学习的汇流瓶颈区可变限速策略研究[J]. 交通运输系统工程与信息, 2015, 15(1): 55-61. [DUAN H, LIU P, LI Z B, et al. Variable speed limit control at freeway merge bottlenecks based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2013, 15(1): 55-61.]

[9] WU Y K, TAN H C, RAN B. Differential variable speed limits control for freeway recurrent bottlenecks via deep reinforcement learning[J]. Transportation Research Part C: Emerging Technologies, 2020, 117(4): 102649.

[10] KE Z M, LI Z B, CAO Z J H, et al. Enhancing transferability of deep reinforcement learning-based variable speed limit control using transfer learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 22(7): 4684-4695.

[11] VATANI R N, CETIN M. Dueling double deep Q network for improving freeway lane reduction bottlenecks throughput via variable speed limit control[C]// Proceedings of the 99th Annual Meeting of the Transportation Research Board. Washington, 2020: 1-19.

[12] YU H Y, IANG R, HE Z B, et al. Automated vehicle-involved traffic flow studies: a survey of assumptions, models, speculations, and perspectives[J]. Transportation Research Part C: Emerging Technologies, 2021, 127: 103101.

[13] LI Y, PAN B, XING L, et al. Developing dynamic speed limit strategies for mixed traffic flow to reduce collision risks at freeway bottlenecks[J]. Accident Analysis & Prevention, 2022, 175: 106781.

[14] YU M, FAn W D. Optimal variable speed limit control in connected autonomous vehicle environment for relieving freeway congestion[J]. Journal of Transportation Engineering, Part A: Systems, 2019, 145(4): 04019007.

[15] HAN L, ZHANG L, GUO W A. Optimal differential variable speed limit control in a connected and autonomous vehicle environment for freeway off-ramp bottlenecks[J]. Journal of Transportation Engineering, Part A: Systems, 2023, 149(4): 04023009.

[16] LI D , ZHAO Y , PRAKASH R , et al. A hybrid approach for variable speed limit implementation and application to mixed traffic conditions with connected autonomous vehicles[J]. IET Intelligent Transport Systems, 2018, 12(5): 327-334.

[17] 何勇, 唐琤琤. 道路交通安全技术[M]. 北京: 人民交通出版社, 2008. [HE Y, TANG Z Z. Road traffic safety technology[M]. Beijing: China Communications Press, 2008.]

[18] WANG Z Y, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]. New York: Proceedings of the 33th International Conference on International Conference on Machine Learning. 2016.

[19] TREIBER M, HENNECKE A, HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Physical Review E, 2000, 62(2): 1805-1824.

[20] 姚志洪, 顾秋凡, 徐桃让, 等. 考虑时延的智能网联汽车混合交通流稳定性分析[J]. 控制与决策, 2022, 37(6): 1505-1512. [YAO Z H, GU Q F, XU T R, et al. Stability of mixed traﬃc ﬂow with intelligent connected vehicles considering time delay[J]. Control and Decision, 2022, 37(6): 1505-1512.]

[21] MILANES V, SHLADOVER SE. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data[J]. Transportation Research Part C Emerging Technologies, 2014, 48: 261-279.

[22] MILANES V, SHLADOVER SE, Spring J, et al. Cooperative adaptive cruise control in real traffic situations. IEEE Transactions on Intelligent Transportation Systems, 2014, 15(1): 294-305.

[23] LIN S, SCHUTTER B D, HRGYI A, et al. On a spatiotemporally discrete urban traffic model[J]. IET Intelligent Transport Systems, 2014, 8(3): 219-231.

[24] NGUYEN V. Bayesian optimization for accelerating hyper-parameter tuning[C]. Sardinia: Proceedings of the 11th IEEE SecondInternational Conference on Artificial Intelligence and Knowledge Engineering, 2019.

[25] PAN T L, GUO R Z, LAM W H K. Integrated optimal control strategies for freeway traffic mixed with connected automated vehicles: a model-based reinforcement learning approach[J]. Transportation Research Part C: Emerging Technologies, 2021, 123: 102987.