基于深度强化学习的道路交叉口生态驾驶策略研究

doi:10.16097/j.cnki.1009-6744.2024.01.008

交通运输系统工程与信息 ›› 2024, Vol. 24 ›› Issue (1): 81-92.DOI: 10.16097/j.cnki.1009-6744.2024.01.008

• 智能交通系统与信息技术 • 上一篇下一篇

基于深度强化学习的道路交叉口生态驾驶策略研究

李传耀¹，张帆¹，王涛²，黄德鑫¹，唐铁桥^*3

1. 中南大学，交通运输工程学院，长沙 410100；2. 合肥工业大学，汽车与交通工程学院，合肥 230000； 3. 北京航空航天大学，交通科学与工程学院，北京 100191

收稿日期:2023-11-07 修回日期:2023-12-18 接受日期:2023-12-21 出版日期:2024-02-25 发布日期:2024-02-11
作者简介:李传耀(1987- )，男，湖南永州人，副教授
基金资助:
国家自然科学基金(72271248，72288101)

Signalized Intersection Eco-driving Strategy Based on Deep Reinforcement Learning

LI Chuanyao¹, ZHANG Fan¹, WANG Tao², HUANG Dexin¹, TANG Tieqiao^*3

1. School of Traffic and Transportation Engineering, Central South University, Changsha 410100, China; 2. School of Automotive and Transportation Engineering, Hefei University of Technology, Hefei 230000, China; 3. School of Transportation Science and Engineering, Beihang University, Beijing 100191, China

Received:2023-11-07 Revised:2023-12-18 Accepted:2023-12-21 Online:2024-02-25 Published:2024-02-11
Supported by:
National Natural Science Foundation of China (72271248，72288101)

摘要/Abstract

摘要： 在互联和自动驾驶环境下，生态驾驶具有显著的潜力，可提高交通效率并降低能源消耗和排放。本文探讨一种基于深度强化学习算法的生态驾驶策略，该算法可优化互联自动驾驶汽车(CAV)的纵向操纵和横向决策；将状态空间分为与车辆动态特性相关的局部变量，以及与信号交叉口相关的全局变量，确保CAV与环境之间的充分互动；奖励函数综合考虑了车辆的驾驶要求，与信号灯的协同作用以及全局节能激励因素；此外，设计一个典型的城市道路场景训练模型。结果表明，在信号灯和智能体输出协同控制下，本文提出的策略可以实现CAV的生态驾驶，并确保CAV准确驶入目标车道；在动态交通环境下进行仿真显示，通过控制多辆CAV引导人工驾驶车辆，本文方法可将交叉路口的通行能力提高约17.90%，并将交通系统的燃料消耗和污染物排放降低约8.76%。

关键词: 智能交通, 生态驾驶, 深度强化学习, 互联与自动驾驶汽车, 信号交叉路口

Abstract: Eco-driving in a connected and autonomous driving environment has great potential to improve traffic efficiency, energy saving, and emission reduction. This paper proposes a prosocial eco-driving strategy based on the deep reinforcement learning algorithm that optimizes the longitudinal manipulation and lateral decision-making of the connected and automated vehicle (CAV). The state space is divided into the local variables related to dynamic vehicle characteristics and the global variables associated with signalized intersection to ensure adequate interaction between the CAV and the roadway environment. The designed reward function integrates the vehicle driving requirements, synergy with signals and global energy saving incentives. In addition, this study developed a typical urban road intersection scenario to train the model. The results show that the proposed strategy can achieve eco-driving of the CAV in collaboration with the signal and output lateral control to ensure the vehicle travels to the target lane. In addition, simulations in a dynamic traffic environment reveal that the proposed method can improve the capacity at the intersection by about 17.90% and reduce the traffic system's fuel consumption and pollutant emissions by approximately 8.76% through the control of multiple CAVs to guide the human-driven vehicles.

Key words: intelligent transportation, eco-driving, deep reinforcement learning, connected and autonomous vehicle; signalized intersection

中图分类号:

U491.2

李传耀, 张帆, 王涛, 黄德鑫, 唐铁桥. 基于深度强化学习的道路交叉口生态驾驶策略研究[J]. 交通运输系统工程与信息, 2024, 24(1): 81-92.

LI Chuanyao, ZHANG Fan, WANG Tao, HUANG Dexin, TANG Tieqiao. Signalized Intersection Eco-driving Strategy Based on Deep Reinforcement Learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2024, 24(1): 81-92.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.tseit.org.cn/CN/10.16097/j.cnki.1009-6744.2024.01.008

http://www.tseit.org.cn/CN/Y2024/V24/I1/81

参考文献

[1] KIM M, KIM H K. Investigation of environmental benefits of traffic signal countdown timers[J/OL]. Transportation Research Part D: Transport and Environment, 2020, 85: 102464[2024-01-11]. https:// doi.org/10.1016/j.trd.2020.102464.

[2] TANVIR S, CHASE R T, ROUPAHIL N M. Development and analysis of eco-driving metrics for naturalistic instrumented vehicles[J/OL]. Journal of Intelligent Transportation Systems, 2021, 25(3): 235-248[2024-01- 11]. https://doi.org/10.1080/15472450.2019.1615486.

[3] 刘显贵, 王晖年, 洪经纬, 等. 网联环境下信号交叉口车速控制策略及优化[J]. 交通运输系统工程与信息, 2021, 21(2): 82- 90. [LIU X G, WANG H N, HONG J W, et al. Speed control strategy and optimization of signalized intersection in network environment[J]. Journal of Transportation Systems Engineering and Information Technology, 2021, 21(2): 82-90.]

[4] 程颖, 张佳乐, 张少君, 等. 大型货运车辆生态驾驶及节油潜力评估[J]. 交通运输系统工程与信息, 2020, 20 (6): 253-258. [CHEN Y, ZHANG J L, ZHANG S J, et al. Evaluation of eco-driving behavior and fuel-saving potential of large freight vehicles[J]. Journal of Transportation Systems Engineering and Information Technology, 2020, 20(6): 253-258.]

[5] XIA H, BORIBOONSOMSIN K, BARTH M. Dynamic eco-driving for signalized arterial corridors and its indirect network-wide energy/emissions benefits[J/OL]. Journal of Intelligent Transportation Systems, 2013, 17(1): 31-41[2024-01-11]. https: //doi.org/ 10.1080/ 15472450.2012.712494.

[6] LIAO P, TANG T Q, LIU R, et al. An eco-driving strategy for electric vehicle based on the powertrain[J/OL]. Applied Energy, 2021, 302: 117583[2024-01-11]. https: //doi.org/10.1016/j.apenergy.2021.117583.

[7] LI J, HE C. A novel dynamic cooperative traffic control algorithm for the reduction of traffic delay[J/OL]. Machines, 2022, 10(10): 831[2024- 01- 11]. https://doi. org/10.3390/machines10100831.

[8] 韩磊, 张轮. 混合交通流环境下基于改进强化学习的可变限速控制策略[J]. 交通运输系统工程与信息, 2023, 23(3): 110-122. [HAN L, ZHANG L. Variable speed limit control based on improved dueling double deep Q network under mixed traffic environment[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(3): 110-122.]

[9] 赵建东, 贺晓宇. 多网联范围下的智能网联车换道决策组合模型研究[J]. 交通运输系统工程与信息, 2023, 23(1): 77-85. [ZHAO J D, HE X Y. A combination model for connected and autonomous vehicles lane-changing decision-making under multi connectivity range [J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(1): 77-85.]

[10] LI G, YANG Y, LI S, et al. Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness [J/OL]. Transportation Research Part C: Emerging Technologies, 2022, 134: 103452[2024-01-11]. https:// doi.org/10.1016/j.trc.2021.103452.

[11] LIU B, SUN C, WANG B, et al. Adaptive speed planning of connected and automated vehicles using multi-light trained deep reinforcement learning[J/OL]. IEEE Transactions on Vehicular Technology, 2022, 71(4): 3533- 3546 [2024- 01- 11]. https:// doi.org/ 10.1109/ TVT.2021.3134372.

[12] SELIMAN S M S, SADEK A W, HE Q. Automated vehicle control at freeway lane-drops: A deep reinforcement learning approach[J/OL]. Journal of Big Data Analytics in Transportation, 2020, 2(2): 147-166 [2024-01-11]. https:// doi.org/ 10.1007/ s42421-020- 00021-0.

[13] PUTERMAN M L. Chapter 8 Markov decision processes [M/OL]// Handbooks in operations research and management science: Chapter 2, Elsevier, 1990: 331-434 [2024-01-11]. https://www.sciencedirect.com/ science/article/pii/S0927050705801720.

[14] ZHOU M, YU Y, QU X. Development of an efficient driving strategy for connected and automated vehicles at signalized intersections: A reinforcement learning approach[J/OL]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(1): 433-443 [2024- 01-11]. https://doi.org/10.1109/TITS.2019.2942014.

[15] GUO Q, ANGAH O, LIU Z, et al. Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors[J/OL]. Transportation Research Part C: Emerging Technologies, 2021, 124: 102980 [2024- 01- 11]. https://doi.org/10.1016/j.trc.2021.102980.

[16] ALSABAAN M, NAIK K, KHALIFA T, et al. Applying vehicular networks for reduced vehicle fuel consumption and CO2 emissions[M/OL]// INTECH Open Access Publisher, 2012 [2024-01-11]. http://www.intechopen. com/books/ intelligent-transportation-systems/ applying-vehicular- networks- for- reduced- vehicle- fuel-consumption-and-co2-emissions.

[17] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[M/OL]. arXiv, 2013 [2024-01-11]. http://arxiv.org/abs/1312.5602.

[18] VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[M/OL]. arXiv, 2015 [2024-01-11]. http:// arxiv.org/ abs/ 1509.06461.

[19] TREIBER M, HENNECKE A, HELBING D. Congested traffic states in empirical observations and microscopic simulations[J/OL]. Physical Review E, 2000, 62(2): 1805-1824 [2024-01-11]. https:// doi.org/ 10.1103/ PhysRevE.62.1805.

[20] KESTING A, TREIBER M, HELBING D. General lane-changing model MOBIL for car-following models[J/OL]. Transportation Research Record: Journal of the Transportation Research Board, 2007, 1999(1): 86- 94 [2024-01-11]. https://doi.org/10.3141/1999-10.

[21] RAKHA H, AHN K, TRANI A. Development of VT-Micro model for estimating hot stabilized light duty vehicle and truck emissions[J/OL]. Transportation Research Part D: Transport and Environment, 2004, 9(1): 49-74 [2024-01-11]. https:// doi.org/ 10.1016/ S1361- 9209(03)00054-3.

基于深度强化学习的道路交叉口生态驾驶策略研究

Signalized Intersection Eco-driving Strategy Based on Deep Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	魏丽英, 吴润泽. 基于鱼群效应的智能网联车队形成与演化机理研究[J]. 交通运输系统工程与信息, 2024, 24(2): 76-85.
[2]	黄玮, 李世昌, 曾海鹏. 基于非冲突合流策略的交叉口灵活信号控制方法[J]. 交通运输系统工程与信息, 2024, 24(2): 86-95.
[3]	张玺君, 聂生元, 李喆, 张红. 基于自注意力机制的深度强化学习交通信号控制[J]. 交通运输系统工程与信息, 2024, 24(2): 96-104.
[4]	唐斌, 刘光耀, 江浩斌, 田宁, 米伟, 王春宏. 基于柔性演员-评论家算法的决策规划协同研究[J]. 交通运输系统工程与信息, 2024, 24(2): 105-113.
[5]	王婉琦, 程国柱, 徐亮. 基于无监督聚类分析的激进换道行为识别方法[J]. 交通运输系统工程与信息, 2024, 24(2): 166-178.
[6]	陈峥, 赵文龙, 郭凤香, 赵志刚, 刘昱, 刘永刚. 计及安全性与舒适性的智能车辆换道轨迹规划研究[J]. 交通运输系统工程与信息, 2024, 24(1): 55-65.
[7]	裴玉龙, 傅博涵, 王子奇, 张杰. 引力理论框架下基于综合竞争力的自动驾驶拟人换道决策模型[J]. 交通运输系统工程与信息, 2024, 24(1): 66-80.
[8]	王丹, 林业. 警告刺激对驾驶员接管性能的影响机理研究[J]. 交通运输系统工程与信息, 2024, 24(1): 106-114.
[9]	李岩, 王泰州, 徐金华, 陈姜会, 汪帆. 面向动态交通分配的交通需求深度学习预测方法[J]. 交通运输系统工程与信息, 2024, 24(1): 115-123.
[10]	任其亮, 徐韬, 刘媛, 程龙春. 考虑载客状态的改进孤立森林浮动车异常数据检测算法[J]. 交通运输系统工程与信息, 2024, 24(1): 124-131.
[11]	袁见, 刘福强, 安琨, 郑喆, 马万经, 俞秋田. 基于交通波特征的车道级车流溯源方法[J]. 交通运输系统工程与信息, 2024, 24(1): 159-167.
[12]	曲大义, 孟奕名, 王韬, 宋慧, 陈意成. 基于分子力场的网联自主车辆跟驰安全特性及模型[J]. 交通运输系统工程与信息, 2023, 23(6): 33-41.
[13]	郭凤香, 黄金涛, 陈昱光, 郭延永, 刘攀. 多种缺失模式下交通数据组合近似填补方法[J]. 交通运输系统工程与信息, 2023, 23(6): 42-50.
[14]	蒋贤才, 徐慧智. 混合驾驶环境下交叉口空间资源动态控制方法[J]. 交通运输系统工程与信息, 2023, 23(6): 63-73.
[15]	李泰国, 张天策, 李超, 周星宏. 基于面部倒立摆模型与信息熵的驾驶员疲劳检测[J]. 交通运输系统工程与信息, 2023, 23(5): 24-32.