基于强化学习的干扰条件下高速铁路时刻表调整研究

doi:10.16097/j.cnki.1009-6744.2023.05.029

交通运输系统工程与信息 ›› 2023, Vol. 23 ›› Issue (5): 279-289.DOI: 10.16097/j.cnki.1009-6744.2023.05.029

• 运输组织优化理论与方法 • 上一篇下一篇

基于强化学习的干扰条件下高速铁路时刻表调整研究

庞子帅^a，王丽雯^a，彭其渊^*a,b

西南交通大学，a. 交通运输与物流学院；b. 综合交通运输国家地方联合工程实验室，成都 611756

收稿日期:2023-06-30 修回日期:2023-08-30 接受日期:2023-09-05 出版日期:2023-10-25 发布日期:2023-10-23
作者简介:庞子帅(1988- )，男，山东菏泽人，博士生
基金资助:
国家重点研发计划(2022YFB4300502)

High-speed Railway Timetable Rescheduling Under Random Interruptions Based on Reinforcement Learning

PANG Zi-shuai^a, WANG Li-wen^a, PENG Qi-yuan^*a,b

a. School of Transportation and Logistics; b. National United Engineering Laboratory of Integrated and Intelligent Transportation, Southwest Jiaotong University, Chengdu 611756, China

Received:2023-06-30 Revised:2023-08-30 Accepted:2023-09-05 Online:2023-10-25 Published:2023-10-23
Supported by:
National Key Research and Development Program of China (2022YFB4300502)

摘要/Abstract

摘要： 研究干扰条件下列车时刻表调整对提高高速铁路实时调度指挥决策水平和行车组织效率具有重要意义。本文基于数据驱动的优化方法研究干扰条件下列车时刻表调整，旨在提升时刻表调整模型实时应用效果。考虑列车运行约束，以列车晚点时间最小为目标，基于强化学习近端策略优化(Proximal Policy Optimization, PPO)模型提出列车时刻表实时调整方法。建立列车运行仿真环境，PPO智能体与环境不断交互贪婪搜索使目标函数最优的策略。分别使用随机干扰案例和我国武广高速铁路实际数据中干扰案例测试PPO模型的性能及效率。结果表明：PPO模型优于其他常见的强化学习模型，以及调度员现场决策方案(由历史数据获得)，PPO模型至少可减少13%的列车晚点时间；PPO模型收敛速度明显优于其他常用强化学习模型；PPO得到解的质量与最优解仅相差约2%，且相比于得到最优解的速度具有明显提升，使其能更好地应用于实时决策。

关键词: 铁路运输, 时刻表调整, PPO模型, 高速列车, 列车运行干扰

Abstract: Research on high-speed train timetable rescheduling under interruption conditions is of significant importance for enhancing the real-time dispatching capabilities of railways and optimizing train operation efficiency. This study employs a data-driven optimization approach, specifically deep reinforcement learning, to explore methods for reconstructing train operation trajectories under interruptions. Using the Proximal Policy Optimization (PPO) model while considering train operation constraints, we propose a train rescheduling approach to minimize train delays. We establish a train operation simulation environment where the PPO intelligent agent continuously interacts with the environment, seeking the optimal strategy with minimal delay. To evaluate the PPO model's performance and efficiency, we conduct tests using scenarios involving random interruptions and actual data from the Wuhan-Guangzhou high-speed railway in China. The verification results demonstrate that the train rescheduling scheme derived from the PPO model outperforms those obtained from other common reinforcement learning models and even the decisions made by on-site dispatchers. It can reduce train delays by about 13%. PPO exhibits significantly faster convergence compared to other commonly used reinforcement learning models. Although the solution quality obtained by PPO is about 2% less than the optimal solution, the PPO model has a significant improvement in the computation speed of obtaining the near-optimal solution. This makes it a more practical choice for real-time decision-making.

Key words: railway transportation, timetable rescheduling, proximal policy optimization (PPO), high-speed railway; train operation interruptions

中图分类号:

U292.4

庞子帅, 王丽雯, 彭其渊. 基于强化学习的干扰条件下高速铁路时刻表调整研究[J]. 交通运输系统工程与信息, 2023, 23(5): 279-289.

PANG Zi-shuai, WANG Li-wen, PENG Qi-yuan. High-speed Railway Timetable Rescheduling Under Random Interruptions Based on Reinforcement Learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(5): 279-289.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: http://www.tseit.org.cn/CN/10.16097/j.cnki.1009-6744.2023.05.029

http://www.tseit.org.cn/CN/Y2023/V23/I5/279

参考文献

[1] 朱子轩, 鲁工圆, 何必胜, 等. 单线铁路网络列车运行智能调度仿真研究 [J]. 计算机仿真, 2019, 36(5): 194- 201. [ZHU Z X, LU G Y, HE B S, et al. Simulation research on intelligent dispatching of train operation on single-track railway network[J]. Computer Simulation, 2019, 36(5): 194-201.]

[2] 杨锐, 牟瑞芳, 闫海峰. 列车运行计划仿真模型研究 [J]. 中国铁道科学, 2017, 38(6): 125-133. [YANG R, MU R F, YAN H F. Research on simulation model of train operation plan[J]. China Railway Science, 2017, 38 (6): 125-133.]

[3] QUAGLIETTA E, CORMAN F, GOVERDE R M. Impact of a stochastic and dynamic setting on the stability of railway dispatching solutions[C]. The Hague, Netherlands: 16th International IEEE Conference on Intelligent Transportation Systems, 2013.

[4] VEELENTURF L P, KIDD M P, CACCHIANI V, et al. A railway timetable rescheduling approach for handling large-scale disruptions[J]. Transportation Science, 2016, 50(3): 841-862.

[5] 邓念, 彭其渊, 占曙光. 干扰条件下高速铁路列车运行实时调整问题研究 [J]. 交通运输系统工程与信息, 2017, 17(4): 118-123. [DENG N, PENG Q Y, ZHAN S G. Research on real-time adjustment of high-speed railway train operation under interference conditions [J]. Journal of Transportation Systems Engineering and Information Technology, 2017, 17(4): 118-123.]

[6] 徐培娟, 张大伟, 彭辉, 等. 多类干扰下的高铁列车运行调整优化模型 [J]. 铁道科学与工程学报, 2021, 18 (7): 1723-1731. [XU P J, ZHANG D W, PENG H, et al. Optimization model for high-speed train operation adjustment under multiple types of interference [J]. Journal of Railway Science and Engineering, 2021, 18 (7): 1723-1731.]

[7] 李智, 张琦, 孙延浩, 等. 高速铁路列车运行图鲁棒性协同优化模型研究 [J]. 交通运输系统工程与信息, 2019, 19(5): 169-176. [LI Z, ZHANG Q, SUN Y H, et al. Research on robust collaborative optimization model for high speed railway train diagram[J]. Journal of Transportation Systems Engineering and Information Technology, 2019, 19(5): 169-176.]

[8] 牛宏侠, 宁正, 张肇鑫. 高铁列车运行图车站与区间冗余时间同步优化模型[J]. 交通运输系统工程与信息, 2019, 19(3): 96-102. [NIU H X, NING Z, ZHANG Z X. Synchronous optimization model of station redundant time and interval redundant time in high-speed railway train working diagram [J]. Journal of Transportation Systems Engineering and Information Technology, 2019, 19(3): 96-102.]

[9] ŠEMROV D, MARSETIČ R, ŽURA M, et al. Reinforcement learning approach for train rescheduling on a single-track railway[J]. Transportation Research Part B: Methodological, 2016, 86: 250-267.

[10] LIAO J, YANG G, ZHANG S, et al. A deep reinforcement learning approach for the energy-aimed train timetable rescheduling problem under disturbances[J]. IEEE Transactions on Transportation Electrification, 2021, 7 (4): 3096-3109.

[11] 代学武, 程丽娟, 崔东亮, 等. 基于强化学习的高速列车群运行调整方法[J]. 中国科学: 信息科学, 2022, 52 (5): 890-906. [DAI X W, CHENG L J, CUI D L, et al. High speed train group operation adjustment method based on reinforcement learning[J]. Chinese Science: Information Science, 2022, 52(5): 890-906.]

[12] 王荣笙, 张琦, 张涛, 等. 基于蒙特卡罗树搜索-强化学习的列车运行智能调整方法 [J]. 中国铁道科学, 2022,43(5): 146-156. [WANG R S, ZHANG Q, ZHANG T, et al. Intelligent adjustment method of train operation based on Monte Carlo tree search reinforcement learning [J]. China Railway Science, 2022, 43(5): 146-156.]

[13] 韩忻辰, 俞胜平, 袁志明, 等. 基于Q-learning的高速铁路列车动态调度方法[J]. 控制理论与应用, 2021, 38 (10): 1511-1521. [HAN X C, YU S P, YUAN Z M, et al. A Q-learning based dynamic scheduling method for high-speed railway trains[J]. Control Theory and Applications, 2021, 38(10): 1511-1521.]

[14] 俞胜平, 韩忻辰, 袁志明, 等. 基于策略梯度强化学习的高铁列车动态调度方法 [J]. 控制与决策, 2022, 37 (9): 2407-2417. [YU S P, HAN X C, YUAN Z M, et al. High speed train dynamic scheduling method based on strategic gradient reinforcement learning[J]. Control and Decision Making, 2022, 37(9): 2407-2417.]

[15] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. Arxiv Preprint ArXiv, 2017: 170706347.

基于强化学习的干扰条件下高速铁路时刻表调整研究

High-speed Railway Timetable Rescheduling Under Random Interruptions Based on Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	吴艺迪, 何世伟, 周汉. 重载铁路智慧港口站作业计划综合优化模型及算法研究[J]. 交通运输系统工程与信息, 2023, 23(6): 215-226.
[2]	张正坤, 朱昌锋, 景云, 邢进. 基于候车时间和效益损失的城轨列车运行计划优化[J]. 交通运输系统工程与信息, 2023, 23(5): 215-226.
[3]	林俊亭, 倪铭君. 基于扩张状态观测器的虚拟编组触发模型预测控制[J]. 交通运输系统工程与信息, 2023, 23(4): 134-146.
[4]	洪治潮, 张锦, 孙文杰, 申皓, 赵钢, 梁宏斌. 区块链背景下的中欧班列网络配流研究[J]. 交通运输系统工程与信息, 2023, 23(4): 270-281.
[5]	彭宏勤, 张国伍. “双碳”目标下高铁与民航高质量发展策略——“交通7+1论坛”第五十九次会议[J]. 交通运输系统工程与信息, 2023, 23(3): 1-5.
[6]	侯涛, 唐丽, 牛宏侠. 基于数据驱动的高速列车速度复合控制研究[J]. 交通运输系统工程与信息, 2023, 23(3): 145-152.
[7]	邰国璇, 黄友能, 李春驰, 王悉. 基于灵活编组的市域快轨时刻表优化方法研究[J]. 交通运输系统工程与信息, 2023, 23(3): 195-203.
[8]	姜正, 王瑞, 杜海宾, 于健, 陈雪. 重载自动驾驶机车长大下坡空电联合制动运用研究[J]. 交通运输系统工程与信息, 2023, 23(3): 311-318.
[9]	田小鹏, 牛惠民, 韩瑛. 面向需求的城际列车时刻表与停站方案同步优化[J]. 交通运输系统工程与信息, 2023, 23(2): 197-207.
[10]	童瑞咏, 毛保华, 杜鹏, 魏润斌, 黄俊生. 重载铁路回程运输组织与定价综合优化[J]. 交通运输系统工程与信息, 2023, 23(2): 217-224.
[11]	徐菱, 徐瞳, 江文辉, 肖骅, 张湘虹. 基于货主偏好的中欧班列定价与补贴机制研究[J]. 交通运输系统工程与信息, 2023, 23(1): 2-9.
[12]	冯佳, 李国威, 付义龙, 彭程, 吴语慧. 共享车辆基地的城轨车底运用与检修计划协同编制[J]. 交通运输系统工程与信息, 2023, 23(1): 131-140.
[13]	徐禾颖, 吕红霞, 吕苗苗, 王萌. 重载铁路装车端空车调配优化研究[J]. 交通运输系统工程与信息, 2023, 23(1): 141-151.
[14]	冯芬玲, 蔡明旭, 贾俊杰. 基于多层复杂网络的中欧班列运输网络关键节点识别研究[J]. 交通运输系统工程与信息, 2022, 22(6): 191-200.
[15]	张海, 吕苗苗, 倪少权. 基于非均匀发车间隔的大小交路时刻表优化模型[J]. 交通运输系统工程与信息, 2022, 22(6): 224-233.