Abstract
Under a wide range of traffic cultures and driving conditions, it is essential that an automated vehicle performs highway merging with appropriate driving styles - driving safely and efficiently without annoying or endangering other road users. Despite the extensive exploration of Meta Reinforcement Learning (Meta RL) for quick adaptation to different environments and its application to automated vehicle driving policies, most state-of-the-art algorithms require a dense coverage of the task distribution and extensive data for each of the meta-training tasks, which is extremely expensive for the automotive industry. Our paper proposes IAMRL, a context-based Meta RL algorithm in which meta-imagination reduces real-world training tasks and data requirements. By interpolating the learned latent context space with disentangled properties, we perform meta-imagination. As a result of our autonomous highway merging experiments, IAMRL outperforms existing approaches in terms of generalization and data efficiency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Rakelly, K., Zhou, A., Finn, C., Levine, S., Quillen, D.: Efficient off-policy meta-reinforcement learning via probabilistic context variables. In: International Conference on Machine Learning, pp. 5331–5340. PMLR (2019)
Zhang, S., Wen, L., Peng, H., Tseng, H.E.: Quick learner automated vehicle adapting its roadmanship to varying traffic cultures with meta reinforcement learning. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1745–1752. IEEE (2021)
Mendonca, R., Gupta, A., Kralev, R., Abbeel, P., Levine, S., Finn, C.: Guided meta-policy search. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Kirsch, L., van Steenkiste, S., Schmidhuber, J.: Improving generalization in meta reinforcement learning using learned objectives. ar**v preprint ar**v:1910.04098 (2019)
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., Levine, S.: Meta-reinforcement learning of structured exploration strategies. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135. PMLR (2017)
Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. ar**v preprint ar**v:1803.02999 (2018)
Wang, J.X., et al.: Learning to reinforcement learn. ar**v preprint ar**v:1611.05763 (2016)
Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., Abbeel, P.: Rl\(^2\): fast reinforcement learning via slow reinforcement learning. ar**v preprint ar**v:1611.02779 (2016)
Wong, S.C., Gatt, A., Stamatescu, V., McDonnell, M.D.: Understanding data augmentation for classification: when to warp? In: 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–6. IEEE (2016)
Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363. PMLR (2019)
Ni, R., Goldblum, M., Sharaf, A., Kong, K., Goldstein, T.: Data augmentation for meta-learning. In: International Conference on Machine Learning, pp. 8152–8161. PMLR (2021)
Yao, H., et al.: Improving generalization in meta-learning via task augmentation. In: International Conference on Machine Learning, pp. 11 887–11 897. PMLR (2021)
Khodadadeh, S., Zehtabian, S., Vahidian, S., Wang, W., Lin, B., Bölöni, L.: Unsupervised meta-learning through latent-space interpolation in generative models. ar**v preprint ar**v:2006.10236 (2020)
Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: learning behaviors by latent imagination. ar**v preprint ar**v:1912.01603 (2019)
Hafner, D., Lillicrap, T., Norouzi, M., Ba, J.: Mastering Atari with discrete world models. ar**v preprint ar**v:2010.02193 (2020)
Lee, S., Chung, S.-Y.: Improving generalization in meta-RL with imaginary tasks from latent dynamics mixture. In: Advances in Neural Information Processing Systems, vol. 34, pp. 27222–27235 (2021)
Lin, Z., Thomas, G., Yang, G., Ma, T.: Model-based adversarial meta-reinforcement learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 10161–10173 (2020)
Mendonca, R., Geng, X., Finn, C., Levine, S.: Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. ar**v preprint ar**v:2006.07178 (2020)
Wen, L., Zhang, S., Tseng, H.E., Singh, B., Filev, D., Peng, H.: Prior is all you need to improve the robustness and safety for the first time deployment of meta RL. ar**v preprint ar**v:2108.08448 (2021)
Higgins, I., et al.: beta-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations (2016)
Chen, R.T., Li, X., Grosse, R., Duvenaud, D.: Isolating sources of disentanglement in VAEs. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, vol. 2615, p. 2625 (2019)
Tokui, S., Sato, I.: Disentanglement analysis with partial information decomposition. ar**v preprint ar**v:2108.13753 (2021)
Burgess, C.P., et al.: Understanding disentangling in \(beta\)-VAE. ar**v preprint ar**v:1804.03599 (2018)
Chen, R.T., Li, X., Grosse, R.B., Duvenaud, D.K.: Isolating sources of disentanglement in variational autoencoders. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Tran, L., Yin, X., Liu, X.: Disentangled representation learning GAN for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1415–1424 (2017)
Montero, M.L., Ludwig, C.J., Costa, R.P., Malhotra, G., Bowers, J.: The role of disentanglement in generalisation. In: International Conference on Learning Representations (2020)
Leurent, E.: An environment for autonomous driving decision-making (2018). https://github.com/eleurent/highway-env
Treiber, M., Hennecke, A., Helbing, D.: Congested traffic states in empirical observations and microscopic simulations. Phys. Rev. E 62(2), 1805 (2000)
Hidas, P.: Modelling vehicle interactions in microscopic simulation of merging and weaving. Transp. Res. Part C: Emerg. Technol. 13(1), 37–62 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, S., Wen, L., Zhuang, H., Tseng, H.E. (2024). Quickly Adaptive Automated Vehicle’s Highway Merging Policy Synthesized by Meta Reinforcement Learning with Latent Context Imagination. In: Sun, F., Li, J. (eds) Cognitive Computation and Systems. ICCCS 2023. Communications in Computer and Information Science, vol 2029. Springer, Singapore. https://doi.org/10.1007/978-981-97-0885-7_17
Download citation
DOI: https://doi.org/10.1007/978-981-97-0885-7_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0884-0
Online ISBN: 978-981-97-0885-7
eBook Packages: Computer ScienceComputer Science (R0)