SpringerLink
Log in

A Process for Scenario Prioritization and Selection in Simulation-Based Safety Testing of Automated Driving Systems

  • Conference paper
  • First Online:
Product-Focused Software Process Improvement (PROFES 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14483))

  • 568 Accesses

Abstract

Simulation-based safety testing of Automated Driving Systems (ADS) is a cost-effective and safe alternative to field tests. However, it is practically impossible to test every scenario using a simulator. We propose a process for prioritizing and selecting scenarios from an existing list of scenarios. The aim is to refine the scope of tested scenarios and focus on the most representative and critical ones for evaluating ADS safety. As a proof-of-concept, we apply our process to two pre-existing scenario catalogs provided by the Land Transport Authority of Singapore and the Department of Transportation. After applying our process, we prioritized and selected six scenario groups containing 51 scenarios for testing ADS in the CARLA simulator.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (Canada)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Uber Self-driving Crash.

  2. 2.

    Tesla Autopilot Crash.

  3. 3.

    NHTSA.

  4. 4.

    List of Pre-Crash Scenario for Crash Avoidance Research.

  5. 5.

    Test Scenarios- European New Car Assessment Programme.

  6. 6.

    List of Scenario Categories for the Assessment of Automated Vehicles.

  7. 7.

    NCSA Tools, Publications, and Data (Traffic Safety facts Annual Report Tables).

  8. 8.

    European Road Safety Data Portal - European Union.

  9. 9.

    Statistics at DfT - United Kingdom.

  10. 10.

    Canadian Motor Vehicle Traffic Collision Statistics: 2021.

  11. 11.

    Stat. for collision with dynamic&fixed objects: Ch. 3\(\rightarrow \)Passenger Cars\(\rightarrow \) Table 42.

  12. 12.

    Stat. for weather and lighting conditions: Goto: Chap. 2\(\rightarrow \)Time \(\rightarrow \)Table 26.

  13. 13.

    Stat. for occupants, non-occupants killed&injured in traffic crashes:Goto:Chapter 1:Trends \(\rightarrow \) General\(\rightarrow \) Table 3.

  14. 14.

    Stat. for crashes by vehicle driving maneuver: see table “Vehicles Involved in Single- and Two-Vehicle Fatal Crashes by Vehicle Maneuver”, State: USA, Year: 2020.

  15. 15.

    List of scenarios - Land Transport Authority of Singapore.

  16. 16.

    List of scenarios - US Department of Transportation - Table 1.

  17. 17.

    GES Crash Statistics - See Table 5: All crashes.

  18. 18.

    https://carla.org/.

References

  1. Abdessalem, R.B., Nejati, S., Briand, L.C., Stifter, T.: Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the 40th International Conference on Software Engineering, pp. 1016–1026 (2018)

    Google Scholar 

  2. Chen, Z., He, F., Yin, Y., Du, Y.: Optimal design of autonomous vehicle zones in transportation networks. Transp. Res. Part B: Methodological 99, 44–61 (2017)

    Article  Google Scholar 

  3. Chen, Z., He, F., Zhang, L., Yin, Y.: Optimal deployment of autonomous vehicle lanes with endogenous market penetration. Transp. Res. Part C: Emerging Technologies 72, 143–156 (2016)

    Article  Google Scholar 

  4. Duarte, F., Ratti, C.: The impact of autonomous vehicles on cities: a review. J. Urban Technol. 25(4), 3–18 (2018)

    Article  Google Scholar 

  5. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1

    Chapter  Google Scholar 

  6. Iso, I.: Pas 21448-Road Vehicles-Safety of the Intended Functionality. Int, Organization for Standardization (2019)

    Google Scholar 

  7. Jha, S., et al.: ML-based fault injection for autonomous vehicles: a case for Bayesian fault injection. In: 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 112–124. IEEE (2019)

    Google Scholar 

  8. Jha, S., et al.: Kayotee: A fault injection-based system to assess the safety and reliability of autonomous vehicles to faults and errors (2019). ar**v preprint ar**v:1907.01024

  9. Karunakaran, D., Worrall, S., Nebot, E.: Efficient statistical validation with edge cases to evaluate highly automated vehicles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–8. IEEE (2020)

    Google Scholar 

  10. Kaur, P., Taghavi, S., Tian, Z., Shi, W.: A survey on simulators for testing self-driving cars. In: Fourth International Conference on Connected and Autonomous Driving (MetroCAD). IEEE (2021)

    Google Scholar 

  11. Komzalov, A., Shilov, N.: Driver assistance systems: state-of-the-art and possible improvements. In: Proceedings of the 20th Conference of Open Innovations Association FRUCT, LETI University, St. Petersburg, Russland (2014)

    Google Scholar 

  12. Mercedes-Benz: Safety first for automated driving (SaFAD) (2019). https://group.mercedes-benz.com/innovation/case/autonomous/safety-first-for-automated-driving-2.html. Accessed 19 Jul 2023

  13. Nitsche, P.: Safety-critical scenarios and virtual testing procedures for automated cars at road intersections. Ph.D. thesis, Loughborough University (2018)

    Google Scholar 

  14. O’Kelly, M., Abbas, H., Gao, S., Shiraishi, S., Kato, S., Mangharam, R.: APEX: Autonomous vehicle plan verification and execution (2016)

    Google Scholar 

  15. Park, S., Park, S., Jeong, H., Yun, I., So, J.: Scenario-mining for level 4 automated vehicle safety assessment from real accident situations in urban areas using a natural language process. Sensors 21(20), 6929 (2021)

    Article  Google Scholar 

  16. Pei, K., Cao, Y., Yang, J., Jana, S.: DeepXplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, pp. 1–18 (2017)

    Google Scholar 

  17. Ponn, T., Gnandt, C., Diermeyer, F.: An optimization-based method to identify relevant scenarios for type approval of automated vehicles. In: Proceedings of the ESV-International Technical Conference on the Enhanced Safety of Vehicles, Eindhoven, The Netherlands, pp. 10–13 (2019)

    Google Scholar 

  18. Ramanagopal, M.S., Anderson, C., Vasudevan, R., Johnson-Roberson, M.: Failing to learn: autonomously identifying perception failures for self-driving cars. IEEE Robot. Autom. Lett. 3(4), 3860–3867 (2018)

    Article  Google Scholar 

  19. Rubaiyat, A.H.M., Qin, Y., Alemzadeh, H.: Experimental resilience assessment of an open-source driving agent. In: 2018 IEEE 23rd Pacific Rim International Symposium on Dependable Computing (PRDC), pp. 54–63. IEEE (2018)

    Google Scholar 

  20. SAE: 3016-taxonomy and definitions for terms related to on-road motor vehicle automated driving systems (2021). https://www.sae.org/standards/content/j3016_202104/. Accessed 15 Jul 2023

  21. Tian, Y., Pei, K., Jana, S., Ray, B.: DeepTest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, pp. 303–314 (2018)

    Google Scholar 

  22. Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1555–1562. IEEE (2018)

    Google Scholar 

  23. Waymo: Waymo safety report: on the road to fully self-driving (2018). https://downloads.ctfassets.net/sv23gofxcuiz/4gZ7ZUxd4SRj1D1W6z3rpR/2ea16814cdb42f9e8eb34cae4f30b35d/2021-03-waymo-safety-report.pdf. Accessed 18 July 2023

  24. **nxin, Z., Fei, L., **angbin, W.: CSG: critical scenario generation from real traffic accidents. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1330–1336. IEEE (2020)

    Google Scholar 

  25. Yamamoto, D., Suganuma, N.: Localization for autonomous driving on urban road. In: 2015 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), pp. 452–453. IEEE (2015)

    Google Scholar 

  26. Zhang, M., Zhang, Y., Zhang, L., Liu, C., Khurshid, S.: DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 132–142 (2018)

    Google Scholar 

  27. Zhou, H., et al.: DeepBillboard: systematic physical-world testing of autonomous driving systems. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 347–358 (2020)

    Google Scholar 

Download references

Acknowledgements

This research was partly funded by the Austrian BMK, BMAW, and State of Upper Austria under the SCCH competence center INTEGRATE [(FFG grant 892418)], the Estonian Research Council (grant PRG1226), Bolt Technology OÜ, and the Estonian state stipend for doctoral studies.

Author information

Authors and Affiliations

  1. University of Tartu, Narva mnt 18, 51009, Tartu, Estonia

    Fauzia Khan, Hina Anwar & Dietmar Pfahl

Authors
  1. Fauzia Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hina Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dietmar Pfahl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fauzia Khan .

Editor information

Editors and Affiliations

  1. FHV Vorarlberg University of Applied Science, Dornbirn, Austria

    Regine Kadgien

  2. Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany

    Andreas Jedlitschka

  3. FHV Vorarlberg University of Applied Science, Dornbirn, Austria

    Andrea Janes

  4. University of Oulu, Oulu, Finland

    Valentina Lenarduzzi

  5. University of Oulu, Oulu, Finland

    **aozhou Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Khan, F., Anwar, H., Pfahl, D. (2024). A Process for Scenario Prioritization and Selection in Simulation-Based Safety Testing of Automated Driving Systems. In: Kadgien, R., Jedlitschka, A., Janes, A., Lenarduzzi, V., Li, X. (eds) Product-Focused Software Process Improvement. PROFES 2023. Lecture Notes in Computer Science, vol 14483. Springer, Cham. https://doi.org/10.1007/978-3-031-49266-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49266-2_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49265-5

  • Online ISBN: 978-3-031-49266-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation