Verification and Validation Utilizing Carla Simulator for Autonomous Driving Development

Abstract

As the level of autonomous driving (AD) functions advances, various events and problems have occurred in many unexpected or unseen situations. Hence, the design of AD systems requires a validation and verification process to guarantee robust operability. A Simulation-Driven Verification and Validation is based on practical test scenarios in a simulation environment. This approach is a cost-effective way to verify the system requirement at the design level and validate the performance of AD functions at the vehicle testing level by utilizing practical test scenarios in simulation. The practical test scenarios are provided by European New Car Assessment Program (Euro NCAP) and they contain harsh conditions. It is also possible to derive numerical values optimized for AD function safety from the simulation results. The performance of AD functions can be robust and enhanced by applying optimized values to the system design. Verification and validation were conducted in a simulation environment through real-field scenarios. Autonomous Emergency Braking (AEB) - Vulnerable Road User (VRU) test scenarios, representative Euro NCAP test scenarios, were implemented. We created the Euro NCAP AEB-VRU test scenario to design an effective AEB function. We used RoadRunner to build the test road and used ScenarioRunner to render the test scenario written by the Association for Standardization of Automation and Measuring Systems (ASAM) OpenSCENARIO format according to Euro NCAP test requirements. The result of AEB-VRU has been investigated under normal conditions and harsh environments as well. This work shows that we can extend the safety of the AEB function by changing the vehicle speed according to situation perception, which indicates the possibility of utilization of a simulator for autonomous vehicle system design. In addition, suitability assessments of the Responsibility-Sensitive Safety (RSS) metrics were presented. RSS is a mathematical model proposed by Intel and Mobileye to provide an Automated Driving Safety Framework. One concern for adapting RSS in real road autonomous driving is that the RSS model may provide too conservative safety distances to nearby cars than necessary in real-world driving. As the result of the RSS simulation, the trade-off between safety margin and traffic flow was shown. If the safe distance is excessively secured, the traffic flow may be slower, and it causes inefficiency in vehicle flow. RSS can cause traffic flow disturbances, demonstrating the need for social acceptance when autonomous vehicles become commonly used in real-world environments.