In order for us to develop our proof-of-concept (PoC), we needed to use an AV (Automated / autonomous Vehicle) simulation package, and some AV control software to act as the system-under-test.
There are many AV simulation packages available so which one should we use? After drawing up our list of needs and wants from a simulation package, we evaluated several of them. Some of the criteria that we used included:
Commercial vs. free, open source. Relevant issues: cost, stability, maturity, support (short-term and long-term) for questions and issues, quality of documentation, under active development or not,
Level of modelling fidelity and accuracy of AV sensors, ego vehicle (e.g. vehicle behaviour and dynamics), other vehicles and the surrounding environment.
Capability of API (Application Programming Interface) to set up and configure driving scenarios via software (i.e. not through manual operation).
Ability to interface simulation software with AV control software.
Level and maturity of control of other vehicles. Can they only move along pre-programmed routes or can they also respond to the surrounding traffic?
How deterministic are the simulation runs. Is a particular simulation perfectly reproducible?
Simulation packages used
We decided to use two simulation packages in order to cover a wide range of tests and simulations:
CarMaker by IPG Automotive, a mature commercial simulator that is widely used in the automotive industry. It has excellent, validated models for vehicle systems, vehicle dynamics and AV sensors, and a powerful and stable API to set up driving scenarios. They provide excellent support for their enterprise customers and their partners.
LGSVL Simulator by LG, a free open source simulator that includes integration with two open-source AV control software platforms: Apollo and Autoware. It is under active development and has detailed and up-to-date documentation. They have excellent support on their forum, with our questions normally receiving a detailed and informed response within 24 hours.
For both simulation packages, our software was able to set up and run complex driving scenarios with no manual intervention, extract the relevant data from the simulation runs, calculate and analyse relevant Key Performance Indicators (KPIs), and set up follow-up simulation runs. We used a range of KPIs to cover objective and subjective criteria including safety, system performance, occupant comfort, and effects on traffic flow.
We had the whole system running on desktop PCs as well as on GPU-enabled virtual machines in the cloud. After investigating a number of cloud providers, it was decided that Azure virtual machines would be the most suitable for our application. Some of the tasks we wanted to perform, on both the PCs and the virtual machines, were slightly non-standard, and we received excellent detailed technical support from both IPG Automotive and LG.
We were able to identify and use powerful AV simulation software to help us develop our own technology. In the end, we decided to use two simulation packages in order to provide more breadth to the driving scenarios and the KPIs that we investigated.
This provided us with invaluable expertise on using and tailoring / modifying both established industry-standard simulation software as well as newer open-source simulation software for AV development. It is important to understand the pros and cons of different tools to achieve different goals. In the longer term, our “directed search” tool will have to integrate and work seamlessly with other tools that our customer is using, including simulation packages, so it is also useful for us to understand any issues that this may cause as early as possible.
We would like to thank IPG Automotive for their help and support in providing the CarMaker software for our development work.