A team consisting of Ezequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, and Paolo Arcaini, submitted the Frenetic tool to the CPS testing competition of SBST 2021 which showed good results in terms of diversity and failure generation rate.
The competition focused on the scenario generation for testing the lane-keeping capabilities of an autonomous driving system, using the BeamNG simulator. The goal was to generate virtual roads that lead to lane-keeping failures (a.k.a. OBEs, for Out-of-Bounds-Errors) within a certain time frame. The submitted tools competed in two distinct configurations, differing in time budget, vehicle configuration, and OBE tolerance.
Frenetic is a genetic approach that leverages a curvature-based road representation, and introduces some nuances aiming at improving diversity of the generated roads.
Implementations were not ranked, but the competition report (in print) states that Frenetic is generally “very effective and triggered many failures“. In fact, Frenetic was among the two tools that “triggered at least 10x more OBEs than the other tools for both configurations”. In particular, “Frenetic exposed the most diverse set of failures on average” and found ”the highest number of OBEs for which the ego-car invades the opposite traffic lane”, which might be of particular interest for car manufacturers.* The Frenetic implementation and a description of the algorithm are available in our repository.