AV Testing Advances Without Standards

The failure of the AV START Act in the United States Senate did more than just delay U.S. federal regulations for self-driving car technology that has yet to progress beyond the pilot-test stage.

It delayed discussions that could have narrowed the almost infinite number of choices automated vehicles (AVs) must be prepared to make by creating guidelines defining what constitutes “safe” operation of a self-driving car. These guidelines would have set de facto technical priorities that could have accelerated development and acted as functional requirements to help make functional safety testing of AVs definitive enough to shave a couple of zeros off the number of AV road-test miles.

Road-test miles are instead being shaved off by the industry, which has some ideas for shortening or virtualizing the safety validation cycle for AVs.

Do we need standards?
The Senate bill (S. 1885), the AV START Act, was supposed to streamline regulation of automated-vehicle testing and production by consolidating rules created by 32 states and extend the Federal Motor Vehicle Safety Standards (FMVSS) to include functions relative to autonomous driving as well as physical safety of the cars themselves.

“We need a good functional safety standard. FMVSS doesn’t say a thing about AVs and there is no functional safety standard for autonomy,” said Philip Koopman, associate professor of electrical and computer engineering at Carnegie Mellon University, who specializes in safety testing and validation of autonomous vehicles. “With or without AV START, in 2019 we need to see the industry come together and see how these things can be made safe. If the industry can get together on safety, that would do more to move things forward than regulations. If it could give the government a realistic idea of direction and timelines, that would get NHTSA closer to mark as well.”

The National Highway Traffic Safety Administration (NHTSA), which supervises the regulations, has produced several sets of voluntary guidelines and requests for feedback from the industry, but is not expected to deliver a firm set of regulations for another year or two.

NHTSA’s efforts so far have been tepid, at best, according to Roger Lanctot, director of Automotive Connected Mobility at Strategy Analytics and author of a 2017 study commissioned by Intel that predicted connected and automated vehicles could save 600,000 lives and add as much as $7 trillion to the global economy by 2050.

“NHTSA is at least two years behind,” Lanctot said. “The government was caught flat-footed on autonomous vehicles and NHTSA doesn’t have a clue what kind of guidance to give. I don’t see that changing.”

Validating validation
It’s not impossible to approach a complex system-of-systems like an AV to validate specific functions that confirm it meets some requirements, or to focus on validation and verification of individual chips or SoCs whose function is critical. It’s more efficient to start somewhere in the middle and move sideways through channels for communication with other components, rather than looking at an entire vehicle from either the very highest level or the very lowest, according to Marques McCammon, VP and general manager of connected vehicle solutions at Wind River Systems, Inc.

Demanding environmental requirements make testing and validation difficult in automotive applications anyway, but it’s usually better to start by validating combinations of smaller systems and their interaction than by evaluating how the whole car behaves, McCammon said.

One of them, until recently, was to put an untested AV on the road and just keep driving it—IRL and virtually—until it racked up so many miles that the number of decisions required to have done it provides a reliable base of data to show whether or not an AV was making decisions based on what its software designers hoped to accomplish or not.

AV developers have been rolling over endless miles of real and virtual road tests since 2016, in response to a Rand Corp. report called “How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?” The report suggested that demonstrating that 100 AV test vehicles could perform 20% more safely than humans in the same situation would require about 5 billion miles.

High repetition testing of stochastic questions makes sense, but the method as applied to most self-driving vehicles produced anecdotal evidence, not proof, Koopman has argued.

“The only way you can tell an autonomous vehicle is going to make the safe decision in every edge case is to road test it or simulate it and watch—but that’s still just anecdotal,” Koopman said. “It could do the right thing a million times, then try to kill everyone the next time because it’s Tuesday morning and it’s raining, and all you know is that it probably won’t.

There’s no alternative to simulation for testing at the system or vehicle level, but simulations have to go beyond just exposing an AV to environmental scenarios and test for potential faults in both software and hardware, according to Tom Anderson, technical marketing consultant at OneSpin Solutions.

“Verification has gotten much more rigorous and standards like 26262 have been updated, but they can’t cover every possible fault, so you have to plan and test for random failure if an alpha particle hits a bit at the wrong time,” Anderson said. “If your smartphone has to reboot, that’s not a big deal; in a self-driving vehicle it could cause a crash.”

It might be possible to shorten the mileage using a framework of criteria appropriate to the location and driving task required, however, according to recent Rand report acknowledged the unwieldiness of infinite road-testing and the need for standards, but suggested refining the results by tracking performance metrics specific to the route and context in which the vehicle is operating (on a closed test course, in simulation, on public roads or without a safety driver, for example).

The framework allows for “leading” metrics that act as proxies for desired, safe-driving behaviors. Lagging measures would count traffic tickets or informal demerits for failing to follow traffic rules or criteria set up by the test methodology. The metrics could shorten testing requirements with results that are more definitive regardless of whether they came from public roads, closed tracks or simulations.

Testers could also try to build public confidence by running AVs through their paces and showing off the racked-up point totals to increase consumer confidence.

One big problem: not only are there no AV safety standards, there is no commonly accepted definition of what “safety” is for an AV:

“For the purposes of this report, we define safety as the overall ability of a vehicle to operate without harm to passengers or other road users within the roadway ecosystem. This definition is broadly consistent with other definitions of safety. It focuses on people. Damage to property or infrastructure and injuries to animals are of secondary concern, chiefly of interest because people could have been injured,” according to the Rand report.

Weighting specific behaviors can help tailor metrics, but there is no replacement for hands-on evaluation of how a system works and whether it’s doing the right thing.

“There are different schools of thought about how to validate artificial intelligence, but a more bottom up approach seems more accurate,” McCammon said. “Some people are talking about using inference to try to verify whether the way it makes complex decisions is valid, but that’s too much. You can’t use AI to predict whether your AI systems are functionally safe. That’s tantamount to asking a baby to plug a laptop into a light socket. There are better ways to get it done.”

Driving all those miles helps AV designers see subtleties and choices that should translate to much higher levels of skill than metrics based on whether an AV ran over anything on a particular trip out, according to a piece written by Bryan Salesky, CEO of Ford AV partner Argo AI in Medium October, 2017.

“For example, the car needs to know when it will have to move over slightly for a large truck to give it more room, or adjust its speed to stay out of another driver’s blind spot…or commit to an action consistently so that other road users can respond correctly,” Salesky wrote. “It’s only from all of these examples and real-world driving that we can learn to predict the micro-maneuvers that turn out to be the leading indicators of the likely actions of other road users.”

It’s important to do more than observe, however, and to analyze performance on a level more detailed than a driver’s-license-test examiner’s checklist, however.

“Validation is the only way to really know that something works, but it’s better to expand beyond one component or system and look at the way they interact – looking at the interaction among lidar, RADAR cameras and the central processor that uses that data to decide whether it is looking at a pedestrian walking a bike or a bike parked by the side of the road,” McCammon said.

“Even at the level of chip validation you need input from those other systems—simulated data from sensors or whatever, output to the actuators—so you have a closed-loop system that includes more than just the electronics,” according to Neil Hand, director of marketing for Mentor, a Siemens Business. “With ADAS you’re dealing with systems of systems so you have vision systems going into the chip, the chip going into the braking system to affect what’s going on. You have to have some insight into that complexity, which goes up an order of magnitude when you start to see the rest of the systems in the car.”

Chips designed to control a braking function or engine management tend to be relatively simple and specialized, not complex and general-purpose like an iPhone component, “so it is realtively simple to make sure their own flows match 26262 or whatever requirement you’re using,” according to Dave Kelf, vice president of marketing for Breker Verification Systems.

“Chips for radar detection or other sensors are more complex and much larger, so people tend to use flows for larger chips that don’t match as well, and machine learning requires all kinds of strange processing methodologies. So trying to use the old flows is breaking down a bit,” Kelf said.

“Machine learning will be more important to autonomous driving, which means you have to rely on the software, not just the hardware to meet systematic and random failure requirements.” Tools and middleware for systematic flow are likely to remain a big market opportunity, Kelf said.

Standards like ISO 26262 don’t map machine-learning processes, leaving “huge” gaps autonomous systems, but there are already efforts underway to modify or expand on it and to widen the pool of data resources like HD maps beyond the impact of OpenDRIVE, a file format that allows digital recreation of road networks, according to Chad Partridge, CEO of simulation-testing provider Metamoto.

“This isn’t a driver-in-the-loop test with a video screen and fake driver’s seat a person would sit in,” he said. “The testing is all high capacity, high throughput, largely cloud-based simulations with system hardware in the loop and software in the loop, with virtual I/O that makes it look to the car as if it is actually performing. Our APIs simulate laser returns and radar data, for example, that trigger the lowest-level firmware in sensors so it will respond as if it is really operating the vehicle.”

Related stories
Who Will Regulate Autonomous Vehicles Best?
How one state is approaching regulation of driverless cars—without addressing whether they are safe.
Auto Chip Test Getting Harder
Each new level of assistance and autonomy adds new requirements and problems, some of which don’t have …
Who’s Paying For Auto Chip Test?
Testing of automotive chips is becoming more difficult and time-consuming, and the problem is only going to get worse. There is more to this …
Auto Chip Test Issues Grow
Semiconductors used in cars have higher quality and reliability requirements than most chips, but they have the …
How Automotive ICs Are Reshaping Semiconductor Test
The test strategy for automotive ICs has two main areas of concern: … Logic BIST testsall logic on chip, captures the response into scan chains, …