Competing Auto Sensor Fusion Approaches

As today’s internal-combustion engines are replaced by electric/electronic vehicles, mechanical-system sensors will be supplanted by numerous electronic sensors both for efficient operation and for achieving various levels of autonomy. Some of these new sensors will operate alone, but many prominent ones will need their outputs combined — or “fused” — with the outputs of other sensors in order to maximize their value.

What’s not obvious at this point is where that sensor fusion will occur within the vehicle, or how exactly the data will be combined, sorted, and prioritized.

“It’s clear there is too much sensor data and not enough processing power in automobiles today, and sensor fusion will happen,” said Andy Jaros, vice president of IP sales and marketing at Flex Logix. “How that will happen is still evolving and will depend on automotive system designers’ decisions for each model.”

Streams from sensors like cameras, radar, and lidar can be combined in a number of ways to provide new or more reliable information. Where that processing happens helps to inform the networking architecture, but it also impacts how artificial intelligence (AI) will operate on the data.

“Raw data is expected to be fused with camera or other sensors and then go into a perception engine to yield potentially better results,” said Willard Tu, senior director, automotive at Xilinx.

There is no universal agreement on how this will work, but there are many opinions. “Honestly, I think it’s a bit religious right now,” said John Swanson, senior product marketing manager at Synopsys.

Sensor fusion basics
Sensor fusion has been in vogue ever since the big electronic sensor push 10 or so years ago. The idea is that by combining the results of several imperfect sensors, one can get a better overall measurement. One way of looking at it is that one sensor can cover for the weak points of another sensor. For example, data from accelerometers, gyroscopes, and magnetometers are combined today to give better position and navigation information.

“The purpose of sensor fusion [in an automobile] is to build the awareness of the vehicle as if it were a human, because humans have sight, auditory perception, and olfactory perception,” said Thierry Kouthon, technical product manager, security at Rambus. “Vehicles don’t have that, so we have to replace it by something that has the same accuracy.”

In electric/electronic (E/E) vehicles, the majority of the focus goes into the vision-related sensors: cameras, radar, and lidar. They’re far from the only sensors in a vehicle, but performance-related sensors — monitors that say how things are going — are more likely to operate on their own.

Those monitors are likely to move into the electronics themselves, although they’ll be completely different from the monitors needed for an internal combustion engine (ICE). “In combustion engines, you have at least 15 external monitors monitoring the engine itself,” noted Gal Carmel, general manager for automotive at proteanTecs. “But in an electrified car, you don’t have an engine anymore. You have a battery, and you have a battery management system. So the trend you see is basically taking all those monitors and putting them inside the electronics.”

The visual sensors are much more complex, and the situations they have to deal with — varying light, weather, a wide range of objects of interest — mean that each has its strong points and weak points. The idea is that by using them together, one can cover all of the situations one way or another.

Fig. 1: Scene understanding & navigation using sensor fusion. Source: Siemens

Sensor fusion can, in principle, work in a couple of different ways. For one approach, all outputs from all sensors are combined all the time, in different measure for different circumstances. In another, the system relies mostly on one sensor, such as the camera, unless circumstances place the quality of the camera data in doubt — like low light. The other sensor options then act as redundancy to step in when needed.

“If you look at all those sensors, yes, they supply some kind of overlap or redundancy, but they also supply different abilities,” said Carmel.

Other data may also be fused in, as well, like GPS and map data. A vehicle’s position may be determined by GPS fused with recognized landmarks in view and the lane markers, but it can be cross-checked against map data to ensure that the calculated position makes sense.

“That can also go all the way to V2X connectivity, meaning infrastructure like traffic lights and railway crossings,” said Rambus’ Kouthon. “Other cars can also communicate with you and tell you, ‘Be careful, the truck you’re trying to overtake has another vehicle right in front that you cannot see.’”

So there are many ways that data can be fused, and the quality of that fusion can be a differentiator.

Are all the sensors needed?
Each of the sensors represents a cost, of course, so using fewer is less expensive. While there are three main visual sensors for now, opinions vary as to whether all three will last. The ability to go from three sensors down to two relies specifically on the quality of sensor fusion and AI.

“When there’s enough data and the AI is smart enough, one of these sensors is going to drop out,” said Vikram Patel, director of product marketing at Infineon. “If these OEMs could get away with using just radar and a camera, I think they would.”

The lidar manufacturers see things differently, of course. “Those guys think that we can have just lidar and maybe radar, or cheap radar, and get rid of the camera,” added Patel.

The goal of sensor fusion at the highest levels is to make decisions using the best possible data. This is critical for those high-level system that need to make life-and-death decisions — advanced driver assistance systems (ADAS) or L4/L5 autonomy systems. The quality of the decision is the end game. What that means specifically depends on where one is in the car.

At the local level — say, in one corner of the car — that may mean deciding what objects are in view. For a side-facing camera, it may mean identifying potential pedestrians or signs. In the front of the car, the camera may need to see farther ahead and be capable of calculating closing speeds.

At the “global” level, this must serve the needs of ADAS and autonomy. This is where inputs from around the car are combined to create a complete picture of the vehicle’s environment. “All measurements, pedestrians, object detection, understand of the landmarks, the lane mark eventually lead to a simple time to collision,” said Carmel.

Three types of processing
Doing this well requires three different kinds of computing, which may be easily confused. First is the simple processing of visual data. That’s basic image-processing work like color conversion, removing noise, and normalizing data, all of which are intended to improve the quality of data used in the downstream computing.

“Every customer I speak to talks about converting the pure raw ADC or FFT data into something a little smarter or cleaner,” said Infineon’s Patel. “The noise has been filtered out, and just the peaks have been hit, and then you move it out into whatever pipe you’re using.”

The next two may be somewhat intertwined. Fusion combines data from different sources, and machine learning (ML) makes sense of the images. Sensor fusion is not ML, and ML isn’t sensor fusion, strictly speaking. But data can be fused before being presented to an ML engine, or ML could be used on individual data streams pre-fusion, and then the ML results could be fused together.

The fusion architecture also must be able to deal with data generated outside the vehicle. “Smart city infrastructure has a similar problem [to internal automotive sensor fusion],” said David Fritz, senior director for autonomous and ADAS at Siemens EDA. “On one side, it detects objects and classifies them. How do I get that information into a vehicle in a way that is easily consumed? If they send the raw data, the 5G bandwidth they eat up is going to be phenomenal. Think about 10,000 intersections in New York City, all streaming terabytes of data simultaneously. It’s going to flood the system. That’s a non-starter.”

“Let’s say they come up with some compressed way to do that,” he continued. “Then the vehicles themselves are going to have to add compute power to decompress the data.”

Having external sources communicate object metadata that the vehicle could natively work with would simplify that data fusion.

Fig. 2: On the left is a scenario where sensor data is delivered directly to the central processor, either completely raw or after some minor data cleanup. It might come on a single connection to the zonal controller or go directly to the central processor. On the right, sensor data is classified locally in the zone with metadata being sent to the central processor. Source: Bryon Moyer/Semiconductor Engineering

If this becomes the way things are handled, then all of this object sharing would require some metadata standardization. “We’re not aware of any movements to standardize across the industry, although it is inevitable,” said Fritz. “It is highly likely to be driven by a de facto standard by whoever gets out there first and has the most market share.”

It’s not an enormous problem to solve, however. “There’s a finite number of things that you actually have to have standardized metadata for,” he added. “And the amount of data for each of those is small.”

Where will all of this happen?
So given all of this computing work, where will it happen within the vehicle? This is one of the considerations driving the design of vehicle architectures. The so-called “zonal” architecture appears to have established itself as a preferred approach, even if not all manufacturers are on board.

“The next-generation architecture is the zonal architecture,” said Robert Schweiger, director of automotive solutions at Cadence. “On each corner of a car, you have a zonal controller that takes care of the sensors that are mounted somewhere in that physical region.”

The idea is that a fair bit of work gets done by a controller in a limited part of the car — say, the front left portion. The results of that work could then be delivered to a central computer, where the local pieces can be assembled into a global view.

“We see the zonal architecture happening simply because of complexity,” said Patel. “It’s easier to manage and move the data around in a zonal scheme.”

Conversations around this tend to focus on raw data vs. processed data, but it’s not quite so clean-cut. “The phrase ‘raw data’ can lead to a lot of confusion, because different people have different definitions of what raw data is,” cautioned Patel. “It’s not purely the completely processed object data that you see today. But you will see partial processing done, and the result being sent down the Ethernet pipe.”

Others agree. “Today, most of the sensor fusion is object-oriented, not raw,” noted Tu. “An object-oriented approach requires less performance. Raw fusion will require more complex processing, and the techniques are novel.”

The zonal approach simplifies systems by distributing work in a way that can be modular and can scale. For example, a low-end car may have two zones, while a high-end vehicle might have six or more. Having a zonal controller as a common element makes it more likely that the controller volumes will be higher — and that costs will need to be kept in check.

“This could be where a middle ground comes out,” said Synopsys’ Swanson. “So in the zonal architecture you do some level of abstraction to give it to the central processing unit.”

A few OEMs — notably the new ones that are starting from scratch — may opt for an architecture that has everything done centrally. But so far at least, those OEMs haven’t attempted to attract a broad swath of the market, focusing instead on the high end.

“If I’m a Tesla, I mainly make high-end vehicles,” said Patel. “I don’t really care about scaling for a low-cost car. GM and VW — they will care.”

He’s not alone in seeing this split. “There is the old trend, the traditional OEMs, the BMWs, Daimlers, and Fords,” said Kouthon. “And there is the new trend — the Aptivs, the Teslas, the Waymos — and they have a dramatically different view of things.”

Tesla, for example, uses an extremely powerful central processing unit. “That chip takes in all the camera feeds, all the ultrasound feeds, and all the radar feeds and processes them centrally,” said Kouthon. “Their reasoning is that we are going to provide redundancy at the level of the wiring between the central processing unit and the sensors so that the chance of failure is reduced.”

That wiring redundancy reflects proprietary work they’ve done both on in-vehicle data delivery and power delivery to the electronics. By using ring architectures, such redundancy can be had without literally doubling the wires in the network.

Reliability is, of course, a big deal for an automobile. “If you use only a single central processor, you’re putting a lot of reliability on one computer,” Swanson noted.

It also could be argued that not all decisions need to be made centrally. “There are probably a lot of decisions that can be made locally,” Swanson said. “You don’t need the supercomputer to control how often the windshield wipers go.”

What goes into the zone?
Given the zonal approach, we can now address the question of what will be done where, although there may be some push and pull by different interests in the supply chain.

Camera-makers, for instance, may wish to differentiate by performing classification themselves and then delivering object metadata. At the very least, they may do much of the heavy image processing pre-ML and pre-fusion.

The zonal controller could implement any of the local processing that the sensors themselves didn’t perform. That could be signal cleanup, it could be local ML, or it could be local sensor fusion. A big driver of this decision is bandwidth for moving data around. Any processing done locally in the zone would naturally compress the raw data. Instead of shipping raw camera data to the central computer, the zone could send object metadata.

The zonal controller could do initial classification of the objects within its view. “The zonal controller will be the device that pre-processes the data coming from those sensors and sends higher-level object data to the central compute unit,” said Cadence’s Schweiger. “You can do a pre-fusion [i.e., an early fusion] in the zonal controller, and then you do the final fusion of all four zonal controllers in the central computer. The pre-fusion will also provide you the benefit of reducing the data.”

This strongly correlates with decisions about which networks will be used to ship all of this data around. Siemens did some work with an OEM where it simulated different setups.

“We did three models of the entire vehicle,” explained Fritz. “On one of them, we would ship the raw sensor data to a central computer over automotive Ethernet. The central computer would do all the sensor fusion, object detection, classification, and decision making. The second thing that we did was to ask, ‘What if we did some of the sensor computation on the edge?’ That reduces the network bandwidth by about 40%.”

In this model, Siemens sent a 3D point cloud from the edge to the central computer. “The third option was to do everything on the edge close to the sensor,” Fritz said. “And then all we have to do is pass the object metadata — its location, its classification, its direction of movements, that sort of thing. We saw that the first case is going to need greater than 10 Gbps to transfer all the raw sensor data,” which is far more than the 1 Gbps available over the current ethernet maximum. “Only the last two were deemed to be feasible. We’re leaning towards object fusion as opposed to any kind of 3D point cloud.”

This suggests a fair bit of work could be implemented in the zone. But how much? One important consideration for the zonal controller would be cost, which could limit the amount of heavy ML hardware that could be included at an acceptable price point. That will be in important factor in partitioning the work.

“From an economic point of view, you cannot put your utmost AI performance into the zonal controller because the silicon area would be too large,” cautioned Schweiger. “So that means you need to have a mix.”

That said, the way the processing is partitioned is likely to remain proprietary as OEMs try to find the best combination of sensor fusion and ML in the zone and in the center. “How they distribute certain tasks in their vehicle may be the secret sauce of the OEM,” noted Schweiger.

Or put simply, no one size fits all. “It all depends on the manufacturer and their philosophy,” added Tom Wong, director of marketing, design IP at Cadence. “And the compute resources that they have available in the core and the amount of power they they’re willing to burn.”

A temporary question?
Still, there is a persistent belief that, given no limitations, the best decisions can be made by taking all raw data to the central computer for computing. By compressing the initial raw data, all of the pre-processing sacrifices some information in favor of reducing bandwidth. It may not be optimal, but it’s good enough for now, especially given that the vehicles will be bandwidth-limited for the time being.

The use of the zones for sensor fusion and ML would, in this view, be a transitional thing until we can more effectively bring all of the data to the center. “Right now, the companies are still developing very smart sensor modules,” said Schweiger. “But later on, that will change and will shift towards the central compute units.”

When this will happen is anyone’s guess, because even as we design faster networks, we’re also adding more data. That suggests the zone could host some of this fundamental computing for a long while to come.

If that transition does happen, it doesn’t necessarily signal the end of the zonal architecture, however. Many tasks unrelated to sensor fusion or ML can still be performed locally, and the zonal architecture provides an efficient and convenient way to do that.

“Maybe there are things where you do not need to have data fusion,” said Schweiger. “You just pre-process something, like beam-forming of a radar sensor. It’s an individual thing for this one sensor, and it’s not driving the decisions.”

So despite the move by OEMs to provide the best mix of sensor fusion and ML computing between the zone and the center, it will migrate to the center over time. That would argue against, for example, performing classification in a camera at the edge, and it might set up a tussle between what the camera makers feel they need for differentiation and what the OEMs want to do.

“Once the OEMs have sorted all this out, everything will be more scalable and easier to maintain,” noted Schweiger. “It can have a cleaner software architecture because you do not need to flash 100 ECUs.”

Kouthon concurred. “By having a network where you bring everything to the central computer, it’s a lot easier for to do over-the-air updates because you have only a couple of things to update every time,” he said.

For the time being, different OEMs are likely to draw the line between the zones and the center in different places. Flexibility will be important, whether that be in software or in programmable hardware. “We have heard arguments for both sides of the many questions around sensor fusion,” said Flex Logix’s Jaros. “We think the answer is to use eFPGAs to allow system designers to have one chip that supports both scenarios, where the fusion is done near the sensor or in a central processing unit.”

But this will be a slow process, given how vehicles are designed. “You’re going to see solutions like this available on the market around 2026,” said Fritz. Everyone will need to watch how this plays out and be ready to modify their strategies if necessary.