ADAS Adds Complexity To Automotive Sensor Fusion

Sensor fusion is becoming increasingly popular and more complex in automotive designs, integrating multiple types of sensors into a single chip or package and intelligently routing data to wherever it is needed.

The primary goal is to bring together information from cameras, radar, lidar, and other sensors in order to provide a detailed view of what’s happening inside and outside of a vehicle. In the newest designs, sensor fusion often occurs in zonal controllers or central compute modules. These devices act as intermediaries, collecting data from different sensors and transmitting it to a central processing unit. The exact location of sensor fusion, as well as which sensors are combined, can vary depending on the vehicle’s architecture and the manufacturer’s approach.

There are two main options for fusing together sensors in a vehicle. “In the zonal ECU, sensor fusion can be performed, which is really a smart switch,” Ron DiGiuseppe, automotive IP segment manager at Synopsys, said. “The other option is to do the sensor fusion on the central compute module, where the data extraction could occur in that central compute module.”

Which option to use is typically is based on cost. “If you do the fusion in the zonal ECU, that application is a chip — an SoC with multiple ports,” DiGiuseppe said. “It handles all kinds of different data types — CAN, MIPI, etc. They all go through that zonal ECU, and then get transferred to central processing so it can be done in the zonal ECU. But generally it’s passed through to the central compute, and that would have a high-performance automotive in-car Ethernet, as well as long channel MIPI ports. Most of the CAN would get converted in the ECU, and the CAN packets would go to the zonal ECU to get extracted and then encapsulated into Ethernet to be transferred back to the central compute.”

A key enabler for effective sensor fusion is the high-speed in-vehicle network, with automotive Ethernet being a common choice. Current production vehicles use 1 Gbps Ethernet, with 10 Gbps systems on the horizon. This high-speed network allows for rapid data transfer between sensors, zonal controllers, and the central processing unit.

The fusion process itself often involves specialized hardware and software. For example, unified DSPs (Digital Signal Processors) can handle data from different sensor types, such as radar, LIDAR, and cameras. Additional accelerators may be used for specific functions like Fast Fourier Transforms (FFT) or computer vision tasks. So while this is more complex than the original sensor fusion concept, but it’s also more mature.

“Like all new technologies there was a rush to a solution,” said David Fritz, vice president of hybrid and virtual systems at Siemens Digital Industries Software. “Almost by definition, every solution was new and differentiated. Today, sensor fusion is well understood. Frankly, we’re seeing smarter perception stacks that make sensor fusion less of a concern. Lidar, radar, camera, and inertia all interplay, but the trend with sensing is more intelligence at the sensor, which makes fusion much less of a concern.”

What gets combined varies by use case. Robert Schweiger, group director automotive solutions at Cadence, described one scenario in which a vision processor also processes radar data, as well as some AI functions. “I want to analyze the point cloud and identify, based on the points, what kind of object I have in front of me,” he said. “There are commercially available cores, with the instruction set of radar and vision cores integrated in a single core. That means it can do lidar, radar, and vision in this one single core.”

Or perhaps the application is a very demanding sensor system with the latest high-definition sensors that that deliver a ton of data to the processor. “You may have an accelerator on top of it to support a DSP and accelerate, for instance, FFT functions on top,” Schweiger noted. [If the processor] includes an AI base functionality up to 2 TOPS, that means with those two cores you have a very powerful system. Some people need more AI performance, so you need a neural processing unit, where one instance can be scaled up to 80 TOPS, and many of those instances that you can put together on a multi core system can be scaled up to hundreds of TOPS or a thousand TOPS.”

All these processors on a chip needs to be connected, as well, which requires a network-on-chip. “A NoC is needed to connect our different IPs into a subsystem, which eventually becomes a chiplet,” Schweiger said.

Then, for the sensor fusion, assume there is a sensor stack. “You have a camera radar, you have a 4D imaging radar for high resolution, you have a front camera, and you have another camera to the side or to the back,” Schweiger said. “These are the different sensor modalities, and now you want to fuse together everything. The corner radar maybe a low-end or mid-range type of radar, so it’s not so demanding, but the 4D imaging radar is a very demanding type of sensor. There’s lots of data, which means you need a lot of processing here. For the front camera, it’s the same thing. The smart camera in the front is normally the most powerful camera, and it is doing a lot of pedestrian detection, image analysis, filtering, and so forth. There may also be a low-end camera for the back. So how does the system look for the sensor fusion? [Our approach is a] unified DSP that can do radar, lidar and baseline AI.”

The same instance used for the radar also can be used for the camera, whether it’s the front camera or the back camera. “There, we process vision time of flight and AI,” he noted. “From the upper DSP, we generate a radar PCL (point cloud), and generate a vision point cloud for the lower part, so we can fuse the front camera and maybe side camera together, or corner radar and front radar. We fuse it together in the core. If we need more performance, we add FFT as an accelerator, or another accelerator that can accelerate FFT functions, as well as computer vision functions.”

Smarter sensors
Increasingly, this will include machine learning, as well. “The types of sensors being used to achieve the granularity of control for an autonomous platform or a heavily driver-assisted function, and the types of data they’re using, are inherently more complex than traditional sensor measurements that you might find in an adjacent sector like the process industry or the medical field,” explained Andrew Johnston, director of quality, functional safety and cybersecurity at Imagination Technologies. “Those sensors might still — and have done — sensor fusion for decades. They’ve relied less on machine learning, and arguably that’s still evolving, but the types of transducers they use are inherently simpler. So you might end up with simple look-up tables and cross-reference algorithms.”

In the automotive landscape, the big change comes down to visual perception and making structure from motion. “To do that well, you need an array of different sensor types,” Johnston said. “You can functionally deliver autonomy with single sensor types, but with mixed results. And because these are high-integrity functions, i.e., safety-critical, to employ levels of redundancy and diversity is a good thing. At the system level, you want to fuse sensors from camera, radar, lidar, ultrasound, and then even couple that to more traditional sensors, like light sensors or temperature sensors. You’re doing that to try and understand what the operational environment is. The problem is that’s an inherently complex functionality and technology space, and at the semiconductor level, it’s even more challenging. In an ideal world, every semiconductor supplier wants to try and build one product that can satisfy a number of use cases. We try not to make use-case specific products because that will have a very niche application, and customers may still not choose your product. So the balancing act that IP and semiconductor providers have is trying to assess use cases systematically and understand what could be done in the hardware software space before they figure out what the transistors on a chip need to do.”

What this means in the automotive space is a lot of machine learning, and a lot of high-bandwidth requirements on machine learning. “You want to execute these algorithms in real-time, and what that means is you’re doing a lot of different types of complex matrix multiplications and data fusion, and you ideally want to do that on a singularity and to do that quickly,” he said. “The reason why it’s doing this quickly is it has to make control-system-based decisions, just like a human driver would in the loop. We do it naturally, and we are analog, so you’re trying to represent an analog world and an analog controller in a digital domain. It’s quite an interesting challenge philosophically.”

Transition to SDV, zonal architectures
As the automotive industry transitions toward software-defined vehicles and zonal architectures, the approach to sensor fusion is evolving. OEMs are working to integrate these new systems with their existing architectures, which presents significant challenges. The goal is to create a more centralized and scalable system that can be easily updated and expanded as technology advances.

It’s worth noting that the specific implementation of sensor fusion can vary significantly between manufacturers, with some opting for more centralized approaches while others distribute processing across multiple nodes in the vehicle.

“Today, sensor fusion is the process of combining data from multiple sensors to create a more comprehensive view of the environment, and typically not based on generative AI and deep machine learning,” said Ted Karlin, senior director for marketing and applications sensing of compute and connectivity products at Infineon Technologies. “In the future, more applications using generative AI and deep machine learning with sensor fusion algorithms will become significantly more intelligent. With generative AI, the sensor fusion process will become more adaptive as these models can synthetically fill in data gaps, simulate potential outcomes, and therefore predict and react to changes in real time. They also will have greater contextual awareness, as historical context can be a part of the algorithm, along with environmental factors and situational awareness for higher-quality sensor fusion outputs. Lastly, machine learning can become more personalized as generative AI could utilize a specific person’s monitoring or diagnostics to drive personalized conclusions.”

Others agree. “The power of fusing imaging radar with camera is well understood,” said Adiel Bahrouch, director of business development for silicon IP at Rambus. “Radar, for example, is not able to read signs, and you cannot train it to read signs. It cannot see different colors, red, color, blue color, while the camera does. You can train cameras with AI to recognize signs, people, objects, lanes and so forth. When you combine those characteristics with all the good things that radar can bring to the table, you can have a very powerful system which can easily beat lidar. Lidar is yet another technology which has a lot of good things, but very, very expensive. For very high autonomous levels, I don’t see that the camera can do the job alone since the camera, in dark or very poor weather conditions, can introduce some safety issues. Radar doesn’t have the capabilities of camera in terms of resolution, image detection, and pattern recognition, but when you combine those two, you have a very powerful system that can bring autonomous driving forward.”

As far as where this resides in the vehicle, Bahrouch points to the evolution of the E/E architecture as a starting point. “The traditional vehicles have their own domain-based architectures where all the ECUs and all the sensors are connected, which means the fusion is a big challenge. That architecture doesn’t support those types of activities. But with the shift toward zonal based architectures, where sensors that are located in the same corner are combined under one big processor, the zonal processor helps to gather all the information from different sensors to start doing the fusion. That means the information is processed locally, so it’s zonal, or through the high-speed Ethernet backbone that collects all the information from different angles and starts doing the fusion centralized in the center of the vehicle. I don’t know which one is going to be the mainstream approach, whether it’s going to be at the edge or on the zonal SoC, or the brain, or a combination of those. Will it do pre-processing before all the information is combined? This is an area that is now evolving. It’s quite a new area, so not all the stakeholders are disclosing their strategy.”

There is widespread agreement, however, that the continued integration of lidar, radar, and camera sensors will make vehicles safer. “Advanced sensors will make a significant impact on ADAS solutions by providing more accurate data and improving safety maneuvers, from lane-keep-assist to auto parking and braking,” said Wayne Lyons, senior director of marketing for AMD’s Automotive Segment. “As the number and different types of sensors within the vehicle, such as cameras, radar and lidar, continues to rise each year, pioneering companies like Waymo have already logged millions of autonomous miles in vehicles that leverage all three technologies. In addition, emerging EV companies in China are leveraging advanced sensors like lidar to differentiate their safety offering in the fiercely competitive EV market. To achieve the real-time performance required so the various sensor data can be used in real-world driving situations, developers will need a flexible architecture that can provide the necessary performance with functional safety, all on a single chip.”

4D radar
The move to 4D radar will further increase the amount of sensor data to be fused, Synopsys’ DiGiuseppe said. “The transition to 4D radar is increasing the amount of DSPs, and it’s also being channelized. Instead of just one-reflection data, you can have channelized data, so you get multiple radar reflections using virtualized channels. That provides a more extensive data set, so it’s no longer just a single radar bounce-back. Since it’s channelized, it’s closer to the lidar approach. In lidar technology, you get a point cloud, which typically gives you much more data. That’s one of its advantages over radar. Lidar gives you higher resolution. Plus, you have 128- and 256-channel lidar. Radar is starting to add channelization, too, so the radar data is getting higher resolution, and that allows you to identify objects using a radar data set, just like a camera image.”

AI processors are used here to identify the image. “It’s a dog, it’s a tree, it’s a person,” DiGiuseppe said. “Radars are starting to be able to give you that capability, as well, although certainly not as high-resolution as a camera system because it’s a different problem to solve. While cameras do have some limitations, like at night, under darkness, or in rain or fog or heavy snow, and they still have gaps in terms of object detection identification. Radar now provides these data sets that are a good complement to cameras under those conditions where cameras are not ideal, so the amount of information that is being extracted from these 4D radars is much more useful for ADAS systems. You can do object detection using radar now, which is new.

Conclusion
While sensor fusion technology has advanced significantly, its implementation in automotive systems continues to evolve. The trend is toward more intelligent, distributed processing, along with standardized communication of object data. These concepts seem clear, although implementation is still evolving. Nevertheless, as vehicles in development today get closer to hitting the road, further details should emerge from the OEMs as to the direction this technology is heading.