Auto Industry Shifts Gears On Where Data Gets Processed

In-vehicle processing is becoming a major challenge in automotive electronics due to the massive amount of data being generated by sensors — especially cameras — and the rapid response time required to avoid accidents.

The initial idea that all data could be sent to the cloud for processing has been shelved, most likely permanently. In its place is a growing recognition that data needs to be processed locally, inside a vehicle, and that the entire communications infrastructure has to be rethought in order to manage and react to that data. This has put a big damper on aggressive autonomous vehicle development schedules, and it has raised many questions for which there are still no obvious answers, such as how to build a high-speed network on wheels with sufficient processing power that is still affordable for consumers.

“Considering that an autonomous vehicle produces as much data in one day as the Hubble Space Telescope does in one year, managing that sheer amount of data between on-prem (the vehicle) and the cloud is a tall order with today’s capabilities, let alone processing it,” said Simon Rance, head of marketing at ClioSoft. “A lot of industry and technology collaboration will be required to tackle these challenges before we truly see fully-autonomous driving everywhere.”

There are four areas to consider with data processing for autonomous driving — data acquisition from the sensors, data storage, data management, and data labeling.

5G proponents argued that a cellular network based on millimeter wave technology could sidestep many of these issues. But reality looks very different than the original plan. Assuming it could be built, the cost of building a parallel communication system would be enormous. On top of that, the mmWave version of 5G is not considered reliable enough to be used for safety-critical applications without a vast infrastructure of repeaters, and even then signals are easily disrupted. Unlike 4G LTE and sub-6GHz 5G, mmWave signals fall off quickly and can be interrupted by anything from objects to weather.

Realistically, 5G cannot address all upload-download needs for autonomous driving without local processing. Even with more computing power in vehicles, there is a critical need for security; computations and communications need to be secured and relied upon. First, vehicle components need to be authentic so car manufacturers can rely upon their safe functionality; this means that components need to be genuine and guaranteed to have been manufactured exactly how the component maker desired. All of these can be guaranteed by secure provisioning services implemented directly at component manufacturing sites, according to Helena Handschuh, a Rambus Fellow.

Also, when the automaker is assured that components are genuine, they must ensure the components are designed into the system in the correct way, booted in a secure way, and used to establish a secure computing and communication environment, she said. “This can be achieved by employing components that contain a hardware root of trust such as a programmable root of trust. Roots of trust are based on immutability principles due to the nature of their hardware implementation layers and can guarantee processes such as secure boot. Rambus believes each vehicle gateway should incorporate a hardware root of trust. With regards to secure computations and communications, cryptographic primitives such as authentication and encryption cores are required to make sure no unauthorized parties can access confidential data, substitute fake data and/or change results of computations. Secure cores are critical for this requirement.” Rambus’ root of trust contains these cryptographic primitives, and also serves as a secure endpoint for communication establishment with the cloud. Additionally, a transport layer security (TLS) endpoint can be hosted by the root of trust.

“After securing the computation environment and the communication channel, the system has to provide guarantees that on-board machine learning and edge computing processes are secured. “Put simply, the sensitive parameters of the neural networks that the on-board computing elements use, need to be secured. A hardware root of trust approach allows the car manufacturer to secure both data at rest and during computation, including the sensitive inputs to neural nets and outputs, which the system relies on, to make safety decisions. Finally, a root of trust also protects the neural network by encrypting its sensitive coefficients, resulting in secure inference on-board the vehicle itself,” Handschuh added.

“In addition to bandwidth concerns, any wireless communication technology is subject to interference, dropout, congestion, and delay,” said Sergio Marchese, technical marketing manager at OneSpin Solutions. “Autonomous vehicles will have to carry enough compute power to make decisions on their own. To that end, we have to look at what companies like Tesla are doing. Just this past year, they announced their Full Self-Driving Chip, which allegedly can process 2,300 frames per second. It is in production and can be even retrofitted on previous models.”

But autonomous technology still cannot guide a car through a city, and it cannot always interpret all of the potential interactions between human- and non-human driven cars. That would require much more compute power, and no one is willing to pay for a supercomputer in the trunk of a car.

“People are the biggest challenge,” said K. Charles Janac, chairman and CEO of Arteris IP. “They’re unpredictable. This is why you won’t see autonomous vehicles in an inner city for a long time unless there is some geofencing involved to keep people and cars separate.”

Nevertheless, there are improvements that can be made for more in-vehicle processing.

The next-gen vehicle design platform
A variety of different ideas are being discussed about what the next-generation vehicle platform should look like. While each is unique in design, they tend to coalesce around some common approaches.

“All of the OEMs are willing to take this on,” said David Fritz, senior autonomous vehicle SoC leader at Mentor, a Siemens Business. “They’re coming to the realization that the next-generation platform is essentially a high-bandwidth, high-connectivity, high-compute network on wheels because they’re moving to as much automotive Ethernet as they can. They’re implementing CAN packets in zone controllers/zone domains with the idea that the communication between a decision that’s made and the actuation to occur, whether it is braking, steering, whatever, is extremely time-sensitive, especially at higher speeds. So just having the decision being made in the cloud and sending it back down never seemed to be something that was ever really going to be feasible.”

Communication is such a significant feature in vehicle design that 5G will play some role in the automotive space as vehicles communicate internally, to other vehicles, and to infrastructure.

Tom Wong, director of marketing for design IP at Cadence, noted that a lot of companies finally realize that full self-driving is extremely complicated, and they are now rethinking how to best use their dollars.

Broadly speaking, there are a number of questions about how 5G will fit into automotive, starting with the carriers. It’s still not clear what adoption and deployment will look like, and which version will roll out in what time frame. It’s also not clear how 5G actually will be used for autonomous driving V2X applications. Will it be based on DSRC (direct short range communications) or cellular V2X (as in a variant of 5G)? And what roles do different special interest groups play in influencing this decision?

“If the decision goes to DSRC (basically 802.11p), then there is probably no recurring revenue stream for valued-added services,” Wong said. “Chip vendors will sell a lot of chips. If the decision goes to cellular V2X, then the carriers will be very happy because they now have a new and recurring revenue stream, which could be like a tax on autonomous vehicles, even though they will have to invest in the infrastructure by leveraging deployment of 5G for smartphones.”

Other considerations around data include how it is gathered or generated, what that data will be used for, and whether the data is applied to AI training in the cloud. “Are we really talking about performing AI inference in the cloud and transmitting the path-finding results via 5G? I’m not sure there is a consensus on which approach works best,” he continued.

Regardless, we are at the beginning of the wave on a number of fronts including 5G deployment, mmWave 5G or sub-6GHz 5G, autonomous driving (data processing) in the cloud or at the edge, and DSRC versus 5G for V2X, among others, Wong added.

Another aspect of data management for autonomous vehicles is the fact that manufacturers must have a large fleet in place and a system to query data, said OneSpin’s Marchese. “For example, Tesla can ask for images of driving conditions where it is raining and get lots of those images to build the knowledge necessary for their vehicles to deal with rain situations. Data is as crucial as the hardware to deliver safe vehicles. Hardware and data go hand in hand in making sure autonomous driving is safe. For hardware, meeting the industry’s ISO 26262 standard is critical. In order for automotive chip makers to compete at this level, many aspects of safety must be assured. This includes failure modes, effects, and diagnostic analysis (FMEDA), which must done quickly and accurately. Manual analysis and fault simulation are too slow and costly, and the ramp-up time is much too high for any company wanting to disrupt the market and bring groundbreaking innovation that can be proven on the roads.”

Shifting the OEM core competence
Despite all of these uncertainties, most experts consider this a huge opportunity for the entire automotive supply chain. The big unknown is who is best positioned to lead the charge.

“The core competence you need these days in automotive is not mechanical engineering anymore,” said Burkhard Huhnke, vice president of automotive at Synopsys. “We should not forget that it’s always hard to manufacture and mass produce cars in high volume. Volkswagen alone has more than 100 factories worldwide, and they all have to run perfectly. That, of course, requires the competence in engineering. But today, the expectations from the customer side in regard to functions and features in the cars are more related to the consumer industry. Everybody is used to the smartphone, and now you want to get this user experience also into a car. That requires a rethinking of the competence of OEMs. They need to shift left, and that means a shift into the core competence of the hardware and software stack. Google and other disruptors are really great at that, but the traditional car manufacturers are not. The increasingly complicated architecture of software and electronics is the major challenge. That requires a cultural shift, and an organizational shift to be future-ready. This is the big challenge, and it requires an investment in new tools.”

There is evidence that this can work, but the magnitude of this shift is enormous. Along with understanding semiconductors, software, and integrated electronic systems, there also are multiple clusters of data that need to be managed.

“Number one is consumer data, such as navigation data, which is usually acquired through a smartphone,” Huhnke said. “That has nothing to do with autonomous driving, but it’s the convenience to get more data, on time, into the car. This functionality itself requires a different way of communication interfaces.”

The enormous amount of data being generated just by camera systems — including HD cameras that run at 60 frames per second and provide an HD bitmap or metrics into the data system — requires an equally enormous amount of processing power.

“On top of this, there may be multiple HD cameras in the car,” he said. “It’s completely different. You must all of a sudden increase the data to 100 or 1,000 times more than before. That requires different bandwidth, different communication channels, different protocols, and different IP to handle all this data. On top of that, everything has to be secure and safe, so additionally you need intelligence to ensure that the data is handled correctly and cannot be manipulated. Then there’s a huge debate about what the data from the sensor is going to look like. Do we require radar and cameras and LiDAR? The debate is not over.”

He’s not alone in seeing that. “There are eight cameras in use in high-end vehicles, as well as four radars,” said ArterisIP’s Janac. “That’s too many, and we’re starting to see developments around that. One company, Vayyar, says its radar can see stationary objects, including plastic and wood. But then you have LiDAR, as well. Is that necessary? Probably not. Ultimately, you need one kind of sensor. Whether LiDAR improves to the point where it can do everything, or whether radar does that isn’t clear. But they won’t both exist in a car.”

Who wins?
It’s also not clear what architecture or approach will work best in the short term and the long term.

“The best solution should not be the low-cost solution or the best-performing solutions that include every available sensor,” Huhnke said. “The key is taking the data stream from the sensors, and then the sensor fusion, and figuring out how you get as much from data generated by the different sensors as possible. You have to fuse all the information and calculate out of multiple sensors specific objects, and then you have to react. This is calculating power based on the data stream that you get. That is hugely different compared to the traditional car, which has one serial, one radar sensor, and maybe a couple of ultrasonic sensors, which in the past didn’t have to communicate to each other because they were all created for their specific task.”

That takes significant domain expertise and engineering skill to replace.

“When you really get down to it, what we’re finding is that if you want to bring out a new, next-generation automotive platform capable of these types of bandwidth and compute requirements to handle any level of autonomy greater than, say, Level 2, first of all it’s a complete rework of the design,” said Mentor’s Fritz. “Second, you need to know exactly what your design is going to be now because it takes five or six years have those design changes intersect your model year roadmap. The real question for the OEMs is, ‘How do I decide this now? How do I know what my compute requirements are going to be? I don’t want to hold a rack of these things in the trunk of the car. So how do I design this?'”

Many believe the way to do this is the way we’ve been doing it for smartphones for a long time — you model it.

“You model at high fidelity. You get the empirical data, then you can say, ‘Okay, our PPA is now going to do this. We’re confident it’s going to cover this whole spectrum of applications that we’ve envisioned. We know how big the printed circuit boards are going to be. We know how much heat is going to be generated, and we know how to cool that. We know what the 3D enclosure measurements are so we can tell the chassis team for 2025 or 2026, this is the size of the box. One needs to be placed front left, one needs to be rear right for redundancy.’ It’s based on engineering technique. It’s based on data, and it isn’t just running a bunch of models on a PC and saying it should work because, as we know, it often does not,” Fritz said.

While individual data acquisition, analysis, capture and management have been done for quite some time, the real question is how to bring it all together.

“That next-generation platform design looks nothing like today’s design,” he said. “It’s not just a conglomeration of ADAS ECUs. It’s not that. It’s something designed to address this new class of problems, which we can now define, and through simulation, collect the information on it. Not only that, the methodology of implementing that next generation platform is nothing like what’s being used today. That’s what’s really throwing off a lot of automotive companies. How do you synthesize an AI inferencing machine from nothing but a high-level spec? You can’t do it. How do you actually synthesize a multi-cluster, multi-core PCB so that it can handle braking, steering and IVI when it’s not busy? You can’t. That means there’s a whole new skill set that has to be involved there. But there are resources out there that have been doing it for decades, and they’re available. And that’s why you’re seeing announcements like the one from Toyota and Denso. They’re going to go off and do their own IP company for automotive.”

Elsewhere, companies such as Volkswagen are bringing all the software in house. “They are realizing they have to do things differently,” he said. “You can’t incrementally get from where they are to where they need to be if they go down that incremental path. Then it’s going to end up taking longer, cost more, and they’re going to miss the window.”

Huhnke agreed a new architecture is needed. “The architecture in the past has been based on, say, 100 ECUs, distributed all over the car. Each ECU has a very specific role and functions to fulfill. And in the meantime, you have to consolidate those little computers into more centralized computers because it requires different bandwidth communication speed, and at the same time calculating power to be able to calculate what’s in this data and information. Additionally, the role of accelerators — like neural networks to run statistical calculations — require very specific customizable solutions, as well. But those are actually available and are being implemented these days now straight into the SoC.”

To fulfill the requirements for fully self-driving cars, top players with the electronics and automotive industries are publicly discussing the path forward. “With the next generation, everybody realizes that the compute power available today is not sufficient to calculate all the multiple sensor inputs that are required. At the same time, the sensors are getting better and better, but that comes with high resolution and even more data,” he said.

Conclusion
There still is not consensus on the specifics of system-level vehicle architecture or how in-vehicle data processing will be done. But there does seem to be agreement that, generation by generation, this will be an evolutionary path toward a high-performance solution.

“Step by step, function by function — that’s what we will see over the next decade,” Huhnke said. “Functions will be launched, but always based on the existing electronics. Imagine you have to design a chip today. This is going to be in the car maybe in 4.5 years, if you refer to the development process, because they make a decision to integrate current chip design. At some point you have to freeze the design. In the next model year, you take the next generation of chipsets. So there is only consensus that nobody has the answer.”

The path forward will require new tools, more tools, faster and smaller devices, and higher levels of integration. And it all has to be done for a low cost using low power. But even with the latest technology, it’s an enormous challenge.

Related Stories
5 Major Shifts In Automotive
How new technology developments will change the trajectory of the automotive industry.
Betting On Hydrogen-Powered Cars
Hydrogen fuel cells will play a role in the transportation ecosystem of the future, but how large a role hinges on infrastructure and application-specific demand.
Preparing For The Great Auto War
The age of the internal combustion engine is ending. Now what?
Planning For Failures In Automotive
Is it better to build expensive parts that are highly reliable, or redundant cheaper parts?
Challenges To Building Level 5 Automotive Chips
The challenges to build a single chip to handle future autonomous functions of a vehicle span many areas across the design process.
Autonomous Vehicles Are Reshaping The Tech World
Even before fully autonomous vehicles blanket the road there is major upheaval at all levels of the industry.
Safety Critical Design In Automotive
Finding faults at the chip and system level.
Automotive System Design
How to build and update chips in cars.
Building An Efficient Inferencing Engine In A Car
How to model a chip quickly, including corner cases.
Automotive Knowledge Center