Gaps Emerge In Test Flows

Gaps are showing up in test flows as chipmakers add more analog content and push into more safety-critical applications, exposing more points at which designs need to be tested as well as weaknesses in current tools and methodologies.

The cornerstone of the IoT, and connected devices such as self-driving cars, is a heavy reliance on sensors generating data that is processed in digital logic. To get there, however, requires more third-party IP content, complex power management schemes, and distributed processing architectures that rely on more processing elements and approaches such as heterogeneous cache coherency. As a result, it’s no longer possible to put a probe at one end of a circuit and give a pass/fail rating depending on whether the signal comes out the other end.

Even defining coverage for what needs to be tested has become problematic. Some of those requirements are still being developed, particularly in safety-critical markets such as automotive, medical and aerospace, where testing must be done in context of various modes of operation, under different physical conditions, and in adherence to standards that are still evolving.

“We looked at the early stages of design verification through lab characterization and production test, and we found there are significant gaps between those worlds,” said George Zafiropoulos, vice president of solutions marketing at National Instruments. “There is a significant challenge in re-creating test cases that were created pre-silicon and in using that intellectual property post-silicon. And once things are up and running in the lab and you go to production test, there is significant rework to bring up the test cases for production. It’s also difficult to reproduce failures.”

While most EDA vendors tend to focus from initial concept through manufacturing, chipmakers increasingly look at test from initial concept to post-silicon, where they can assess in-the-field failures and prevent problems in future revisions.

“The real challenge is that pre-silicon testbenches tend to be too ‘ideal’ in all worlds” said Marvin Landrum, validation engineering manager at Texas Instruments. “There are unlimited sources and syncs, they may or may not include the package, and it definitely does not include the post-silicon hardware world. In addition to that, mixed-signal stimulus and response can be a challenge. Pre-silicon testbenches are waveforms, and post-silicon testbenches deal with abstracted instruments.

Landrum said the design and and test engineering communities need to come together in much more standards-driven ways. “If testbenches are apples and oranges, then tools will not be able to help.”

That sentiment is being echoed throughout the chip world. Eike Ruttkowski, head of RF system verification for Intel’s RF product development group, said a much more holistic approach is required. “For us, a central, single-source specification is one of the key items that needs to be shared, from the design phase with pre-silicon verification—whether that is virtual prototyping or RTL simulation—and then go onto hardware, where we also want to test. One of the big challenges is combining pre- and post-silicon and maintaining the abstraction layer. We want to test for both worlds.”

Defining the problem
These issues have been brewing for some time, but they have really come to a head over the past 18 months with the explosive growth of automotive electronics, which numerous industry executives describe as the first real IoT implementation.

“ISO 26262 is taking hold among tier-one vendors, and they are pushing hard for coverage metrics for the whole chip, including the mixed-signal portion,” said Steve Pateras, product marketing director at Mentor Graphics. “At this point there is nothing out there. Right now we are trying to understand the metrics. The next step will be to provide certified coverage. We need to develop the ability around fault defect coverage in mixed signal that is the same as what we have in the digital world. With the IoT, there is potentially a lot of liability.”

That isn’t going to be easy, however. “There is a lot of complexity in the industry already,” said Michael Thompson, RF solutions architect at Cadence. “It gets even more complex when you move to the car with radar, Lidar, and imaging systems, all combined in this giant case rolling down the road. Behind the engine, the wiring harness is the second heaviest part of the car. You’re going to be looking at all sorts of wireless technologies, whether that’s Bluetooth or 5G wireless LANs, which will be handling these systems. So when you think about design for test, you have to start thinking about how you’re going to test all of these things working together, communicating with other vehicles, and dealing with other companies that may not be designing to the same standards.”

There are even questions about whether the test equipment and the software are using the same algorithms, said Thompson. “If they’re not the same, that’s one additional source of error.”

And there are questions about how to deal with all of the data being produced throughout the design through manufacturing process. As complexity increases, so does the amount of data being produced. In that past, that data was generally confined to individual steps within the flow, but with the great shift left, more of that data is required in places it was never utilized before.

“You need access to data to figure out what’s going on and how to optimize it, and then feed it back into this giant process,” said David Park, vice president of worldwide marketing for Optimal+. “The main thing is you want to collect all of it. And if you can simplify the test process, that would be better. Historically, the only reason you did the test process was to make sure a part worked. It’s now possible to use that same data to optimize the process. There is an enormous amount of data that will be available not just from the IC and package space and process data. There is performance, use, test, test rework and reliability data. And all of this can be in the same repository.”

Beware of the moving parts
Not all of that data gets passed along in predictable ways. This is partly due to the fact that everyone in the semiconductor supply chain is so busy trying to solve problems in narrow areas that it’s hard to get a good understanding of all of the different pieces. One of the key developments in the design world over the past couple of decades has been the “design and conquer” approach, whereby engineering problems are broken up into smaller pieces and then reassembled later in the flow.

But there are so many pieces interacting these days that pulling them all back together is becoming problematic. And that’s becoming increasingly true on the test side, where reuse and efficiency—particularly in mixed-signal designs—have never been a requirement.

“People interpret specifications for their own purposes and do not always test a device in the same way,” said NI’s Zafiropoulos. “It’s also very inefficient, because the tests that are created early on are not always used later in the flow. And then you have homegrown versions, which makes it difficult to compare results.”

Those homegrown pieces have come into existence to a large degree because chipmakers have had to fill in the blanks in the test flow by themselves. That certainly has allowed some very complex chips to get built, but it’s hard to re-use that technology or to integrate it with other tools.

“If we don’t get some alignment around the flow and better standard, then the knowledge breaks down,” Zafiropoulos said. “Virtually every semiconductor company I’ve seen has a methodology internally that includes some homegrown tools and some third-party tools. But the problem is becoming so enormous that it’s overwhelming their process.”

Cadence’s Thompson agrees. “Part of the problem is that all of the elements can be moving within the design flow within a day or two of them going to tape-out. Frequently you will find they’ve built something at the last minute, and no one knows what they’ve actually built. Just being able to verify that and ensuring they are building what they should be building is a first step before you even get to the testing.

An efficient test flow needs to incorporate many moving parts inside of groups that so far have been insulated from each other, from embedded software to IP integration to verification and manufacturing process. While this is a good idea, it’s tough to achieve regardless of whether a chip is being developed by an IDM or a fabless semiconductor company.

Interim testing strategies
What companies view as the biggest challenges, and how they deal with them today, highlight the complexity of the problem as well as what they view as the solutions. And this becomes part of the challenge in developing a consistent test flow across companies. Challenges are complex, but they also vary from one company to the next.

TI’s Landrum pointed to a single executable spec as the biggest missing piece today. “We’re centering on something that we call a shared compliance matrix. That results in a set of requirements for different domains. We collect all of those requirements on spreadsheet-based tools. The output of that process is shared across the company. As a replacement for that spec, we align our cross-discipline matrix. That’s our interpretation of the spec.”

Eike said Intel is facing the same problem. “Once you come close to having that spec, though, you’re generating a whole lot of data. You always want to have the comparative versus the spec so that you can make generational correlations. So we need to employ big data analytics to tackle these gigabytes of data. That’s one of the key problems.”

Data overload is a recurring theme in the test world. And while there are big data techniques to manage some of this, much more automation will be required to deal with IoT systems. “The biggest challenge is being able to do machine learning,” said Optimal’s Park. “Some of these companies are capturing terabytes of data. It’s too much to analyze by hand. You may have 10,000 parameters connected to 10,000 other parameters.”

Working in silos
What’s clear is that test is a multi-faceted, multi-disciplinary problem that spans from initial design through manufacturing. Engineering groups typically do not have that kind of reach, however, and individual engineers generally do not have that kind of training.

“When you look at yield, you need to take into account diagnostic locations and calibrate them with yield,” said Mentor’s Pateras. “The question in analog is whether you can tolerate a deficit. These kind of gaps have to be solved within the next year, though. We don’t have a choice. Ultimately, it will be a prerequisite for any design in automotive and other markets, but right now there is no infrastructure or standards there.”

In effect, this comes down to what is considered good enough, and particularly in mixed-signal designs that’s a big unknown at this point. But even if these problems could be solved, who would be in charge of solving them?

“We talk about throwing it over the wall for test, but that’s not the greatest scheme,” said Park. “But finding companies that are willing to deal with this is like looking for a needle in a haystack.”

Zafiropoulos said he met with the vice presidents of engineering and test at one company in Japan. The vice president of engineering never heard of any the test vendors, and the vice president of test never heard of any of the engineering vendors. “Neither one could care less about each other’s problem. But the new world order will be that to get a product out the door, someone will have to own that. The companies that are enlightened about that will be successful. These separate silos can’t survive.”

Making those changes isn’t easy, though.

“No matter who you make responsible for the whole chain or concept or design through verification, your structure, methodologies and tools have to allow this person to exist,” said Intel’s Ruttkowski. “The problems are getting so complex that there’s a good reason people roll back into their silos. It’s hard to grasp and hard to handle. Moving forward, it’s only possible to remove those boundaries if we have the processes and the tools in place to give whoever is responsible for the test in the end the ability to do that job.”

Related Stories
New Drivers For Test
Pressure is mounting to reduce test costs, while automotive is demanding more ability for circuits to test themselves. Could this unsettle existing design for test solutions?
Tech Talk: ISO26262
What can go wrong in designing to this automotive standard.
Are Chips Getting More Reliable?
Maybe, but metrics are murky for new designs and new technology, and there are more unknowns than ever.