Applying Lessons Of Mass Production To Verification

I’ve recently been experiencing that time-honored tradition of helping an elderly family member as they go through one surgery after another attempting to restore worn-out, miscellaneous body parts. What’s most surprising, beyond the costs, is that shopping for a knee or disc replacement is much like shopping for a car. Do you go for the high-performance knee, which maybe hasn’t been tested on many patients (and is priced like a hand-built Italian sports car)? Or, are you more conservative, and go for the tried and true daily cruiser knee, which has been produced in massive quantities, and like a Toyota, is priced reasonably and isn’t as likely to be recalled? Nobody wants to experience a medical device recall – especially when getting it serviced is even less pleasant than a visit to the car dealer (yes – it is possible).

With cars, it is clear that the more cars of a given model are produced and delivered, the more likely that all the bugs will be worked out and you’ll be able to drive home without having its electronics shut the car down at random. With verification, the more a design is tested the less likely a bug will be missed. Unfortunately, knowing when verification is ‘done’ is always a challenge and the final decision isn’t as black and white as you would hope. Additionally, with signoff criteria, sometimes when striving to hit every single milestone we forget the grey areas.

Code coverage, for example, is a common signoff criteria that comes in various forms (line coverage, branch coverage, toggle coverage, etc.) and yet in the end it is often viewed as a checklist item. The same can be true for functional coverage as well. The grey areas come into play when we look at the details behind those checkmarks. With line coverage, for example, one can compare the number of hits for various lines. Are they all hit about the same? Obviously that’s not likely to be true. Some state machine states, for example, may only get hit a few times relative to other parts of the design, and therefore other lines of RTL. That may be okay – depending on the state machine. The question is: How do we know which lines are more important than others? And can we find out early enough to do something about it?

Fortunately tools such as Atrenta’s SpyGlass and BugScope provide intelligent static analysis that can help identify the most important lines of RTL to test. These products rank various components of a design by their relative complexity. Remember, bugs are most likely to be hiding in the most complex parts of the design. With this knowledge, we can analyze test results to see if additional focus is warranted and where.

This is when timing comes into play. While many simulators can go beyond simply checking the box and actually showing how often each cover point (line, branch, node, etc.) is exercised, this can come at a great cost to performance. As a result, this is typically done at the very end of the verification cycle (if at all), when there’s little time to benefit from the results. This is where BugScope fits in. With BugScope, engineers can track relative coverage throughout the project. This enables engineers to fine tune their test suites – taking a simple check box and turning it into a means to intelligently balance verification focus. Ultimately, this means that we end up with a more thoroughly verified design.

With embedded electronic design becoming a facet of every aspect of daily life, verification quality is ever more critical. If the trend holds true, a few years from now, a lot of our designs will be used by the medical field – taking biotech from simple knee replacements to whole limb replacements. Imagine the day where interrupting your mom’s breakfast to demand she serve you first, and then complaining that the eggs are cold, would uncover a state machine sequence coverage hole that caused all of her implanted bionic replacements (now interconnected thanks to IoT) to take control and turn her into an uncontrollable Terminator. This may sound like something for the distant future, but I’m pretty sure I experienced this firsthand last Saturday.