Using Software Approaches In Hardware Verification

Agile methodologies, created to improve quality in software code, increasingly are being applied to hardware verification.

This is less of a drastic shift than it might first appear. Developing a verification testbench is largely software, and similar methodologies can be used for reducing bugs in hardware.

“A testbench is nothing more than a big software project, and it makes perfect sense to adopt the Agile development process here,” said Harry Foster, chief verification scientist at Mentor, a Siemens business. “[SystemVerilog] took Verilog and augmented it with new object-oriented constructs, which facilitated modern code that has to be integrated and developed into the testbench. Prior to that, we created tests—and still do—out of C, C++, and SystemC.”

It’s not quite that simple with Agile development methods, though. “There are a lot of hardware groups that want to adopt Agile into hardware development, but Agile was never initially thought about in terms of hardware, so there are differences,” Foster said.

To understand why, it helps to keep the key constructs of Agile development in mind:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

“The hardware development process involves a lot of tools and processes, and this is done to ensure repeatability and quality,” Foster said. “As such, that first key aspect of Agile, individuals and interactions over processes and tools, becomes a problem. Software development has never really had that much repeatability. I won’t be able to push a button and get the same result. It’s more of a creative process, and so there are fewer tools for software development and less formalized processes than hardware development.”

Even so, a number of projects have incorporated aspects of the Agile development process in the hardware world, including military, and high-end servers. “I know of a few hardware projects that have adopted the scrum Agile framework, and specifically sprints, so that when they’re creating their RTL code they go off and do little sprints to add little features at a time,” he said. “That’s been pretty successful. The other thing that has really been successfully adopted is continuous integration.”

Although continuous integration has seen some success in the hardware world, there are still issues. Specifically, before continuous integration, engineering teams would develop the RTL, then synchronize multiple check-ins, so there would be multiple IPs at the end of the day or every couple days. Then the integration nightmare begins because some things that are integrated break other things, and the engineering team often has no idea why.

With continuous integration, when when something is checked in, it goes through regression testing. If it doesn’t work, other people can continue working, which simplifies the debug process.

“[Continuous integration] is a natural fit with Agile,” said Larry Melling, product management director in the System & Verification Group of Cadence. “As you are adjusting and making these changes, what really fuels the ability to do that and not lose control is a continuous integration, where there is a very solid testing and regression verification approach. As changes are happening, you can quickly validate to see that you’re not breaking things, or taking the design to an unstable place. If you make too many changes without verifying along the way, you can really take yourself to a place of instability and dig a hole that you have a tough time getting out of. To this point, Agile and contInuous integration or something close to continuous integration is almost a must to deploy it successfully.”

Fortunately, even though adoption in the hardware world may be relatively new, work here has been happening for the better part of the last two decades, said Frank Schirrmeister, senior group director for product management and marketing for emulation, FPGA-based prototyping and hardware/software enablement at Cadence. “Once the industry had a term, it became much easier to talk about it, but it has been in the use model of a number of semiconductor companies for quite some time. It’s the notion of being able to start integratIon of hardware and software early, and then at every change, validate that you didn’t break any of the previous tests, then refine the tests further as you go down in the level of abstractions, add more tests which require timing, add more tests which require even gate level type timing, things like that. It’s now much better captured in continuous integration.”

This bodes well for adopting and applying Agile methods.

“From an Agile perspective, we do see things like scrums where the engineering team, for instance, is focusing on feature development in a product,” Shirrmeister said. “They may say, ‘For this week, we’re focusing in this sprint on those three features to be looked at.’ In that case, we have the situation that in verification, you have this verification plan. In Agile, there are also focal points of verification as they change throughout time, where you can say, ‘I’m now focusing on the video subsystem. Some of it can run in parallel, but to other subsystems to be verified.’ Then there are focal points very similar to what you see in scrums or sprints in Agile methodology. But even more critical than in the software portion, you need to make sure you don’t break any of the previous tests so the regressions grow as you’re doing that.”

Sergio Marchese, technical marketing manager at OneSpin Solutions, noted that his company has adopted many Agile techniques for R&D. “On the hardware side, while there is some consensus that Agile can reduce development waste and foster a more co-operative working environment, there has been limited adoption. One of the exceptions is verification and, in particular, formal verification.”

As a verification engineer, he had a number of successful experiences applying Agile principles. “A key benefit of Agile hardware development is that the turnaround time to identify and fix a bug, and verify the new RTL version, can be reduced dramatically, from days to minutes in complex projects,” he said. (See Fig. 1.)

Fig. 1: Agile’s advantages. Source: OneSpin Solutions

Formal verification tools enable this type of process. “Designers can use formal tools themselves to execute automated checks and quickly explore intricate design functions without the need of a simulation testbench,” Marchese said. “They also can work side-by-side with verification engineers to develop end-to-end assertions in a form of pair programming aimed at delivering correct code iterations. The ultimate measure of success is that traditional, non-Agile verification should then find few or no bugs at all.”

Challenges
Still, this isn’t always so easy.

“While the hardware community has an increasing interest in Agile methodologies, fueled by its success in software projects, it is clear that Agile methodologies cannot be applied to hardware projects without some adaptation,” said Marchese. “Furthermore, companies aiming to fully embrace Agile need to change their engineering culture, which is a gradual and complex process.”

There are other challenges, as well. Meilling said there needs to be a safety net in place, which is the continuous integration piece. In addition, it’s important not to lose sight of the roadmap for a design.

“That is one of the real risks with Agile,” Melling said. “You get into the day-to-day operational practice of saying, ‘We need to do this, we need to do that,’ and it’s very, very important that the top-level context not be lost. The team should say, ‘Wait a minute, we’re creating this product, the use features all have to be there.’ Don’t fall into the trap of having lost sight of the objective and redouble your efforts. And the verification plan needs to tie back to requirements traceability. Those are the kinds of things that can keep you grounded on the roadmap front.”

Another enabling component to make continuous integration work is the Portable Stimulus, Schirrmeister said. “The notion of verification reuse is huge in this context, because you can’t just start rewriting the same tests in a new environment once you are another three months through the project flow. The amount of verification reuse is certainly something that will be even more critical going forward, which enables this agility to be able to re-use the verification tests from before, refine them and enhance them.”

Conclusion
To be sure, Agile processes are making their inevitable way into semiconductor design and verification.

“Commonly, iterative design and test components are integrated into continuously running regression environments,” noted Dave Kelf, chief marketing officer at Breker Verification Systems. “Repositories such as GitHub allow for a degree of automation and cloud-based environments will further improve this process. Providing abstract testbenches that operate at the system level and are portable to different verification engines will form the backbone of Agile regression environments. Individual elements of an intent specification may be selected to target specific design components while collecting global coverage metrics.”

This is just starting to make a dent in the design world, after years of discussion.

“Agile is here today in verification, and if people aren’t doing it, they ought to look into it,” Mentor’s Foster said. “It is a very effective way to speed up, be more productive, and deliver exactly what the customer wants in terms of creating the testbench.”