Rapid Evolution For Verification Plans

Verification plans are rapidly evolving from mechanisms to track verification progress into multi-faceted coordination vehicles for several teams with disparate goals, using complex resource management spread across multiple abstractions and tools.

New system demands from industries such as automotive are forcing tighter integration of those plans with requirements management and product lifecycle development. As a result, today’s verification plan must encapsulate the entire development methodology.

“Verification has become a massive challenge,” says Colin McKellar, vice president of verification platforms for Imagination Technologies. “The big shift we have seen fairly recently is that the planning needs to be more aligned with the product requirements, be that requirements from a safety point of view or product requirements. It includes asking questions such as, ‘Does my software run well on this device?’ It is no longer hardware verification. It is product validation.”

This is on top of what verification teams have been doing in the past. “They continue to plan for regular coverage,” says Boris Gommershtadt, group director for R&D within Synopsys. “Then they need to plan for software validation, for power management, and now for fault coverage. For all of these they have to track progress across different tools. In the past, people would get the coverage results from the simulator and it would annotate the verification plan. That was the advanced users. They now have to import coverage result from multiple sources.”

But have the prime objectives changed? “The fundamentals of verification planning have not changed,” asserts Rob van Blommestein, head of marketing for OneSpin Solutions. “The chip development team must extract from the design specification all the features to be verified, and enter them in the verification plan, identify which tools will be used to verify which features, annotate the verification plan with the results from all the tools, and collect coverage results from all the tools to make an accurate assessment of verification thoroughness.”

Fig. 1: Multiple levels of verification. Source: Neil Johnson; Mentor, a Siemens Business

That has evolved over time. “Historically, when people were faced with a complex problem in the design process they looked at simplifying the problem,” says Prakash Narain, president and CEO for Real Intent. “That is done by defining a methodology. Then you can build analysis on top of that methodology to help solve the problem effectively. By enabling continuous integration and by shortening the verification part of the cycle, we have been able to left shift by bringing certain problems into the design domain.”

There are both top-down and bottom aspects to the problem. “Fundamentally you have to ask, ‘Is my product the right product?’” says Imagination’s McKellar. “That is everyone’s real question. Many people get hung up in massive amounts of detail. Do they have 99.7% rather than 99.8% coverage? We still do a lot of unit verification, and that problem space has not disappeared. The biggest challenge with that is answering when we are done. But unit verification is not the biggest problem space today. A better way of looking at it is top-down as opposed to bottom-up.”

The new plan
A modern verification plan is a lot more complex than in the past. Larry Melling, product management director for Cadence, breaks it into 3 areas:

• Engine expansion. Formal, simulation, emulation, prototyping — and now post-silicon — become interesting in terms of data that can be gathered and collected and rolled into the verification picture.
• Requirements and the traceability of requirements. These are the core functional requirements of the design and what it has to do. Now there is an expansion into the realm of negative testing, as required for safety and security. Requirements include how to ensure that if something bad happens, that the problem is contained.
• Methodologies. There is a lot of expansion of methodology toward continuous integration, regression management and multi-site development, plus expanding from in-house compute platforms into the Cloud. Methodology combined with compute resource availability and deployment is creating a series of issues around how do you get all of the tests run on the different facilities, how often do you run them, when do you run them and how do you get the data back and collect it.

Most teams can no longer avoid software. “There are multiple things to verify,” says Synopsys’ Gommershtadt. “You verify the design correctness against your functional specification. You verify your software correctness per its functional specification, and you verify the power according to the power requirements spec. There are various specification and requirements, and the verification plan details what and how you verify, and how you measure what you verify.”

Complicating things even further are multi-generational aspects to products. “When it comes to things like performance, the notion from the ’90s that you do performance analysis and then you change something to optimize and make it better is actually performed over multiple projects,” says Frank Schirrmeister, senior group director for product management and marketing at Cadence. “You may do specific fixes, but you are planning beyond one chip typically. You plan for certain levels within the current chip, but certain items will be left for the next project where you will look at power and performance based on actual chip data from the previous project.”

Ultimately, verification is not an absolute, it is risk management. “A lot of the planning is getting the best answers as early as possible,” says McKellar. “At the architectural stage, can I get within some tolerance of acceptance in terms of performance and PPA? If I can, then I reduce the risk levels towards the end. Can I, early in the process, plug software into an early model of the hardware that is available at the time? Can I start asking questions and making sure we have minimized the chance of problems at the high level?”

The role of abstraction
Abstraction allows us to concentrate on the important elements before the details clutter the picture and inevitably lead to much slower execution. The addition of software makes abstraction essential. “When you buy a processor from Arm, you plug it in and do integration testing and you are done,” says Simon Davidmann, CEO of Imperas Software. “Arm has done 10¹⁵ cycles of verification on that core. You know it is an Arm core and you know the RTL does what is in the spec. You can’t have a cycle-accurate reference model because you just can’t write them. You can create it from the RTL, but that is after the fact, not before the fact. They are also too slow. So if we can get a system up and running with all of the software, it means that when they do try to run it on the RTL they will have eliminated most of the easy problems. You have to do verification at multiple levels.”

It also requires a separation of issues. “There are two processes here,” says Schirrmeister. “The notion of verifying at a higher level of abstraction kicks in at the hardware-software interface. The second is when someone says I will build a full-chip SoC model in SystemC and use high-level synthesis (HLS). There is some divide-and-conquer going on, but from a verification planning perspective you need to consider the whole chip. It is becoming interesting when you bring software developers into the picture, where we need a higher level of abstraction, such as using a hybrid solution. That needs to be part of the plan, but there are challenging organization issues.”

High-level synthesis plays a critical role here. “One of the key value propositions of HLS is that you don’t have to do everything in RTL, which is pretty late in the design, and after waiting for various tools,” said Maximillian Odendahl, CEO of Silexica. “You can do everything at the C level. And because you can do faster simulation, you also can iterate faster. You can iterate at the algorithmic level. So now you already have a good idea how everything is going to behave and how it’s going to fit in terms of area and performance.  You still need to do the end verification of the hardware after place-and-route and everything else, but hopefully that’s the last piece — and not the core piece. You don’t want to have to do the core piece over and over again.”

HLS also helps understand how various components will impact the overall system once they are integrated into it. “People still do pieces of the design in isolation, and then at the end they think they can throw everything together and it will be okay,” Odendahl said. “That may have worked at one time, but it doesn’t work anymore.”

Hybrid emulation, which may be a new concept to many teams, is another take on the same theme. “When someone runs emulation, they would usually verify either the real software, or they want to verify some aspects of power,” explains Gommershtadt. “When they reach the emulation stage, they do not expect to find bugs in the RTL, especially for the big IPs such as the processor or the GPU. It does not make sense to slow down the emulation or to increase the capacity and put all of this inside the emulator, even though it is possible and people are doing that. So those pieces are put into a virtual prototype that can run fast enough and keep up with the emulator.”

The virtual pieces are not identical to RTL, so does hybrid means that anything has been verified? “If we run the code, we do not need to revalidate that at the RTL,” says Davidmann. “You do not validate Linux on RTL. You may run it on silicon and you may run a little bit on RTL. You do not need to run on the RTL, and you make the assumptions that the software compilers map to the ISA and the ISA implements what the compiler expects.”

Again it goes back to risk levels. “The user may decide they need to run 90% of the verification content on the hybrid model and the remaining 10% with the full RTL, which will verify with complete accuracy,” adds Gommershtadt. “The hybrid models are not cycle accurate, but even the processor would usually run asynchronously from the rest of the system. So the system with respect to the processor does not have to be cycle-accurate, and the behavior exhibited by the behavioral model is still a legal behavior.”

For those who want added confidence, “you can look at the increase in the use of prototyping to say that people do want to make sure that the software results they are seeing on hybrid and virtual platforms hold up against the real RTL,” notes Cadence’s Melling. “The push for a faster, more accurate platform, to get the additional check of the software running on timing-accurate hardware, or cycle-accurate gives you that checkpoint. It all comes down to the evaluation of how much the abstraction separates you from reality and how do you close that gap. People will find ways to do that.”

New products are coming to market to address that. Aldec has just announced a hybrid between emulation and prototyping that utilizes processor cores within an FPGA. “Our new hybrid co-emulation platform for large ASIC and SoC designs brings together our hardware emulation system and an embedded system board,” says Zibi Zalewski, general manager of the hardware division at Aldec. “The embedded system board features a Xilinx FPGA, which contains a quad-core ARM Cortex-A53. A bridge between the two products allows the ARM cores to be shared with the emulator, meaning that software team members can prototype on fast clock-speed hardware and benefit from fast system boot-up.”

Adding power and performance
Verifying power and performance are still nascent problems. “Power is more containable,” claims McKellar. “You have to get representative vectors and put that through the right tooling. The hope is that those vectors are representative. Getting estimates for power at RTL is easier than it has even been. It is easier, using emulators or implementation tools, to get reasonably acceptable numbers about likely rate of change and effect of IR drop across devices. However, we are still a level away from where the end product guys would like us to be.”

The structural verification of power is relatively simple and often driven from the UPF document. “You can generate coverage based on the UPF,” says Gommershtadt. “You want to make sure that you have covered all the power domains and power states and switches between them. That is the activity from a correctness point of view. The whole point of UPF is to improve the power. People will need to generate data from the simulation, either through side formats or through tools that measure power consumption of a chip under realistic software load. People can get those diagnostic measurements and then plan for it. So they may want to see the power consumption to be less than a certain amount under these test cases.”

The industry is looking toward Portable Stimulus (PSS) and similar in-house tools as a way to create those representative vectors.

Challenging the status quo
As new demands are placed on projects, they must be constantly vigilant that they are using the best techniques available and that they have not been replaced by newer, more efficient methods. “We need to be asking, ‘Is that the right test?’” says McKellar. “Am I prioritizing my tests the right way to maximize my progress and minimize the chance of engineering mistakes being caught later on? Unless you embrace this type of question you will never really progress. You will just be mindlessly adding stuff.”

Formal provides many such examples for prioritization. “The increased capacity of today’s formal engines means that a wider range of design features can be selected for verification primarily by formal tools,” says OneSpin’s van Blommestein. “Some specific features can be covered by formal apps for connectivity checking, floating-point verification, RISC-V verification and compliance, and other aspects of the design. In all cases, the results from formal tools must be annotated back into the verification plan for a unified view of verification progress.”

Focused tools have solved some tough challenges. “Both CDC and RDC solve meta-stability problems,” points out Real Intent’s Narain. “Traditionally, these are unsolvable in both static timing analysis and functional simulation. The RDC problem space has become larger with increasing sizes of SoC designs that include power management. Because of the advancements in design methodologies and practices, new failure modes are emerging.”

In the last 5 to 10 years we have seen hardware design and verification groups take advantage of continuous integration and hybrid agile methods to automate and enhance workflows. “The ever-increasing design complexity coupled with the need to transition between coverage driven, testplan driven and requirement driven verification methodologies has brought project management challenges that only modern life-cycle management solutions can address,” says Mark Carey, product engineer for Mentor, a Siemens Business. “With traceability in place, it provides the insight for optimizations to take place during the lifetime of the project, ensuring the best usage of time and resources, boosting productivity, and reducing time to market.”

Conclusion
Early verification plans provided a formalized mechanism for defining what the testbench should look like, and for tracking progress toward the goals that had been established. Today, that plan defines most of the important aspects of the design and verification methodology, encompassing all phases of development, and has to ensure continuity between product cycles. It requires close attention to ensure that it remains efficient and effective.