Practical Processor Verification

Custom processors are making a resurgence, spurred on by the early success of the RISC-V ISA and the ecosystem that is rapidly building around it. But this shift is amid questions about whether processor verification has become a lost art.

Years ago custom processors were common. But as the market consolidated around a handful of companies, so did the tools and expertise needed to develop those processors. As the pendulum swings back toward application-specific designs, the industry is struggling to fill the void.

This is especially difficult with processors because of the required design flexibility. Processors can implement almost any function that can be expressed in software because the hardware has to be designed not knowing what software it eventually will execute. But that also means any bug, no matter how small or seemingly inconsequential, can result in a catastrophic outcome — and the processor world is littered with such examples.

But this also raises a number of important questions. For example, when a processor is deeply embedded and only has to run enough software to perform the intended function in a device, is near-exhaustive verification necessary? This is built into the cost of a commercial core, but can the bar be set lower?

“People with RISC-V designs have talked about the compliance suite that says that you conform to the definition of the ISA,” says Paul Cunningham, corporate vice president and general manager at Cadence Design Systems. “That ensures your compatibility with the ecosystem probably more than anything else. That’s the only thing as a minimum you actually have to verify. After that you are making a judgment call about how much more verification it is worth doing.”

For those wanting to sell a commercial core, or when the core is accessible to user-level software, this is not adequate. “A good processor verification flow needs to include all the industry standard techniques,” says Daniel Schostak, architect and fellow for Arm’s Central Engineering Group. “This includes coverage driven constrained random simulation, formal verification, and all kinds of hardware prototyping, each used in a complementary way to ensure no stone is left unturned. Today’s complex processor designs require projects of the order of hundreds of person years’ effort, and most of this effort is verification-related rather than design-related.”

“Verification is an essential element in our processor design methodology,” said Roddy Urquhart, senior marketing director at Codasip. “We use consistency checks between the instruction accurate and cycle accurate descriptions of a processor core and we generate a UVM environment to thoroughly verify the processor RTL against the ISS golden reference. We are convinced that processors need a high level of functional and RTL code coverage, and we use both direct tests and generate random assembler programs to achieve this.”

Run all software
But how much verification depends on the application. Many of the users of RISC-V cores are building deeply embedded systems. “If you’re building a deeply embedded processor, it’s really only your team that’s going to see it — your software guys,” says Simon Davidmann, CEO of Imperas Software. “If your team builds their own RISC-V core, or you build a proprietary core — or you even build or buy it in for an IP provider — it’s never going to be as well tested and fleshed out as a core from IBM, MIPS or Arm.”

Changes in the software can have a big impact on that equation. “You probably have to consider what happens if you make post-silicon upgrades,” says Cadence’s Cunningham. “The software you’re verifying on pre-silicon may not be what is ultimately involved, especially if it’s an embedded part that will be in the market for many years. If you’re building something that goes into a consumer appliance, you could be selling it for 5 or 10 years after you make a chip.”

This is part of project planning. “Limiting verification would make a sizable difference because you can manage risk better,” says Colin McKellar, vice president of verification platforms for Imagination Technologies. “As long as you’re prepared to downgrade some features or potentially spend some extra money, time and effort on software fixes, that would be a lot more containable. You still need to have some level of quality, but it does contain the problem and you don’t need to be telling the world about any bugs. You can protect your brand. You can focus on the really important stuff, and this would be a good way of reducing the cost associated with the product.”

The minimum requirement is being compliant. “There’s this whole concept of compliance, because you might be getting the tool chain from one vendor, and the processor IP from another,” says Imperas’ Davidmann. “To make use of the ecosystem, you’ve got to make sure that your core is compatible with it.”

In addition to being compliant, it has to be fit for a purpose. “In your verification plan, it may say that I need these software workloads to be covered,” says Cunningham. “If that defines the point at which I am comfortable that the CPU is functioning correctly, then what is the return on investment (ROI) associated with additional verification? People talk about the asymptote in terms of coverage closure. You may pick a point much sooner on the curve and say, ‘I’ve done enough.’ A lot of the verification cost is associated with that very, very long tail. The ROI is not the same for me because I’m just using it for one embedded application. I could spend another million dollars over the next month to do additional verification, but when have I done enough?”

That question is difficult to answer because requirements can change. “You start with a minimal set of scenarios to make sure that you are able to run meaningful programs that the processor is likely to be subjected to in the application space,” says Hagai Arbel, chief executive officer for Vtool. “In one custom processor application we were recently involved with, they started with the assumption that nobody was going to be touching the code, but over time they wanted to relax that restriction. In the end, they wanted the software teams to be able to develop more software and address new applications after the silicon would arrive. Even though it was a custom processor and they were not selling it as a general-purpose processor, it still ultimately required the same level of maturity and verification as an Arm processor.”

Risk, flexibility and cost must be balanced. “When you’re defining your verification strategy and doing validation, you don’t necessarily have the apps you will be running in the final product available,” says Imagination’s McKellar. “You need to have some level of flexibility, but from flexibility comes complexity, and that usually require more time and money to verify correctly.”

Defining the strategy
The industry has lots of experience verifying complex IP blocks and integrating those into even more complex systems. What makes processor verification different? “An average piece of hardware has a block-level spec and you can understand the requirements very well,” says Vtool’s Arbel. “But when it comes to a processor, you can’t just have the architect write the specification and know that it provides you with the performance that you wanted. There tend to be a lot of misunderstandings between the teams — the hardware implementation team, the software team, as well as within the end system.”

Those block-level approaches and methodologies continue to apply to the individual blocks or interfaces within a processor. “At the unit and interconnect level, fast high-capacity formal tools excel at verifying any aspect of the design that has provably correct protocols and/or a high degree of concurrency,” says Duaine Pryor, chief technologist for Mentor, a Siemens Business. “Coherency protocols, ordering transformations, and various aspects of the memory subsystem fall into this category. Formal is also valuable when the unmodified and modified elements of the instruction set can be formally specified.”

Having multiple teams involved also means multiple models. “It is necessary to put an emphasis on the system aspect of the design and verification process,” says Jens Michael Warmuth, group manager for functional safety & system reliability verification at Fraunhofer IIS’ Engineering of Adaptive Systems Division. “The verification has to be done throughout the complete design process in order to prevent breakages and gaps. This can be accomplished by using a system level model from concept phase up to the validation of a prototype and by reusing test cases from prior design steps. The design and verification process has to be done as efficient as possible.”

And it takes all of the available tools to accomplish that. “Processor verification has a lot in common with other large complex designs, but there are unique aspects, as well,” says Nicolae Tusinschi, design verification expert at OneSpin Solutions. “Multi-instruction sequences present special challenges and require automated formal verification techniques. Corner cases related to exceptions, pipeline hazards, cache coherency protocols, and IEEE 754 floating-point operations are highly unlikely to be exercised in simulation, so only full formal proofs can provide the confidence needed. Even baseline ISA compliance, which sounds simple, can only be certain if proven formally.”

The tools supplied by the industry today may not be enough. “While the emergence of open source ISA’s, like RISC-V, with support for custom extensions gives an incredible amount of freedom to processor designers, it poses a very interesting verification challenge of making sure that all the designs are compliant and functionally correct,” says Shubhodeep Roy Choudhury, co-founder of Valtrix Systems. “This calls for a shift in the way the test generators are designed in that they need to be highly configurable to allow verification of custom features along with legacy/baseline features.”

And that requires knowledge about the implementation. “You can code up the ISA instructions and you can start generating randomized programs from it,” says Cunningham. “But how many randomized programs, and how much compute would you have to run? How many cycles of randomized instruction before you’re comfortable that you’ve covered enough? You get into this realm of needing to be smarter. You want to guide randomization and you want to structure it in certain ways. Guiding it could be based on the microarchitecture specification of the CPU. That’s good, because you can try to generate instruction sequences that are designed to exercise different parts of your microarchitecture. But you may not be covering everything.”

Other tools are not well suited to the problem either. “Existing verification tools all have their strengths, but they’ve all got big weaknesses,” says McKellar. “The world of emulation takes too long because compile times are problematic, especially as design sizes increase. The CPUs that you’re running simulation on are flat in terms of performance. So we haven’t really increased performance very much. The big problem with the industry is the lack of unified coverage. We had to develop our own verification management infrastructure on top of point tools that we buy from the vendors. We also have to work around the limitations of the state space capacity for a lot of the tools. We’ve taken the formal tools and added ways and methods of driving them. We also have done the same for things like fault injection and fault analysis. The tools are too weak in terms of state space ability, so we developed our own methodologies to make them work in our environment.”

Two verification phases
In general, people talk about two general approaches to processor verification. The first uses a traditional UVM approach, while the second utilizes a software generator. In both cases, a reference model is used to verify results.

Darko Tomusilovic, verification director for Vtool, explains the principle of a UVM-style verification methodology. “The diagram shown in Figure 1 presents the verification environment architecture, with an active VIP being used to preload processor instruction memory with meaningful opcodes, feeding the reference model. A bunch of active and passive VIPs were instantiated around the processor in order to collect and react to the transactions generated by the CPU, feeding the checker that compares them against the model expectation. During this process, after the reference model was built, the majority of effort was invested in debugging the hardware bugs on one side, and the bugs within the reference model itself on the other side.”

Fig 1. Block level testing environment: Source: VTool

The active VIP also could be a stream generator, such as the RISC-V generator being made available by Google.

Several models and execution environments can be built for this type of verification. “The verification flow for custom processors, such as RISC-V, would need to include an instruction set simulator (ISS), RTL simulator, FPGA-based emulator and FPGA prototypes,” says Louie De Luna, director of marketing for Aldec. “Reuse and portability of the test environments from one to the other would be play a critical role. Comparing the results between ISS (running the reference processor models) and RTL simulator is a good starting point, but would soon hit a speed wall due to the large number of processor configurations that need to be verified — especially with the addition of custom instructions and registers. Some applications with stringent timing requirements, such as those in automotive and avionics, would increase the duration of simulation drastically.”

If performance is an issue, emulation and prototyping have to play a role. “Processor design flows must start at the C algorithm level, followed by efficient high-level synthesis, coupled with hybrid verification and analysis at multiple abstraction levels, where emulation is front and center,” says Lauro Rizzatti, a verification expert. “FPGA prototyping, upgraded to handle multi-billion gate designs for full-system validation before tape-out, completes the design flows.”

The second phase involves software. “We introduced a high-level, C-like domain-specific language, implemented within a C++ environment,” says Vtool’s Tomusilovic. This is shown in figure 2. “This was compiled into binary files to be run by the processor. A software simulator, also written in C++, takes the binary file as an input, simulating it according the processor specification and generating the expected results, with software execution logs.”

Fig 2. System level testing environment. Source: VTool

When software is being executed, fast execution of the RTL becomes even more important. “Software applications and firmware can be tested using FPGA prototypes,” says Aldec’s De Luna. “Re-using the same FPGA-based board from emulation to prototyping would not only save money but would also allow easy migration of the RTL especially if it needs to be partitioned into multiple-FPGAs. Verification of custom processors will not be a simulation only effort, it will require powerful tools including hardware-assisted environments interconnected with each other.”

The two-phase approach highlights some of the issues with coverage. “Coverage was mostly emphasized on the block level in the usual UVM test bench,” says Tomusilovic. “There we used coverage to prove that all the random scenarios will be exercised. On top of that, we define some system level scenarios where our cover point is whether a certain system scenario is exercised or not. So, instead of defining some complex cover points, we simply rely on the fact that a certain system scenario was run and passed.”

Conclusion
A significant portion of the value that processor companies provide is the extensive verification environment that they have built up over the past two decades or more. The EDA market stopped making tools for them explicitly because the market size became too small. Instead, the processor companies continued to develop the necessary tools and methodologies.

Extensible processor architectures have been around for a long time, but they were within large EDA companies that had the necessary tool development expertise. The market is now changing rapidly, and tools are being created to fill some of the voids. It is not yet clear if these will become open-source, or if the EDA industry will take up the necessary support of them over the long term. In the interim, knowledge and experience in processor verification has become a very hot commodity.