Components For Open-Source Verification

Defining an open-source verification methodology is a lot more difficult than just developing an open-source simulator. This is the reality facing open-source hardware such as RISC-V. Some people may be asking for the corresponding open-source verification, but that is a much tougher problem — and it is not going to be solved in the short term.

Part one examined the reasons why open-source verification may make sense and the large amount of commercial software that is needed to do this today. This article will examine the role that various organization can play in the creation or support of a verification flow and the progress being made towards pieces of that becoming open-source.

It is useful to consider this in the context of RISC-V, because the infrastructure surrounding that is layered. Each layer has a role to play and adds something to the final development of a solution. At the top is RISC-V International, which defines the ISA and the extensions that are considered to be standard. The organization has a specific set of needs for verification. In addition, a number of consortiums, including lowRISC, OpenHW, OpenTitan, and Chips Alliance, are building open-source extensions to RISC-V or implementations of them that satisfy certain sets of requirements.

Myriad companies utilize the output of RISC-V or the consortiums to create custom implementations for people to integrate into their chips. Many of those companies also want open-source components for bus interfaces, memory subsystems, and more. Some companies, such as Western Digital, have developed implementations and have fed those back into the consortiums.

Almost the entirety of that structure currently uses proprietary tools for verification. But the Chips Alliance has plans to change that. “The Chips Alliance has a pure open-source mindset where they’re trying to make their tools open-source as well,” says Simon Davidmann, CEO for Imperas Software. “So they’ve got an open-source Verilog simulator, they’ve got an open-source ISS in there. They’re trying to use Python as a testbench language. Spiritually, they are trying to build an open-source EDA ecosystem where everything is open and free.”

But the industry cannot wait for open-source tools to become available. “I would love it if there was an open-source SystemVerilog simulator,” says Mike Thompson, director of engineering for the Verification Task Group of the OpenHW Group. “We really support the push that’s going on in Chips Alliance to bring Verilator up to a more capable SystemVerilog simulator. But you cannot do complete 100% end to end, top to bottom, front to back verification of a RISC-V core using Verilog 95. There’s no free lunch in verification. You either pay with money for commercial products, or you pay with time, which is basically money for salaries.”

You may be surprised where additional support could come from. “There is always a place for open-source tools in any environment, whether that be EDA or outside,” says Neil Hand, director of marketing for design verification technology at Mentor, a Siemens Business. “If there is sufficient support for that, it will help push the industry forward. A driving factor may be something like RISC-V compliance testing, where there is clearly a need, there is clearly a demand, and you have lots of open-source providers. If we can find a way to help them get that commonality it will be good for the world.”

Established companies are not the ones that appear to be pushing for this. “Some relatively large companies are looking to see if some of the open-source tools are a way for them to optimize their costs, as well as that being fruitful for the open-source development community,” says Colin McKellar, vice president of verification platforms for Imagination Technologies. “But if you’re doing high-end compute devices with lots and lots of verification requirements, and you got very complicated IP, I’m pretty sure you’re spending enough on resourcing ad people that a small amount of investment into tools is justified.”

A view from the top
Even an organization like RISC-V has layers of verification need. “We look at it as a pyramid,” says Mark Himelstein, CTO for RISC-V International. “You have device certification at the bottom. Then there’s compliance test, and then there are various application binary interface (ABI) tests. There are standards in the industry where performance may be important, so we have to look at correctness as performing correctly, as well. And then, ultimately, the application has got to run. You aren’t guaranteed, even if you pass verification and compliance, that everything else will pass. The world doesn’t work that way.”

RISC-V International is concerned with verification that is implementation-independent. A completely different set of tests are required for implementations. “Device verification is left for either the companies to do themselves, or the consortium and system organizations that concentrate on that stuff,” says Himelstein. “Places like lowRISC, OpenHW, OpenTitan, and Chips Alliance, all work on various pieces. They have chosen to share that between those consortium members. And then there are some people who do stuff themselves because they either need in it a more timely fashion, or their secret sauce is so different that they really can’t show it.”

One of the concerns about open-source is responsibility and response. “A big part of risk is knowing who to beat up when things don’t work,” says Mentor’s Hand. “They can hold your feet to the fire and know that something will get fixed, and it will get fixed within their project timeline. It will get fixed in a way that they know that they can sign off their design. It’s one of the reasons why, even for the open-source hardware, you start to see companies doing commercial support.”

For others, that is one advantage of open-source. “With open-source, when it doesn’t work you can fix it,” says Tim Saxe, CTO for Quicklogic. “Admittedly, there may only be a few people who have that capability, but once it’s in the open-source tools you enable a lot of researchers and other companies to poke at things and to develop them. It’s better if we move them to the open-source tools in the long run, away from your proprietary.”

That takes a commitment from the open-source community to be successful. “You cannot guarantee that you’re going to think of all the possible things but, what you have to guarantee is you’re going to respond,” says Himelstein. “That’s all you can do, and hopefully you set things up so you can respond well.”

The current approach
A common processor verification technique to test the complex states and extreme corner cases is to employ a random instruction stream generator, such as the Google open-source project, RISCV-DV ISG as a test source, which can found on GitHub. By setting up the SystemVerilog test environment to run the tests in a side-by-side configuration, with the DUT and reference model, a step-and-compare methodology can be enabled, as shown in figure 1.

Fig. 1: A common verification configuration. Source: Imperas Software

In this case, the stream generator is open-source and the reference model, while proprietary to Imperas, is provided free of charge for the cores made available by the OpenHW Group. “We make our model free, but you can’t get access to the internals of the simulator and stuff like that,” says Davidmann. “We make the models very configurable, so that as long as you’re adhering to the RISC-V ISA. Then they have all the configuration options they need. If you extend the RTL, you extend the model, you extend the testing and you write new tests. You have an incremental change to do in two or three places.”

More is needed. “Comparing the results between the reference models and RTL simulator is a good starting point, but you would soon hit a speed wall due to the large amount of processor configurations that need to be verified — especially with the addition of custom instructions and registers,” says Louie De Luna, director of marketing for Aldec. “Some applications with stringent timing requirements, such as those in automotive and avionics, would increase the duration of simulation drastically. FPGA-based emulation can augment the flow. UVM-based tests and environments used in RTL simulation can easily be ported to an emulator with the help of the SCE-MI standard. Emulators provide cycle-based precision with signal-level accuracy and support designs with multiple asynchronous clocks.”

This can be used for other tasks, as well. “We also target FPGAs,” says OpenHW’s Thompson. “These will be in an FPGA implementation long before we finish the simulation and verification, or have real silicon. We’re not looking to that as a verification technique for the RTL. We’re looking at that as something that would basically help bring up the hardware, validate the tool chain, that sort of thing.”

There is a certain amount of contention about reliance on proprietary elements, particularly for the reference model. “We actually provide a number of instruction set simulators at different levels,” says Himelstein. “I will caution that each of the simulators are at different levels and have different capabilities. If you’re doing RTL it’s one level. If you’re doing functional, instruction-level simulation it’s another level. And then if you’re doing what I would call functional operating-system level it’s a different level. Qemu and Spike are at the functional operating system level. The SAIL model that we provide is at the functional instruction level. So when you’re doing some complex things like heavy duty synchronization with hypervisors, you probably don’t want to do that with a SAIL model. You probably want to do that with Spike, because it has built-in hypervisor support.”

Adding formal
Some people prefer the exhaustiveness which comes with formal verification. “We take the SAIL model and update that to match the specific core that we’re working on,” says Thompson. “The beauty of the SAIL model is it’s extensible in the same way as the RISC-V ISA. We then execute a formal verification of that core to create a set of SystemVerilog assertions that can then be used to do a formal verification of the RTL. The tool that does the formal verification comparing the assertions to the RTL is a proprietary tool, and the techniques that will be used to do that would be proprietary to each vendor.”

“The input to any formal tool is SystemVerilog assertions,” says Kiran Vittal, product marketing director for verification at Synopsys. “The lowest common denominator should be some basic setup and flow or methodology which can be extended and the open-source hardware provider can do that. There are other kinds of static checks that can very quickly catch corner case issues or even issues before you write functional tests or even do synthesis. These are simple linting checks.”

Fig. 2: Formal verification of a RISC-V core. Source: Synopsys

The methodology starts with the formal proof kit. “The general strategy for checking relies on observing the path of all instructions from the point of issue to the point of completion using observation models, where non-determinism allows for other instruction inter-leavings to intersperse between the issue and completion of the specific instruction being checked,” says Ashish Darbari, CEO for Axiomise. “This ensures that we can catch all kinds of bugs that are control- and concurrency-dominated, and can cause an incorrect result to occur.”

When applying formal to an implementation, additional properties have to be written. “The assertions are dependent on that microarchitecture, so they would be very difficult to re-use,” says Thompson. “For a four-stage pipeline, where each stage can stall the upstream stage, given a set of conditions, those conditions are different for that core than they would be for, say, a six-stage processor. Once you identify and write the assertions, they’re now specific to that core. The assertions can be written in SystemVerilog and they can be open-source, but they’re only ever going to be used for that core.”

Formal is certainly the preferred way to go for some companies. “Given the implementation transparency of open-source hardware, it is easier to employ white-box verification using formal verification such as property proving or equivalence checking,” says Bipul Talukdar, director of applications engineering for SmartDV in North America. “Formal methods from EDA companies may come with licensing complications, but those tools might be more useful compared to open resources because monetizing such a model would be a challenge and restrict widespread adoption. The solution could be opening up formal tool licensing mechanisms for third-party formal method development.”

Stream generators
The one area seeing a lot of attention from the open-source community is instruction stream generators. “Anything RISC-V produces or makes available has to be open-source,” says Himelstein. “So, for example, the functional compliance test suites will be open-source. The functional compliance test suites will be done with a test generator and easily modifiable.”

Configurability is key. “While the emergence of open-source ISAs like RISC-V with support for custom extensions gives an incredible amount of freedom to processor designers, it poses a very interesting verification challenge of making sure that all the designs are compliant and functionally correct,” says Roy Choudhury, CEO and co-founder of Valtrix Systems. “This calls for a shift in the way the test generators are designed. They need to be highly configurable to allow verification of custom features along with legacy/baseline features.”

The best known is the stream generator from Google. “When you’re building a processor, you’ve got to generate lots and lots of tests for it, and instruction sequences,” says Davidmann. “Traditionally, people have used constrained random test sequences. In the RISC-V world everybody needs an instruction stream generator, and there’s actually about seven or eight of them. It’s all done with SystemVerilog constraints and functional coverage. They’ll use commercial tools for simulation, but they will write an open-source Random Test Generator.”

Covering the implementation
There is a lot more to a processor than just the instructions. “Verification is something that people have grossly underestimated and I’m as guilty of underestimating the verification efforts for a CPU,” says Greg Tumbush, an engineer for EM Microelectronics. “It’s a huge job, a multi-man-year effort. If you’re just worried about instructions, you can generate a bunch of instructions. With RISC-V you’re throwing in debug, exceptions and interrupts, so the scope of the problem is way beyond whether it executes instructions.”

As with any verification project, it has to start with a plan. “You must have a very thorough verification plan,” says McKellar. “We have become too fixated on automated coverage. It all comes down to reviewing what you’ve done diligently and asking, ‘Is that enough?’ And if that’s not enough for the risk of throwing away chips, you need to invest a little bit more money in tools, or someone independent to review or verify it for you.”

This isn’t simple. “We have complete verification plans and we execute to those plans and build verification environments” says Thompson. “We apply techniques like constrained random stimulus generation, functional coverage to track completeness, code coverage, checklist to make sure that we’ve done everything that we said we do, and review the code to make sure that was done at the right level of quality. And that’s both the test bench code and the RTL code.”

Coverage can still be a weak point, though. “The industry is quite quite evolved in terms of stimuli generation,” says Darko Tomusilovic, verification director for Vtool. “Also, for checkers, we have well-defined standards, reference models, etc. We have a long way to go to achieve a similar level of quality in the area of coverage. The notion of functional coverage in UVM and SystemVerilog is oriented to whatever is easy to cover. We see a lot of cover points inserted simply because they were easy to add, and they do not provide any meaningful value. On the other side, we have a bigger problem. There are plenty of times when real corner cases where not there simply because it was very difficult to express them in functional coverage language.”

Conclusion
There is no open-source verification flow available today, and while some pieces are being worked on, it will be a long time before we can expect to see open-source verification tools with even close to the same levels of capabilities or performance as the commercial tools. While some may want tools that only have the capabilities required to verify a RISC-V core, most people then want to integrate that into a much larger system, and suddenly the cost of verification tools becomes less significant.

Bodies like RISC-V International are being realistic. “We’re trying to make sure that the ecosystem that enables people to do everything from device verification to debuggers to compilers and optimizers or middleware databases, and runtimes, interpreters and so on — they are all important. Our job is around the open-source piece, around the implementation independent piece. The stuff that we do is necessary, but not sufficient, for a customer or member to successfully create a solution for their customers.”

So far, few companies have shown a willingness to put up the necessary amount of money that it would take to create a suite of open-source verification tools. The best bet the industry has is within the Chips Alliance, and a lot of people are wishing them success. The EDA industry, meanwhile, remains on the fence.

“If you invest in technology, you have to innovate, you need to charge for it,” says Synopsys’ Vittal. “How can I give you an open-source tool and not charge for it and still come up with innovative technologies?”

McKellar agrees. “Everyone needs to be given a reasonable ability to make money, a reasonable markup, or things will collapse.”