The design freedoms of RISC-V offer developers flexibility for innovation – now processor IP verification quality is also a flexible option.
By Lee Moore and Simon Davidmann
The RISC-V open standard ISA (Instruction Set Architecture) offers developers the opportunity to configure the features and functions of a custom processor to uniquely address their target end application needs and requirements. RISC-V has a modular structure with many standard instruction extensions for additional dedicated hardware features such as Floating Point, Bit Manipulation, DSP, Cryptographic, Vectors, and many others currently under development. In addition, custom instructions can be added to further optimize the design. As RISC-V is an open standard specification, many implementations have been developed as commercial projects whilst others are made available as open-source hardware IP. Thus, SoC developers have several options in selecting a processor core, from developing a new design, selecting from the available cores, to modifying and enhancing any of these starting points.
The existing SoC design verification flows have been established for designs built from the integration of processor IP cores from the established mainstream suppliers. The industry SoC design verification (DV) methodologies are well established with UVM based on SystemVerilog test benches and block-level design verification. The current SoC DV methodologies share a common assumption as a starting point, that of ‘known good’ processor IP. These advanced verification techniques for processors are a closely kept internal trade secret and the quality of the deliverables is a trusted brand value of the suppliers.
RISC-V is not just expanding the design freedoms for the system architects to develop new optimized designs, RISC-V is also expanding the scope of work for the verification teams.
The SoC verification with UVM and SystemVerilog is founded on simulation-based flows, and as more SoC design teams adopt RISC-V it is natural that the SoC DV team will therefore adapt the established methodology to address the complexities of processor verification. This article covers some of the options and latest trends in RISC-V verification and simulation.
The open standard ISA of RISC-V defines the boundary between the hardware of the processor and the software that will run on it. An ISA specification is fundamental to the processor design and operation. As a processor is designed to run a software program, it is perhaps obvious that part of any design verification plan is to test the operation with software. Indeed, getting any software to run is a useful test during the design and development process. In some cases, such as a research project or a proof-of-concept design, the basic processor may not be the main focus. In some cases, the processor is only to provide a front-end interface to a more complex accelerator such as a vector processor for an AI or ML project. In this case, it is not required or expected that the RISC-V processor will need to run the full range of all possible instructions or use cases. In other situations, the processor functionality will need to be extensively tested across a broad range of application scenarios. Whilst some basic levels of testing might be acceptable for a simple test chip on a multi-project wafer, additional verification is required for a design targeted at high volume or a high-reliability use case applications.
Ultimately, a decision on the required design quality is part of the design process and the cost of verification is just another option to consider in the feature and functionality tradeoffs – a processor hardware specification or feature list can be evaluated on the impact to the project based on a review of these key factors:
RISC-V is changing the traditional processor IP design tradeoff decisions from a discussion on “P–P–A” metrics to “P–P–A–V”. A minor design feature that only requires a small number of gates might appear as an attractive choice with just a ‘PPA’ based assessment as the feature provides some useful functionality with what appears to be only marginal design and implementation costs. However, in considering the ‘PPAV,’ the cost to verify an optional feature could dramatically affect the total verification cost, effort, and complexity. As a basic assumption, each optional feature could double the verification scope of work, to cover all the use-cases for both with and without the option enabled.
Getting any software to run on a new processor implementation proves something is working, but “hello world” is just the start and not an exhaustive functional test. During the design and development of a RISC-V processor, it is natural to test a new implementation with some simple code snippets. Getting any software to run marks some progress during the design phase. Some classic milestones in processor development are getting some software to run such that it produces a response that is visible via an output device or file. This is typically a simple “hello world” message that indicates some internal operational activity on the processor produced an expected output response. As the design progresses, another early milestone is the first boot prompt for an OS or RTOS. These achievements, while significant in the design progress, offer very little coverage to be considered part of a functional verification plan, but they at least show the design is ready for a more robust test plan with software as input stimulus that can now exercise the design more completely.
Fig. 1: Simple log or signature file comparison method with free riscvOVPsim from GitHub.
The first step to verify a processor is to run some test cases and compare the outputs against a quality reference model or a self-testing signature, as shown in figure 1. The same software input stimulus can be run in an RTL simulation on the new RISC-V processor implementation. The RTL of the processor Design Under Test (DUT) is simulated with Verilog, or in verification test benches with SystemVerilog, with the resulting outputs saved to a log file for analysis. The same input is also applied to a reference model and the outputs are also saved to a file. A comparison between the two files may show some discrepancies. However, as a processor is a complex state machine, the two separate simulations may not accurately align for adequate verification purposes.
Another approach with the test program is to build in a self-test or signature result to indicate a satisfactory conclusion. While a simple go-no-go result might appear useful, the reality of a DV plan requires a method for analysis and debugging of a problem from the observed point of failure back through the design to uncover the source of the functional error.
The RISC-V international architectural compliance test suites are configured as tests with an integrated signature to indicate a pass. The test suites are designed to illustrate a basic level of conformance to the RISC-V specification without any attempt at detailed functional verification.
To support this simple comparison-based approach the reference simulator riscvOVPsim is available for free and has been adopted by many academic and industrial teams. It is available via GitHub [1]. As riscvOVPsim supports all of the standard ratified RISC-V extensions, it can be used for simple tests for the basic architectural validation testing. However, this signature file comparison approach is not a robust DV solution due to the deficiencies that are addressed with the following methods.
Fig. 2: The post-simulation trace-compare method with the free riscvOVPsimPlus.
A more sophisticated method of output comparison is to extend the comparison over full instruction trace-level analysis. The flow shown in figure 2 includes a test generator to stimulate both the RTL under test and the reference model and compare the output trace log files. In this example, the Instruction Stream Generator (ISG) is the open-source RISCV‑DV originally developed by Google and available on GitHub [2]. Since a purely random sequence of instructions would include a significant number of illegal and invalid combinations, an ISG generates random instruction but is constrained to provide both legal and valid instruction sequences. Exercising a processor with unexpected sequences helps to target corner case scenarios to extensively test the full operational range of the processor. To support this trace-based flow Imperas have made available the free reference model riscvOVPsimPlus, [3] which is available from OVPworld.org [4].
The RISC-V open-standard ISA is by definition intended to be flexible not just in the options and features supported in the current revision of the specification, but also in the versions as the specifications themselves evolve over time. It may be expected that future implementations of a RISC-V processor could be based on different release versions of the spec for base features, debug, and any combination of the various optional extensions.
riscvOVPsimPlus supports both the configuration options and also the version references for all of the specifications to cover the full envelope of requirements as a verification reference model.
The trace comparison approach can be applied with increasing levels of analysis starting with the program flow, basic PC (Program Counter) monitoring, program data, and monitoring all of the processor registers and memory changes.
However, the trace-based comparison method has some drawbacks, the trace files can become very large to manage and analyze. Also, discrepancies or test failures could occur early in the tests, which can become an inefficient use of resources with unnecessary simulation beyond the point of failure. While supporting some key DV objectives, trace-compare has efficiency limitations and is only considered suitable for simple processors. Also, when there is a trace comparison difference, it can be hard to ascertain the cause as all that is visible is trace files with no ability to perform any more detailed drill-down interrogation.
Fig. 3: The data-path lockstep method for step-and-compare verification.
To provide a comparison-based flow without the drawback and inefficiencies of the trace-compare post-processing, the reference model can be simulated in synch with the RTL DUT within the same SystemVerilog test bench. This flow is shown in figure 3. The reference model is encapsulated in the SystemVerilog testbench and provides not only an immediate indication should a discrepancy be found, but also an interactive debug environment to investigate divergence. A processor verification plan is typically focused on the key metrics for coverage but the efficiency to identify and investigate a divergence is an important factor in selecting a verification methodology.
The data-path lockstep-compare method overcomes the trace file size issues and some of the inefficiencies of the earlier options. It provides significant DV support and usability but still does not cover all of the operational events that a processor will need to manage, such as asynchronous events and interrupts. This data-path lockstep approach is a robust verification plan for some basic levels of processor core functionality.
Fig. 4: The OpenHW CORE-V-VERIF test bench as used for CV32E40P.
RISC-V is being adopted for some of the most sophisticated design projects with state-of-the-art processor architectural features such as hardware multithreading, out-of-order (superscalar), multiprocessor, and heterogeneous arrays, plus the flexibility of the RISC-V Vectors specification for AI and DL compute engines. The asynchronous lockstep-compare method shown in figure 4 fully exercises the multi-state behaviors of a complex processor and also supports asynchronous events such as interrupts and debug modes.
The asynchronous lockstep-compare method builds on the earlier approaches to provide a quality and synchronized DV environment between the DUT and the reference model. The testbench is adapted to introspect the running activities on the processor operation and provides the synchronization across events, registers, interrupts, and debug to align at the instruction boundary with the reference model.
Currently, in RISC-V verification the asynchronous lockstep-compare is the gold standard in dynamic (simulation based) verification and represents the best DV environment currently available for RISC-V. By using instruction by instruction analysis, the complete processor execution flow including internal state can be inspected and analyzed. The interaction between the comparison and debug stages allows efficient trace and analysis to identify root cause failure analysis. However, in addition to the initial test bench set-up, the overall verification effort for a processor remains a considerable task. A complex processor verification plan may involve test capacities in the range of a peta of instructions (1015) for full coverage across all use cases and operational scenarios. This async lockstep-compare approach is truly a robust verification plan for the most complex of processor functionality.
The OpenHW group [5] was formed to develop open-source RISC-V core IP with industrial strength verification utilizing the best available tools and flows. The first core to be released is CV32E40P and the test strategy and plans are all available as part of the open-source deliverables on GitHub [6]. The OpenHW contributing members behind the verification work, known as the ‘CORE-V Verif’ project, recognized that the value in the verification project was not just to fully test the processor IP but to document and provide all the tests, frameworks, and scripts as open source. Any adopter of an open-source core should set-up the test bench and environment to fully qualify any IP contribution at the start of a project. During the project development, this helps support regression tests and maintains the quality of the processor IP throughout the design phase which is especially important if modifications or further extensions are added.
While the test bench and frameworks were targeted initially at the RCV32E40P core development, it will be expanded to cover the future roadmap of cores including CV32E40X, CV32E40S, CVA6 32bit and 64bit application cores plus more to come.
Not all projects have the same goals and objectives, as in all areas of engineering the correct application of resources in design time and effort is balanced against the overall design goals and objectives. So, this is equally true in the general case for SoC verification. However, in the past, the processor IP quality from the mainstream suppliers was not an option and it was mandatory to provide the highest quality IP, but with RISC-V, developers can approach the design verification task with the same level of flexibility as all the other aspects that RISC-V now enables. In some cases, a simple comparison-based trace analysis is sufficient to confirm some level of basic operation of the processor, while designs targeted at volume production and/or high-reliability applications will justify significant additional verification efforts. The challenge of RISC-V is not just the verification task, but the correct selection of the quality level appropriate for the end application.
The Imperas [7] RISC-V reference model is being used by some of the most sophisticated design projects with state-of-the-art processor architectural features such as hardware multithreading, out-of-order (superscalar), multiprocessor, and heterogeneous arrays, plus the growing adoption of RISC-V Vectors for AI and DL compute engines. Active users include NVIDIA Networking, Seagate, NSITEXE/Denso, Google Cloud, Chips Alliance, lowRISC, OpenHW Group, Andes, Valtrix, Nagra/Kudelski, Silicon Labs, multiple RISC-V International Working Groups, and many others that are yet to be made public.
RISC-V is enabling processor design freedoms to the broad community of SoC developers, but this also represents a migration of the processor design verification task from the teams at a few mainstream IP providers to all RISC-V adopters. Developers that accept the design freedoms of RISC-V must also undertake the appropriate level of verification responsibility.
References
Simon Davidmann is founder and CEO of Imperas and initiator of Open Virtual Platforms (www.OVPworld.org). Prior to founding Imperas, Davidmann was a VP in Synopsys following its successful acquisition of Co-Design Automation, the developer of SystemVerilog. Prior to founding Co-Design Automation, Davidmann was an executive or European GM with 5 US-based EDA startups including Chronologic Simulation, which pioneered the compiled code simulator VCS, and Ambit. Davidmann was one of the original developers of the HILO logic simulation system, and co-authored the definitive book on SystemVerilog.
 |
 |
 |
 |
 |
 |
 |
 |
 |  |
… and my favorite is step #5 Formal Verification using a reference model, cpu RTL, SVAs, etc. which is what I’m doing for a RISCV I’m designing. 700+ property assertions working so far and about 15 min. to verify