Showing that a processor core adheres to a specification becomes more difficult when the specification is extensible.
The open-source RISC-V instruction set architecture (ISA) continues to gain momentum, but the flexibility of RISC-V creates a problem—how do you know if a RISC-V implementation fits basic standards and can play well with other implementations so they all can run the same ecosystem? In addition, how do you ensure that ecosystem development works for all implementations and that all cores that claim to be RISC-V have implemented the specification correctly?
RISC-V chip developers need to delve into conformance testing. The good news is the RISC-V Compliance framework is being developed to ensure that a product claiming to be a RISC-V core will run a common set of applications, tools and system software. But conformance testing skills in each EDA team will need to be developed.
Making sure an implementation can work with other implementations is the role of conformance testing, which is not something the EDA industry has a lot of experience with. In the past, the EDA industry has generally deemed one implementation to represent golden behavior that all others must meet, while many other standards use plug-fests to ensure compatibility. But with so many possible implementations, configurations and adaptations that can be made to a RISC-V core, this presents a substantial challenge.
“RISC-V is an open-source standard ISA with exceptional modularity and extensibility,” says Allen Baum, system architect at Esperanto Technologies and chair of the RISC-V Compliance Task Group. “Anyone can build an implementation and there are no license fees, except for commercial use of the trademarks. A basic instruction set is augmented by extensions to support special functions such as atomic and compressed instructions, and additional data types such as floating point, vector and bit operations.”
The potential success of RISC-V isn’t simply that the architecture is open, but that any implementation can run applications and system software written for the ISA. “Implementers are free to add custom extensions to boost capabilities and performance, while at the same time don’t have to include features that aren’t needed,” adds Baum. “Unconstrained flexibility, however, can lead to incompatibilities that could fragment the customer base and eliminate the incentives for cooperative ecosystem development.”
Software compatibility is the key. “The RISC-V ISA defines the interface between software instructions that can be executed by the hardware resources of the processor,” says Kevin McDermott, vice president of marketing at Imperas Software. “It defines the essential boundary between hardware and software.”
Compliance versus verification
Compliance, in the context of microprocessors and microcontrollers, ensures a level of compatibility so that software, development tools and operating systems can be used across a range of devices.
“Anyone who wants to develop a RISC-V device will wish to ensure they have a correct implementation that is compliant to the ISA standard in order to benefit from the reusability of software and tools and enjoy the full benefits of the RISC-V ecosystem,” says McDermott. “The risk of fragmentation can occur when incompatibility develops across devices that requires software modifications to such an extent that software reuse becomes limited and thus stalls widespread investment.”
This is a new challenge for the industry. “There is no established method in the industry for dealing with compliance,” adds Simon Davidmann, CEO for Imperas. “For all other ISAs, they are controlled by a single company, so ensuring they ship the right product out of the door is a very different problem. With RISC-V, multiple companies will be designing products, and they and their customers must have confidence their implementation fits within the envelope of the specification.”
Compliance is not about RTL verification or trying to prove that the RTL is correct. Compliance is trying to ensure that it meets the standard. “Compliance is a critical new verification issue,” says Dave Kelf, CMO for Breker Verification Systems. “The RISC-V Consortium is producing a compliance suite to aid in this effort. However, running this is just the beginning.”
How far should compliance go toward verification? “Compliance is absolutely critical in steering the ecosystem in a consistent direction, which ultimately allows the industry to benefit from shared investments,” says Tim Whitfield, VP of strategy for Arm’s Embedded & Automotive Line of Business. “We firmly believe that the right level of compliance is just as critical. This is reflected in our resource and investment that has been key to the proliferation and success of Arm architectures. There is a balance to strike in compliance as sometimes it can be too little, leading to problems with ecosystem compatibility, or too complex leading to unnecessary cost and time.”
The tools and techniques used for compliance and verification may also be different. “Testing for compliance is different than standard SoC verification in that it focuses on the instructions and valid sequences rather than using techniques like constrained random,” says Neil Hand, director of marketing for Design Verification Technology within Mentor, a Siemens Business. “While there is some place for traditional constrained random, compliance needs to focus on the ISA itself. This does introduce many new methodology challenges, especially since different cores can have different microarchitectures that will affect the timing of data. This means that compliance largely becomes a software problem to ensure valid and invalid instruction sequences give the correct result.”
While verification methods rely on coverage, compliance has a similar issue. “How to measure compliance to the RISC-V ISA has become a source of increasing concern and confusion to many companies considering adopting the platform,” explains Chris Jones, vice president of marketing for Codasip. “Unlike legacy architectures, there is no one method to assure compliance. And until there is a compliance standard, the RISC-V ecosystem risks fragmentation. Testing each of the RISC-V permutations that are possible, given the increasing number of standard extensions, is not a trivial task. Before we can even discuss compliance we need to agree upon what compliance means and what sort of framework should be implemented. Given that there are both cooperative and competing parties within the ecosystem, it is essential that whatever compliance framework is adopted be impartial and independent.”
The RISC-V Foundation and its Compliance Working Group are developing and maintaining test suites, framework and methodology to enable its ecosystem to ensure ISA compatible processors meet the demand of the software user base for RISC-V. “The RISC-V compliance framework has three major elements,” says Esperanto’s Baum. “A modular set of test suites that exercise all aspects of the ISA; golden reference signatures that define correct execution results; and frameworks that select and configure appropriate test suites based on both platform requirements and claimed device capabilities, run the tests and compare the results with reference signatures, reporting success or failure.”
Fig 1. RISC-V consortium compliance framework. Source: Esperanto
The test suites and the framework elements shown in Figure 1 are based on standard assemblers, compilers, loaders and scripting languages. Reference signatures are generated by reference models (some formal) of the architecture. Test generation is aided by a standard format that includes macros that simplify creation, report results and aid debugging of failures. Additional tools can measure test coverage using a variety of metrics to ensure result quality.
Current status
The compliance working group was one of the earliest groups established, despite being dependent on the work product of the other committees that are creating the RISC-V specification. Baum explains the current status of the compliance committee’s efforts:
“The RISC-V Foundation recently held a ‘design a RISC-V soft core competition’ and all the finalists successfully ran their designs through the compliance suites,” says Baum. “Looking further out, tests and tools will be needed for features that are not strictly ISA-related, such as debug and trace standards, non-deterministic behavior found in memory models, and highly concurrent systems, and timing related security issues.”
Understanding the level of completeness for the compliance suite is critical. “Coverage metrics used in RTL design and verification are not applicable, as they are often functional and connected to specific microarchitectures,” says Davidmann. “Code coverage provides coverage of model source, which is useful to see how much of a model is exercised by tests but does not show how much of the specification is covered. Fault simulation provides instruction decode coverage which explores decodes of the instructions and mutates legal bits and detects that there is a test that stimulates and observes each bit.”
Davidmann provides results from their tools that demonstrate the existing level of compliance coverage.
Fig 2. Compliance coverage for RISC-V rv32i instruction set. Source: Imperas
Taking compliance further
With verification, demonstrating that a design does something positive is half the task. The other requirement is proving it does not do anything wrong. “In addition to compliance of the base ISA, you have to ensure that any modifications, extensions or features must also ensure the base ISA implementation remains unperturbed with later additions,” says McDermott.
There are two parts to this—verification of the extensions, and ensuring those extensions did no harm. “Perhaps the real verification issue is ensuring that a processor model remains compliant even when the instruction set has been extended,” says Kelf. “As other configurable processor companies discovered, the issue with instruction extensions is verifying not just the new instructions but ensuring that the existing instruction functionality is not broken. Any RISC-V verification platform must allow the easy inclusion of tests on top of the compliance suite that test the entire extended processor ISA thoroughly and completely. Reusable test systems such as Portable Stimulus are required for this.”
Formal verification often has been seen as a better way to show that things are not broken. “Full compliance must ensure not only that the core does what it is supposed to do, but also that it does not do what it is not supposed to do,” says Sven Beyer, product manager for Design Verification at OneSpin Solutions. “This latter requirement, which requires the exhaustive analysis of formal verification, is essential to detect hardware Trojans and other trust and security vulnerabilities.”
Again, the question can be raised if compliance involves things that are not defined by the specification, but are equally important to users of a core. Security is one example. “The question of security is an interesting one, since it is outside of the specification,” says Mentor’s Hand. “This is therefore not a compliance issue. This does not mean it should be ignored. But testing for the security needs a different approach, and there are existing tools and technologies such as formal verification that can be used here.”
This goes well beyond compliance. “Customers want ‘compliance plus’,” asserts Codasip’s Jones. “They want to adhere to the RISC-V ISA, but at the same time extend the instruction sets. The challenge becomes how to provide adequate coverage while keeping the test pool growth and maintenance effort at acceptable levels.”
And it may involve more than just the instruction set. “A critical component of the RISC-V open source architecture is the cache-coherent interconnect fabric, TileLink,” says Deepak Kumar Tala, chairman for SmartDV Technologies. “Ensuring compliance with the TileLink specification requires a complete verification solution that supports fast testbench development, a complete regression suite, and interactive analysis and debug.”
Conclusion
The RISC-V consortium and its members are taking compliance very seriously, but it’s not simple. They have to invent a lot of technology along the way. In addition, the industry has to step up with relevant tools, models, and methodologies to prevent fragmentation from happening.
The industry appears to be confident that it can make this happen, but all of this takes time.
Related Stories
Building Security Into RISC-V Systems
Part 2: Emphasis shifting to firmware, system-level architectures, and collaboration between industry, academia and government.
Open-Source RISC-V Hardware And Security
Part 1: The advantages and limitations of a new instruction set architecture.
RISC-V Inches Toward The Center
Access to source code makes it attractive for custom applications, but gaps remain in the tool flow and in software.
RISC-V: More Than A Core
Interest in the open-source ISA marks a significant shift among chipmakers, but it will require continued industry support to be successful.
 |
 |
 |
 |
 |
 |
 |
 |
 |  |
Leave a Reply