Confusion Persists In Verification Terms

I find it amazing that an area of technology that attempts to show, beyond a reasonable doubt, that a design will work before it is constructed can be so bad at getting some basic things right. I am talking about verification terminology.

I have been in this industry for over 40 years and it is not improving. In fact, it is getting worse. The number of calls I have with people where they have to spend time defining what they mean by basic terms is ridiculous. Some of this confusion goes back a long time. Some of it was started and perpetuated by a large semiconductor company. And some of it has been caused by marketing people who want to make something appear better than it actually is.

Let’s start with the earliest forms of confusion. This resulted in terms such as tests and testcases and testbenches. Forty years ago, simulation was a tool for the production test industry. Designers didn’t simulate. Testers were huge, expensive machines and you needed to develop the tests that would run on those testers in an off-line manner. A set of vectors was developed that would be loaded into the tester, fed into the device under test, and the results captured and compared to golden vectors. The coverage model was the stuck-at model, because it had been shown that while not all manufacturing faults resulted in something being stuck to either 0 or 1, it provided a reliable metric to gauge the quality of the test.

Then along came the semiconductor industry, and the notions of verification evolved from the tester industry. Some companies correctly tried to change the name to verification-bench, verification-cases, and design under verification, but those terms are almost never used. Terms like verification and validation were defined by many organizations and started to appear in high-tech dictionaries. While there was some variation between those definitions, they were all essentially the same. Verification is making sure that an implementation matches a specification, and validation is establishing if the specification was correct. Very few companies did validation back then. That was what markets did. Companies either bought your product or they bought one from your competitor, which had developed a slightly different product.

There are some grey areas with verification. For example, when mapping a design into an emulator or FPGA prototyping system, you are creating an implementation that is not the one you eventually will implement in silicon. So when you verify that implementation, which you don’t actually care about, are you doing an act of verification or validation? The implementation is a proxy for the eventual implementation. You are trusting the tools that they are functionally equivalent, so long as all of the rules have been followed, and thus you are verifying the RTL that represents the design. The RTL has not suddenly become a specification. It is an implementation, and the other implementations are derived from it. It is still an act of verification.

What about a processor? This has become a lot more of an active topic these days because of RISC-V, and again it is actually quite simple. The dividing line is the ISA, and that is the specification. So are you validating that the ISA is actually the one that you meant, or are you verifying that your implementation matches the ISA specification? New terms such as conformance and qualification also come into play here, and these are loose terms defined by an organization to say you passed our defined set of test cases.

If you do not modify the ISA, you do not do validation. That has been done for you by RISC-V International. You are verifying an implementation. If you run software on that processor, you are doing it to see if the processor works. The only time it becomes validation is if you are assessing if the processor can execute that software efficiently, or if by modifying the ISA its performance would be improved. If you identify a bottleneck in the processor while running the software, that is verification because it relates to an implementation. Your implementation is not optimal and needs to be fixed.

When you attempt to apply the terms to the software itself, it does become a little more difficult because software is rarely specified. There is only the implementation, so what are you comparing it to? Without a specification or a golden model, they really are doing test — poking the software and looking to see if it does the right thing. Incidentally, shmooing is actually a term that comes from the test industry.

I also have heard the validation term used when related to standards, such as a new memory interface or other high-speed interface. People say they are attempting to ascertain if the specification is correct, when most of the time they are really just asking, “Do we know how to implement it, or can it be implemented using the technology available to us today?” If they do change the specification based on attempts to implement, I guess it could be defined as validation, even though for the wrong reason. The specification may have been correct, but may not be useful if nobody can implement it.

Then there are terms like functional coverage, which is a joke. There is nothing functional about functional coverage. It is an implementation observation that a human has ascertained is a proxy for a certain piece of functionality having been executed. It is a hideous metric and actually tells you nothing about completeness. At least with most implementation metrics, like code coverage or path coverage, they actually can tell you something meaningful in a negative sense — they tell you when you are not done. They identify holes in your verification. Functional coverage tells you nothing. It is not a good implementation metric, nor is it in any way tied to the coverage of the specification.

And while we are on the subject of specifications, here is one of my favorites – Portable Stimulus. This name was devised to ensure it didn’t sound competitive to SystemVerilog within the standards body. It is a model of intended functionality, and as such is perhaps the first true verification environment that the industry has attempted. SystemVerilog is nothing more than a mishmash of pieces from which a somewhat deficient verification environment can be created. Portable Stimulus also defines a credible coverage model – not the one added to it to make it look like SystemVerilog, but the natural one based on graph coverage – something that mathematicians have worked on for a long time.

So if you ever think about using the term validation, make sure you are really ascertaining if the specification is correct. Most people and most companies don’t do validation. Their marketing department does that for them in very loose terms.