中文 English

Confusion Persists In Verification Terms

Some of it is marketing, some of it technological change, but the problem is getting worse.

popularity

I find it amazing that an area of technology that attempts to show, beyond a reasonable doubt, that a design will work before it is constructed can be so bad at getting some basic things right. I am talking about verification terminology.

I have been in this industry for over 40 years and it is not improving. In fact, it is getting worse. The number of calls I have with people where they have to spend time defining what they mean by basic terms is ridiculous. Some of this confusion goes back a long time. Some of it was started and perpetuated by a large semiconductor company. And some of it has been caused by marketing people who want to make something appear better than it actually is.

Let’s start with the earliest forms of confusion. This resulted in terms such as tests and testcases and testbenches. Forty years ago, simulation was a tool for the production test industry. Designers didn’t simulate. Testers were huge, expensive machines and you needed to develop the tests that would run on those testers in an off-line manner. A set of vectors was developed that would be loaded into the tester, fed into the device under test, and the results captured and compared to golden vectors. The coverage model was the stuck-at model, because it had been shown that while not all manufacturing faults resulted in something being stuck to either 0 or 1, it provided a reliable metric to gauge the quality of the test.

Then along came the semiconductor industry, and the notions of verification evolved from the tester industry. Some companies correctly tried to change the name to verification-bench, verification-cases, and design under verification, but those terms are almost never used. Terms like verification and validation were defined by many organizations and started to appear in high-tech dictionaries. While there was some variation between those definitions, they were all essentially the same. Verification is making sure that an implementation matches a specification, and validation is establishing if the specification was correct. Very few companies did validation back then. That was what markets did. Companies either bought your product or they bought one from your competitor, which had developed a slightly different product.

There are some grey areas with verification. For example, when mapping a design into an emulator or FPGA prototyping system, you are creating an implementation that is not the one you eventually will implement in silicon. So when you verify that implementation, which you don’t actually care about, are you doing an act of verification or validation? The implementation is a proxy for the eventual implementation. You are trusting the tools that they are functionally equivalent, so long as all of the rules have been followed, and thus you are verifying the RTL that represents the design. The RTL has not suddenly become a specification. It is an implementation, and the other implementations are derived from it. It is still an act of verification.

What about a processor? This has become a lot more of an active topic these days because of RISC-V, and again it is actually quite simple. The dividing line is the ISA, and that is the specification. So are you validating that the ISA is actually the one that you meant, or are you verifying that your implementation matches the ISA specification? New terms such as conformance and qualification also come into play here, and these are loose terms defined by an organization to say you passed our defined set of test cases.

If you do not modify the ISA, you do not do validation. That has been done for you by RISC-V International. You are verifying an implementation. If you run software on that processor, you are doing it to see if the processor works. The only time it becomes validation is if you are assessing if the processor can execute that software efficiently, or if by modifying the ISA its performance would be improved. If you identify a bottleneck in the processor while running the software, that is verification because it relates to an implementation. Your implementation is not optimal and needs to be fixed.

When you attempt to apply the terms to the software itself, it does become a little more difficult because software is rarely specified. There is only the implementation, so what are you comparing it to? Without a specification or a golden model, they really are doing test — poking the software and looking to see if it does the right thing. Incidentally, shmooing is actually a term that comes from the test industry.

I also have heard the validation term used when related to standards, such as a new memory interface or other high-speed interface. People say they are attempting to ascertain if the specification is correct, when most of the time they are really just asking, “Do we know how to implement it, or can it be implemented using the technology available to us today?” If they do change the specification based on attempts to implement, I guess it could be defined as validation, even though for the wrong reason. The specification may have been correct, but may not be useful if nobody can implement it.

Then there are terms like functional coverage, which is a joke. There is nothing functional about functional coverage. It is an implementation observation that a human has ascertained is a proxy for a certain piece of functionality having been executed. It is a hideous metric and actually tells you nothing about completeness. At least with most implementation metrics, like code coverage or path coverage, they actually can tell you something meaningful in a negative sense — they tell you when you are not done. They identify holes in your verification. Functional coverage tells you nothing. It is not a good implementation metric, nor is it in any way tied to the coverage of the specification.

And while we are on the subject of specifications, here is one of my favorites – Portable Stimulus. This name was devised to ensure it didn’t sound competitive to SystemVerilog within the standards body. It is a model of intended functionality, and as such is perhaps the first true verification environment that the industry has attempted. SystemVerilog is nothing more than a mishmash of pieces from which a somewhat deficient verification environment can be created. Portable Stimulus also defines a credible coverage model – not the one added to it to make it look like SystemVerilog, but the natural one based on graph coverage – something that mathematicians have worked on for a long time.

So if you ever think about using the term validation, make sure you are really ascertaining if the specification is correct. Most people and most companies don’t do validation. Their marketing department does that for them in very loose terms.



7 comments

Tudor Timi says:

Using the term “regression” instead of “test suite” is my pet peeve. A “regression” is what happens when the previously developed and tested system doesn’t work properly after a change.

Rajeev Ranjan says:

Hello Brian –
Couldn’t resist writing a couple of items that I disagree with:
(a) You write — “For example, when mapping a design into an emulator or FPGA prototyping system, you are creating an implementation that is not the one you eventually will implement in silicon. So when you verify that implementation, which you don’t actually care about, are you doing an act of verification or validation?”
I am puzzled — as to why would there even be a confusion treating “emulation” and “prototyping” process as “validation” — in your original definition? If one agrees that “validation” is meant to verify specification – then there should not be any discussion around traditional RTL simulation, emulation, prototyping be even considered a “validation” step. Just saying.
(b) This one is mind boggling to me. Not sure how you are indicating that “functional coverage” is a joke. I agree with your statement that variety of code coverage metric would tell you what you are missing. That is true in general. Coverage metrics should be used — more for what is being missed — vs what has being covered. But “functional coverage” as a metric is indeed useful. This is independent of underlying verification technique – simulation / formal verification / emulation. When you have not covered a “functional event” in your verification process — that’s one huge piece of important information. Why is it less important than the traditional line/branch coverage? That simply does not make sense.

Brian Bailey says:

Hi Rajeev – On your first point, so many people think that as soon as software is involved, that you are no longer verifying the hardware. You may also be verifying the software, but you continue to be verifying the hardware as well, except under very specific conditions. As for the second point, it is because it takes intelligence, and a lot of it, to make a good coverage model. That I deem to be a bad metric. Also, there is no easy way to go from a coverage hole to the creation of a scenario that would cover it and even more difficult to constrain a UVM testbench to attack it. When you talk to verification teams, one of the things they struggle with is the creation of a good coverage model, and that is just the ones who realize what they are doing. Ask Harry Foster what he thinks about people who check off – using functional coverage – to mean they have 10 coverage points for a large design.

Yoav Hollander says:

Hi Brian

It is, indeed, annoying that verification terminology does not seem to converge.

Here is a related post I wrote a while ago, which you may find entertaining: https://blog.foretellix.com/2015/08/09/terminology-for-multiple-verification-tribes/

Kumar says:

Can you please comment about the usage of sequencer in uvm environment?

Gaurav Jalan says:

Good summary on the illusion we work on. Interestingly the base of verification itself is : No proof of BUG & not Proof of no BUG 🙂

Had posted on similar lines –
https://whatisverification.blogspot.com/2010/10/proof-of-no-bug-or-no-proof-of-bug.html

Tom Melham says:

“Symbolic Execution” used to refer to pathwise symbolic exploration of software executions.

Leave a Reply


(Note: This name will be displayed publicly)