Is increased verification effort paying off?
“The more you know, the more you know you don’t know.” ― Aristotle, 4th C. BC
When Aristotle uttered this humble aphorism, he wasn’t telling us to throw up our hands and not bother with learning. He was encouraging us to continue digging deeper, to get answers and ask questions of those answers — that the thrills and rewards of study are truly without end. This is a big part of our motivation behind the roughly bi-annual Wilson Research Group Functional Verification studies: to increase our understanding of this vital industry so that we, and you, can make better decisions about how to go about our business.
This article is the final installment of a four-part series that presents the findings of the 2018 study. In this final article we discuss various IC/ASIC language and library adoption trends, low power trends, and data relevant to schedules, number of required spins, and classification of functional bugs. Finally, we will take a closer look at two interesting aspects of functional verification: non-trivial bug escapes and safety critical designs.
Language and Library Adoption
First we will look at language and library adoption trends. Figure 1 shows the aggregated adoption trends for languages used to create RTL designs across all market segments and all regions of the world. We see continual interest in SystemVerilog for RTL creation. It is not uncommon for IC/ASIC projects to use multiple languages when constructing their testbenches, often due to legacy code as well as purchased verification IP.
Figure 1. IC/ASIC Languages Used for RTL Design
Figure 2 shows the adoption trends for languages used to create IC/ASIC testbenches. The data suggest that the adoption rates for all languages used to create testbenches are either declining or flat. There is a continual increase in design teams working on designs less than 100K gates, which is an additional factor we need to consider when examining the findings in Figure 2. This can yield some interesting results in the study: very small projects typically do not apply advanced verification techniques, which can bias the overall industry verification technique adoption trends to appear as flat or declining. However, even factoring this bias into the results we find that SystemVerilog adoption is still starting to saturate or level off in the upper-70s range since most IC/ASIC projects are mature in their simulation processes for IP and subsystem verification.
Figure 2. IC/ASIC Languages Used for Verification (Testbenches)
Figure 3 shows the adoption trends for various IC/ASIC testbench methodologies built using class libraries.
Figure 3. IC/ASIC Methodologies and Testbench Base-Class Libraries
Here we see a decline in adoption of all methodologies and class libraries with the exception of Accellera’s UVM, whose adoption continued to increase between 2014 and 2018. Furthermore, our study revealed that UVM is projected to continue its growth over the next year. However, like SystemVerlog, it is expected to saturate or level off in the mid- to upper-70 percent range.
This was the first survey we asked the question concerning the new Accellera Portable Test and Stimulus Standard (PSS) language. PSS was not officially approved until late June 2018, and our study had started prior to its approval. We should be able to get a better sense of its true adoption rate with our next study.
Figure 4 shows the IC/ASIC industry adoption trends for various assertion languages, and again, SystemVerilog Assertions seems to have saturated or leveled off. This is just another indication of a mature industry, which has adopted standard processes for IP and subsystem verification.
Figure 4. IC/ASIC Assertion Language Adoption
Low Power Management
Now we will turn to IC/ASIC design and verification power trends. In Figure 5, we see that about 71 percent of today’s design projects actively manage power with a wide variety of techniques, ranging from simple clock-gating to complex hypervisor/OS-controlled power management schemes. This is essentially unchanged from our 2014 study.
Figure 5. ASIC/IC Projects Working on Designs That Actively Manage Power
Figure 6 shows the various aspects of power-management that design teams who actively manage power must verify. Many projects, since 2012, have moved to more complex power-management schemes that involve software control. This adds a new layer of complexity to a project’s verification challenges, since these more complex power management schedules often require emulation to fully verify.
Figure 6. Aspects of Power-Managed Design Verified
Since the power intent cannot be directly described in an RTL model, alternative supporting notations have recently emerged to capture power intent. Figure 7 shows the various standards used to describe power intent that have been adopted. You might note that the 2018 study was the first time we tracked the new UPF 3.0 standard.
Figure 7. Notation Used to Describe Power Intent
Effectiveness of ASIC/IC Verification
In an earlier article in this series, we provided data that suggest a significant amount of effort is being applied to IC/ASIC functional verification. An important question the various studies have tried to answer is whether this increasing effort is paying off. To help answer that question, let’s look at the survey findings in terms of schedules, number of required spins, and classification of functional bugs.
Figure 8 presents design completion time compared to the project’s original schedule. The bottom line is that meeting the originally planned schedule is still a challenge for most of the industry.
Figure 8. Design Completion Compared to Original Schedule
Other trends worth examining relate to the number of spins required between the start of a project and final production. Figure 9 shows this industry trend from 2012 through 2018. The data suggest that achieving first silicon success is getting worse, while achieving second silicon success has improved.
Figure 9. Required Number of Spins
Figure 10 shows various categories of flaws that are contributing to respins. Of course, multiple flaws can trigger a respin.
Figure 10. Types of Flaws Resulting in Respins
If you look at the root cause of bugs contributing to IC/ASIC respins you will note that both functional flaws and clocking flaws have improved since 2012. In other words, it is other factors shown in this graph that has led to a decrease in first silicon success, such as: fast or slow timing paths, IR drop, and mixed-signal issues.
Now there has been a common theme throughout this article series concerning the IC/ASIC market segment. That is, for the most part, the IC/ASIC market has done a remarkable job of maturing its functional verification processes over the past 15 years. And the data in Figure 10 suggest that this is paying off in terms of reduced logic and functional flaws.
Figure 11 examines the root cause of these logical or functional flaws. Design errors remain the leading cause of functional flaws. In addition, problems associated with changing, incorrect, and incomplete specifications are a common theme often voiced by many verification engineers and project managers.
Figure 11. Root Cause of Functional Flaws
Non-Trivial Bug Escapes and Safety Critical Designs
For this year’s study, we decided to do a deeper dive related to the following:
The study results show that the IC/ASIC market has matured its verification processes over time, and we see the FPGA market also starting to mature its processes. This maturity is likely due to the growing complexity of designs and related efforts to control cost and engineering headcount through the adoption of FPGA design and verification solutions that increase productivity.
Perhaps the most concerning finding from this year’s study relates to the number of FPGA projects with non-trivial bug escapes into production. However, we did find an interesting correlation between the improvement of reduced functional flaws contributing to non-trivial bug escapes and the maturing of FPGA projects’ functional verification processes. The data suggest that projects that are more mature in their functional verification processes will likely experience fewer bug escapes.
To test this claim, we partitioned the study participants into two separate groups: FPGA projects with no bug escapes and FPGA projects that experienced a bug escape. We then examined the percentage adoption of various verification techniques. The results are shown in Figure 12. The verification techniques related to bug escapes versus no bug escapes does not sum to 100 percent because we performed our analysis in terms of technique adoption independently on those two groups.
Figure 12. Process Maturity and Non-Trivial Bug Escapes
These findings are statistically significant in that the group with no bug escapes tended to have a higher adoption of various verification techniques, which suggest they are more mature in their verification process. However, what we are unable to measure from our study is how effective a project was in adopting any of these processes. For example, a project that experienced a bug escape could claim that they have adopted functional coverage, yet the fidelity of their functional coverage model might be poor due to their inexperience. From our study data, we are unable to assess successful or effective adoption for any particular verification technique.
The second aspect of our 2018 study that we decided to examine a little deeper relates to safety critical designs and their silicon success. Intuitively, one might think that the rigid and structured process required to adhere to one of the safety critical development processes (such as, DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601 for medical, etc.) would yield higher quality in terms of preventing bugs and achieving silicon success.
Figure 13 shows the percentage of FPGA projects that claimed to be working on a safety critical design.
Figure 13. Percentage of FPGA Projects Working on Safety Critical Designs
We partitioned the findings into two groups: projects working on non-safety critical and safety critical designs. Then, as shown in Figure 14, we examined the percent of bug escapes and no bug escapes for projects working on safety critical designs.
Figure 14. Non-Trivial Bug Escapes for Safety Critical vs. Non-Safety Critical FPGA Designs
Clearly, the data suggest that a development process adopted to ensure safety does not necessarily ensure quality. Perhaps this is non-intuitive. However, to be fair, many of the safety critical features implemented in today’s designs are quite complex and increase the verification burden.
This concludes the findings from the 2018 Wilson Research Group Study. We hope you found this information helpful in making technology, language, and methodology choices in your functional verification endeavors. And we hope to see you in 2020 with the next industry-wide Wilson Research Group Functional Verification Study. As another very smart man said:
“The more I learn, the more I realize how much I don’t know.” ― Albert Einstein, 20th C. AD
Can’t wait to learn more about the latest industry trends in ASIC/IC and FPGA functional verification? The Wilson Group study will be featured at this year’s DAC 2019 Verification Academy booth – view all of our scheduled sessions here.
Check out the first three articles in this series:
Part 1 – Trends In FPGA Effectiveness: The 2018 Wilson Research Group Functional Verification Study
Part 3 – The Weather Report: 2018 Study On IC/ASIC Verification Trends
Source for all images: Wilson Research Group and Mentor, A Siemens Business, 2018 Functional Verification Study © Mentor Graphics Corporation
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply