Verification Effectiveness In The Face Of FPGA Complexity: The 2020 Wilson Research Group Functional Verification Study

Making informed decisions backed by good data is the key to success in highly competitive, robust markets such as FPGA design and verification.

Helping our community in that endeavor is the motivation behind the worldwide Wilson Research Group Functional Verification Study. We also use that information to make sure our research and development efforts continue to deliver the solutions our customers need today and tomorrow.

In this space over the next four months, a series of four articles will summarize the key findings from the 2020 Wilson Research Group Functional Verification Study. The first two will focus on FPGA trends; the third and fourth on the IC/ASIC market. We will begin with an overview of the study itself and the procedures we followed to maintain the integrity, validity, and inclusiveness of the survey.

The 2020 Study: Sampling and bias mitigation

The 2020 Wilson Research Group study results are a continuation of a series of industry studies on functional verification that began in 2007 and continued biennially starting in 2010 through 2020. Each of these studies was modeled after the 2002 and 2004 Collett International Research, Inc. studies and focused on the IC/ASIC market. We began surveying the FPGA market in 2012.

For the purpose of our study, a randomized sampling frame was constructed from multiple acquired industry lists. This enabled us to cover all regions of the world and all relevant electronics industry market segments. While Siemens EDA architected the study in terms of questions and then compiled and analyzed the final results, we commissioned Wilson Research Group to execute our study. After data cleaning the results to remove inconsistent, incomplete, or random responses, the final sample size consisted of 1492 eligible participants (i.e., n=1492).

Figure 1 compares the percentage of 2016, 2018 and 2020 study participants (i.e., design projects) by targeted implementation for both IC/ASIC and FPGA projects. It is important to note that targeted implementation does not represent silicon volume in terms of the global semiconductor market since a single project could account for a significant portion of semiconductor market revenue.

Fig. 1: Study participants by targeted implementation.

Since all survey-based studies are subject to sampling errors, we attempt to quantify this error in probabilistic terms by calculating a confidence interval. For our study, we determined the overall margin of error to be ±3% using a 95% confidence interval. In other words, this confidence interval tells us that if we were to take repeated samples from a population, 95% of the samples would fall inside our margin of error ±3%, and only 5% of the samples would fall outside.

A big concern when architecting any type of study relates to addressing three types of bias: non-response bias, sample validity bias, and stakeholder bias.

The 2020 study demographics saw an 11% decline in participation from North America (figure 2) but an increase in participation from Europe and India. The shift in demographic balance can introduce potential non-response biases in the findings that need to be considered. For example, regional shifts in participation can influence the findings for design and verification language adoption trends.

Fig. 2: 2018-2020 study demographics.

Sample validity bias occurs when not every member of a studied population has an equal chance to participate. For example, when a technical conference surveys its participants the data might raise some interesting questions, but unfortunately, it does not represent members of the population that were unable to participate in the conference. To safeguard against this kind of bias, the sample frame for our study was not derived from Mentor’s customer list, since that would be a biased study. For our study, we carefully chose a broad set of independent lists that, when combined, represented all regions of the world and all electronic design market segments. We reviewed the participant results in terms of market segments to ensure no was inadvertently excluded or under-represented.

Stakeholder bias occurs when someone who has a vested interest in survey results can complete an online study survey multiple times and urge others to complete the survey in order to influence the results. To address this problem, a special code was generated for each study participation invitation that was sent out. The code could only be used once to fill out the survey questions, preventing someone from taking the study multiple times or sharing the invitation with someone else.

Drivers behind growing FPGA complexity

FPGAs have recently grown in complexity equal to many of today’s IC/ASIC designs. Some of the more interesting trends in the 2020 study related to FPGA designs are as follows.

• The FPGA market continues to struggle with non-trivial bug escapes into production.
• The FPGA market is rapidly maturing its verification processes to address growing complexity.
• FPGA projects that are more mature in their functional verification processes are likely to experience fewer bug escapes.

The IC/ASIC market in the mid- to late-2000 timeframe underwent growing pains to address increased verification complexity. Similarly, we find today’s FPGA market is being forced to address growing verification complexity. With the increased capacity and capability of today’s complex FPGAs and the emergence of high-performance SoC programmable FPGAs (e.g., Xilinx Zynq UltraSCALE+, Intel Stratix, and Microsemi SmartFusion2), traditional lab-based approaches to FPGA verification and validation are becoming less effective. We will quantify the ineffectiveness of today’s FPGA verification processes in terms of nontrivial bug escapes into production. But first, we will look at some of the contributors to complexity in FPGA designs: an increase in embedded processors, asynchronous clock domains, and security and safety features.

The number of FPGA projects that have incorporated a RISC-V processor in their design was 23%. In addition, the number of FPGA projects that have incorporated some type of AI accelerator processor (e.g., TPU, etc.) was 19%.

Fig. 3: Number of embedded processor cores.

In figure 4, we see that 92% of designs being implemented as FPGAs contain two or more asynchronous clock domains. Verifying requirements associated with multiple asynchronous clock domains has increased both the verification workload and complexity. For example, a class of metastability bugs cannot be demonstrated on an RTL model using simulation. To simulate these issues requires a gate-level model with timing, which is often not available until later stages in the design flow. However, static clock-domain crossing (CDC) verification tools have emerged and are being adopted to help identify clock domain issues directly on an RTL model at earlier stages in the design flow.

Fig. 4: Number of asynchronous clock domains.

Today we find that 43% of FPGA projects add security features to their designs. Examples of security features include security assurance hardware modules (e.g., a security controller) that are designed to safely hold sensitive data, such as encryption keys, digital right management (DRM) keys, passwords, and biometrics reference data. These security features add requirements and complexity to the verification process.

In 2020, we find that 40% of all FPGA projects are working under one of multiple safety-critical development process standards or guidelines. In figure 5 we show the specific breakdown for the various standards for those projects working under a safety-critical development process standard or guideline. Note that some projects are required to work under multiple safety standards or guidelines, which is why the percentage adoption sums to more than 100 percent. For example, IEC61508 and IEC61511.

Fig. 5: Safety critical development standard used on FPGA projects.

Verification effectiveness in the face of complexity

With this growing complexity, how have FPGA project results faired in terms of verification effectiveness? IC/ASIC projects have often used the metric “number of required spins before production” as a benchmark to assess a project’s verification effectiveness. Unfortunately, FPGA projects have no equivalent metric. As an alternative, our study asked the FPGA participants “how many non-trivial bugs escaped into production?” The results, shown in figure 6, are somewhat disturbing. In 2020, only 17% of all FPGA projects were able to achieve no bug escapes into production, which is worse than IC/ASIC in terms of first silicon success, and for some market segments, the cost of field repair can be significant. For example, in the mil-aero market, once a cover has been removed on a system to upgrade the FPGA, the entire system needs to be revalidated.

Fig. 6: Non-trivial FPGA bug escapes into production.

Figure 7 shows various categories of design flaws contributing to FPGA non-trivial bug escapes. The percentage of “logic or functional flaws” remains the leading cause of bugs. New flaws being tracked in the 2020 study are associated with safety (8%) and security (6%) features. Obviously, multiple flaws can contribute to bug escapes, which is the reason the total percentage of flaws sums to more than 100%.

Fig. 7: Types of flaws resulting in FPGA bug escapes.

Figure 8 demonstrates the root cause of logical or functional flaws by various categories. The data suggest design errors are the leading cause of functional flaws, and the situation is worsening. In addition, problems associated with changing, incorrect, and incomplete specifications are a common theme often voiced by many verification engineers and project managers.

Fig. 8: Root cause of FPGA functional flaws.

In addition to bug escape metrics that we used to determine an FPGA project’s effectiveness, another metric we tracked was project completion compared to the original schedule, as shown in figure 9. Here we found that 68% of FPGA projects were behind schedule. One indication of growing design and verification complexity is reflected in the increasing number of FPGA projects missing schedule by more than 50 percent during the period 2012 through 2020.

Fig. 9: Actual FPGA project completion compared to original schedule.

What lies ahead?

So far we’ve looked at the foundations of an inclusive, comprehensive, and unbiased survey and shared our findings on FPGA verification effectiveness. In the next article, we will look deeper into FPGA trends in terms of project resources and verification technology adoption. We will close that article with some of our conclusions regarding various aspects of the FPGA market based on this year’s study.

If your eager for more right now, check out the full paper: 2020 Wilson Research Group Functional Verification Study FPGA Functional Verification Trend Report.