Trends In FPGA Verification Effort And Adoption: The 2018 Wilson Research Group Functional Verification Study

As contributors and pioneers in the digital revolution, we are often so busy creating and innovating that we are compelled to focus on the trees, never mind the forest. But as we are all aware, the more we know about the bigger picture, context, historical and projected trends, or simply how other people are doing the same thing, the more efficiently and successfully we can do our jobs.

Providing this kind of information is the reason we conduct the worldwide Wilson Research Group Functional Verification Study every two years. Its findings offer the FPGA and ASIC communities the knowledge they need to make the best methodology and tool choices for their design flows and business goals. It also informs the EDA industry in how to best assist and sustain those needs.

While multiple studies on IC/ASIC functional verification trends have been published, surveys specifically focused on FPGA design and verification trends do not exist. To fill this research gap we began studying the FPGA market in 2012. Only in 2018 did we have sufficient multi-year data points to identify verification trends and draw significant conclusions, the highlights of which we present here as part of a four part series on the Wilson Research Group 2018 study.

FPGAs have recently grown in complexity equal to many of today’s IC/ASIC designs. In this unbiased survey we quantify the impact of this growing complexity in terms of verification effectiveness and effort.

In our first article (see Trends in FPGA Effectiveness), we shared our findings on FPGA verification effectiveness. In this article, we will look deeper into FPGA trends pertaining to verification effort. We will also share our findings on the factors behind growing design complexity and the trends in verification technology adoption. We will close with some of our observations regarding various aspects of the FPGA market before turning to the ASIC market in the third and fourth articles.

Time, effort and productivity
There is no simple answer to the question, “How much effort was spent on verification in your last project?” In fact, it is necessary to look at multiple data points derived from multiple questions to truly get a sense of effort spent in verification. To try to assess the effort spent in verification, let’s begin by looking at one data point: the total project time spent in verification.

Fig. 1 shows the percentage of total FPGA project time spent in verification. You can see two extremes in this graph. In general, teams that spend very little time in verification are typically working on designs with a good deal of existing pre-verified design IP, which is integrated to create a new product. On the other extreme, projects that spend a significant amount of time in verification often have a high percentage of newly developed design IP that must be verified.

Fig. 1. Percentage of FPGA project time spent in verification

Overall, we found an increase in the average percentage of FPGA project time spent in verification during the period 2014 through 2018. This is an indicator of growing design and verification complexity.

Another verification effort indicator is engineering headcount. This is also significant because one of the biggest challenges in FPGA today is to control cost and engineering headcount. The most effective way to do this is to identify FPGA design and verification solutions that increase productivity. Therefore, to illustrate the need for productivity improvement, we looked at patterns of increasing engineering headcount.

Fig. 2. Mean peak number of FPGA engineers working on a project

Fig. 2 shows the mean peak number of FPGA engineers working on a project. While, on average, the demand for design engineers is growing at about a 4% CAGR (which is similar to that of IC/ASIC), the demand for verification engineers is growing at about a 10% CAGR. But verification engineers are not the only project stakeholders involved in the verification process. Design engineers spend a significant amount of their time in verification too, as shown in Fig. 3.

Fig. 3. Where FPGA design engineers spend their time

In 2018, design engineers spent, on average, 56% of their time involved in design activities and 44% of their time in verification. However, the data indicates a trend that FPGA design engineers are spending less time involved in verification tasks. There are two reasons for this trend. First, many FPGA projects have added verification engineers to their teams, which means design engineers can focus most of their effort on design. Second, in general there has been increased adoption of larger, more complex FPGAs, which has increased the design engineer’s workload.

Growing design complexity
One industry driver that has had a substantial impact on FPGA design and verification complexity is the emergence of new layers of design requirements (beyond basic functionality), which did not exist years ago. These include clocking requirements, security requirements, safety requirements, and requirements associated with hardware-softwareinteractions.

A significant change in FPGA designs in the last fifteen years is the movement toward SoC-class designs. For example, 63% of all FPGA designs contained one or more embedded processors, as shown in Fig. 4. Furthermore, 36% of all FPGA designs contain two or more embedded processors, while 14% include four or more embedded processors. SoC-class designs add a new layer of verification complexity to the verification process due to an increased number of design requirements. For example, SoC-class designs often require verification of hardware and software interactions, new coherency architectures, and complex network-on-a-chip interconnects.

Fig. 4. Number of embedded processor cores

In Fig. 5, we see that 90% of designs implemented as FPGAs contain two or more asynchronous clock domains. Verifying requirements associated with multiple asynchronous clock domains increases both verification workload and complexity. For example, a class of metastability bugs cannot be demonstrated on an RTL model using simulation. To simulate these issues requires a gate-level model with timing. However, static clock-domain crossing (CDC) verification tools have emerged to help identify clock domain issues directly on an RTL model at earlier stages in the design flow.

Fig. 5. Number of asynchronous clock domains

The number of projects implementing security features is increasing, as shown in Fig. 6. Security features such as security-assurance hardware modules that safely hold sensitive data (such as encryption keys, digital right management keys, passwords, and biometrics reference data) add requirements and complexity to the verification process.

Fig. 6. FPGA design projects implementing security features

Safety-critical design also increase requirements that contribute to complexity. Fig. 7 shows the increase in the number of FPGA projects working under one of multiple safety-critical development process standards or guidelines. Some of these safety-critical projects are further required to work under multiple safety standards or guidelines.

Fig. 7. FPGA design projects working on a safety-critical design

Tool, language and methodology adoption
To address growing design and verification complexity, we find that many FPGA projects have been forced to mature their verification processes. We found that the FPGA design space is about where the ASIC/IC design space was seven years ago in terms of verification maturity—and it is catching up quickly.

Fig. 8 shows the adoption trends for languages used to build testbenches and the projected adoption for next year based on study participants’ responses. It is not uncommon for FPGA projects to use multiple languages when constructing their testbenches. This practice is often due to legacy code as well as purchased verification IP.

Fig. 8. FPGA project verification language adoption

Historically, VHDL was the predominant language for FPGA testbenches, yet it has leveled off to about 45% adoption, and we see an increase in SystemVerilog adoption. Today, it is not unusual that the RTL design was created using VHDL, and the testbench was created using SystemVerilog. The increase in “Other” in 2018 is due to the recent adoption of various scripting languages by various FPGA projects (e.g., Python). Finally, the downward trend in VHDL and Verilog adoption in 2016 was probably an anomaly in the study. It is more likely that adoption has only slightly declined or remained flat.

The adoption trends for various base-class library and methodology standards are shown in Fig. 9. We found that the Accellera UVM is currently the predominant standard adopted to create FPGA testbenches. You might note that we are not showing historical trends for the Open Source VHDL Verification Methodology (OSVVM) and the Universal VHDL Verification Methodology (UVVM), since 2018 was the first year we tracked these methodologies.

Fig. 9. FPGA project methodology and base-class libraries adoption

Finally, FPGA project adoption trends for various assertion language standards are shown in Fig. 10, where SystemVerilog Assertions (SVA) is the predominate language. Similar to languages used to build testbenches, it is not unusual to find FPGA project teams creating their RTL in VHDL and then creating their assertions using SVA.

Fig. 10. FPGA project assertion language adoption

In general, formal solutions (i.e., formal property checking combined with automatic formal applications) are one of the fastest growing segments in functional verification technology. The adoption trends for formal property checking (e.g., model checking) and automatic formal applications are shown in Fig. 11. The adoption of formal property checking on FPGA projects is growing at an impressive 21% CAGR, and the adoption of automatic formal applications is growing at a 29% CAGR. Historically, the formal property checking process has required specialized skills and expertise. However, the recent emergence of automatic formal applications provides narrowly focused solutions that do not require specialized skills for adoption.

Fig. 11. FPGA project formal technology adoption trends

Fig. 12 shows the FPGA project adoption trends for various simulation-based techniques from 2012 through 2018, which include code coverage, functional coverage, assertions, and constrained-random simulation.

Fig. 12. FPGA project simulation technique trends

With maturity comes confidence
The study results show that the FPGA market has matured its verification processes. This maturity is likely due to the growing complexity of designs and related efforts to control cost and engineering headcount through the adoption of FPGA design and verification solutions that increase productivity.

Perhaps the most concerning finding from this year’s study relates to the number of FPGA projects with non-trivial bug escapes into production. However, we did find an interesting correlation between the improvement of reduced functional flaws contributing to non-trivial bug escapes and the maturing of FPGA projects’ functional verification processes. The data suggest that projects that are more mature in their functional verification processes will likely experience fewer bug escapes.

To test this claim, we partitioned the study participants into two groups: FPGA projects with no bug escapes and FPGA projects that experienced a bug escape. We then examined the percentage adoption of various verification techniques. The results are shown in Fig. 13.

Fig. 13. FPGA simulation technique adoption vs non-trivial bug escapes

These findings are statistically significant in that the group with no bug escapes tended to have higher adoption of various verification techniques, which suggest they are more mature in their verification process. However, what we are unable to measure from our study is how effective a project was in adopting any of these processes. For example, a project that experienced a bug escape could claim that they have adopted functional coverage, yet the fidelity of their functional coverage model might be poor due to their inexperience. From our study data, we are unable to assess successful or effective adoption for any particular verification technique.

Now that we’ve covered trends in FPGA functional verification, we will turn to ASIC/IC market trends in two articles appearing in the coming months.

Can’t wait? Learn more about trends in FPGA verification in the on-demand webinar Staying Competitive by Evolving Your FPGA Verification Methodologies. We also offer several courses on the latest in FPGA verification challenges and solutions, check out FPGA Verification Academy Course.