The verification challenges that are addressable by ML and the techniques and algorithms that show promise.
By Dan Yu, Harry Foster, and Tom Fitzpatrick
Welcome to the era of EDA 4.0, where we are witnessing a revolutionary transformation in electronic design automation driven by the power of artificial intelligence. The history of EDA can be delineated into distinct periods marked by significant technological advancements that have propelled faster design iterations, improved productivity, and furthered the development of intricate electronic systems.
Notably, the advent of EDA 1.0 was ushered in by the introduction of SPICE (simulation program with integrated circuit emphasis) at the University of California, Berkeley, in the early 1970s, which revolutionized circuit design.
In the 1980s and early 1990s, EDA 2.0 emerged as a result of the development of efficient place-and-route algorithms. This period, also known as the RTL era, witnessed a transition from gate-level design to higher-level abstractions, with RTL design enabling circuit descriptions at the register-transfer level, thereby improving simulation performance. This period witnessed a significant milestone with the introduction of logic synthesis.
The rise of system-on-chip (SoC) designs in the late 1990s and early 2000s marked a pivotal moment that led to EDA 3.0. This era witnessed the emergence of an IP development economy coupled with design reuse methodologies. EDA tools and standards were developed to support the design, verification, and validation of SoCs, enabling engineers to manage the escalating complexity of SoC-class designs.
In many aspects, EDA 4.0 aligns with the broader trends of the Industrial Revolution 4.0, which is rapidly changing the way companies manufacture, improve, and distribute their products, led in part by the digitization of the manufacturing sector. EDA 4.0 has evolved to facilitate the design of intelligent and connected devices, harnessing the potential of cloud computing and artificial intelligence (AI) and machine learning (ML) capabilities.
EDA tools now incorporate machine learning, virtual prototyping, a digital twin, and system-level design methodologies to bring accelerated verification, automated verification workflows, and increased verification accuracy to EDA products. The EDA 4.0 era promises optimized product performance, reduced time-to-market, and streamlined development and manufacturing processes.
In this article, we delve into the state-of-the-art implementation of ML solutions specifically tailored for functional verification. We explore the challenges addressable by ML and present novel techniques and algorithms that hold relevance in this domain.
Table 1 summarizes the topics and subtopics applicable to functional verification when including all functional verification topics in the general perspective of programming code verification. Italicized text indicates subtopics that were unexplored in other general survey publications.
Table 1: Topics of ML applications in functional verification.
Requirement engineering in functional verification is the process of defining, documenting, and maintaining verification requirements, which is critical to ensure the excellent quality of the underlying IC design.
Requirement definition involves translating ambiguous natural language (NL) verification objectives into verification specs with formality and precision. The quality of the translation directly dictates the correctness of the verification. Traditionally this process is laborious and consumes substantial design cycles with several iterations of manual proofing to ensure quality.
Two groups of classic approaches have been proposed to automate translation. One group of approaches is to introduce constrained natural language (CNL) to formalize the specification drafting, followed by a template-based translation engine. This approach requires significant upfront investment in developing a powerful CNL syntax and a comprehensive compiler/template system to ensure it is powerful enough to address most of the requirements encountered in functional verification. Furthermore, it burdens developers with learning an additional language, which prevents the idea from becoming widely accepted.
The other group leverages the classic natural language process (NLP) to parse NL specifications and extract relevant key elements to formulate formal specifications.
The advance of ML translation in the NL domain has made fully-automated machine translation commercially feasible and sometimes exceeds the performance of average human translators. It has kindled the hope of leveraging large-scale trained NL models with up to billions of parameters to directly translate NL specifications to verification specs in SystemVerilog Assertions (SVA), Property Specification Language (PSL), or other languages. Several attempts to do a successful end-to-end translation have been observed, but none have been made production ready. The major hurdle to this approach is the scarcity of available training datasets that pair NL specifications with their formal translation. The most extensive datasets are merely about 100 sentence pairs. The number pales in comparison to their NL peers, which routinely come in millions or even billions of sentence pairs.
Converse to requirement definition, summarization looks at the code and translates it into a human-understandable NL summary. It assists developers in reading less ideally maintained code or understanding complex logic. An ideally implemented code summarization can insert inline documentation into code blocks or generate separate documentation. With its help, the maintainability and documentation of the code can be significantly improved.
The application of ML on code summarization has been experimented with in more popular computer languages, e.g., Python and JavaScript. Several groups of approaches have experimented with various degrees of success. Information retrieval (IR) based approaches focus on applying NLP to source code and looking for similar code with existing summarization in place. This group of approaches heavily relies on the quality of the existing code with summaries. Its use is only possible within a close organization where many existing code repositories are readily available. Heuristics-based approaches instead try to define specific rules based on heuristics identified in a module’s definition, e.g., a module with many submodules of basic read / write command lines might be considered a memory module. Therefore, a summary can be constructed from a predefined pattern for the memory module.
At the time of this writing, code summarization in IC design verification has not yet been reported in any literature. It is reasonable to be optimistic that the success of other languages can be realized in IC design and verification, which has yet to be confirmed by the research community. In particular, recent progress with cross-language models might help transfer learned knowledge from other programming languages to IC design. However, in addition to the challenges general to ML on code summarization, the intrinsic temporal parallelism in IC design and verification code can present challenges uncommon in other programming languages.
Specification mining has been a long-term software engineering topic. As an alternative to manually drafting specifications, it extracts specifications indirectly from the design under test (DUT) execution. ML can be applied to mine recurring patterns from simulation traces. It can help automate either simulation-based coverage closure or formal verification. It is assumed that commonly recurring patterns might be the expected behavior of the DUT. Alternatively, an event pattern rarely occurs in the traces can be regarded as an anomaly; therefore, it can be used for diagnostic and debugging purposes.
ML has been applied in pattern discovery and anomaly detection across many domains where temporal data of a complex system are available. Azeem et al. propose a general software engineering approach where ML is used to discover formal specifications from observing protocol traces and finding the possible problematical implementation of the protocol. Successful experiments have inspired interesting follow-up research projects in specification mining with ML.
As the cost of fixing a bug grows exponentially along the stages of IC development, static code analysis offers an attractive option to improve code quality and maintainability at an earlier stage of design development.
Code smell refers to suboptimal design patterns in source code, which might be syntactically and semantically correct, but violate best practices and can lead to poor code maintainability. A particular example is code duplication, where the same function is implemented multiple times across a project or the entire code base. Some copies can have a particular bug fixed in a relatively short period, while the same bug goes unnoticed in other copies.
Classic code smell detection relies on defined heuristic rules to identify patterns in the source code. Instead of manually developing these rules and metrics in the static code analysis tools, an ML-based approach can be trained on a large amount of available source code to identify code smells. Research has proven that smell detection with ML can lead to universal code smell detection and significantly fewer pattern implementation efforts. The resulting smell score can then be used for code quality assessment and help developers improve product quality consistently. Furthermore, ML-based code refactoring might provide helpful hints on improving code smell or even further some candidate changes.
The application of ML in functional verification is not yet visible, and the unavailability of large training datasets has prevented existing research from fully exploiting this solution’s potential.
Developers working on IC design can be most productive when the proper tools are provided. Simple code completion is a standard feature in the modern Integrated Development Environment (IDE). However, more advanced techniques involving deep learning were proposed and are maturing quickly. It is now possible to train ANNs with billions of parameters from many large-scale open-source code repositories to give reasonable recommendations of code snippets from developers’ implementation intent or the context.
ML might also help IC developers stay productive with semantic code search, which allows retrieving relevant code by NL queries. As code is usually full of various abbreviations and technical jargon, semantic searches can be more effective in finding relevant code snippets without correctly spelling the key variable, function, or module names. While similar to semantic search in many existing search engines, semantic code search is able to help find abbreviated and highly technical code with vague concepts. The mean reciprocal rank of the best model can already achieve usable scores of 70%.
Although theoretically the same ML techniques applied to other programming languages can be applied to IC design, no research has been published yet on coding assistance.
Recent surveys indicate that functional verification is still the most time-consuming step in IC design, and functional and logic errors are still the most important cause of a respin. Any improvement in the speed of functional verification will significantly impact the quality and productivity of IC design. ML has been used in both formal and simulation-based verification for their acceleration.
Formal verification uses formal mathematic algorithms to prove the correctness of a design. Modern formal verification orchestration employs formal algorithms to target designs of different sizes, types, and complexities. Experience and heuristics can help developers select the most appropriate algorithms from the library for a specific problem.
As a statistical method, ML cannot directly address formal verification problems. However, it has been proven to be very helpful in formal orchestration. With its prediction of computing resources and probability of solving a problem, it is possible to schedule formal solvers to best use these resources to shorten the verification time by first scheduling the most promising solvers with lower compute resource consumption. The Ada-boost decision tree-based classifier can improve the ratio of solved instances from the baseline orchestration from 95% to 97%, with an average speed up of 1.85. Another experiment was able to predict resource requirements of formal verification with a 32% average error. It iteratively applies feature engineering to carefully select features from DUT, properties, and formal constraints, which are then used to train a multiple linear regression model for resource requirements prediction.
Contrary to formal verification, simulation-based verification usually cannot ensure complete correctness in the design. Instead, the design is put under a test bench with certain random or fixed-pattern input stimuli applied, while the outputs are compared to the reference outputs to verify if the design’s behavior is expected. While simulation is the bread and butter of functional verification, simulation-based verification can also suffer from long verification times. It is not uncommon for the verification of a complex design to take weeks to complete.
A promising idea being discussed and experimented with is to use of ML to model and predict the behavior of a complex system. The Universal Approximation Theorem proves that a multi-layer perceptron (MLP), a feed-forward ANN with at least one hidden layer, can approximate any continuous function with arbitrary accuracy. Whereas normalized recurrent neural networks (RNNs), a specialized form of ANN, are proven to approximate any dynamic system with memory. Advanced ML accelerator hardware has made it possible that ANNs can model the behaviors of some IC design modules to accelerate their simulations. Significant acceleration may be achieved depending on the capability of AI accelerators and the complexity of the ML models.
Besides manually defined test patterns, standard techniques employed in simulation-based verification include random test generation and graph-based intelligent testbench automation. Due to the “long tail” nature of coverage closure, even a tiny efficiency improvement can easily result in significantly reduced simulation time. Much research on the application of ML to functional verification has focused on this area.
Extensive ML studies have demonstrated that they can do better than random test generation. Most research employs a “black box model,” assuming that a DUT is a black box whose inputs can be controlled and outputs monitored. Optionally, some testing points can be observed. The research does not seek to understand the behavior of the DUT. Instead, the focus is spent on reducing unnecessary tests. They employ various ML techniques to learn from historical input/output/observation data to tune the random test generators or eliminate tests that are unlikely to be useful. In a recent development, a reinforcement learning (RL) based model was used to learn from a DUT’s output and predict the most probable tests for a cache controller. When the reward given to the ML model is the FIFO depths, the experiments are able to learn from historical results and hit the full target FIFO depths in several iterations, while the random test generation-based approach is still struggling to hit more than 1. An ML architecture with a much finer granularity requires that an ML model be trained for every cover point. A ternary classifier is also employed to help decide if a test shall be simulated, discarded, or used to retrain a model further. Support vector machine (SVM), random forest, and deep neural network are all experimented on a CPU design. It can close 100% coverage with 3x to 5x fewer tests. Further experiments on FSM and non-FSM designs have demonstrated 69% and 72% reductions compared to directed sequence generation. However, most of these results still suffer from the limitation of the statistical nature of ML. A more comprehensive review of ML-based coverage-directed test generation (CDG) gives an overview of several ML models and their experiment results. Bayesian Network genetic algorithms and genetic programming approaches, Markov model, data mining, and inductive logic programming are all experiments with various degrees of success.
In all the approaches discussed, an ML model can make a prediction based on the learning from historical data it has gathered but has the minimal capability to predict the future, i.e., which test might be a more promising option for hitting an uncovered test target. As this kind of information is not yet available, the best they can do is pick the tests that are the most irrelevant to the historical tests. Another promising experiment explored a different approach, where the DUT is regarded as a white box, and the code is analyzed and converted to a Control / Data Flow Graph (CDFG). A gradient-based search on a trained Graph Neural Network (GNN) is used to generate tests for a predefined test target. The experiments on IBEX v1, v2, and TPU achieved 74%, 73%, and 90% accuracies at coverage prediction when trained with 50% cover points. Several additional experiments also confirm that the gradient search method employed is insensitive to the GNN architecture.
It is noted that due to the unavailability of training data, most of these ML approaches only learn from each design without exploiting any prior knowledge from other similar designs.
Bug analysis aims to identify potential bugs, localize the code blocks containing them, and give fix suggestions. Recent surveys found that verification of an IC spends roughly the same amount of time as it does in design and that functional bugs contribute to about 50% of respins for an ASIC design. Therefore, it is critically important that these bugs can be identified and fixed in the early functional verification stage. ML has been employed to help developers detect bugs in designs and find bugs faster.
Three progressive problems need to be solved to speed up bug hunting in functional verification, namely, clustering of bugs by their root causes, classification of root causes, and suggestion of fixes. Most research focuses on the first two, with no research results on the third one available yet.
Semi-structured simulation log files can be used for bug analysis. It extracts 616 different features from metadata and message lines from log files of undisclosed designs. The experiment on clustering achieved Adjusted Mutual Information (AMI) of 0.543 with K-means and agglomerative clustering and 0.593 with DBSCAN even after feature dimensionality reduction, far from ideal clustering when AMI achieves 1.0. Various classification algorithms were also tested to determine their accuracy in solving problem 2. All the algorithms, including random forest, Support Vector Classification (SVC), decision tree, logistic regression, K-neighbors, and naïve Bayes, are compared on their power to predict root causes. The best score was achieved by random forest with 90.7% prediction accuracy and 0.913 F1 scores. Another approach proposes to use a labeled dataset from code commit to train a gradient boosting model, where more than 100 features about authors, revisions, codes, and projects were tested until 36 were selected for the algorithm. The experiments show that it is possible to predict which commits are most likely to contain buggy code and potentially reduce manual bug-hunting time significantly.
However, due to the relative simplicity of the ML techniques adopted, they are not able to train ML models that can consider rich semantics in code or learn from historical bug fixes. Therefore, they cannot explain why and how the bugs occur nor suggest revising the code to eliminate bugs automatically or semi-automatically.
Significant breakthroughs in ML techniques, models, and algorithms have been witnessed in recent years. Our research found that very few of these emerging techniques are adopted by functional verification research, and we optimistically believe great success will be possible once they are used to tackle challenging problems in functional verification.
Transformer-based large-scale NL models with billions of parameters trained on the enormous amounts of text corpus achieved near-human or exceeding-human level performance in various NL tasks, e.g., question answering, machine translation, text classification, abstractive summarization, and others. The application of these research results in code analysis has also demonstrated these models’ great potential and versatility. It has been demonstrated that these models can truly ingest a large corpus of training data, structure the learned knowledge and provide easy accessibility. This capability is instrumental in static code analysis, requirement engineering, and coding assistance for several popular programming languages. Given enough training data, it is reasonable to believe that these techniques can train ML models for various functional verification tasks.
Until recently, it was hard to apply ML to graph data due to the complexity of their structure. Graph neural network (GNN) advances have promised a new opportunity for functional verification. One such approach converts a design into a code/data flow graph, which is then further used to train a GNN to help predict the coverage closure of a test. This kind of white box approach promises previously unavailable insight into the control and data flow in a design, which can generate directed tests to fill potential coverage holes. Graphs can represent rich relational, structural, and semantic information encountered in verification. The rich information from training an ML model on graphs can afford many new possible functional verification tasks, e.g., bug hunting and coverage closure.
EDA 4.0 is transforming electronic design automation through the power of artificial intelligence and delivers several key technologies that will help engineers realize the revolutionary changes of Industry 4.0. In this article we provide a comprehensive survey of the potential contributions of machine learning in addressing various aspects of functional verification. The article highlights the typical applications of ML in functional verification and summarizes the state-of-the-art achievements in this field.
However, despite the application of diverse ML techniques, current research primarily relies on basic ML methods and is limited by the availability of training data. This situation is reminiscent of the early stages of ML applications in other advanced domains, indicating that ML applications in functional verification are still in their nascent phase. There remains significant untapped potential for leveraging advanced techniques and models to fully exploit the capabilities of ML. Furthermore, the utilization of semantic, relational, and structural information in today’s ML applications is still not fully realized.
For a more detailed exploration of this subject, we invite you to refer to our white paper titled A Survey of Machine Learning Applications in Functional Verification. In this white paper, we delve deeper into the topic, offering insights from an industrial perspective and discussing the pressing challenge posed by the limited availability of data. The full paper also includes exhaustive references to the fascinating research and writings that inform much of this article.
Harry Foster is Chief Scientist Verification for Siemens Digital Industries Software; and is the Co-Founder and Executive Editor for the Verification Academy. Foster served as the 2021 Design Automation Conference General Chair and is currently serving as Past Chair. He holds multiple patents in verification and has co-authored six books on verification. Foster is the recipient of the Accellera Technical Excellence Award for his contributions to developing industry standards and was the original creator of the Accellera Open Verification Library (OVL) standard. In addition, Foster is the recipient of the 2022 ACM Distinguished Service Award, and the 2022 IEEE CEDA Outstanding Service Award.
Tom Fitzpatrick is a Strategic Verification Architect at Siemens Digital Industries Software (Siemens EDA) where he works on developing advanced verification methodologies, languages, and standards. He has been a significant contributor to several industry standards that have dramatically improved the functional verification landscape over the last 25 years, including Verilog 1364, SystemVerilog 1800, and UVM 1800.2. He is a founding member and current Vice-Chair of the Accellera Portable Stimulus Working Group and currently serves as the Chair of the IEEE 1800 and Accellera UVM-AMS Working Groups. Fitzpatrick is a long-time member of the DVCon US Steering Committee and is the General Chair for DVConUS 2024. He is also a member of the Design Automation Conference Executive Committee. Fitzpatrick holds Master’s and Bachelor’s degrees in Electrical Engineering and Computer Science from MIT.
 |
 |
 |
 |
 |
 |
 |
 |
 |
Great article!
Really interested to see how ML training models can free up verification time while better identify test to hit errors earlier in the process.
Excited to see what’s next from Siemens EDA