The Verification Mindset

The practice of semiconductor verification has changed substantially over the years, and will continue to do so. The skillset needed for functional verification 20 years ago is hardly recognizable as a verification skillset today, and the same should be expected moving forward as design and verification becomes more abstract, the boundary of what is implemented in hardware versus firmware and software continues to move, as well as the adoption of new technologies. A move toward a complete verification spectrum is a constant response to the constant increase in complexity and time-to-market pressures.

This means the definition of a good verification engineer changes over time.

“A good verification engineer 20 to 30 years ago understood designs, how to build simple functional test plan checklists, and wiggle wires roughly at the design level in simulators with bonus points for building models that could detect functional failures,” noted Chris Giles, design solutions product manager at Siemens EDA.

A decade ago, good verification engineers were proficient in object-oriented coding. They were able to build an entire testbench infrastructure that generated constrained-random stimulus, monitored interfaces, monitored results for correctness, and addressed coverage versus the joint design/verification-defined test plan.

“Today’s good verification engineers understand system functionality, software/firmware functionality, the hardware/software interface, and build object-oriented testbenches that are portable and flexible to run in a myriad of technologies from simulation through emulation and on to prototyping,” Giles said. “On top of that, a good verification engineer today can use formal verification, as well, such that the right tool is applied at the right time to get an entire system built right-first-time.”

Others point to similar changes. “Today, a good verification engineer is almost an all-knowing engineer,” said Zibi Zalewski, general manager of the hardware division at Aldec. “It is very good to have a design background, which allows them to build better hardware-oriented testing scenarios. They need a knowledge base of different verification methodologies, HDL languages, and libraries. It is no longer enough to code in VHDL or Verilog. UVM or OSVVM provide new possibilities, but also expectations from the verification engineer to learn it and use it efficiently. Such skills are always beneficial for the verification team managers.”

This can be extremely challenging work. “Verification at the high end is unbelievably hard,” according to Simon Davidmann, CEO of Imperas Software. “It is far more complex than developing an architecture or writing software. Those are not easy. But actually demonstrating that they have few bugs, if any, is much harder. Hence, the software world has built formal linting tools, and the hardware world has built formal tools. It’s very hard to do verification, to know when you’re done, and know you’re doing it correctly. In processor verification, it’s a black art, so it’s proprietary. People don’t know outside of Intel, Arm, Tensilica how to do verification of a processor.”

Picking the right people
And those people are in short supply. Finding the right verification engineer for a particular team requires thinking outside the box.

Hagai Arbel, CEO of Vtool, said what he looks for in a verification engineer is different than what others might look for. “I am on the extreme side of looking for engineering potential — understanding problem solving versus knowledge. I’m also on the extreme side in terms of how important it is to know the language well. For me, it is maybe 10% knowledge on languages. I cannot tell you that a brilliant engineer without any experience with, let’s say, two months of experience will have more productivity than a medium engineer with 10 years of experience. But in the long run — and this long run will come very quickly — I always prefer the approach, the ability, the mindset that may be personal qualities that will make a good verification engineer. Given the right circumstances, the learning curve can be amazing. I have several engineers in the company that we nourished from the very beginning, and after 6 to 12 months of work, they are capable of doing things that are rarely seen in people with 3,4, or even 5 years of experience.”

When interviewing an engineer for potential employment, Arbel said the best indicator of a good verification engineer is not the answers they give, but the questions they ask, and how quickly.

“Really brilliant verification engineers immediately grasp the uncertainty of what is not defined,” he said. “You ask them a question, you present something to them, and they immediately ask what is missing. It may or may not have been missed on purpose, and could be interpreted on several levels. Some people are able to see this very quickly. Equating this to a talent, it’s like RAM in a computer. They have an ability to hold a complex abstract image in their head. This is a very good talent. Also, good verification engineers are by definition people with a very wide angle. They like to know a lot of things about a lot of things. They want to know everything around — maybe not necessarily the deepest bits or bytes of a transistor, but the world around it. This is another characteristic of a good verification engineer. I find good verification engineers more similar in their thinking process to architects than to designers. And of course, the softer quality is they have a very strong sense of responsibility and accountability. They are the type of people who say, ‘Yes, I will do it. Count on me. I’m willing to take the risk and the responsibility that comes with it.’”

Most of this is not technical, either. “A lot of this is the way you look at life,” Arbel said. “Some people concentrate on their small world, and a verification engineer needs to be the opposite. You need to like to work with people. I don’t think that I can name one excellent verification engineer that is either selfish, likes to work on their own, and doesn’t collaborate with others.”

But a technical foundation is, of course, essential.

“More than just a resume, I am looking for people that have a sense of DIY,” said Philippe Luc, director of verification at Codasip. “If they are playing with a Raspberry Pi, if they have made a project outside of the scope of their job or university project, I’m looking to see how large their natural creativity is, along with problem solving abilities. I am more interested in people who know several different computer languages like Python, C, C++, rather than people who are explicitly quoting UVM and SystemVerilog. This speaks to the wideness of their spirit and their knowledge. A verification job for me is not only designing the testbench. It’s also making a little script to run them on the cloud, a little script to gather all the results, a little script to measure results from functional coverage. And this is not, strictly speaking, UVM or SystemVerilog.”

The knowledge of multiple languages fosters the drive for a verification engineer to think differently about how to approach things, he said. “It’s something that happens with natural language polyglots that speaks to the brain’s ability to learn complexity. It’s a positive sign.”

During an employment interview itself, Luc looks to see if the candidate has both the theoretical knowledge gained from university, as well as the ability to solve complex questions on the spot. “My favorite question is to propose an RTL/design block where I provide the schematic, some waveforms, and ask a candidate how to verify this block. This is a very large question. For junior people it’s quite a complex question. For people with more experience, it’s a simpler question. I am looking to hear all of what the candidate says. Do they look to the testing of the normal case? Do they look at checking the corner cases? Ideally, there would be some mistakes in the specification, some missing information. Some candidates manage to find all the missing information, and this is a positive sign for me, whereas some candidates just don’t sweat it.”

During non-pandemic times, Luc sometimes offers a metal brain teaser to candidates. But with interviews online now, he has employed creative techniques like asking for the solution to a chess problem. “I have tested this particular chess problem. I spent hours and didn’t find the solution, and this guy solved it in five minutes. First, it was impressive. Second, it was a good sign that he didn’t cheat on his CV, which sometimes happens.”

Open-source and software misconceptions
Verification pain comes in a variety of forms, especially when it comes to open source which has also been at times misconstrued as ‘free.’

Imperas’ Davidmann noted that some people view a self-testing program as verification. “This is a simple program which says if 3 + 3 = 6, everything’s got to be good. But that tests such trivial stuff. Similarly, in the area of open source, some people think this about, for example, RISC-V compliance testing, and that it’s good enough verification. (It’s not.) One of the fundamental problems when you make things more open source is that it becomes applicable to many more people. What we find is huge numbers of people have no clue about the complexities of verifying a processor or an SoC. If you ignore RISC-V, in open source that wasn’t really a problem in the past. Anybody who’s designing a chip would look up and say, ‘It’s going to cost me $30 million. I’m going to need a team. I’m going to have to have 50 people in verification. I’ve got to buy a million dollars of Verilog licenses. I’ve got to buy a $5 million emulator. And I need people who understand all that stuff.”

Likewise, many software engineers view hardware and system verification through the lens of software, and vice versa. “Software-based TLM interfaces provide the access layer to build high-level test models or test benches, even to connect virtual platforms that mimic a part of the system with a processor(s) and drive the design under test,” said Aldec’s Zalewski. “A hardware background with software development experience would be a perfect match for such complicated testing platforms, which means C/C++/Python experience is a big advantage for a verification engineer.”

From a high level, there are certain characteristics that define good verification engineers. To Juergen Jaeger, product management group director at Cadence, what makes a great verification engineer are the following qualities:

• Organization. In verification it’s all about structure, discipline, accurate documentation and the desire for perfection.
• Curiosity. Good verification engineers need a desire to understand the to-be-tested product, and to understand how an end-user would use the product.
• Persistence. Never give up finding the elusive bugs.
• Creativity and courage. Find new ways of doing things, even if it’s been done before.
• Non-obsession with perfection. Knowing when enough is enough is critical, because nothing will ever be 100% perfect.

In addition, verification isn’t something engineers can merely read about in a book. It needs to be practiced.

“Yes, read the books, but if you haven’t really done it, it’s a challenge,” Davidmann said. “Being a good verification engineer is a mindset. It’s the way the person’s mind works. They are inquisitive. You’ve either got it or you haven’t. Certain people are attracted to it because they want work out that something is working. If you show them something is working, they want to pull it apart, and say, ‘Why is it working?’ or ‘Why is it not working?’ These are people for whom it’s all about facts, not assumptions. And it’s not anything about politics. They’ll say how it is because they’re only interested in the truth. It’s a certain type of person who is a good verification engineer. They are precise, they are slow, they really are engineers’ engineers. They don’t believe anything easily. They assume nothing. Everything’s got to be proved. They’re really methodical, and are able to keep track of everything without guessing, jumping steps or assumptions. It’s all detail oriented. And finally, it’s no problem if they’re wrong.”

It’s also good when verification engineers have experience with different verification approaches. “You need to be able to pick a proper verification approach, depending on the design architecture, complexity and functionality,” said Zalewski. “Is it more classical, like bit-level testing or transaction level with high-level code that needs to be ready for in-hardware verification modes? That means an extensive knowledge of the tools like simulators, emulators or prototyping hardware. Each of the areas is a really wide domain, and it is very hard to find a verification engineer who can do all of it. This is why a lot depends on the project requirements and proper organization of the verification team to use the best available skills for the proper stage of the verification process. Experienced verification engineers with hardware and software skills, and a high-level view on the whole process and ability to use the experience properly, are key to the success of verification — and the whole project’s success.”

Conclusion
The skill set for today’s verification engineer doesn’t necessarily include design knowledge, but it should. “Verification engineers need to understand the sequential nature of hardware — something which is not necessarily apparent to someone who approaches the discipline from a pure software background,” Giles said.

Verification also has to be about more than just verifying functions. “Unless the verification engineer understands the potential signatures of failure in hardware, it is always a risk that such failures will slip through verification undetected,” Giles noted. “A good example is a clock domain crossing, where a violation may not be modeled in the simulation environment — or perhaps data integrity hardware, which will only activate if a soft error occurs. The push into object-oriented coding for testbenches 10 to 20 years ago created a scenario where the need to build teams that could efficiently build constrained-random testbenches resulted in a knowledge gap, with verification teams focusing on the software side and lacking hardware and design fundamentals.”

Ultimately, a good verification engineer leverages the foundation of fundamentals and theory in a way they can be adapted quickly over time. “Added to that is an intellectual curiosity to continue to learn new things and adopt new skills. Finally, the good verification engineer is polished by several years of experience. The result is that of a ‘renaissance engineer,’ understanding everything from hardware design, failure mechanisms, and constraints, to system requirements and software/firmware development, object-oriented coding, formal methods, and the differences between the capabilities of various technologies to be used, such as simulation, emulation and prototyping. This engineer has the fundamentals and curiosity to continue to evolve and adapt to the ever-changing discipline, which explains why a good verification engineer is hard to find and in great demand,” Giles said.