The Trust Burning Debug Cycle From Hell

As bad as The Trust Burning Debug Cycle From Hell sounds, it’s worse than you think. Most of us don’t realize it exists. In my first 10 years as a hardware developer I wrote code like it could never exist! But then came the realization. It’s a cycle that preys on us all. It tempts me constantly.

Most of us in hardware development are used to seeing bugs as annoyances at a minimum, though more critically blocks and showstoppers. They grow in databases and become part of the checklist that stands between a team and its delivery objectives. Ultimately their damage materializes as schedule overrun. On the surface this makes sense because schedule is where the money is. But damage goes beyond the cost of schedule overruns.

To truly understand The Trust Burning Debug Cycle From Hell I had to stop thinking of bugs in terms of simple schedule overruns and start thinking about damage using a metric that’s always made me nervous: trust. Measuring damage in terms of trust means I start asking myself questions like “do my colleagues, managers and clients trust me to do my job well?” No one ever wants the answer to that question to be ‘no’, me included.

Unfortunately, trust is easy to burn. When verification engineers build VIP for other teams, we tend toward a code-n-release approach: very soon after the VIP code is written it’s released to users. We usually forego, or at least delay rigorous testing, thinking it’ll be more effective to resolve issues in situ. Users typically hit a lot of bugs. Every bug costs them debug time. Every frustrating debug minute is a burden that gives them the opportunity to question our ability to deliver.

Depending on who you’re working with, a few bugs probably ends up being acceptable. When it becomes to more than a few though; when bugs appear as part of a cycle; when it’s a constant debug-fix-and-release, that’s when the questions start and users start losing trust.

The good news is that early testing of VIP can help us build and keep trust with users. Basically, you hit the bugs in your code before anyone else does to avoid passing them along. Two techniques I use are test-driven development (TDD) and unit testing. Both techniques are borrowed from software development, both apply wonderfully in hardware development and both have helped me avoid The Trust Burning Debug Cycle From Hell.

Or maybe I should say almost avoid it…

…because the bad news is that even with the right techniques it can still be oh-so-tempting to hurry right past the testing and deliver straight to users. I hate to admit it, but for all the practicing, writing and time I’ve dedicated to TDD, there are times where I defer on quality under pressure. I dream myself into the situation where faith in my untested code is good enough to avoid The Trust Burning Debug Cycle From Hell. But faith in my code has never been enough to turn my dream into reality. Invariably, my untested code becomes a menace to the team. The time I save in development is easily consumed by frustrated users. Worst of all, the decent code I do deliver using TDD is overshadowed by the buggy code I produce without.

And peoples’ trust in me? Gone. At least that’s the way it feels. I’m disgusted with myself every time I try and save time at someone else’s expense. Regrettably, it remains something I can’t help but try when it seems I need it, even when I know exactly how it’ll turn out.

The moral of this story: seeing the improvements I’ve made with TDD and unit testing when building VIP helped me realize the damage I do to a team–and to myself–without it. It’s plain as day. Bugs are more than items in a database or weeks added to a schedule; they’re personal to me now. I build them; they’re my fault.

That may sound like a stressful way to work but it’s only stressful when I’m tempted into cutting corners with the old code-and-release. Because every time I slip into that old habit, waiting for me around that corner is The Trust Burning Debug Cycle From Hell… grinning deviously… happy to have me back.

Related Stories
Debug: Last Bastion Of Automation
Folklore erroneously claims verification consumes 70% of development time. But does debug really consume 50%?
Debug Becomes A Bigger Problem
EDA companies have been developing more integrated debug flows that bring execution engines and hardware and software closer together, but is that enough?
Getting Formal About Debug
Formal technology is playing an increasing role in the debug process, and some fundamental new directions are being explored to make debug more automated.