The Trust Burning Debug Cycle From Hell

Is my untested code a menace to the team? It gets personal.


As bad as The Trust Burning Debug Cycle From Hell sounds, it’s worse than you think. Most of us don’t realize it exists. In my first 10 years as a hardware developer I wrote code like it could never exist! But then came the realization. It’s a cycle that preys on us all. It tempts me constantly.

Most of us in hardware development are used to seeing bugs as annoyances at a minimum, though more critically blocks and showstoppers. They grow in databases and become part of the checklist that stands between a team and its delivery objectives. Ultimately their damage materializes as schedule overrun. On the surface this makes sense because schedule is where the money is. But damage goes beyond the cost of schedule overruns.

To truly understand The Trust Burning Debug Cycle From Hell I had to stop thinking of bugs in terms of simple schedule overruns and start thinking about damage using a metric that’s always made me nervous: trust. Measuring damage in terms of trust means I start asking myself questions like “do my colleagues, managers and clients trust me to do my job well?” No one ever wants the answer to that question to be ‘no’, me included.

Unfortunately, trust is easy to burn. When verification engineers build VIP for other teams, we tend toward a code-n-release approach: very soon after the VIP code is written it’s released to users. We usually forego, or at least delay rigorous testing, thinking it’ll be more effective to resolve issues in situ. Users typically hit a lot of bugs. Every bug costs them debug time. Every frustrating debug minute is a burden that gives them the opportunity to question our ability to deliver.

Depending on who you’re working with, a few bugs probably ends up being acceptable. When it becomes to more than a few though; when bugs appear as part of a cycle; when it’s a constant debug-fix-and-release, that’s when the questions start and users start losing trust.

The good news is that early testing of VIP can help us build and keep trust with users. Basically, you hit the bugs in your code before anyone else does to avoid passing them along. Two techniques I use are test-driven development (TDD) and unit testing. Both techniques are borrowed from software development, both apply wonderfully in hardware development and both have helped me avoid The Trust Burning Debug Cycle From Hell.

Or maybe I should say almost avoid it…

…because the bad news is that even with the right techniques it can still be oh-so-tempting to hurry right past the testing and deliver straight to users. I hate to admit it, but for all the practicing, writing and time I’ve dedicated to TDD, there are times where I defer on quality under pressure. I dream myself into the situation where faith in my untested code is good enough to avoid The Trust Burning Debug Cycle From Hell. But faith in my code has never been enough to turn my dream into reality. Invariably, my untested code becomes a menace to the team. The time I save in development is easily consumed by frustrated users. Worst of all, the decent code I do deliver using TDD is overshadowed by the buggy code I produce without.

And peoples’ trust in me? Gone. At least that’s the way it feels. I’m disgusted with myself every time I try and save time at someone else’s expense. Regrettably, it remains something I can’t help but try when it seems I need it, even when I know exactly how it’ll turn out.

The moral of this story: seeing the improvements I’ve made with TDD and unit testing when building VIP helped me realize the damage I do to a team–and to myself–without it. It’s plain as day. Bugs are more than items in a database or weeks added to a schedule; they’re personal to me now. I build them; they’re my fault.

That may sound like a stressful way to work but it’s only stressful when I’m tempted into cutting corners with the old code-and-release. Because every time I slip into that old habit, waiting for me around that corner is The Trust Burning Debug Cycle From Hell… grinning deviously… happy to have me back.

Related Stories
Debug: Last Bastion Of Automation
Folklore erroneously claims verification consumes 70% of development time. But does debug really consume 50%?
Debug Becomes A Bigger Problem
EDA companies have been developing more integrated debug flows that bring execution engines and hardware and software closer together, but is that enough?
Getting Formal About Debug
Formal technology is playing an increasing role in the debug process, and some fundamental new directions are being explored to make debug more automated.



Lars Asplund says:

The FPGA team I joined in May have been using TDD and unit testing for a few years now and are known internally as the team not releasing bugs. That’s not entirely true but compared with all the SW teams in the company, which also use TDD and unit testing, they are doing well and they do a lot better than they did before adopting these practises. These ways of working are great regardless if you’re into HW or SW coding.

hcarter333 says:

There should be an AA style meeting for this kind of thing 🙂 I’ve always held it as a point of pride that I document what I want my code to do before implementing it. I occasionally backslide, and my code suffers for it every time.Same thing with testing. I know I’ll wind up saving hours or even days if I just build the damned test harness, but the temptation is huge just to ‘crank out the code in a few minutes, and get on with life.’

nosnhojn says:

hamilton, you’re right on the AA-style meeting! great analogy! I know it’s wrong, but I can sometimes convince myself that skipping the testing is ok. maybe what’s missing is the support group 🙂

Leave a Reply

(Note: This name will be displayed publicly)