Finding The Unexpected In High Performance Designs

It was growing dark as I drove a winding road on Mt. Hood, deep in the American northwest forest. The firs were thick, creating a lot of shadows and making it tough to see things clearly. Then out of the corner of my eye, I swear I saw a 10-foot “man” covered with brown fur. It looked a lot like a Wookie. But everyone knows Wookies aren’t real. It had to be Bigfoot!

I slammed on the brakes and turned around. I slowly drove back to where I thought I saw something. Nothing. Nothing at all.

I parked on the side of the road and got out. I did not see a path through the forest. I did not see footprints. They would have been big and hard to miss. No broken branches to indicate something big had pushed its way through recently. No tufts of Bigfoot fur snagged on the thorns of the salmonberry brush. I listened carefully and heard only the lonely howl of the wind through the tops of the pine trees. But I swear I saw something!

Don’t you ever wish you could go back in time and take a closer look at something? Tivo did that for television. If you are not paying attention and want to see something, just back up and watch it again. I love my Tivo.

For software debug, Greenhills invented the time machine — a feature of their debugger that lets you step the debugger backwards and see what the software was doing back in time. Here at Mentor Graphics we have Codelink, an embedded debug environment that lets you run your software forward or backwards. And we just brought out a new feature called “Veloce Deterministic ICE,” an App for our Veloce emulation systems that allows you to record and replay an ICE run.

Here’s how it works. When running an ICE emulation — that is, an emulation where you have stuff from the outside world connected to your design in the emulator through I/O cables — Veloce watches and records all the input signals and stores them off-line in a database. In a later run, you can use that recording as input to the design under test. That way you can run the design to exactly the same state it was in when a bug occurred during the ICE run.

Why is this cool? It really helps with debug. You see, when running emulation while connected to the outside world, you cannot stop the design — or even slow it down, not even a little bit. If you can’t stop it, you can’t examine it, and you can’t upload waveforms. Which makes it really hard to debug. By saving the inputs and being able to rerun them you can stop and take a closer look at something that screamed past during the original run that you did not get a good look at.

A lot of bugs that folks uncover in emulation are found when attaching the design to the outside world and running it for hours and hours with realistic stimulus. While you can set a trigger to go off when “the bad thing” happens, sometimes one buffer full of trace data is not enough to fully debug the problem. Without the Veloce Deterministic ICE App, you can only restart your design and hope you see the problem again. What if you don’t — well, let’s not think about that.

So how does all this tie into my world, which is all things software? Most designs have processors these days. When looking at system-level problems (like when you hook up your big SoC to the outside world and run gobs of realistic stimulus on it), you’re probably going to have to look at what the software is doing. In the past, we did not have a way to hook up Codelink to designs running on Veloce in ICE mode. Codelink could be used only in TestBench Xpress (TBX) mode or in our VirtuaLAB virtual environment. The only way to debug code on a design running in ICE mode was to use a JTAG probe. But JTAG probes are hugely intrusive and most of these types of problems disappear when intruded upon (much like the shy northwest Bigfoot).

There is another place where this pays off for the Codelink user — multi-core designs. Codelink operates on a trace of the processor to deliver its debug views. Tracing the processor involves shoveling a lot of data off the emulator. 1 or 2 cores, no big deal. 8 or 10 cores, we start to slow things down. 100 or 200 cores — well, that’s a problem. It works, but performance slows to a crawl. Here’s the thing — no human can look at 200 debuggers at the same time and make any sense of it.

Being able to deterministically rerun the design means you can upload a few cores worth of debug data at a time. So you would start by looking at what the main processor is doing. Then follow the execution flow to several other processors. If the debug path leads you to a core that you don’t have debug information for, you can just run a quick replay and log that core.

I don’t have any customers running 200 cores in a design…yet. So far the largest design I’ve worked on had 34 cores, but with core counts going up all the time I don’t think it will be too long now.

Veloce Deterministic ICE is like having a dashboard cam for your design. It lets you go back and examine in detail what happened when something out of the ordinary happens.

As for those strange encounters on shadowy byways, I’m off to buy a dashboard camera. Bigfoot is not going to elude me twice!