You Can’t Walk Straight Blindfolded

Let’s examine the first part of the title of this blog. It is stated as a given. But is it true that you really can’t walk straight when blindfolded? That is what my children and I set out to investigate one sunny afternoon in October (yes we live in California).

We looked for a nice open field with little to no surrounding sound, so that you cannot use the sound to set your bearing. We found one close by at a school soccer field surrounded by an empty school on a Sunday, and a farm, which was also deserted on a Sunday afternoon. My daughter started first, using the eye patches that you receive in United Business class (being a frequent flyer has the advantage of being upgraded sometimes) as a blindfold. And indeed she had a hard time walking straight and ended her first walk in the shape of a nice quarter circle.

My son’s first walk turned out quite similar. Then it was my turn. After four and a half minutes of walking and convincing myself that I was going reasonably straight, this is the walking pattern that the GPS in my pocket, also known as my iPhone, plotted:

As it turns out, you cannot walk straight when blindfolded. And the good news is that in almost all situations you have to, which explains why we can have straight sidewalks.

However, when developing software it turns out that it is hard to get a good view of the entire system. Most of the pieces are black boxes to the software developers, and they have to rely on their experience and high-level knowledge to gain visibility into the software execution of the system to debug issues.

Rather than just seeing the world through the MMU of a particular CPU, virtual prototypes using SystemC models offer developers a bird’s-eye view on the whole system. This means each and every CPU and peripheral register can be inspected. Even hardware signals between peripherals can be probed and watched, such as an interrupt line coming from the timer to the interrupt controller.

Next to that, the programs under execution can be observed or multiple programs on multiple cores can be debugged synchronously. Thus, when advancing the time, the developer will see multiple program code windows updating the contents based on the advancement of the CPU’s program counter. At the same time the developer can observe the interaction with the hardware by being able to look at the platform memories and registers.

Using a debugger, a software developer can validate the state of the system. But once he or she has figured out that the system is not behaving according to the specification, the first question that arises is, “What is going wrong?” Once there is clarity about the “What”, the next thing to figure out will be, “Why are things going wrong?”

“What is going wrong” can be a tough question to answer, however. If you are debugging an OS kernel, it becomes hard to understand what is going wrong from a system standpoint. The OS kernel may get stuck somewhere during the boot. Using a debugger you can observe a multitude of processes that have been scheduled and executed “OK” by the kernel. But if you were able to step away from debugging the implementation and the current situation and get a more global view of the system behavior, also in the past, it may become clear that the OS kernel is trapped in a particular kernel thread waiting for an interrupt.

To make this debug/analysis process more efficient, virtual prototyping tools can provide an OS aware system level analysis framework. As an example you can access a trace of the operating system in the context of one or multiple CPUs over time. This will let you know which processes are completed and in which order this is done. Once you figure out a suspicious situation, such as a data abort during the boot, you can get to a function level trace to get clarity about which function or even which instruction is causing the problem.

In order to analyze the integration between multiple software stacks, you can observe the shared memory communication between multiple CPUs. OS specific extensions to the framework allow you to visualize internal kernel data in a meaningful way such as the kernel debug messages, even before the console drivers are working.

The presented analysis capabilities work without instrumentation or modification of the embedded software. Probes in the virtual prototype models provide the necessary analysis data to the analysis framework.

So if you can get this level of visibility, why would you ever settle with developing software blindfolded?