The Rise Of Complex Debug On Heterogeneous Multicore SoCs

When projects move away from discrete development of loosely coupled systems to an integrated heterogeneous environment, elephantine debugging challenges are created. These challenges do not exist during discrete development because developers are able to design, develop, test, and optimize within the confines of their own device.

But when consolidating heterogeneous systems, developers and testers need a method to view the system as a whole. They need to understand how each operating system and application environment is working. They need to understand where there might be shared resource contention, or saturation of processors, busses, or devices. Developers require an approach that allows them to see how certain behaviors in one part of the system affects, or is affected by, other parts of the system.

One of the challenges we face today is that even if we consolidate our system into a single heterogeneous SoC, different people are still working on different parts of the system. It’s possible that only one or two people are responsible for ensuring the entire consolidated systems works as expected.

This is not necessarily a good thing.

The traditional debug of a multicore system

For demonstrative purposes, let’s use the NXP i.MX 6SoloX as our heterogeneous SoC, which has multiple, non-identical cores such as the ARM Cortex-M4 and the ARM Cortex-A9. In this scenario, to make full use of the cores, we are utilizing two very different operating systems: Mentor Embedded Linux and Mentor Nucleus RTOS.

Now, how would you normally debug a system such as this? One approach is to look at one OS at a time.

Figure 1: Heterogeneous debug of a multi-OS embedded system.

To debug our Linux application, we might use gdbserver (Figure 1). To debug the Linux kernel or kernel modules, the use of a JTAG or something like eKGDB would suffice. For the RTOS, debug is often performed by using a JTAG or some sort of software debug agent. Most boards have only one JTAG connection so you would need a JTAG that is capable of attaching to all required cores.

This JTAG also needs to be configurable. For example, you may only want the RTOS cores to halt when you hit a breakpoint and not the cores running Linux. Linux or the RTOS may be running in SMP mode across multiple homogeneous cores. In order for homogeneous cores to halt quickly, you want to use hardware cross-triggering so your JTAG must support that too.

To make things even more challenging, you may have to use a completely different set of development and debugging tools for each operating system.

Now that your system has been consolidated in a single SOC, you’ll need to debug the entire integrated system, not just the individual pieces, and you should be able to do this from a single IDE. Also, if you are debugging your RTOS-based application using JTAG on one core and you want to debug your Linux application at the same time, you better make sure that hitting a breakpoint in the RTOS doesn’t halt the Cortex-A9 the Linux is running on.

The modern debug approach

Mentor Embedded Sourcery CodeBench enables simultaneous debugging of the Linux kernel, user space, Nucleus RTOS, and even bare metal applications from a single IDE. The IDE provides numerous debug interfaces including gdbserver, Nucleus, debug agents, and third-party JTAGs such as ARM DSTREAM and Segger J-Link.

In the example depicted in Figure 2, the ARM DSTREAM JTAG is used to debug both the sending of messages from the Nucleus RTOS using rpmsg and receiving of messages on Linux via the rpmsg Linux kernel module. If a developer hits a breakpoint in the Linux kernel, the Linux kernel halts, but the Nucleus remains unaffected. Conversely, if a developer hits a breakpoint in Nucleus, it halts – but the Linux kernel remains unaffected.

Figure 2: Heterogeneous debug with Sourcery CodeBench and ARM DSTREAM.

 Deep system visualization

Debugging only gets you so far. In a heterogeneous system, you need to be able to see what’s happening across all the operating systems and inside all of the cores – on one synchronized time scale. What’s happening on one core, may be affecting what is expected to happen on another core. You can’t see this type of behavior by stepping through code. Mentor’s Sourcery Analyzer provides a timeline view for analysis of application and kernel level events across all your operating systems running on all your cores on a perfectly synchronized time line.

Figure 3: Visualization deep into a system – developers are able to analyze and troubleshoot CPU utilization.

In Figure 3, you can see activities such as CPU utilization, which includes process and threads states for both operating systems on a single, aligned time scale. Developers are able to zoom in and scroll to find out what exactly is happening when the CPU utilization is high on one core and lower on the other cores.

Conclusion

Today’s multicore systems produce vast volumes of performance, trace, log, and monitoring data. To stay competitive, developers require a customizable visualization tool that performs myriad capabilities and enables the creation of unique performance dashboards. Debugging heterogeneous multicore systems is a difficult task – even among the more seasoned developers. Knowing that tools exist to assist in these complex debugging endeavors is welcome news for all parties involved.

For a more detailed discussion, please download my white paper “Addressing Debug Challenges on ARM-based Heterogeneous Multicore SoCs.”