Continuous Integration

In this article, I tackle the classic question engineers developing software for custom integrated circuits (ICs) grapple with constantly:

How do I test my software before the hardware team gets me a working silicon chip?

No ‘one size fits all’ solution is provided here (look for that alongside my pet unicorn); instead I detail an easy-to-use yet powerful approach to solve this problem for a particular development scenario.

If you are developing software for ICs and you or your team does any of the following:

• Develops on Linux or Windows
• Utilizes continuous integration
• Loses time dealing with merge/integration issues

then read on. If not, keep reading anyway, as this methodology can be used effectively in a variety of situations.

Background and Approach
Continuous integration is a software development practice where team members merge their work at least daily, and each merge is checked by an automatic build process to identify errors. This enables developers to avoid the dreaded ‘merge-commit hell’ that occurs when attempting to combine heavily edited and interwoven applications together, not to mention the possibility of inducing errors from not properly stress-testing code. See the ‘Why CI is important’ article for an in-depth look for why continuous integration is so important when developing software. Developing software for ICs, however, adds another layer of complexity in choosing what platform to run the relevant software on.

Many options exist in this space: Emulation, FPGAs, racks upon racks of hardware development boards, virtual prototypes, etc. Each has their benefits and drawbacks in certain development situations. To implement continuous integration a solution that enables fast software run-times, inexpensive scaling for large teams, and easy integration with continuous integration platforms is essential. Based on these metrics I chose virtual prototypes as the hardware model, specifically Arm’s Fast Models. Keep an eye out for a future blog on why this is the best choice, as a thorough explanation is beyond the scope of this blog.

The solution proposed here is a methodology to integrate virtual prototype simulation into an existing continuous integration framework, such as Jenkins or Bamboo. This allows IC software developers to stop worrying about physical hardware and focus their energy on writing excellent code. The Component Architecture Debug Interface (CADI)—an API for simulation control and debug access to software running on Arm virtual platforms—allows python to interact with Arm Fast Models and provides a connection between hardware and software status. Python, using CADI commands, can control the execution of applications, set breakpoints, read and write to memory, access registers, view test variables, and more. This information can then be utilized to create comprehensive software tests and can interface with an existing continuous integration framework. This article will cover how to use python’s CADI platform (cleverly named PyCADI), including (1) how to setup the development environment and (2) a walk-through of example code with results.

The example use-case here is for a team that uses continuous integration in a Windows environment to develop software for an Arm Cortex-M4. To enable automated software verification, it is standard practice for them to use a variable in every test to indicate the test result. This variable, dubbed ‘test_pass’, defaults to 0 and is set to 1 if the test passes without error. This technique helps each test plug into the team’s general regression scripting.

Let’s get started.

Environment Setup
This section is separated into the ‘First-time setup only’ and the ‘Possibly reoccurring setup’ for steps that may need to occur before testing can be run, depending on what the goal is and what variables change across tests. This blog was developed on Windows, with only minor adjustments needed for Linux users.

First-time Setup Only
The first step to running software on Fast Models is to get a Fast Model! Click the button below to obtain an evaluation license if not already downloaded. Follow the installation guide specified in the downloaded package for Windows or Linux.

Fast Models Downloads

The PyCADI library requires python 2.7.13 (very specific I know), so ensure that python 2.7.13 is installed (along with an IDE of choice). See the button below to download the correct version of python. Make sure to add ‘python’ and ‘pip’ to your system’s Path environmental variable as well. Note: Using more recent versions of python 2.7 should work but is untested. Use with caution!

Python 2.7.13 Download

To implement breakpoints on function names and specific lines in files, a python library called pyelftools is required. A simple pip install command will do the trick:

• pip install pyelftools

Possibly Reoccurring Setup
An executable application is of course necessary for testing said executable application. The details of how to create apps is not covered in this blog for software application developers. Note the absolute path to whatever application is going to be tested as PyCADI will need to locate it.

A Fast Model representing the end hardware must also be provided. Creating a Fast Model system from scratch is beyond the scope of this blog, but see the reference guides located under <install_directory>\FastModelsPortfolio_<version>\Docs for tutorials on getting up and running with Fast Models. In this example, a simple Cortex-M4 core is created in System Canvas with a clock source and memory.

One important note for compiling a Fast Model system for python scripting is the build settings. With the relevant system open in System Canvas, navigate to “Project > Project Settings > Targets” and ensure that ‘CADI library’ is checked and nothing else. Then, when applying these changes and selecting “Build”, System Canvas will generate a “.so” file in Linux and a “.dll” file in Windows, both having the name “cadi_system_<system_type>”. Building with the settings in the screenshot below, the model name will be “cadi_system_Win64-Release-VC2013.dll”.

Code Walk-through
Now that the setup is complete and a model + an application are in hand, let’s dive right into an example going line by line. Recall that the team’s objective here is to check if the test passed in their regression environment. The variable ‘test_pass’ should default to 0 and change to 1 if the app checks out without error. Accordingly, ‘test_pass’ should be checked at the test start and test end.

Up first, importing the necessary modules and setting the relevant path for PyCADI’s python. This path setting is primarily for Linux based systems, and fails if the tools for Fast Models are not sourced properly. fm.debug is imported from the PyCADI module, and the python elf tools are loaded from the previously installed pyelftools python library.

import sys, os
# Set python path to Fast Models 
try:
    sys.path.append(os.path.join(os.environ['PVLIB_HOME'], 'lib', 'python27'))
except KeyError as e:
    print "Error! Make sure you source all from the fast models directory. Try something like this:"
    print "$ source /FastModelsTools_11.0/source_all.sh"
    sys.exit()
import fm.debug
from elftools.dwarf.descriptions import describe_form_class
from elftools.elf.elffile import ELFFile

Next the test variables are specified, with the absolute paths to the model and the application hardcoded. Also indicated is a register and variable to track during the test. R15 is a register of interest as the Cortex-M4’s Program Counter, which will verify where breakpoints are hit in memory. The variable is ‘test_pass’ to check if the test passes, and is specified by its location in the virtual platform’s memory. This is done to enable consistent tracking of the variable throughout the test, with the variable being saved to a location not used by the program to avoid overwriting. In this example that place is at the hexadecimal address ‘0x20005000’.  To place this—or any—variable at that location in your software code at initialization, use the uncommented syntax as opposed to the commented syntax (in C):

/* unsigned int test_pass; */
#define test_pass (*((unsigned int *) 0x20005000))

The variable ‘test_pass’ is identical to a typically defined variable in your code, with the only difference being that it now has an explicitly defined memory address location at 0x20005000.

Finally the three different types of breakpoints are initialized: (1) At a hexidecimal address (location in program memory), (2) on a function’s start (SCS_init in this example), and (3) at the line number of a file. Note that setting a breakpoint on a file line number will only work if it is breakpointable, which sounds obvious but does not allow setting on an empty line, one with just a bracket, etc. If a mainstream debugger can set a breakpoint there, so can PyCADI.

# Path specifications
model_path = 'C:\Users\zaclas01\itm_integration_windows\Model\cadi-Win64-Release-VC2013\cadi_system_Win64-Release-VC2013.dll' 
app_path = 'C:\Users\zaclas01\itm_integration_windows\Applications\itm_app\startup_Cortex-M4.axf'
# Variable and register to check
cpu_reg = 'R15' 
app_var = '0x20005000' #test_pass
# Where to set breakpoints
BPT_on_file_line = ['main.c:100']
BPT_on_function = ['SCS_init']
BPT_on_memory = ['0x122']

It should be noted that the method of matching the function names and line numbers to their respective locations in program memory is done using my solution utilizing easy-to-use python tools. PyCADI natively supports setting breakpoints in these cases: When program execution reaches a memory address, when a memory location is accessed, and when a register is accessed. Setting breakpoints on function names and line numbers requires some mapping to occur, with a reusable example shown in this blog.

In the code snippet below python starts interacting with the CADI port of the model. The module fm.debug creates a model object, from which a target object ‘cpu’ is obtained (the first, and only cpu, in this model), and the specified application is loaded onto the cpu.

# Load model
model = fm.debug.LibraryModel(model_path)
# Get cpu
cpu = model.get_cpus()[0]
# Load app onto cpu
cpu.load_application(app_path)

Now onto setting breakpoints. All breakpoints here are set using the ‘cpu.add_bpt_prog()’ command, stopping when program execution reaches an address in program memory. This functionality is built into the fm.debug module of PyCADI. As all breakpoints need a known address, more processing is required to map a function name or file + line number to its corresponding address in memory. The pyelftools library helps with this by obtaining the formatted DWARF debug information from the application file:

# Obtain DWARF debug information using Python ELF tools
with open(app_path, 'rb') as f:
    elffile = ELFFile(f)
    dwarfinfo = elffile.get_dwarf_info()

Utilizing this object ‘dwarfinfo’, I created functions to map function names to memory addresses and file + line number to memory addresses. Here is the mapping from function names to memory, taking the ‘dwarfinfo’ above as an input with a list of functions to map:

def decode_funcname(dwarfinfo,BPT_on_function):
    func_lineno_dic = {}
    # Go over all DIEs in the DWARF information, looking for a subprogram entry with an address range that includes the given address.
    for CU in dwarfinfo.iter_CUs():
        for DIE in CU.iter_DIEs():
            try:
                if DIE.tag == 'DW_TAG_subprogram':
                    lowpc = DIE.attributes['DW_AT_low_pc'].value
                    # DWARF v4 in section 2.17 describes how to interpret the DW_AT_high_pc attribute based on the class of its form.
                    highpc_attr = DIE.attributes['DW_AT_high_pc']
                    highpc_attr_class = describe_form_class(highpc_attr.form)
                    if highpc_attr_class == 'address':
                        highpc = highpc_attr.value
                    elif highpc_attr_class == 'constant':
                        highpc = lowpc + highpc_attr.value
                    else:
                        highpc = 'Invalid highPC class'             
                    func_name = DIE.attributes['DW_AT_name'].value    
                    for BPT_funt in BPT_on_function:
                        if func_name == BPT_funt:
                            # Store in dictionary
                            func_lineno_dic[func_name] = (hex(lowpc),hex(highpc))
            except KeyError:
                continue
    return func_lineno_dic

Here is the code connecting file + line numbers to memory, with the same ‘dwarfinfo’ as an input alongside a list of file + line number entries to map:

def decode_filelinenum(dwarfinfo,BPT_on_file_line):
    hex_to_filelinenum_dic = {}
    # Go over all the line programs in the DWARF information, looking for one that describes the given address.
    for CU in dwarfinfo.iter_CUs():
        # Look at line programs to find the file/line for the address
        lineprog = dwarfinfo.line_program_for_CU(CU)
        if lineprog==None:
            continue
        for entry in lineprog.get_entries():
            # We're interested in those entries where a new state is assigned
            if entry.state is None or entry.state.end_sequence:
                continue
            # Extract relevent data
            file_name = lineprog['file_entry'][entry.state.file - 1].name
            line_numb = str(entry.state.line)
            file_linenum = file_name+':'+line_numb
            address = hex(entry.state.address)
            # Check if file_linenum is supposed to be monitored:
            for BPT_fln in BPT_on_file_line:
                if file_linenum == BPT_fln:
                    # One file/line can span several instructions; enter in the first occurance
                    if not file_linenum in hex_to_filelinenum_dic:
                        hex_to_filelinenum_dic[file_linenum] = address         
    return hex_to_filelinenum_dic

With these two functions in hand, creating the desired breakpoints only requires a few lines. All breakpoints are added to a breakpoints list for easy processing later.

# Track all breakpoints set in list
bpts = []

# Set bpts on hexidecimal locations specified
for hex_loc in BPT_on_memory:
    bpt = cpu.add_bpt_prog(int(hex_loc,16))
    bpts.append(bpt)
    
# Create dic of specified functions and their start & end addr
# Set bpt on the start of the functions
func_dic = decode_funcname(dwarfinfo,BPT_on_function)
for key in func_dic:
    bpt = cpu.add_bpt_prog(int(func_dic[key][0],16))
    bpts.append(bpt)
    
# Create dic of specified file & line numbers and their locations
# Set bpt on the mapped location
filelinenum_dic = decode_filelinenum(dwarfinfo,BPT_on_file_line)
for key in filelinenum_dic:
    bpt = cpu.add_bpt_prog(int(filelinenum_dic[key],16))
    bpts.append(bpt)

Finally the moment we have all been waiting for: Running the application! This example is set up to run the model until a breakpoint is hit or until 3 seconds passes with no breakpoint, whichever happens first. Here is a simple breakdown of the code flow:

1. If a breakpoint is hit:
   1. Hit breakpoint is identified and printed out
   2. Specified variable (‘test_pass’) is obtained and printed out
   3. Specified register (‘R15’) is obtained and printed out
   4. The model starts to run again
2. If 5 seconds passes:
   1. Specified variable (‘test_pass’) is obtained and printed out
   2. Specified register (‘R15’) is obtained and printed out
   3. Test ends

This code sample runs the application in blocking mode; the app can also be run in non-blocking mode to allow for additional computation during app runtime. The timeout time can also be changed.

while 1:
    try:
        model.run(timeout=3)
    except:
        # On timeout, no breakpoints have been hit and the model will hang at the end of the application
        model.stop()
        break      
      
    # Test stopped due to a breakpoint, identify it
    for bpt in bpts:
        if bpt.is_hit:
            print "breakpoint %s was hit" %bpt
    

    # Get variable status
    var_status = cpu.read_memory(int(var_check,16),count=1)[0]
    print 'Variable %s value is: %s' %('test_pass',var_status)

    # Get register status
    reg_status = str(hex(cpu.read_register(reg_check)))
    print 'Register %s value is: %s' %('R15',reg_status)  

    # Back to start of loop, restart model

# Model done running
	#-----------------------------------------------------------------
# Get variable status
var_status = cpu.read_memory(int(var_check,16),count=1)[0]
print 'Variable %s value is: %s' %('test_pass',var_status)

# Get register status
reg_status = str(hex(cpu.read_register(reg_check)))
print 'Register %s value is: %s' %('R15',reg_status)  
    
print 'Monitoring complete, clean exit'

That is the entirety of the sample code! While being simple and easy to digest, the results are powerful. Here is the output when running the above code (with some additional print statements included for clarity):

Here it is clear to see that ‘test_pass’ indeed starts set to 0 and returns 1 once the test is completed, indicating a successful test. This status can be sent to other applications or a continuous integration platform in a variety of different ways, such as creating a verification text file or returning the status to another program that called the PyCADI script.

Wrap-up
Using PyCADI with Fast Models offers a lightweight yet powerful methodology for streamlining the continuous integration process for integrated circuit software developers. As opposed to solutions involving racks of hardware boards, FPGAs, or hardware emulators, PyCADI with Fast Models is inexpensive, simple, and fast. It can also integrate effectively with existing continuous integration platforms. Ultimately utilizing this virtual hardware scripting system enables a more powerful use of the continuous integration practice, leading to reduced merge conflicts and faster time to market. All that leads to a more enjoyable workflow and more money in the bank.

Further, because this method of scripting is done with python, all of python’s modules and resources are available when scripting. This opens the door to endless possibilities when integrating with other environments and analyzing data. There is full API documentation of PyCADI which details all possible fm.debug object parameters and settings, but for the sake of convenience I put together a quick ‘cheat-sheet’ of useful commands to get started with PyCADI and Fast Models right away. That sheet, along with a sample Cortex-M4 model, is provided for free in the zip folder below. Using the files in this zip folder along with Fast Models, python, and your own applications, you can start improving your continuous integration system in a matter of hours.

Download now, and happy coding. Rather, download now for happy coding!