The Growing Software Challenge: From Stacks To SMP

By Ann Steffora Mutschler
Building a system now includes software, but defining the software stack is a mounting challenge for engineers. What used to be almost exclusively drivers now includes RTOSes and OSes, executable files, middleware, firmware, IP, embedded software and applications.

With millions of different embedded products, all with different sets of software, it comes down to product requirements in terms of what the product has to do. And that plays a lot into what kind of software stack you need, said Cadence architect Jason Andrews.

“How much does the user see? How much do they need to be exposed to? On one hand, you have products where the user sees nothing. The software is completely automatic and invisible. The other extreme is a product where the users can add their own applications. Then you have to build something that is quite a bit more flexible. It comes down to the requirements of the products,” he explained. “Also, for most software engineers, they think about the hardware they have. A lot of times they are given hardware and they say, ‘Go find the software stack and get all the software to do certain requirements on a given hardware.’ So the hardware isn’t always flexible. There may be only a certain amount of memory or a certain kind of processor they have to work with.”

Up the stack
Technically speaking, the software stack depends on the IP at every layer. “The OS layer needs to implement the driver that interfaces with the IP at the lowest level (registers),” said Achim Nohl, technical marketing manager for Synopsys’ solutions group. “The OS driver provides rudimentary services to the upper layer that can now control the IP (e.g., In case of a GPS driver: open GPS device, write control data to the GPS device, read control data from the GPS). The OS is shielding the hardware details such as registers of the GPS.”

The next layer is the middleware, which is largely a set of libraries. In the case of Android, it has a huge middleware component consisting of libraries that provide easy-to-use interfaces to the application layer to control all aspects of a phone. The middleware provides more comprehensive services to the application layer to control the IP. For example, in the GPS example above it will allow to get the coordinates, get the direction, among other things.

“This stacking makes a lot of sense as now the application gets more independent of the OS. The middleware is shielding the details of the OS. This way, the same application can run on OMAP or Qualcomm Snapdragon. Even though both platforms run different configurations of the OS, the middleware allows the application not to worry about it. The middleware takes care that the functions such as get coordinates are translated into the driver commands for OMAP or Snapdragon. Thus the middleware is typically platform-specific at the driver level. It talks to the drivers, not to the hardware directly,” Nohl said.

Next is the application layer. Applications (executables) can be native or interpreted. “Native” means they are compiled for a specific platform. This is very efficient, but it makes the application executable platform-specific. In order to create the executable OS, libraries and middleware libraries need to be considered. Thus, the application executable gets OS- and middleware-specific and can run only on those platforms using the same processor, such as an ARM A9 only, an ARM A8 only, or a MIPS processor only.

Finally, applications in the Android Market Place are Java-based and can run on any phone that runs the Android software stack. Obviously, it is a challenge to make sure all those layers in the software stack work well together and are tested well enough. There can be dozens of variants of hardware platforms.

Selecting and defining the stack
In terms of the selection of the software stack, it depends on what kind of software you need to run.

“In a deeply embedded device, you want to run a software stack with a very small memory profile and very little overhead so that power is saved and not much memory needs to be spent.” Nohl said. “In the case of a DSP-type subsystem you may run an RTOS, as you have real-time constraints to fulfill such as processing signals from a sensor every 100ms. In the case of security applications, you might run a so-called hypervisor underneath the software stack. This is a kind of virtualization that somehow watched what the OS tries to do with the hardware and can interfere in case of a strange thing going on. In the case of a consumer application, you will choose an OS such as Android. Other vendors have their own locked OS such as RIM or Apple. In the case of automotive, special software stacks such as AUTOSar have been specifically designed for automotive applications.”

After defining the software stack, ideally, it should be analyzed. ARM has a software tool, Streamline, which is meant for performance analysis but can also perform stack analysis, according to Javier Orensanz, product manager for debug tools in the system design division of ARM. Streamline can be configured to generate a static analysis report on the stack usage for the different parts of the software and from this report, you can get a pretty good idea of what kind of stack you need, he said.

“In reality, you should always compliment your analysis of the stack with some dynamic analysis to see how the stack is used as the software runs. In this area there are several things that can be done. The quick, easy and dirty way that most people do it is to fill the memory with a pattern, run the code, then check the memory to see how low the stack went. This analysis is pretty good because as long as you cover the typical usage of the software it gives a dynamic way of how low the stack is going to go. So it is very useful. But this doesn’t tell you which functions make use of the stack or just how low the stack goes. This is why some alternative information is good to complement it, so if you see an unexpectedly high use of the stack you need more information to make informed decisions on how to improve your software,” Orensanz explained.

Key considerations
At a higher level, there are a variety of issues that can be considered ‘key,’ but what is most important depends on the perspective.

“I think if you ask software people, ‘what’s your biggest challenge,’ there are probably two things: All engineers will say the requirements keep changing, which is always a struggle. Besides that, I think the number one thing everyone is facing now is the gigantic complexity crisis because what happens is that the software stack has gotten so big that you’re assembling a lot of code that you didn’t write,” said Andrews. “The stack is like layers of abstraction and they get all the pieces and it sounds good, but it’s just like any kind of re-use story. It sounds good when you get to re-use things, but as soon as something breaks then you have to peel back all those abstraction layers and figure out what’s broken underneath. Pretty much universally everybody I’ve been talking to on the software stack and the system verification say, ‘We get all the pieces, we put it all together, it mostly works, and then all of a sudden the user interface starts bogging down and it’s slow. Now what?’ So they have to go way down in the bottom of the stack somewhere and find some driver, which came from somebody else and is misbehaving, and it takes a really long time because of the complexity. This is the biggest challenge—software stack complexity. A lot of mobile devices contain 10 million-plus lines of code and you wrote almost none of it.”

One way to manage this complexity is with simulation-based techniques in virtual platforms that aim to help engineers understand how the system is running, more quickly find where the problems are, analyze what parts of software are taking time, why the battery is draining so quickly – basically to get more information out. Historically this was done on FPGA boards or prototype boards and the visibility was very poor.

“Quality is very complicated to achieve,” said Victor Reyes, technical marketing manager in Synopsys’ solutions group. “We are dealing now with huge software stacks with very complex software running on very complex hardware platforms with multicore systems, and basically the integration issues between software and hardware are so complicated it’s not easy to understand all the issues that the integration produces.”

Synopsys is trying to create solutions to help engineers improve quality by improving software debug by giving global visibility on the complete system, including the hardware platform, the software executed on top, as well as the environment around the device.

Too many cooks
Another issue gumming up the works is that there are more stakeholders contributing to an embedded software stack

“This has changed quite a bit from the past,” Nohl said. “Previously, it was the mobile phone manufacturers that were providing the firmware and applications. Today it is many-fold. The semiconductor companies need to establish an operating system running on their specific application processor. Then there are the application spec providers like Google with Android, and those application software stacks need to be ported now by the mobile phone vendors to the specific mobile phone, running on the operating system provided by the semis. And at the end, you have a whole community of software developers that are creating apps for this specific application—all running on an embedded device. All the functions are highly distributed and things are intertwined.”

Outside of the consumer realm, Reyes said that automotive applications demand safety and robustness as the amount of software in the car is growing exponentially.

The impact of SMP
In addition to quality, complexity, time-to-market, performance, power and the usual suspects creating problems, the true realization of symmetric multiprocessing is having a huge impact in the industry.

“Multicore and multithreaded software support is becoming a crucial element as we go forward,” said Mark Throndson, MIPS’ director of product marketing. “It takes a lot more sophistication in terms of the software development and such, but it’s basically being driven by performance demands to achieve the Web-connected experience, driving down into everyday products, not just the PC. For better or worse, the software needs to be able to take advantage of that, and frankly it’s been easier historically to expand the hardware in this area and software is challenged to catch up effectively. It’s a combination of SMP-based operating systems as well as fundamentally developing code that’s more threaded in nature to take advantage of it.”

Learning how to make simultaneous multithreading and simultaneous multiprocessing work well together, then designing software threads so that when running on a multicore they make optimal use of the resources, for example, and balancing load equally between processors – all of these skills should be next on the software engineer’s to-do list.