What Is Functional Accuracy?

What it means to be functionally accurate in the context of virtual platforms varies greatly, depending upon whom you ask and even when you ask them. But that doesn’t mean that functional accuracy isn’t useful.

Jon McDonald, technical marketing engineer for the design and creation business at Mentor Graphics, expects to see a lot more activity in this area in the near future because engineering teams have to figure out how to define what is the minimum functionally accurate system. “We’ve done a lot of work in moving up in abstraction, in how we’re describing the design, but there hasn’t been nearly as much thought put into moving up in abstraction what is correct in the design and what is a verified design, because it’s going to be different at different levels of abstraction for different people.”

Frank Schirrmeister, group director of product management and marketing in the System and Verification Group at Cadence, agrees that the quest of trying to move the golden reference model upward has been going on for years, and in some areas, is possible now depending on the scope of the system.

So what exactly is functional accuracy in the context of a virtual platform? “It means that if you take a specific synchronization point, and this works very well with video, for example — let’s say frame by frame, it really decodes H.264 or whatever it is. You functionally represent that correctly in the virtual platform and it may be a C function or it may be abstracted even into a middleware-type function that you find on the workstation on which you execute the virtual platform. You may not even do H.264 decoding functionally correct in a C model yourself. You may just call an API, which your host workstation may provide, but at the end of the day functionally accurate means that at a specific synchronization point the data you see at an output is correct. That would be functionally accurate on a per-block level,” Schirrmeister offered.

The meaning will be rather different from a 14-bit fixed point signal processing perspective, however. “If I have it refined, now it is micro-functionally accurate because now literally every bit would be correct,” Schirrmeister said. “It would not be that the video by itself is correct, calculated in floating point, but now I would have every bit absolutely correct representing the functionality. Then, you get away from the space of functionally accurate in a context of timing and so forth. How do I now transmit this data? That’s becoming an implementation aspect.”

Another side of functional accuracy is this desire to have a fully correct executable specification, which is even more difficult to achieve, he explained, “because now you elevate this from a block level, let’s say the video decoder, to the full system aspect and now you have to essentially represent all the scenarios a system could go through correctly. That is very hard to build and there is simply sometimes no need for a fully executable specification because you end up, for verification, typically not using the full system anyway. You define the subsystem or the sub-components of the design that you need to verify. Functional accuracy is definitely important for the verification aspect and of course the ultimate goal in that context has always been to come to one model from which you can drive the verification and simulation model and use the same model for for implementation. I think we’re getting closer. There are now paths out there.”

Another definition of functional accuracy in the context of a virtual platform involves properly modeling higher-level mechanisms, according to Benoit de Lescure, director of application engineering at Arteris. Multiple levels of mechanisms in a system — cycle-by-cycle, precise protocol type of transactions, for example — will have to be cycle- and bit-accurate. But there are also mechanisms, such as those that rely on measuring quantities of something transferred over a large period of time, like quality of service.

“You expect the network on chip (NoC) to implement mechanisms like quality of service, and using those mechanisms you will be able to share a resource like target or link, after an arbiter, according to some settings for, say, I want this guy to get 30% of the bandwidth. But 30% of the bandwidth means you’re measuring transfer over a larger period of time. In that context, I want to properly model that path of the system, so I’m looking at long simulations. There are a bunch of details I don’t care about, and it’s okay to abstract them. But what I care about is to be functionally accurate for my quality of service. What kind of detail can you get rid of? And is acceptable to abstract in that context, for instance, anything that doesn’t interfere directly with the system and the considerations like error management? You will assume your system has no errors, and even if there is a way to capture an error. All you want is a functionally accurate model of the system when it’s in steady state.”

Measuring performance
Drew Wingard, chief technology officer at Sonics, noted that functional accuracy also could be referred to as performance accuracy in the context of the two commonly deployed virtual platform approaches. One approach is focused on building a model that allows software development to proceed in advance of hardware being available. “In that case, the notion of time as something that needs to be modeled accurately is not often considered in most of the work. The work is about getting the software functionally correct. The virtual platform models that are created typically are focused on trying to get the fastest simulation speed they can get, so they are intentionally cheating with respect to modeling time.”

The microprocessor on which that software is running needs memory access to store data structures and receive instructions. “They don’t model that at all,” he said. “They basically just use the host memory on the CPU on which they are running the simulation very directly to increase the speed of the software simulation by orders of magnitude by not trying to model that. As soon as you do that, a lot of things around time go out the window. What matters to getting the software functioning correctly typically boils down to ordering. If this event has to happen before that event, there are situations in which you can’t convince yourself the software is going to work unless the virtual platform model can let you say that. That boils down to things like synchronization, not really about time modeling.”

Wingard observed that people have tried to build those models where they are ‘loosely timed,’ but in his experience many design teams give up on the value of that.

The other style of virtual platform modeling is even more abstract. He cited the old CoWare’s deal with Motorola to solve a specific problem. The call centers were getting calls from owners of phones, but there were so many models of phones that everyone in the call center had to have a box full of Motorola phones so that when they were trying to guide somebody through a step, they could power on the specific model of phone and walk the caller through the system. CoWare created a high-level virtual platform model that could run the actual software in close to real time so that they had a computer in front of them that had a virtual version of the phone.

“When we get close to topics around performance,” Wingard said, “what we find is that there’s a different kind of platform model that’s often created and that’s one that tries to abstract away the function of the chip, and instead focus on the performance of the chip. Why would you do that? We do worry about how long it takes for the simulation to run. In the performance domain, a lot of time the data that would be transmitted between two blocks is kind of meaningless — all that matters is how much data is moved and over what period of time and to where, and things like that.”

This leads to the topic of memory modeling, because in many of these systems most of the communication between components happens via external shared memory in the form of DRAM. That means you have to model the throughput of the memory system, and the latencies associated with it are dependent upon the addresses, the burst lengths, the read-write mix, and things like that,” he said. “The characteristics of the memory can vary widely, or wildly even, based upon those because DRAMs are built out of pages and banks, and things like that. What we found in that style of modeling is it’s really important to model the network and the memory subsystem in a cycle accurate fashion. It turns out that anything that’s not fully cycle accurate in that modeling ends up generating errors which tend to accumulate so you can’t make good performance determinations. The tradeoff we make is, we take what most people think of as a valuable IP block—the CPUs and the GPUs—and we replaced those with black boxes and we think of them as just traffic generators. As long as you can get relatively accurate patterns, and burst lengths and behavior in the time domain, then you can combine that information together and build useful things.”

A tricky shift
While the benefits might seem clear, many times the shift to a higher level of abstraction can be tricky. McDonald said that during a recent project with Altera, its engineers knew Mentor Graphics was creating a TLM platform for them but they kept trying to verify it the way they would verify RTL. “They were trying to look at things cycle-by-cycle or instruction-by-instruction, and it just doesn’t apply at a transaction level. We need to be able to say what are the critical things that have to be correct; what are the critical things that have to be in the proper order, and it’s not going to be everything because the models are just different.”

But this too present challenges as there isn’t really an agreement in the industry of what those things are.

“I’m sure there’s probably a way of tagging those things or expressing what are the critical things,” McDonald said. “Part of the problem is I don’t think you necessarily can say across the board, for example, that the interrupts have to happen in the same order. Somebody might say it’s not functionally correct if the interrupts don’t come in in the same order. Another person might say, ‘I don’t care what order the interrupts come in in as long as my outputs are coming out appropriately, and my responses to stimulus may be different.’”

The fact that many of the people who are using virtual platforms are software-focused also complicates matters because software is typically a lot simpler in terms of its concurrency requirements, McDonald explained. “The software people might be thinking about multiple threads, but not at the same level you would in hardware. And you’re typically thinking about a single thread of execution, but when you look at a complex system with multiple threads, interrupts coming in asynchronously, resource collisions across processes running on different cores, and even changes in the behavior of the system based on more load on different cores—all of these things can change where things get processed, what order they get processed in, and you want your system to be able to support that. So there may be multiple correct sequences of actions that need to be able to be accepted as correct.”

Clearly, much needs to be done to sort this all out for the engineering community. But at least the problems and discrepancies are being identified, which is a good first step.