Will New Processor Architectures Raise Energy Efficiency?

Data centers continue to heat up as new processors consume more energy than ever before. Cooling is the primary weapon against the heat these processors generate, but it won’t be able to keep up forever with traditional processor architectures. New ones may be necessary.

There are opportunities today to make well-known architectures more energy-efficient, but the number of options for substantial changes is dwindling. As a result, new architectural ideas are being considered, with some moving to commercial availability. Some are very different from what exists today, even to the point of trying to recycle energy within circuits.

But processors don’t exist in a vacuum. They have broad ecosystems, including support for operating systems, coding, compilation, testing, and debugging. Would any architectural changes be compelling enough to force a large-scale infrastructure change?

Is the era of general computing over?
Embedded processors have long been tuned for specific applications. “In the past, people chose off-the-shelf DSPs and CPUs and used them as is, but at the expense of power consumption,” said Prakash Madhvapathy, product marketing director for Tensilica audio/voice DSPs at Cadence.

Data centers and PCs have employed general-purpose processors because the workloads can be incredibly varied. But the tradeoffs made for generality may be too much of an impediment to power reduction, even in the data center.

“The history of the system on a chip has been one of progressively more specialization,” said Steve Roddy, chief marketing officer of Quadric.  “The SoC era began with the first such chips having only one programmable processor on chip — a CPU.  Over the years, each new major category of processor core has been additive — a new type of processor added in addition to the CPU (or CPUs).  The creation of each new category has been a reaction to growing demand for a specific class of compute revolving around data types and computational intensity that CPUs are not well-suited to address.”

Roddy noted there is a style of computing that CPUs do particularly well, and a style that benefits more from GPUs. “CPUs are designed to chase pointers in C code,” he said. “They are not efficient at the highly repetitive task of drawing polygons, no matter how far you try to stretch performance with customizations.”

Second-generation NPUs with support for more AI graph operator instructions quickly replaced the first generation, but “a CPU with helper accelerators is still just a pointer-chaser wielding a set of crutches with roller skates,” he continued. “In 2025, we see the emergence of fully C++ programmable neural processing units [NPUs] in SoC designs that completely shift the AI workload off of the legacy CPU and onto a purpose-built processor tailored to the task.”

Tuning such an architecture requires extensive workload profiling. “When you run workloads that represent your final use cases, you have to make sure that the choices you make for instructions are tuned to your application,” said Madhvapathy. “You balance the area increase that each instruction provides with the performance benefit that it gives and the power consumption that entails. When you add area, you also increase the dynamic power a little bit, and you also increase static power because static power is proportional to area.”

This also is happening in the data center. Instead of replicating a single general-purpose architecture throughout, different groupings will be summoned for different workloads. This is already the case for AI, but it may go beyond that, allocating servers according to the needs of the workload.

Data-flow processor as an alternative
One way to approach this is to look into how the data flows through the processor. For example, Efficient Computer claims its data-flow architecture can reduce general-purpose computing power by a huge amount. “We’re getting compared to other parts in the market doing embedded computing, and we’re seeing 10X to 100X improvement in the energy consumption compared to those other offerings,” said Brandon Lucia, CEO and co-founder of Efficient Computer.

Efficient’s architecture uses an array of computing tiles that have no instruction fetch, no cycle-to-cycle configuration of the sort a standard processor must perform as instructions change, and no indirect data movement. Instead, the compiler statically assigns instructions to specific tiles. After loading, those instructions remain with the tile.

Fig. 1: Efficient’s data-flow processor (top) as compared with a typical one (on bottom). Most of the blocks in the traditional processor go away with Efficient’s architecture, as do some of the standard elements in the main computing tiles. Source: Efficient Computer

A network-on-chip (NoC) connects the tiles, but unlike typical NoCs, this one is circuit-switched (like old phone systems), not packet-switched. Once the network is configured, data simply flows while the connection remains open.

The energy efficient on-chip network allows one of the tiles to talk to any of the other tiles. “At configuration time, you set up a tile to execute an instruction, and then you set up the routes between the tiles so that when an operation runs on a tile, it produces an output that gets routed directly to the input port of a downstream instruction,” Lucia said.

This is simple in concept, and not entirely novel. Deeply embedded code has limited workloads and always does the same thing. But making it more general-purpose requires some enhancements.

Reloading on the fly
It might seem that this approach would work only for programs that can fit onto the array in their entirety. The first product configuration is a 12 x 12 array, so that’s not a lot of tiles. The company is working on larger devices, and the tiles can scale, but Efficient also must scale the memory and I/O subsystems, which requires more work. So at least for the time being, this isn’t a huge array of cores.

To avoid limiting the program size, the company has included a reconfiguration capability that can swap in new code. “It’s like continuation passing,” explained Lucia. “We reconfigure in a pipelined way so that as the first continuation is finishing, the second one is getting configured onto the fabric and can begin running as soon as inputs are available from the previous continuation.” They employ a “distributed doneness” algorithm to determine when to reconfigure.

Reconfiguration is partial in that any tile that’s complete can be reconfigured. However, arbitrary partial reconfiguration in the absence of a “done” signal isn’t possible.

Execution is nominally asynchronous in that a tile begins computing when its inputs are ready (according to a ready/valid protocol). Explicit scheduling isn’t necessary.

“That simple protocol allows us to do unicast, multicast, and broadcast communication from one tile to any other tile very efficiently, without the need for synchronization or centralized data structures,” explained Lucia. “You distribute the responsibility of synchronization between the tiles, and it simplifies the design considerably.”

The other big requirement for general computing is the ability to handle conditional execution. That means branching, but it also affects loops, which must test against iteration limits to determine when they’re done. The simple answer is to convert everything into a steering operation. Based on an input signal (a decider), the output is on one or the other path.

But using up an entire tile only for such a basic operation would be inefficient. Instead, it turns out that each tile can fuse instructions, so conditionals can be fused with other instructions. “On a given tile, you might have three or four or five operations encoded in the configuration of the tile, and they’re fused together,” explained Lucia.

As with function units in conventional cores, different tiles have different capabilities. “Some of them can do things like multiplication. Some of them can do things like 32-bit floating points, and other ones can’t,” said Lucia. Code profiling during chip development determines which tiles receive which instructions. The number of fusible instructions depends on the tile and the resources that remain in the tile as the fused instructions claim them.

Such an architecture can exploit parallelism as long as the compiler can detect it. The company says no special programming techniques are necessary, and that standard code will work. “Unlike accelerators, we’re able to do this general-purpose thing even for really freaky code,” said Lucia, who said an old copy of a Doom program ran successfully.

Energy recycling
An even more revolutionary approach to processing comes from Vaire Computing. Its underlying philosophy is based on thermodynamics. “In CMOS, we represent bits using voltage levels on circuit nodes, and there’s some signal energy associated with those voltages,” explained Mike Frank, staff scientist at Vaire Computing. “When we transition to a new voltage level, we dissipate the entire signal energy that’s used to encode a bit. That’s because of the way that switching is done conventionally, by connecting circuit nodes to constant voltage sources. Anytime you throw away digital information, you have to throw away the energy that’s used to represent that information. When you’re throwing away information, you’re just converting it to entropy, and the energy associated with that information is converted into heat.”

Vaire has developed what it calls reversible computing. After performing an operation, that operation can be “reversed” to recapture much of the energy used. “In reversible computing, we try to avoid throwing away information,” said Frank. “Instead, we transform information in reversible ways so no information is really getting lost. This enables you to recycle the energy that was used to represent those bits.”

The chip employs a resonant power supply, which adds an analog element. Each computing node has something akin to an LC tank, from which it pulls energy. Then it returns most of the energy. Inductors are on-chip. The company says losses can be made arbitrarily small and are typically associated with leakage and the Q factor of the resonators.

“It’s an AC supply rather than a DC supply,” said Frank. “Instead of flowing from the power supply to the circuit and then turning into heat, we push energy into the circuit to change its state, and then we pull energy out of the circuit to change its state again.”

The overall architecture remains digital, but slew rates are managed to be slower, allowing for adiabatic (that is, involving no gain or loss of heat) switching. “It’s still fundamentally a digital computing paradigm, and it has the reliability of digital,” said Frank.

It’s possible to completely reverse a set of instructions to recover intermediate states, although that’s not the company’s goal. It wants to allow standard programming techniques, which don’t involve reversing. But it may be possible to further optimize a program by taking advantage of this.

Slower clock, more parallelism
Clock rates won’t be as fast as typical CPU clocks today. They will be closer to GPU clock rates. “GPU clock speeds are much lower, so we’re not that far behind compared to GPUs,” said Rodolfo Rosini, CEO of Vaire Computing.

That’s partly because the energy recapture takes time, adding to the clock cycle so it all happens within a cycle. “Within that clock cycle, parts of the circuit are going forward, parts are going backward, de-computing things,” said Frank. The overhead cuts the clock speed by around 50%.

“This technique works really well when you have ridiculously massive parallelism, and fortunately, today’s large language models are of that nature,” said Frank. This can make up for some of the performance loss. One may be able to save yet more power by slowing the clock and doing more in parallel.

There’s a whole body of research around this topic, and much of it is relatively esoteric, leveraging so-called Toffoli gates and ancilla bits. Vaire appears to be the first to attempt a commercial offering. That means coming up with practical, usable, and affordable implementations of what otherwise have been abstract notions, including ones that don’t involve literal Toffoli gates, which are economically inefficient.

“There are certain rules you have to follow, like not turning on a transistor when there’s a voltage drop across it and not turning off a transistor when there’s current through it,” said Frank. “It turns out that if you obey those rules, it’s logically reversible and it’s highly efficient. It’s just a matter of carefully sequencing the transitions in the circuit and arranging the circuit topology in such a way that you perform a certain series of transitions on the driving clock signals that are controlling that circuit. And if you do everything in the right sequence, everything’s reversible and fully adiabatic.”

Still, Vaire is a couple years away from commercial availability, so architecture and circuit details are not available.

Can new architectures succeed?
Despite all the revolutionary changes the semiconductor industry has brought to the world, it remains uncomfortable with revolutionary new ideas. It’s more comfortable with evolutionary changes, which is why tweaks to existing architectures are always preferred. It takes a compelling problem and an equally compelling solution to drive more substantial change.

Processors require an enormous range of tools, many of which come from third parties. So even though software has been the Achilles’ heel of new AI architectures, it represents a big development must-have for a new processor architecture to achieve mainstream commercial traction.

“What we currently see as a big problem for new architectures is that you also need a big bundle of software on top of it, and that makes it so expensive,” said Andy Heinig, group leader for advanced system integration and department head for efficient electronics at Fraunhofer IIS’ Engineering of Adaptive Systems Division. “A good hardware solution is maybe 20% or 30% of what you need in total. This is often something we see with startups. They ignore the software part, and then it’s clear to us that they don’t have any chance in the market.”

If companies such as Efficient Computer and Vaire Computing can develop their own necessary tools and work with the ecosystem to ensure support, they could have a chance — particularly if software developers don’t really need to understand the underlying architecture to program it. Both companies are trying to make their chips programmable using standard coding techniques.

It ultimately boils down to an economic question. “If we have enough pressure from the data-center side that the energy consumption is too high, then we will see new architectures, because then it also makes economic sense,” said Heinig.

Success depends on a number of factors, though. “There’s no shortage of new system architectures being proposed that are purpose-built to focus on improving performance, cost, and power-efficiency for specific applications,” said Steven Woo, fellow and distinguished inventor at Rambus. “Some subsystems are used for rapid development of differentiated architectures, like CPUs and GPUs. But for some critical subsystems like memory, a key challenge is finding the volumes necessary to enable new architectures due to the high development cost. Markets must be large and offer good return on investment, often leading to a chicken-and-egg dilemma of how to enable a new market with critical technology when the technology needed to do so hasn’t been implemented yet.”

From a pure technology standpoint, big energy savings are possible. It’s just not clear yet how to practically make that happen, or if processor designers must be satisfied with more measured approaches that will still help reduce power, but less dramatically.

Related Reading
Crisis Ahead: Power Consumption In AI Data Centers
Four key areas where chips can help manage AI’s insatiable power appetite.
What’s The Best Way To Sell An Inference Engine?
The hardware choices for AI inference engines are chips, chiplets, and IP. Multiple considerations must be weighed.
When To Expect Domain-Specific AI Chips
With the intended application evolving faster than silicon can be developed, optimizing hardware becomes a delicate balance.