New Approaches Needed For Power Management

Power is becoming a bigger concern as the amount of data being processed continues to grow, forcing chipmakers and systems companies to rethink compute architectures from the end point all the way to the data center.

There is no simple fix to this problem. More data is being collected, moved, and processed, requiring more power at every step, and more attention to physical effects such as heat, accelerated aging, and signal interference. The greater the transistor density and the higher the utilization, the greater the effort required to contain these effects, and the greater the number of tradeoffs that are required. This is especially evident with AI/ML applications, where the primary focus is performance and maximum utilization of processing elements with extremely fast data throughput.

This is in addition to other power-related problems that have been compounding since at least the 40nm process node, when gates began leaking power even in the “off” state. That was solved briefly at 16/14nm when a fin was added to transistor gates to control static leakage, and it worked remarkably well until about 7nm, when current leakage began creeping back. That led to gate-all-around FETs, which are being implemented starting at 3nm, with plans for complementary FETs somewhere in the angstrom range. And all of that is aggravated by increasing dynamic density at each new node, which results in more heat in a progressively smaller area, and forces chipmakers to shift their key metrics from processor frequency to performance per watt.

“Now that you have to bring in and dissipate more power in a small space, all kinds of interesting things can happen,” said Steven Woo, fellow and distinguished inventor at Rambus. “Over time, the heat that’s being dissipated can have effects on the chip, and you may need to worry about thermal cycling. As the chip does a lot of work, part of the chip stops, then continues to do more work. These rapid cycles of dissipating a lot of power, then not, cause local heating, cooling, and thermal stresses that all chips are subject to. Sometimes you worry about things like the solder balls on the package cracking over time, or PCBs warping, along with other mechanical stresses. You can imagine when you have a very big chip, toward the edges of the chip it will expand more than in a small chip. That can add stress too. This means you have to really be careful about how you cool things. Memory is no different. With memories, there are very specific things to worry about, such as that the ability to retain data depends on how hot the chip is.”

This is why many of the most advanced chip designs have large heat sinks, and increasingly some type of liquid cooling. It also explains why some of the most advanced designs are pushing processing to the edge, in sensors and/or memory, to limit the amount of data that needs to be moved. This is essentially advanced load balancing at the nanoscale. It’s more efficient and particularly useful for edge devices powered by a battery.

“Lower cost and lower power consumption are extremely important in edge devices,” said Tony Chan Carusone, CTO at Alphawave Semi. “Those are areas where custom silicon becomes more important. One of the things that’s next for computer vision is edge AI silicon solutions that help lower the power, lower the cost.”

This certainly helps, but it still doesn’t solve everything. In the case of image sensors, for example, large amounts of data in the form of streaming video or images may need to be processed. But these sensors are very temperature sensitive — not so much in their device characteristics, but more because of their polarity.

“If you want an image sensor, you have this large chip and you want to do the image faithfully in very high resolution,” explained Marc Swinnen, director of product marketing for the semiconductor division of Ansys. “You can’t have that image sensor bending and warping because of temperature. Even a slight bowing of the image sensor will leave your image distorted. The problem is that most configurations stack image sensors on top of electronics chips in order to get immediate processing of the image. But those electronics get warm, and that will cause the image sensor to warp if you’re not careful. You need to do extremely accurate and very sensitive thermal analysis to make sure that the image sensor remains flat and doesn’t warp or bend because of thermal mechanical stress.”

Fig. 1: Thermal simulation of a graphics card. Source: Ansys.

Multi-chiplet issues
Add heterogeneity into the mix, particularly with chiplets, and this becomes much more complicated.

“Especially when we talk about heterogeneous integration, you don’t get the thermal conductivity that you would in a straightforward, monolithic design,” said Dan Lee, product management director at Cadence. “There’s a bit more thought and planning that needs to be a part of this, because aging and heating are related in that heat impacts how a chip ages. All things being equal, if you’re operating in a very hot environment, you’re going to expect a lower lifespan.”

All of this together presents a challenge to designers, as sometimes conflicting specs have to be balanced. In the case of automotive chips, for example, heat may come from inside or outside the chip or package.

“Power optimization is very important for our automotive customers, because the ambient temperature ranges are so different,” said Suhail Saif, principal product manager at Ansys. “Because of safety ratings, they have to design for the worst-case scenario analysis, which can be very different from normal cases. The range for which they are designing is so huge that they have to make a lot of compromises on power and performance. They deploy power optimization at every stage. For example, they put in extra logic. Even if it takes more battery power, that’s okay. In a car, you can easily supply power, so they compromise on that, but then they build more power controllers into their chip than you would find otherwise in order to manage the power later on. These kinds of design compromises are made at every stage to make sure that the power is under control and the power envelopes are always met.”

More data, more types of chips
In addition to carefully balancing designs using existing components, the industry is exploring incorporating novel chips. This is especially evident in the image sensor world, which is one of the main causes for the increase in data. In cars, for example, streaming images from the road are forcing automotive engineers to consider a variety of alternatives, from overall system-of-system architectures to new types of chips. In an electrified vehicle in particular, compute efficiency is considered essential to maximizing range, and heat is largely wasted energy.

This is evident with the introduction of event-based vision sensors (EVSs), also known as neuromorphic sensors. Yole Research predicts the market will reach $2.9 billion by 2034. Based on the neuromorphic idea of a “silicon retina,” EVSs offer a lower-power solution to vision sensing. They can handle many of the same functions as traditional computer vision sensors, with a fundamental rethinking of the method of operation based on how the brain processes vision, rather than what’s convenient for a digital chip.

The biggest stumbling block to the wide adoption of neuromorphic sensors is uncertainty about when they will rise beyond niche status, not whether academic ideas can scale. Most neuromorphic sensors use a newer form of neural networks, called spiking neural networks (SNNs), which are unlike current computer vision devices that use convolutional or transformer neural nets.

“There is very little in the way of published research or production-ready neural nets utilizing neuromorphic techniques,” said Steve Roddy, chief marketing officer for Quadric. “For the conventional neural network models, there are quite literally thousands of variants of object detectors, scene segmenters, object classifiers, pose detectors, and more modalities of proven effective NNs. There are even scoreboards keeping track of hundreds of variants ranked by accuracy, parameter size, and compute intensity. Thus, if one chooses a conventional image sensor and a conventional NPU or GPNPU to run the ML inference on the sensor output, there are plentiful solutions to choose from in building a successful product.”

By contrast, in the area of neuromorphic computing, there are a dozen or more research toolsets to explore SNNs. But there are few, if any, useful repositories of models, Roddy said. “The most prominent thing one finds is a bunch of tools to take conventional NNs and convert them into neuromorphic form for further research and refinement. That’s great if you are a researcher, but not great if you are trying to build a product.”

Still, interest in neuromorphic sensors remains strong in the research and start-up communities because of the low power benefits, which are based on a re-thinking of image processing. The roots of traditional CMOS vision processing go back as far as Muybridge’s late-nineteenth century studies of horses in motion. His work introduced the idea of parsing continuous motion into discrete still “frames,” taken at a particular rate. This principle still underlies most common approaches to computer vision, which means a vision sensor records everything in the scene, without selecting for salience.

This redundant approach also affects energy consumption. Christian Brändli, now CEO of Sony’s Advanced Visual Sensing AG, which is developing a neuromorphic sensor, summed up the contrast in his doctoral dissertation, “While this way of sampling a scene uniformly in space and time allows employing uniform and easily developed processing routines, it is inefficient. This does not matter as long as time and energy are not key to an application…But in systems that interact with the real world, latency becomes an issue and if the systems run only on battery power, power consumption also becomes a critical aspect.” ^1.

To solve the problem, neuromorphic visual systems take as their inspiration the way the optic nerve processes information, by determining salience through luminance changes, and thus discarding repetitive input. A familiar comparison (in theory, not execution) would be JPEG image compression, in which the size of image files is reduced by condensing redundant data, such as a blue sky that does not change from frame to frame.

While JPEG image processing is done after image acquisition, and usually introduces some information loss, a neuromorphic sensor is more akin to a “sophisticated redundancy suppression device at the deposition level,” said Luca Verre, CEO of Prophesee, which has a research sensor that uses SNNs, and a commercial EVS sensor that does not. “Like the biological retina, our sensor is not capturing images at a fixed point in time. By contrast, the regular sensor is capturing a sequence of frames. The problem with this type of acquisition principle is that you end up capturing a lot of redundant information because often part of the scene is fully static, so you keep acquiring images with a lot of redundancy. While a biological retina, in the human eye in particular, is not capturing an image and sending an image to the brain at a fixed frame rate, but only capturing what is changing in the time continuous fashion.”

The pixel captures when the light contrast changes, without providing an intensity value. “It does not provide a gray level or color information. It’s only telling you if there’s been an increase or decrease of light by a certain relative amount. And this is typically associated with some dynamic in the scene to some motion in the scene. Where nothing is happening, then our sensor would be completely silent, while still acquiring the light in the scene. But as long as the light doesn’t cross a certain relative sensitivity threshold, then it will not send any information. This enables that sensor to be very low power. As soon as there is some change in the scene, then the sensor will wake up and then cut through this change at a very high speed because we are able to capture these changes with the time precision of microseconds.”

In both neuromorphic engineering and neurobiology, the narrowing of information is called “sparsity.” In neuromorphic engineering, sparse designs emulate how biological neurons select for salience by reducing the amount of information that passes from the retina to higher-order neurons in the brain, through a series of neuronal “spikes” (firing). These principles led to the development of spiking neural networks, which underlie most events-based sensors, as opposed to the older convolutional neural networks that underlie frame-based sensors.

As imec describes them, “SNNs mimic the way groups of biological neurons operate – firing electrical pulses sparsely over time, and, in the case of biological sensory neurons, only when the sensory input changes.”

Put in perspective, this is one more tool for engineers to reduce power and heat, and to potentially reduce the number of compute elements required to process more data. And that data can be processed locally, which can further reduce the overall power budget.

Prophessee has both SNN-based research projects and a non-SNN sensor already in the market. Meanwhile, imec is working on events-based neuromorphic-inspired processing architectures, for processing and fusion of various sensors (camera, radar, lidar, etc.), while trying to ensure they can work with current semiconductor process technologies and industry-compliant design methods. “We are inspired by the brain and by the biological neuron, but we also want to make devices that are practical for our customers to use in semiconductor chips,” said Christian Bachmann, program director, wireless sensing and edge AI at imec. “What we call the digital neuromorphic mimics a neuromorphic design, but uses standard chip design tools and technologies for implementation. Our work on event-based neural networks makes use of temporal sparsity, also the sparsity in the input data or in the weights of the neural network. We exploited those qualities both on the algorithm side but also on the hardware side in the circuit and architecture implementation on the chip to make neuromorphic designs easy to use by our industrial partners. Our designs aren’t based on exotic materials or techniques that are still decades away from actual fabs. They can be used today.”

And while it isn’t a perfect imitation of the biological neuron but rather an inspiration, the most important aspect is to exploit event-based processing, Bachmann said. “Similar to what the brain does, you only process parts of the neural network if something happens. Sparsity means, for example, that there are gaps in the data or in the weights of the neural network that you don’t need to calculate.”

It’s this sparse approach that should lead to low power benefits. Imec claims its SNN prototype uses up to a hundred times less power than traditional chips and offers a tenfold reduction in latency.

Conclusion
Power is expensive in terms of initial generation, the amount of power required to move data, the resulting heat associated with increased processing, and the effects of that heat on circuit aging and overall performance. Solving these issues requires more than a giant heat sink, though. It requires rethinking every aspect of chip design, where that processing occurs, and what actually gets processed.

How the market for chips unfolds may vary greatly from one application to the next, and from one workload to the next. But the entire chip industry is focused on solving these issues, as well as new ones, and that almost certainly will lead to some fundamental changes in low-power, high-performance design over the next decade.

—Ann Mutschler contributed to this report.

References

1. Brändli, C (2016). Event-based neuromorphic stereo vision, doctoral dissertation for Zurich University

Related Reading
Running More Efficient AI/ML Code With Neuromorphic Engines
Once a buzzword, neuromorphic engineering is gaining traction in the semiconductor industry.
Fundamental Issues In Computer Vision Still Unresolved
Industry and academia are addressing next steps.