Where Power Savings Really Count

Experts at the Table: Semiconductor Engineering sat down to discuss why and where improvements in architectures and data movement will have the biggest impact, with Hans Yeager, senior principal engineer, architecture, at Tenstorrent; Joe Davis, senior director for Calibre interfaces and EM/IR product management at Siemens EDA; Mo Faisal, CEO of Movellus; Trey Roessig, CTO and senior vice president of engineering at Empower Semiconductor. This discussion was held in front of a live audience at DAC. To view part one, click here.

[L-R]: Tenstorrent’s Yeager; Siemens EDA’s Davis; Movellus’ Faisal; Empower’s Roessig

SE: For a long time, the general consensus was that data centers accounted for about 2% of the electricity being generated. But according to the Electric Power Research Institute, by 2030 data centers will consume as much as 9.1% of all electricity generated. Clearly we have a problem to solve, but how much of this can be done at the chip level?

Yeager: There are a lot of things we can do at the chip level to improve efficiency. One of the reasons power doesn’t scale very well is that higher current densities are causing worse voltage droop issues, which then we have to margin. People on this panel are working on techniques to help mitigate that, but we’ve had 150-plus millivolts of droop for more than 10 years on designs, and Vcc_max for reliability and Vcc_min are getting closer and closer together. So getting rid of margin is one of the biggest things that we can do at the at the chip level, as well as revising our hardware-software contracts to get more efficient.

Davis: But with every one of those technology improvements, like backside power, what do you do with that?

Yeager: Backside power delivery will maybe be a little bit of a positive blip. But with 3D stacking, there’s a cost for that in power delivery. It’s going to come back to macroeconomics. Eventually, it’s going to cost you more to do something with ChatGPT — or whatever the future equivalent is — than what you’re willing to pay to just do a couple of Vim commands or to do it manually.

Davis: As chip designers we get efficiencies, but we don’t keep them. We spend them to make something faster, because this is how we win the socket. So there’s Moore’s Law, and there’s Koomey’s Law, which was very similar up until about 2000 where it was doubling the efficiency of joules per compute about every 18 months. Then it went to about 2.5 years. Today, we’re getting chiplets and new techniques for that. At the chip level we’re going to work on making things more efficient, but the big wins are going to be at the system level, or the communication between dies, or in the packaging.

Faisal: In his [DAC] keynote, [Tenstorrent CEO] Jim Keller talked about a pyramid. It’s the same thing at the chip level. The transistor level is at the bottom, and at the top is whatever light bulb is on for a few minutes. The pyramid has to be connected up and down so we can read the sensors on the chip and see what’s happening to the power supply. And based on that, you send certain data and figure out how to run it. That requires more algorithmic and higher-level programming. But a lot has to be done at the bottom of that pyramid, because 1% efficiency in every single gate multiplies up, and it becomes a very big number very quickly.

Roessig: I echo the need for system-level efficiencies, but at the end of the day the processes will get better, and they’re going to suck down whatever charge they need per bit. And that’s how it’s going to work. There will be some companies that can do the system-level voltage scaling. Back in the day, only Intel was doing it, and then AMD caught on. Now, a lot of customers are saying, ‘I don’t want to deal with that. I’m the hardware guy here and I’m designing this part, and somebody else is going to be responsible for knowing how much power my chip is going to take. Where do I put the voltage?’ There is a system-level skill set that a lot of pure silicon companies don’t have yet, but they will get those because it’s going to become a limiting factor.

Davis: With 3D, every chip company is becoming a system company.

Yeager: Absolutely. There’s another interesting dynamic, as well. As people try to get to the market quicker with a new algorithm and a high-level strategy, they tend not to do as good of a job with optimizations at a lower level. So there are going to be some tradeoffs. You’re going to see differentiation in the market where some people will do much better than others depending on how well they do with those efficiency statements, top-to-bottom, from architecture all the way down.

Roessig: This goes back to where in the flow do you start planning the power. As a power guy, I’m accustomed to the standard customer request for how to power a design. Companies that start planning for that up front get to market with a better solution.

Yeager: And companies that don’t are going to get to the end and realize they don’t have a product.

Davis: The details matter.

SE: It’s getting a lot more expensive to push to the next process nodes, and the power and performance benefits go down with each new node. If you pack all of this into a 2.5D or 3D-IC package, how much of an impact will that have on power?

Roessig: Right off the bat, where the power gets converted makes a big difference. If I am inside the package and converting from 3.3 volts at 1,000 amps, and I’m dropping down to 1.6 volts at 2,000 amps, I just saved 1,000 amps by laterally moving across the board and into the chip. Where that power conversion happens relative to the PDN and the bottlenecks is going to be a huge deal. There are a couple companies that can do that on their own chips, but that’s harder for the broad market. But as you get into 2.5D and 3D, that sort of thinking becomes a lot easier.

Davis: [AMD President and CEO] Lisa Su, in a keynote last year, talked about AMD’s new GPUs — 4X improvements in performance and 2.2X for efficiency. They attributed more than half of it to the fact that they went to 2.5D packaging, putting multiple chiplets together and reducing the cost of the communication between components. If you reduce that cost, you get the performance. But if you notice, the efficiency wasn’t 4X. It was performance, because that’s what wins you the socket.

Faisal: 2.5D does bring a lot of efficiencies, but it brings a lot of complexity, as well. If you look at who has chiplet-based designs, it’s just a few companies in the world. What about the rest of us? With AI, we’re going to need thousands of chips. And if they’re chiplet-based, that’s better for us. However, the complexity that’s required to build a 2.5D or 3D-IC system puts a lot of pressure on the tools vendors, as well as creating new opportunities. 3D-ICs is a lot of fun, but it will require heat removal. From a really over-simplified description of modern chips, you pump power into it, you put pump information into it, you get heat out, and you get some new information out. That’s what a chip does. So 2.5D is awesome, and 3D-IC is more awesome, but heat removal will be a big challenge. There is a lot of innovation there. More and more granular management will be required. What Trey said about putting a chip in the package and converting voltage in the package — but also on the chip because sometimes you have to control the voltages and power more granularly — is important. But the only way to do that is if we have access to the on-chip capabilities, as well. That’s even more important when it comes to chiplet-based systems and 3D-ICs.

Davis: Being able to sense things on the chip can have some advantages, too. Thermal affects performance, which affects leakage. From a EDA perspective, now you’ve got not just one corner that affects the whole die, but different parts of the die in different corners. And dynamically, where do you put that chip in your system? Is it next to something hot? That’s not something I model. But if you have a sensor, you can accommodate it. So having that dynamic ability in your design can help you manage things instead of adding more margin, because the alternative is always adding more margin.

Yeager: For true 3D, there are going to be some costs for that. When you look at what’s driving it, it’s that modern nodes are really expensive. So you want to put all your compute and everything on the expensive die, and attach that to a less-expensive die that will just have RAM or power delivery, or something like that. But when you use TSVs and go through silicon, there’s a penalty to pay. There’s not just IR losses and inductance and droop. You’ve got a ton of design rules and TSV densities to deal with. How are you going to fit SRAMs in there? There is low current density around that so you don’t blow out the thermals. It’s a really complicated problem with a lot of permutation options, and there are no tools to help people figure that out at the architectural phase today. People are literally having to go do swags, design them, build hacks of SPICE models for this stuff to try to figure out what might work. But we’re not able to do justice to any of those options with practical amounts of resources to really optimize them.

Davis: The 3D markets are evolving so quickly with the foundries and the OSATs putting things together, but where are the models? We’re a couple years away from that, but we haven’t gotten there yet. That makes it really difficult. There’s also different mechanical stresses with different technologies and materials and TSVs. And now you’ve got different technologies different put together here, and we’ve thinned them out so much that you actually have to care about that now.

Yeager: You can have two companies with kind of the same strategy and they do very different physical implementations and architectures. One will work, the other will be DOA. They’ll get to the end, do the extraction, and figure out this is crazy and it’s not going to work, so don’t even bother. It’s too late and they’ve spent all the NRE. That’s where the big challenge is going to be.