Technical, political, and supply chain challenges are complicating data center buildouts around the U.S.
Key Takeaways:
The bid for more U.S. data centers to meet the insatiable demand for compute power brought on by the widespread use of AI has the full backing of tech giants, but there’s no shortage of obstacles for the industry.
Hyperscalers are looking to satisfy that demand with state-of-the-art server farms, but they are facing mounting political and public opposition to new construction due to their anticipated effects on regional power grids, water supplies, and other environmental impacts. As the industry speeds toward a future defined by AI, what the infrastructure supporting that future will look like, who will build it, and when exactly it will get here remain to be seen.
The demands of AI create bottlenecks
The surge in demand for data center capacity has led to four major supply chain choke points for data center buildouts, all of which have intensified over the past year:
“A year ago, TSMC was clearly the near monopoly for AI accelerators,” said Geoff Tate, a technology consultant. “But a year ago their statements on supply/demand were more that they were able to keep up, versus recent comments of not being able to meet demand. Memory a year ago was not on most people’s roadmap as a bottleneck. This has changed dramatically over the last year. Power was a clear issue a year ago. Lasers were not on most people’s roadmap as a bottleneck, as the transition to optics was unclear. This changed dramatically in the last six months.”
Any one of these issues can have a big impact on the speed at which AI capacity can increase. “These bottlenecks interact in that the worst one will gate demand for all the others,” Tate said. “For AI to grow, they need all the bottleneck technologies to grow. You can’t run a data center with only three out of four.”
Those constraints are especially evident in the power equation, where the demand for more AI capacity is colliding with the limits of existing utility infrastructure.
Energy supply challenges
Legacy power management solutions cannot meet the demands of this AI moment, so data center operators are seeking designs that deliver better token-per-watt performance, according to Davood Yazdani, executive vice president and chief product officer at Endura Technologies.
“At the end of the day, what’s important in a data center is the token per watt that you are generating,” Yazdani said. “What are the ways to do it? One way is increasing the efficiency. Increasing the efficiency means you have more power. That means, because you have more power, you can generate more tokens. That’s one way. Number two, you can have lower efficiency, but if you run the last stage, the regulation stage, at a higher frequency, and you can respond faster, you can make your processor actually work in the most efficient way, and give you the tokens that it needs. That equation is much more powerful.”
The development of integrated voltage regulators will be a key technology development to monitor for hyperscalers seeking power efficiency, Yazdani said.
“The consumption of these data centers is going at a completely different rate than the utility grid can support, so you will always have a cap, and you want to maximize the throughput that you have based on the budget that you’ve been given,” Yazdani said. “That’s why everyone is invested in IVR these days, and they’re excited about the benefit that it brings, but of course it has its own challenges. It’s a new technology to the market. It takes time for them to adopt it. But at the end of the day, in any data center, you have a cap.”
Power supply demand brushes against some of the most persistent drivers of public opposition to data centers, with hyperscaler buildouts linked to higher utility bills for ratepayers.
“Local politics is the most important non-technical challenge,” Tate said. “In many parts of the United States, voters and politicians are taking measures to constrain data center growth because of their perceived impact on high electricity prices. Smart operators avoid using the grid and generate power off-grid if they can.”
With that political climate in mind, nuclear energy powered by small reactors is suddenly a viable option again.
“The nuclear portion of it is going to be an option that many of them are considering when it comes to the power generation, because the normal expansion of the grid is not going to happen,” Yazdani said. “All the data center guys are looking at these microreactors/small nuclear power plants.”
Microreactors aren’t being used for data center power just yet, and not every nuclear-powered data center will draw power from one at all.
“They aren’t being used yet because the new technology is moving through licensing and demonstration,” according to James Walker, CEO at NANO Nuclear Energy. “People sometimes get ahead of themselves assuming every data center is eventually going to have its own microreactor, but I don’t see it that way. Some facilities will always make more sense connected to the grid, especially in areas where there’s already plenty of generation and transmission capacity. But there are also places where developers are struggling to get enough electricity to build new campuses, or they’re being told they’ll have to wait years before the grid can support them, and that’s where advanced reactors become much more interesting.”
Nuclear power is also gaining interest with data center operators because of its technical reliability.
“The amount of electricity being requested already, and the projected power required, is on a completely different scale, and utilities are trying to figure out how to serve that demand without compromising the rest of the grid,” Walker said. “If you’re running a facility around the clock, you also need generation that doesn’t depend on the weather or the time of day, can provide baseload power, has location deployment flexibility, and has predictable low downtime annually. That’s why nuclear is showing up in so many conversations. Existing plants can provide that dependable electricity today, and advanced reactors could give operators even more flexibility in where they build and how they secure power. I don’t think it’s a coincidence that we’re seeing companies that weren’t talking about nuclear five years ago now signing long-term agreements with nuclear generators.”
Public perception
That assumes the hyperscalers have their way, however, because 7 in 10 Americans say they would oppose an AI data center opening in their community, according to a recent Gallup poll. Also, lawmakers in both red and blue states have come out in favor of new restrictions on hyperscalers.
“This is no longer like a conversation among policy wonks,” according to Morgan Scarboro, vice president at policy analysis firm MultiState. “People sit at their dinner tables and talk about AI. They talk about data centers. It is fully in the public consciousness now. When you read a quote about data centers, you don’t know if it’s Steve Bannon or Bernie Sanders speaking, which sums up the whole thing for me. We track thousands of issues here, and I can’t think of a single other issue that is this much of a circle in terms of the legislation that’s going on. When you have legislation enacted in Florida that’s similar to what they’re trying to do in New York, it’s shocking.”
New York paused some environmental permits for new hyperscale data centers for up to a year, but outright construction stoppages are still being debated. Scarboro identified three common types of legislation pushed by state lawmakers over the last year:
Still, government officials are balancing constituents’ data center anxieties with the economic opportunities they offer.
“I would not be jealous of a policymaker who’s in the position of needing to attract economic investment in their state while also hearing a lot from constituents about how they don’t want data centers,” Scarboro said, noting that investment opportunities in other industries are extremely limited.
The public’s distrust of data center operators has complicated the industry quickly, but it has made operating responsibly a key part of making a buildout successful, according to Kyle Illgen, head of product, compute, at Iceotope. “Data center management went from being a niche area of a largely quiet compute market to one of the most complicated in less than 24 months. As if that wasn’t enough, communities and regulators are rightly scrutinizing new developments much more closely after community disasters like Colossus in Memphis. Questions around electricity demand, water consumption, noise, and environmental impact have become central to planning decisions, with billions of dollars in projects delayed or canceled because of local opposition. That means sustainability is no longer just a corporate objective—it has become a prerequisite for expansion.”
Developing and implementing capable thermal management technology will be another important challenge for the industry amid outsider concerns over environmental and grid impacts.
“Thermal management is directly connected to all the protests about power usage, water consumption,” Illgen said. “Every major challenge facing AI infrastructure today—performance, operating cost, sustainability, water use, community acceptance, deployment flexibility, and future scalability—connects back to thermal management. And with every new generation of hardware, the thermal challenge is growing.”
Data center specialization
Just as the increasing complexity of AI processes has led to supply chain bottlenecks, it also has resulted in major operators leaning into specialization in their data center buildouts. That specialization has led to an increasingly fragmented semiconductor IP ecosystem, according to Manmeet Walia, director of product management for mixed-signal PHY IP at Synopsys.
“The CSPs and the hyperscalers are becoming big enough that they can live in their own world, and their world is big enough to support them,” Walia said. “What they desire is very custom solutions. As a result, their type of compute and AI needs are different. Their rack structures are different, which then dictates what type and how much memory they’re going to need, and what series ranges, speeds, and performance metrics they need. They’re disaggregating dies, so all of these solutions are spawning IPs and getting very fragmented.”
At the same time, this fragmentation is being tempered by supply chain issues. DRAM shortages are putting a crimp on how systems are designed.
“There are some consistent drivers behind what all of these guys are doing, and one of them is HBM,” Dan Wilkinson, distinguished engineer at Synopsys, said. “As each new generation comes and the speeds go up, obviously they’re tracking that, and while their workloads may be different, ultimately, they’re using the same memory ecosystem. So obviously, those speeds go up, they need to operate their compute to keep up, and so on and so forth, and they need to stitch these things together with more bandwidth. I would say memory considerations are a big driver across the board.”
Rather than going with traditional architectures, hyperscalers are looking for different workarounds. In the past few years, many of these approaches relied on standard elements, such as Nvidia GPUs and CUDA, to bring solutions to market faster, supplemented with specialized accelerators that were designed internally. But with much more compute horsepower, the cost of moving and storing data has become a much bigger challenge than in the past. The result is a push toward more customized solutions that will likely include GPUs, but which also will include other elements, as well, particularly with the rollout of agentic AI.
“Choice number one is that they can use Nvidia, which gives them the highest performance, the highest raw performance, and it gives them time to market and that peace of mind to get their things done on time,” Walia said. “Their other option is that they either go down the ASIC path, or they go and build their own chips with the COT path. The benefits they have there are lower cost and control of their destiny, and optimizing these chips for their workloads and their racks. That’s where Synopsys would come in with IP, and work with either the aggregators or ASIC vendors and provide solutions. We work with all the major hyperscalers directly on their major SoCs. These are purpose-built IPs, specific for these types of SoCs.”
If enough major players go down the custom ASIC route, it will represent a troubling trend for Nvidia’s position at the top of the marketplace, according to Yazdani.
“Nvidia is number one when it comes to the GPUs, but it’s not going to stay that way because all the hyperscalers have ambitions to control their own destiny. They want to have a solution that is more optimized for the application, and that’s definitely a threat to Nvidia,” Yazdani said. “The GPUs they have are excellent for the type of multiplication they are doing in their system. It works perfectly for certain models, but it’s not necessarily optimized for the workloads that Meta or Google or AWS have. That’s why all of them started developing their own silicon. They are still relying on Nvidia, but over time they will have their own products. If you look at Google, for example, they have their own TPUs. Their volume is growing significantly, and over time, is Nvidia going to keep its market share of more than 80% to 85%, or is that going to reduce?”
Still, Nvidia isn’t going away. “Nvidia is leading this race to get to a 1 megawatt rack between now and 2029 or 2030,” Walia said. “That’s the end goal, and the way that goal is going to be achieved is with higher and higher layers of integration, which means that not only are we doing multi-die solutions, but these multi-die solutions are going to be done in 3D, 3.5D, 4D, and so on. It’s an extra degree of freedom that is going to give us the new options and the new level of complexity.”
Related Articles
AI Data Centers And Auto Industry Converge On Same Issues
The EV revolution relies on battery innovation, while AI data centers need a range of new energy solutions to play nice with the grid. Both sectors are taking notes.
AI Is Rewriting The IP Playbook
As the semiconductor ecosystem pivots to AI, it is transforming how IP is created, verified, managed, and sold.
Liquid Cooling Gains Traction In Data Centers
There are numerous ways to remove heat from chips, and more are on the way.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply