Each standard serves a specific use case, so chip architects are choosing more than one for a single design.
Key Takeaways:
As chips become more complex and packaging options multiply, designers have more choices than ever for connecting system components.
Fast, efficient data flow is essential, particularly in AI systems where the volume of data that needs to be moved between processors and memories is enormous. Data needs to move fast enough to feed the processors without delay, which requires high bandwidth and very low latency. Choosing the wrong interconnects or interconnect architectures can lead to memory bottlenecks, thermal hotspots, and/or signal degradation. Engineers need to weigh options around interface protocols and standards, physical input/output (I/O) buffers, and interconnect wiring and pathways — both for on-chip and off-chip networks.
“The broader trend is that different protocols are being used at different physical ranges, from chiplet and package-level connectivity to rack-level systems,” said Satish Radhakrishnan, head of go-to-market for semiconductors and electronics at Vinci. “That makes interconnect selection less of a pure protocol decision and more of a system-level implementation decision.”
Interface domains and standards include:
While interface terms such as I/O and interconnect are sometimes used interchangeably, there are distinct differences.
“Interconnect is what connects to chips at the physical layer,” said Priyank Shukla, director of product management for interface IP at Synopsys. “When people talk about what interconnect they’re using, PCIe or UALink could be an interconnect technology. But when you say I/O, the typical understanding is that it’s low-speed and general-purpose. You’re not focusing on a controller or anything else, just the physical input output. Meanwhile, I/O chiplets take data in or out from the core die.”
With all the interconnect options and specifications, it can be confusing for designers to pick the correct one.
“As IP providers, we’ve had to explain where each of these protocols fits in,” said Mick Posner, senior product marketing group director for chiplets and IP solutions at Cadence. “We often get the question, ‘Which one should I choose?’ We have to say, ‘Please explain what you’re trying to do,’ because there is a bit of overlapping.”
Each interconnect is tailored to a specific purpose, so one cannot replace all. “Why are you going to sacrifice performance, power, area, and latency for your use case to support a generic interface that’s really not going to differentiate you?” Posner added. “These vendors don’t want a generic interface.”
Rather than winner-take-all, the industry is converging on a layered approach. “You can standardize the building blocks and still have different fabrics for on-package communication, in-rack expansion, and rack-scale connectivity, because the constraints are different at each tier,” said Arif Khan, vice president of product management and marketing for the Silicon Solutions Group at Cadence. “The practical outcome for designers is more choice, but also clearer roles — one set of links for broad compatibility, another for memory semantics and pooling, and others for ultra-low-latency scale-up domains. The systems that win will be the ones that blend these cleanly.”
More options bring more challenges. “The proliferation itself is part of the problem,” said Ashish Darbari, CEO of Axiomise. “Five years ago, you picked an interconnect, picked a package, and got on with it. Recently, I was talking to an architect and learned that these days it is common for architects to simultaneously evaluate UCIe 2.0 for the compute-to-memory link, BoW for cost-sensitive I/O, EMIB-T for the high-bandwidth bridge, CHI on the host side, and a non-coherent NoC for the streaming accelerators — all in the same design. There is a reason for each of them, but the challenge is that no one entity owns everything, and you’re not integrating one protocol. You’re stitching a stack of them. The consequence of this is that bugs will live between the seams of this stack, not necessarily in any one of them.”
A lot of interconnect innovation is focused on achieving amazing performance for a single AI workload. “We’re seeing networks evolve, and all these technologies, I/Os, links, and connectivity protocols evolve along with that,” said Saurabh Gayen, chief solutions architect at Baya Systems. “It’s very fast-moving and a lot of things are emerging and fighting for dominance.”
Choosing the right interfaces is about the engineers’ comfort level, along with product definition. “There might even be hedging of bets. Customers say, ‘We need something that’s able to support both, because we don’t know what’s going to win,’” said Gayen. “You have to make your bets now, and a year from now is when your product comes out. If you pick the wrong horse, you’re done. There’s a lot of bottom-up technical assessment, but that’s not the only thing that goes into it. We’ve seen technology win out that might not be technically the best, but it had the momentum behind it. It had the ecosystem coalesce around it. All these top-down and bottom-up considerations are what we’re seeing from our customers.”
In fact, there may be too many choices. “The real challenge is that the use cases overlap just enough to create confusion, while the implementation costs do not overlap at all,” said Lou Ternullo, senior director of product management for silicon IP at Rambus. “If the design needs ubiquitous compatibility and predictable bring-up, PCIe remains the default choice. If the bottleneck is memory capacity and utilization, CXL becomes compelling because it changes what the fabric can do, not just how fast it moves bits. If the system is a tightly coupled accelerator pod chasing peak bandwidth at very low latency, that is where specialized scale-up interconnects tend to earn their keep. Most modern systems end up mixing these, because one link rarely checks every box.”
In a crowded field, not all standards can hold equal ground. “UCIe, HBI, and high-speed SerDes fabrics are rising in popularity due to disaggregation economics,” said William Wang, CEO of ChipAgents. “Proprietary die-to-die interfaces are declining as standardization accelerates ecosystem interoperability.”
Andy Nightingale, vice president of product management and marketing at Arteris, agreed that bespoke, one-off, die-to-die, and ad-hoc proprietary interfaces are losing favor because they don’t compose well across vendors or product generations. “Every custom link is a permanent tax on validation, bring-up, and supply-chain optionality,” he said.
The overall trend is for short-reach, high-bandwidth interconnects through die-to-die and 2.5D interposers, or 3D die stacking, said Hee Soo Lee, high-speed digital design segment lead at Keysight EDA. “They deliver extremely wide buses at higher speeds.”
UCIe for chiplets and 3D
Today’s market is driven by chiplets and multi-die architectures, giving designers lower power and high bandwidth densities. A chiplet is an electrical die that is designed to do one function. Among emerging interconnects, there is an industry effort to reduce vendor lock-in through standardized chiplet interfaces such as UCIe or BoW.
“Standardized die-to-die and memory-centric interconnects are winning because they reduce ecosystem friction and verification burden,” said Arteris’ Nightingale. “UCIe is explicitly aimed at multi-vendor chiplet interoperability with a defined stack and compliance testing, which is the kind of boring standardization engineers secretly love.”
However, UCIe is yet to have the same dominance in the chiplet space as PCIe has in the board-to-board component interconnect space.
“Chiplet-based design, fundamentally, should be exactly the same as chip-to-chip, except now it’s die-to-die,” said Cadence’s Posner. “So why in the world is it more complex when you take these dies and package them in the same area versus going across chip-to-chip? It really comes down to the die-to-die interfaces. When you’re going from chip to chip, protocols like PCIe make a huge amount of sense because you have a dedicated protocol. Both sides understand it. And we’re in this interim area where multi-die is exploding, but there isn’t a PCIe of die-to-die. Don’t get me wrong, UCIe has come in and is trying to provide that gateway and make a standard that everyone can utilize. But when you’re looking at an application such as the data center, what are you doing? Is this a CPU-to-CPU connection? Is it GPU-to-CPU? Is it GPU-to-CPU to memory?”
Despite the complications, chiplets are here to stay. “One of the benefits of chiplets is a mix and match,” said Keysight’s Lee. “Instead of having to get everything from integrated IDMs, you can include off-the-shelf components. You can get a chiplet, plug it into your system, and make the entire system work. That speeds up ecosystem adoption.”
Fig. 1: Example of 3.5D packaging with a 3D die stack connected to another 2D die with an interposer and UCIe interface. Source: Synopsys
PCIe and CXL
Many AI applications leverage traditional CXL PCIe links to enable disaggregated compute, so systems can dynamically share memory, storage, and accelerators across multiple compute nodes, according to Rambus’ Ternullo.
PCIe is widespread outside of the AI space, as well. “PCIe and Ethernet still dominate applications such as mainstream data centers, enterprise IT, and personal computing,” said Kent Orthner, principal solutions architect at Baya Systems. “They’re getting faster with a lot of these external protocols. Memories have become faster and faster in recent years, using the HBM in-package memory solution, and that’s moving forward to give super-fast, low-latency access to external memory.”
CXL will always live in a particular use case. “CXL was designed for CPU-to-memory,” said Cadence’s Posner. “It enables memory sharing and memory pooling. No other protocol does that.”
Fig. 2: Examples of HPC chiplets using UCIe IP for die-to-die connectivity and PCIe 7 and CXL to connect CPU to memory. Source: Synopsys
Arteris’ Nightingale agreed. “CXL continues to pull more of the memory and accelerator attach story toward coherent fabrics and pooled or shared memory models — handy when AI workloads are dominated by data movement and memory bandwidth, not just raw MACs,” he noted.
The AI data center/HPC race
The main competitors to Ethernet for high-speed GPU scale-up inside the rack are NVLink and UALink.
“NVLink and UALink are getting more traction as the amount of compute and memory within each GPU or accelerator increases and more data needs to be transferred between GPUs,” said Vinci’s Radhakrishnan. “NVLink is the primary protocol for NVIDIA GPU-based systems, while UALink is gaining attention as an open standard for accelerator-to-accelerator communication. Until photonics-based CPO matures, NVLink and UALink are two of the main protocols AI chipmakers are relying on to connect multiple GPUs in parallel and move data efficiently between them.”
Even with these definitions, it’s not always clear which one engineers should choose. “We often get asked, ‘Should I be using NVLink Fusion, or should I go with UALink?’” said Cadence’s Posner. “NVLink Fusion is the public version of Nvidia’s NVLink. They’ve extended it because they want people to hook up into their environment, so that’s the CPU-to-GPU connection. UALink is spearheaded by AMD, based on their Infinity fabric, looking for GPU-to-CPU connectivity and GPU-to-GPU, but within their ecosystem. Again, it’s public.”
PCIe used to be the CPU-to-accelerator interface of choice. “It now competes with NVLink Fusion and UALink because they serve the same purpose, but the latter ones likely have better metrics and are more tailored to that use case,” said Posner. “However, PCIe is never going to go away. Intel uses it in its own architecture. We’re just going to have to live with all these different protocols inside the die and outside the die.”
Baya’s Gayen agreed that multiple standards are here to stay. “There’s an interesting tension in the industry today where hyperscalers are much more comfortable with Ethernet-based technologies. Because of that, Ultra Ethernet and the Open Compute Project’s ESUN (Ethernet for Scale-Up Networking) initiatives have started up. They say, ‘We’re happy with Ethernet. We can keep going with that with just a few tweaks.’ Then everybody is comfortable knowing these are the same lower-level stacks and infrastructure that they’re familiar with. If your customers are happy with the technology, then that’s a huge tailwind. UALink has more fundamental bones, built ground up for exactly what AI needs, whereas Ethernet has repeatedly won. This has happened in the past, as well, where Ethernet has evolved to meet needs, and so it’s unclear what’s going to win.”
With scale-up and scale-out, and the plethora of protocols available, the challenge is deployment. “Systems need to talk to each other, so there needs to be enough systems that speak the same protocol,” said Cadence’s Khan. “Then, for these scale-up protocols to work, you need the switching infrastructure in between, so who’s developing the switches for which one? That results in fragmentation. In some cases, SoC designers are having to put multiple solutions on their SoCs because they don’t know what the market ecosystem will be like when systems are actually deployed.”
ESUN-based switches already have the same Ethernet switch infrastructure. “They just have to update the protocol a bit, and then you have a new switch available, whereas if you’re building something for a totally new protocol, it’s a lot more work,” said Khan. “There’s some economics in terms of whether you want the change to be revolutionary or incremental. New protocols have both system implications and software programming implications. It is always challenging for new software to take advantage of the promised benefits of a new protocol.”
Fig. 3: PCIe 7.0 switch IP with time division multiplexing. Source: Rambus
Optical, CPO, and high-speed SerDes for the future
Many AI data center interconnects are expected to be optical within about five years [1] and TrendForce predicts the global shipment volume of optical transceivers will triple from 26.5 million units in 2023 to more than 92 million units by 2026. [2]
“We are hearing more about optical interconnects, especially moving on to CPO (co-packaged optics),” said Keysight’s Lee. “This is changing the electrical interconnect to optical to overcome power issues. With AI chips, where a lot of ASICs are needed, the system configurations pull a lot of power, which is one of the main issues in AI data centers. Using CPO or silicon photonics will allow us to go lower power. You don’t have to use ASICs in the system. Distance-wise, since you don’t use copper, it saves you a lot of signal loss; therefore, the system is going to be much more effective, and also reduce thermal limits to the system.”
Optical technologies are gaining traction because electrical links face a fundamental limit for how much bandwidth can be delivered through the limited chip-edge area. However, optical is still a niche option rather than default plumbing right now. “Optical I/O is creeping from ‘research darling’ toward selective deployment as packages get bandwidth-hungry and power-limited,” said Arteris’ Nightingale.
Conclusion: no clear winner
Every interconnect protocol has benefits and drawbacks, and there is no short answer for which I/O and interconnect protocol is the best for a given application.
“Standardization is great if it works for everyone, but where standardization has an issue is that you have a lot of legacy infrastructure,” said Cadence’s Khan. “In a data center, if somebody comes up with a new standard — let’s pick UALink as an example — for it to be successfully deployed, you now have to introduce all of these UALink switches for scale-up, and so on. It’s a hefty lift in terms of expense. We saw that with CXL. The reason CXL adoption has been slow is that all the infrastructure required to actually deploy it is not in place. So designers ask, ‘Can I achieve the same thing with a tradeoff on some amount of performance or latency with an existing solution?’”
The number of solutions to solve related problems is comparable to the EM (electromigration) technologies in the market 10-plus years ago. “There were so many different technologies, including FEM (finite element method), FTTD (finite difference time domain), MoM (method of moments), and far more,” said Keysight’s Hee-Soo. “The question was, why isn’t there a single algorithm that solves all the different types of problems? They are all solving one thing, which is Maxwell’s equation. But every technology has pros and cons. One may be really good at solving narrow-band problems, while the other is good at solving wide-band design problems. So, I don’t think there will be only one interconnect standard that governs all different types of interfaces.”
However, there are some clear losers. “In-memory systems, a pair of buses, single-ended, is very sensitive to noise issues,” said Lee. “It also has many yield timing-related issues, and those are also very, very difficult. We are also hearing less about long-reach copper because it’s very expensive with higher loss.”
References
[1] All AI Data Center Interconnects Will Be Optical Within 5 Years (Semiconductor Engineering)
[2] AI Optical Interconnect Boom Drives U.S. Firms to Expand Southeast Asia Outsourcing, Opening the Door for Cross-Industry Entrants, Says TrendForce
Related Articles
Scale Up, Scale Out Get A New Partner
For reaching farther into another data center, developers are now talking about scale-across.
New Data Center Protocols Tackle AI
UALink scales up, while Ultra Ethernet scales out.
Optimizing Data Movement
Problems and solutions for improving performance with more data.
Multiple AI Scale-Up Options Emerge
As data center infrastructures adapt to evolving workloads, parts of Ethernet can be found in scale-up approaches.
NoC Coherency Challenges Balloon With AI SoCs And Chiplets
Complex chips need coherent and non-coherent sub-NoCs to ensure efficient data paths. Correct hierarchy is essential.
Data Boom Puts Pressure On NoCs, Fabrics
New adaptive, mesh NoC topologies are enabling chip designers to optimize data movement in complex SoCs and multi-die systems.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply