Scaling NoCs across chiplets requires earlier validation of coherency, congestion, thermal effects, and fault behavior.
Key Takeaways:
The speed gains from leading-edge AI accelerators and multi-die architectures can be lost if on-chip networks are not optimized to handle diverse types of data traffic moving between memories and processors.
The challenges are numerous. They include everything from determining which traffic needs to be coherent, ensuring there is sufficient margin to reroute signals as data paths age unevenly under different workloads, and ensuring that the right interconnect protocols are deployed in the right locations. In addition, engineering teams need to decide how thermal, noise, and structural integrity can impact the movement of that data.
Networks-on-chip (NoCs) play a pivotal role here. They have evolved from relatively simple on-chip network layouts to a complex, hierarchical orchestration layer that manages and maintains the flow of data throughout a heterogeneous system over its expected lifetime.
“A network on chip was designed to address wire congestion layout, but its new challenge is how to facilitate that across-die connection, ignoring the physical connection,” said Mick Posner, senior product marketing group director for chiplets and IP solutions at Cadence. “What’s the protocol to the system? Does that look like a single network-on-chip? Are there two, so they have their own memory space? Those are the new boundaries that need to be addressed.”
Traditionally, NoCs didn’t care what was going on outside the chip, because that was a link across a standard protocol such as PCIe. “Now that it’s die-to-die, you really want that network-on-chip to extend transparently across the die, and you’ve got a whole new area of innovation to address that,” said Posner. “For example, in the space of Nvidia GPUs, Arm wants to be the CPU next to it. How do you solve that? They’re looking at die-to-die interfaces utilizing AMBA CHI C2C (chiplet-to-chiplet), and that is a cache-coherent interface which solves their problem.”
Verifying coherency
The mix of coherent and non-coherent traffic is particularly challenging.
“A coherent request reordered behind a non-coherent stream, or a credit returned on the wrong virtual channel, is not going to show up as a functional failure in simulation,” said Ashish Darbari, CEO of Axiomise. “It will show up as a deadlock at bring-up, months later, when the cost of finding it is enormous. NoC architects understand this on-die. The difficulty now is that with chiplets, the NoC boundary has moved to the package edge, where timing, retry semantics, and error handling are fundamentally different.”
Compared to I/O coherency, cache coherency is more difficult to implement. “It’s also heavier to verify, because you have many more issues that are hard to find,” said Frank Schirrmeister, executive director of strategic programs for system solutions at Synopsys. “Defining all the cases is completely non-trivial. You must be really careful about how you build the environment around that.”
Massive multi-physics simulations, enabled by almost unlimited compute power, have made this more tenable over the past few years, but with some caveats. “The NoC tool has all the inputs, masses, initiators, and targets,” explained Schirrmeister. “These are the connections. You look at it functionally, you simulate it, and then verification tools help figure out what the actual timing is. That’s the physically aware bit that comes from the NoC provider, and it is where we see one of the significant challenges — the functional verification of coherency aspects. If you look at the differences between cache coherency and I/O coherency, it took the smartest brains, from an architecture perspective, to articulate the test cases.”
Much of this has pushed farther left in the design flow in recent years. “Instead of having the user go in and say, ‘Here’s a test case,’ which is hard to create manually, you say, ‘Write something into this memory, figure out how much is in the cache, move it over to the other side, and now have an access coming from the software running on the second core to the same memory region,'” Schirrmeister said. “‘Then, determine whether the representation within the NoC knows where the last cache element is, and who has the last cache element.’ It’s a very intricate process.”
This system for integrating different processors and automatically generating tests based on constraints is roughly in the same vein as Portable Stimulus, although it’s not based on the PSS standard and focuses on cache coherency.
AI forces I/O considerations
Regardless of whether a NoC is cache-coherent CPU-to-CPU, or non-coherent I/O only, it must be considered in tandem with I/Os. “The more AI you pack in, the more you’re forced to treat the I/O perimeter as a first-class system resource. Bandwidth density, latency determinism, congestion control, quality of service, and error containment become architectural constraints, not post-layout problems,” said Andy Nightingale, vice president of product management and marketing at Arteris. “That’s where a scalable on-chip fabric and a system-level interconnect approach earn their keep. They make connectivity predictable across many endpoints, while giving engineers knobs for topology, buffering, ordering, and isolation so more I/O doesn’t automatically mean more chaos.”
Fig. 1: Example of a setup for Arm cache coherency validation. Source: Arteris
The problem space that needs to be addressed is huge. It involves huge numbers of lines, or physical wires, all of which need to be captured and characterized in simulations of NoCs and I/Os.
“In the grand scheme of things, interconnects are not grossly difficult designs, but they’re big,” said Matt Commens, senior director of product management at Synopsys. “The computer-aided design is not really difficult, as it is typically on the larger catch meter mark and larger node size, but the number of lines is off the charts. What we’re seeing is capacity, capacity, capacity. People want to simulate the entire interconnect of chiplets to study as many lines as possible, if not all. We’re hearing directives from companies where every line has to be simulated. It can be done, but they tend to be pretty big simulations.”
A single die or NoC can be simulated with one power map and clean boundary conditions. But that has to be set in the context of what else is happening in a chip or system.
“A system of multiple dies and chiplets has superimposed, spatially-distributed power maps that thermally couple,” said Satish Radhakrishnan, head of GTM for semiconductor and electronics at Vinci. “Thermal activity in one chip raises its neighbors’ baseline, and the global hotspot may not coincide with any single chip’s local maximum. You cannot simply add up independent single-chip results. Therefore, capturing crosstalk and the effects of neighboring chips and stacks is essential. The boundary condition and cooling solution must also be evaluated as a whole system to capture the full heat transfer path.”
This is where traditional simulations run out of steam. “The number of scenarios that matter at the system level — across different power states, stack configurations, and boundary conditions — makes sequential, specialist-run solves impractical as a design workflow,” Radhakrishnan explained. “The ability to run physics continuously across the full chip-to-system stack, with deterministic and reproducible results, changes what is actually feasible to evaluate before architecture is locked.”
Simulating NoCs is a multi-physics challenge
NoCs typically have distributed heat sources, and that heat is transient in nature, dependent on workloads, chip and system architectures for dissipating that heat, and whether various cooling options are deployed. But it also is dependent on neighboring dies in a package, or chips in a rack, which can cause crosstalk, thermal or substrate noise, and electromagnetic interference, as well as mechanical stress that can result in warpage or turn latent defects into real defects.
“You cannot isolate one node and expect the result to hold,” Radhakrishnan said. “Warpage, low-K dielectric cracking, and delamination are also concerns driven by the system’s transient thermal loading. The deeper challenge is that an accurate but slow solver can answer one physics question, but the NoC problem involves a large, uncertain design space of transient, coupled, multi-scale scenarios. Running enough of those cases to understand the design space — before committing to an architecture — requires a fundamentally different throughput model.”
And since simulation is done at both the transaction level (SystemC) and RTL, it affects the ability to measure exact cycle-accurate bottlenecks. “SystemC or transaction-level models (TLMs) are useful for early architectural exploration and high-level performance tradeoffs,” said Axiomise’s Darbari. “But RTL-level formal verification is required to prove cycle-accurate microarchitectural properties, such as exact credit behavior, arbitration delays, CDC (clock-domain crossing) safety, and packet progress. Simulation-side VIP is used for performance and directed testing, while formal must be used to handle exhaustive corner-case closure. This matters because many real NoC bugs arise from single-cycle races and implementation details that are invisible in transaction-level models. A balanced methodology uses TLM for architecture studies and RTL formal plus RTL simulation for implementation signoff and bottleneck diagnosis.”
Formal verification
Formal verification also can play a key role in finding bugs in the state space for cross-protocol corner cases. “For example, UCIe link states, combined with CHI snoop transactions, combined with NoC arbitration is not something you can simulate your way through,” said Axiomise’s Darbari. “Formal verification can prove credit conservation, deadlock freedom, ordering properties, protocol compliance at every interface. We’ve been shifting formal left of simulation for exactly this reason. By the time a multi-die system is in emulation, a coherence or deadlock bug costs you a tape-out slip.”
Advanced packaging options also bring challenges. “EMIB (Intel’s embedded multi-die interconnect bridge, CoWoS (TSMC’s chip-on-wafer-on-substrate), and hybrid bonding impose real constraints on bump pitch, redistribution, and thermal envelope,” said Darbari. “Teams used to treat packaging as something you handed off downstream. That doesn’t work anymore. The choice between 2.5D and 3D affects the floorplan, the power delivery network, and this has serious implications for validation and verification. It has to be in the conversation from day one.”
Further, when verifying NoCs, designers must ensure the chosen topology correctly connects all nodes and that packets follow the intended shortest or adaptive paths.
“Formal verification can be used to prove structural connectivity and reachability for every legal source-destination pair in a mesh, torus, or custom topology, rather than sampling selected routes in simulation,” said Darbari. “It can be done easily via the connectivity checking app exhaustively. For deterministic shortest-path routing, SystemVerilog assertions, properties, and models can be written so that each hop reduces the routing metric defined by the specification, such as Manhattan distance in a 2D mesh, while still preserving legal progress toward the destination. For adaptive routing, the routing logic can be constrained and proven to choose only among the architecturally permitted minimal or escape paths, ensuring that adaptive choices never become illegal or inconsistent with the routing policy. Symmetry-based reduction and induction on hop count are the practical techniques that make these proofs scale beyond toy topologies.”
Flow control mechanisms, such as credit-based flow control, must also be verified to ensure that buffers do not overflow under saturation. “Credit-based flow control can be verified by proving that the number of in-flight flits targeting a downstream buffer never exceeds the credit budget or physical FIFO depth,” he said. “Properties should ensure that credits are returned exactly when storage is freed, never early, never lost, and never double-counted, because misalignment in credit accounting is a common source of latent NoC bugs. These proofs can be run under saturation-focused assumptions so that even maximal legal traffic cannot cause overflow, underflow, or silent packet loss through backpressure failures.”
Network behavior must also be simulated and verified during a fault, such as a buffer error or a dead router node. “Fault scenarios such as a dead router, stuck link, or buffer error can be modeled formally as non-deterministic fault injections within a specified fault model, allowing the verification environment to explore all placements and timings of the fault within scope,” Darbari noted. “Formal is great for fault modeling, as inherently it runs unconstrained and the non-determinism comes naturally to formal — something needed for fault injection, propagation and detection.”
Testing and emulating NoCs
Chips are becoming so complex that previously discrete steps in the flow are now overlapping. Test is already complex for AI chips, but achieving full coverage is becoming more challenging because the whole ecosystem is undergoing continuous change.
“This includes assessing and fine-tuning your AI fabric to ensure that you’re minimizing GPU idle times,” said Razvan Arhip, product manager for AI and network test solutions at Keysight Technologies. “Another factor is to observe the impact of scaling traffic to hundreds of ports on your AI fabric and on your protocol deployment.”
Fig. 2: Challenges and tasks to test AI fabrics. Source: Keysight/Collective Communication Library (CCL), ECN (Explicit Congestion Notification), PFC (Priority-based Flow Control), RoCEv2 (RDMA over Converged Ethernet version 2)
Impairment testing is a vital part of the process, particularly at the data plane. “We can delay, lose, or drop packets,” said Arhip. “We can add jitter to the packets with the various combinations of patterns into the actual traffic to see how the actual protocols on top behave.”
One of the main differences between the pre-AI era and the AI-era type of traffic is the frequency of incast network bottlenecks, which occur during many-to-one communication.
“There’s a lot of incast inside the AI fabric in data centers, because after the GPUs take the data and perform operations on the data, they exchange traffic in various patterns, such as all-to-all, or reduce ring, and so on,” said Arhip. “When they start, there is a trigger that starts the traffic movement across the network, and there’s a very high probability of seeing incast, of seeing congestion at the particular entry point inside the device that’s being generated by that. We specifically want to emulate these particular moments in the network. First, we deploy the GPU clusters. Then we fire traffic at a specific moment in time. But we do not perfectly synchronize it, because there’s no such thing in the real world. We randomize things between the endpoints, so although the traffic is synchronized between them, it’s not at the nanosecond level. We have options to do that, but by default the traffic is not nanosecond-synchronized between endpoints.”
NoC reliability: data center vs. auto
Network-on-chips must operate optimally in data centers to support high-speed data movement and reduce heat generation. But in automotive applications, lives may be at stake.
“To become automotive-certified and to play in the automotive space, the methodology behind the design and verification, and your support of things like redundancy of error detection, of data protection, all of that goes much higher in automotive,” said Kent Orthner, principal solutions architect at Baya Systems. “With a lot of the big data centers that are doing AI, they anticipate that eventually, over time, chips are going to wear out. But you can’t have that in your car. If you’re doing self-driving cars, it really needs to recognize a pedestrian. You can’t have any possible failures there. A lot of the challenge when it comes to automotive is making sure that the design of the NoC has extra bits on every data path to be able to detect errors and to notify the control plane immediately. Or, in some cases, there’s redundant logic, so that you’re running the same thing multiple times in parallel and comparing it. Then, if anything ever goes wrong, you can recover within nanoseconds. As we see more self-driving and driver-assist features, you’ll see the inference engines running in cars pretty regularly.”
Overall, many automotive data flows are similar to those in data centers. “This may not be quite on the scale that you see in data centers, which are huge refrigerator-sized boxes — but from the chips’ point of view, they’re still very capable chips,” said Orthner. “It’s just that safety now becomes front and center.”
Fig. 3: Integrating various NoCs into a unified fabric that functions like an intelligent global NoC in multi-die systems. Source: Baya Systems
Conclusion
AI training and inference are only as fast as the networks transporting the data between dedicated processors and memory. Likewise, a software-defined vehicle is only as reliable as the networks that support communication between the zonal architecture.
Designers are developing a range of new solutions to verify, validate, simulate, and test network-on-chip designs to ensure that data traffic movement is not the reason a system fails. But the challenge of ensuring unhindered traffic movement in the context of a multi-die assembly and constant routing and rerouting of different types of data looms big in increasingly complex devices, particularly under varying workloads and in mission- and safety-critical markets.
Related Articles
Verifying Networks On Chip
Challenges increase significantly with chiplets and coherency.
I/O Design Challenges Grow In AI Data Centers And HPC Clusters
Physical I/Os can be a chokepoint for high-performance chips and high-speed interconnect protocols, requiring design tradeoffs and extra reliability measures.
Confusion Grows With More Interconnect Options And Tradeoffs
Each standard serves a specific use case, so chip architects are choosing more than one for a single design.
Swapping Out Chiplets: I/Os Vs. Compute
Multi-die assemblies give chip architects the option to change some dies while keeping the rest of the system intact, but which is best to keep?
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply