Data Movement Is the Energy Bottleneck of Today’s SoCs

In today’s AI-focused semiconductor landscape, raw compute performance alone no longer defines the effectiveness of a system-on-chip (SoC). The efficiency of data movement across the chip has become just as important. Whether designed for data centers or edge AI devices, SoCs must now prioritize data transport as a core architectural consideration. Moving data efficiently across the silicon fabric has become a central challenge as application workloads grow in complexity and scale across distributed resources. Inefficient data movement is often the primary bottleneck impacting overall system performance and power consumption.

This bottleneck becomes especially evident in AI workloads. Applications such as large language models and generative AI rely on trillions of data transactions per second. These models often require simultaneous access to memory, caches, and AI accelerators. Without an interconnect solution built to support high-throughput communication, these systems are quickly overwhelmed by congestion, increased latency, and power inefficiencies. This results in underutilized compute, excess energy consumption, and reduced overall system efficiency, even in high-performance designs.

Traditional interconnects, such as buses and crossbars, cannot handle today’s SoCs’ dynamic data flows and bandwidth demands. This has driven a shift toward packet-based network-on-chip (NoC) architectures. A NoC provides a structured and scalable approach to transporting data between the growing number of IP blocks in a chip. Instead of fixed wiring paths, NoCs send data in packets, which allows greater flexibility in routing, reduces physical wire count, and minimizes power consumption, all while improving performance.

Supporting AI Scalability and Chiplet Integration
The versatility of NoC IP lies in its ability to support multiple interface protocols, including AXI, AHB, OCP, and even custom implementations. Each intellectual property (IP) block connects through a Network Interface Unit (NIU), which adapts to that IP’s specific protocol, data width, or clock frequency. This allows the NoC to support communication between heterogeneous IP blocks, regardless of the mixing and matching of vendors or interface requirements. The modular nature of NIU-based connectivity supports reuse, simplifies clock and power domain crossings, and enables scalable integration without the need to re-architect the entire SoC.

As the industry continues shifting to chiplet-based architectures, the value of NoC design becomes even more apparent. Chiplets distribute compute, memory, and I/O functionality across multiple silicon dies to improve yield, reduce cost, and enhance modularity. However, this also introduces a new layer of interconnect complexity, particularly at the boundaries between dies. Maintaining high-speed, low-latency communication between dies is essential for chiplet systems to operate as cohesive, high-performance SoCs.

As SoCs scale beyond monolithic designs, the interconnect must do more than move data. It must bridge system-level architecture with physical implementation. This requires early attention to layout constraints such as floorplanning, congestion, wirelength, and timing closure. When combined with architectural flexibility, this physical awareness is essential for enabling high-speed, low-latency communication across dies. As chiplet-based systems become more prevalent, the ability to align architectural intent with layout constraints has become a defining requirement for scalable interconnect design.

Easing the Memory Bottleneck with Cache Coherency
Another persistent challenge in SoC development is the imbalance between processor performance and memory bandwidth, often called the memory wall. Even with cache architecture advances, data must still traverse significant distances across the chip to reach memory. These round trips introduce latency, which can significantly impact workloads that depend on timely access to large volumes of data, such as AI inference or high-throughput computing.

Cache-coherent NoCs provide a powerful way to address this. In systems that utilize write-back caches at the L1 and L2 levels, the latest copy of a data item may exist in cache rather than in main memory. A centralized directory tracks which core holds the most recent version and coordinates the necessary update or invalidation requests. This approach prevents unnecessary main memory access, synchronizes data across processing units, and helps reduce overall memory traffic.

To make cache coherence viable in practice, the interconnect must support efficient, real-time coordination across multiple initiators, often spanning a mix of CPU cores, AI engines, and domain-specific accelerators. This requirement has transformed coherence from an optional performance enhancement into a foundational element of SoC design. Ensuring consistency without degrading performance in systems with heterogeneous compute elements operating on shared data sets is essential.

Connecting Architecture to Implementation
Designing a high-performing NoC is not just about logical structure. Success also depends on how well the interconnect maps to the chip’s physical layout. Factors such as wire length, routing congestion, and timing closure must all be considered early in the design process. A well-architected NoC can still fall short if it is not aligned with the physical realities of implementation.

This is where physically aware NoCs make a significant impact. Tools like FlexGen smart NoC IP from Arteris directly incorporate floorplan constraints, congestion analysis, and timing goals into the development workflow. This allows the interconnect topology to evolve parallel with the floorplan, rather than being introduced late in the design cycle. The result is better timing closure, fewer routing issues, improved power efficiency, and faster convergence through shorter critical paths.

Automation plays a significant role in enabling these capabilities. Machine learning heuristics within FlexGen can evaluate design requirements, suggest optimal topologies, and generate interconnect structures that meet bandwidth and latency targets. These automated approaches often outperform manual implementations, allowing teams to complete complex designs within tighter schedules.

In addition to FlexGen, Arteris offers FlexNoC interconnect IP for non-coherent interconnect and Ncore cache coherent interconnect IP for cache coherent implementations. FlexNoC provides a configurable, packet-based transport fabric that reduces wire count and congestion while enabling clock and power domain crossing. Ncore extends this capability by supporting full cache coherence with directory-based management, which is ideal for advanced AI and high-performance applications requiring synchronized access to shared data across heterogeneous processors.

As SoCs grow in complexity, the ability to sustain efficient data movement has become a defining factor in system success. Once viewed as a low-level detail, data transport is now central to system-level optimization. Interconnect technology, particularly packet-based NoC architectures, is pivotal in addressing this evolving constraint. NoCs help reduce bottlenecks that would otherwise limit responsiveness and scalability by optimizing how data moves within and between modular components. With the right interconnect, design teams are well-equipped to meet today’s workload demands while achieving the performance and power goals that advanced applications require.