tag: latency

Rethinking Ethernet For The AI Scale-Up Era: Inside ESUN
By Abhinav Kothiala - 13 Jul, 2026 - Comments: 0

Every generation of AI infrastructure has redefined what "the network" means. In today's training clusters — scaling from hundreds to hundreds of thousands of accelerators — the interconnect is no longer a supporting actor. It has become a first-order determinant of system throughput, utilization, and cost per token. Accelerators inside these systems don't just move data; they synchronize o... » read more

Arm Neoverse CMN-700: Performance Analysis Methodology
By Arm - 08 Jul, 2026 - Comments: 0

System performance on modern System-on-Chip (SoC) platforms is increasingly determined by behavior beyond the processing elements. While processor microarchitectures continue to deliver high instruction throughput, overall application performance is often constrained by shared system resources such as caches, memory controllers, and the coherent interconnect that connects them. On Arm Neover... » read more

Impact Of On-Chip SRAM Size And Frequency On Energy Efficiency And Performance of LLM Inference (Uppsala Univ.)
By Technical Paper Link - 03 Jan, 2026 - Comments: 0

A new technical paper titled "Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling" was published by researchers at Uppsala University. Abstract "Energy consumption dictates the cost and environmental impact of deploying Large Language Models. This paper investigates the impact of on-chip SRAM size and operating frequency on the energy efficiency and per... » read more

Dynamic KV Cache Scheduling in Heterogeneous Memory Systems for LLM Inference (Rensselaer Polytechnic Institute, IBM)
By Technical Paper Link - 28 Aug, 2025 - Comments: 0

A new technical paper titled "Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System" was published by researchers at Rensselaer Polytechnic Institute and IBM. Abstract "Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity red... » read more

PCIe Over Optics
By Ed Sperling - 03 Feb, 2025 - Comments: 0

Moving data through a chip or package, and between packages and systems, is becoming a much bigger challenge as the volume of data continues to explode, and as more compute resources are deployed to work on data-intensive problems such as training AI algorithms or running long and complex simulations. There is more data to process in more places, more levels of data storage and access, and any ... » read more

Photonics: The Former And Future Solution
By Karen Heyman - 14 Mar, 2024 - Comments: 0

Experts at the Table: Semiconductor Engineering sat down to talk about where photonics is in the hype cycle and its secure role in data centers, with James Pond, fellow at Ansys; Gilles Lamant, distinguished engineer at Cadence; and Mitch Heins, business development manager for photonic solutions at Synopsys. What follows are excerpts of that conversation. [L-R]: Ansys’ Pond, Cadence�... » read more

Integration Challenges For RISC-V Designs
By Ed Sperling - 27 Feb, 2024 - Comments: 0

One of the big draws of RISC-V is that it allows design teams to create unique chips or chiplets and to make modifications to the instruction-set architecture. That extra degree of freedom also creates some issues when it comes to integrating those designs into packages or systems because they may require non-standard connectivity approaches. Frank Schirrmeister, vice president of marketing at ... » read more

LLM Inference on GPUs (Intel)
By Technical Paper Link - 02 Feb, 2024 - Comments: 0

A technical paper titled “Efficient LLM inference solution on Intel GPU” was published by researchers at Intel Corporation. Abstract: "Transformer based Large Language Models (LLMs) have been widely used in many fields, and the efficiency of LLM inference becomes hot topic in real applications. However, LLMs are usually complicatedly designed in model structure with massive operations and... » read more

What’s Changing In DRAM
By Ed Sperling - 22 Jan, 2024 - Comments: 0

More data requires more processing and more storage, because that data needs to be stored somewhere. What’s changing is that it’s no longer just about SRAM and DRAM. Today, multiple types of DRAM are used in the same devices, each with its own set of tradeoffs. C.S. Lin, marketing executive at Winbond, talks about the potential problems that causes, including mismatches in latency, and high... » read more

Very Short Reach SerDes In Data Centers
By Ed Sperling - 13 Dec, 2023 - Comments: 0

Speed is critical inside of data centers, and the distance that signals have to travel can have a big impact on time to results. But there are a number of variables that need to be considered, including what is an acceptable loss, how much power can be dissipated in a server rack, and what are the various connection options being used. Keivan Javadi Khasraghi, staff technical product manager at... » read more

← Older posts

  • Sponsors

About

Navigation

Connect With Us

Copyright ©2013-2026 SMG   |  Terms of Service  |  Privacy Policy