PCI Express 5.0 Takes Center Stage For Data Centers

The demands on servers at the heart of data centers continue an inexorable rise. Responding to these demands, new platforms keep coming that deliver greater computing performance, have more memory and use faster interconnects. On the way at the end of this year and early 2022 are new server platforms that will take performance to a new level. These new platforms will transition to DDR5 DIMMs for main memory and PCI Express 5.0 (PCIe 5.0) for system interfaces.

PCIe 5.0, as the latest generation of the PCIe standard, will be key to enabling the continued advancement of high-speed computing and processing in the data center. Critically, its bandwidth performance provides the necessary speed of connection between the network interfaces of servers and switches. It is also the key interface connection between CPUs and AI accelerators. Further, more storage is moving away from SAS/SATA and towards non-volatile memory express (NVMe) implemented over PCIe.

PCIe 5.0’s predecessor, PCIe 4.0, was first announced in November of 2011 with the final 4.0 specification released in June of 2017. Offering a top speed of 16 Gigabits per second (Gbps) in a x16 implementation, PCIe 4.0 can deliver full duplex aggregate bandwidth of 64 Gigabytes per second (GB/s). But in a world of exponentially rising data traffic, that’s performance that is already falling behind the power curve. With server network interfaces transitioning from 100 Gigabit Ethernet (GbE) to 400 GbE in the not-so-distant future, 64 GB/s just isn’t enough.

PCIe 5.0 doubles the data rate to 32 Gbps and the resultant full duplex bandwidth of a x16 interface to 128 GB/s, sufficient for 400 GbE links. A 400 GbE link operating at full duplex requires 800 Gbps of bandwidth. Converting to bytes, that’s 100 GB/s of aggregate bandwidth needed which x16 PCIe 5 can support within its performance envelope. But of course, the demand for bandwidth is insatiable, and 800 GbE announced earlier this year will require another speed upgrade. The PCI-SIG is committed to a 2-year cadence of new generations to advance the performance of the standard in support of that need.

Network bandwidth isn’t the only catalyst driving the adoption of PCIe 5.0. The rapid shift in processing workloads, with AI/ML leading the charge, is having a profound impact. For such advanced AI/ML workloads, parallel processing is needed for enormous datasets requiring heterogeneous computing. Specifically, it requires massively parallel architectures, which is why these workloads are being offloaded from the main CPU to a co-processor (AI accelerator), whether it be a GPU, FPGA, or even a purpose-built ASIC. In turn, heterogeneous computing puts a critical demand for bandwidth on the link between CPUs and AI accelerators – a PCIe 5.0 link in the next generation of AI/ML hardware.

If doubling the speed of the PCIe link entailed only a doubling in the complexity of implementation, that would be a pretty good deal. Unfortunately, complexity rises at a higher, non-linear rate with speed, driven in no small part by the growing number of signal and power integrity issues that emerge.

Another major area of design complexity is bridging the transition from the mixed signal to the digital domain. An integrated interface solution of PCIe 5 PHY and digital controller greatly simplifies this challenge for chip designers. The Rambus PCIe 5 interface is just such an integrated solution, with validated PHY and controller, and provided with a complete reference design and test bench for ease of use. It’s specification compliant-backward compatible with PCIe 4/3/2/1, supports root port, endpoint and dual mode implementations and offers optional scatter-gather DMA support.

PCIe 5 is among the key interface technologies needed for the continued advancement of computing and networking performance needed in the next generation of data centers. With Rambus’ PCIe 5.0 interface solution, designers can depend on a robust, high-performance platform for the implementation of their new PCIe Gen 5 ASICs.

Additional Resources:
Solution Brief: PCI Express 5.0 Interface Subsystem
Website: PCI Express 5.0 PHY
Website: PCI Express 5.0 Digital Controller