PCIe 6.0 Takes Data Center Performance To The Next Level

Looking back at 2022, we saw a major update to the PCI Express (PCIe) specification. PCIe 6.0 brought with it some of the most fundamental changes yet seen by the specification, resulting in some exciting capabilities that are set to take data center performance to the next level in the years ahead.

PCIe has been the interconnect of choice in computing for two decades now. Its ongoing advances in bandwidth and power efficiency have met the industry’s ever-growing need for a high-speed, low-latency and scalable interconnect solution. Delivering performance and cost effectiveness, PCIe technology has, therefore, been widely adopted across a broad landscape of data-intensive markets.

Following the introduction of PCIe 3.0 in 2010, each new generation of the standard has offered double the signaling rate of its predecessor. The PCIe 6.0 specification, released in January 2022, boosts signaling rates to 64 gigatransfers per second (GT/s), twice that of PCIe 5.0. For bandwidth-hungry, data-intensive workloads, the extra bandwidth offered by PCIe 6.0 will certainly be a game changer!

Initial designs incorporating PCIe 6.0 will be where bandwidth demands are most intense right now: in the heart of the data center. PCIe 6.0 will be essential for system-on-chip (SoC) designers creating devices that require the movement of large amounts of data. This includes artificial intelligence/machine learning (AI/ML), and high-performance computing (HPC) applications.

Let’s take a closer look at some of the key changes in the PCIe 6.0 specification.

One of the biggest changes is the shift to PAM4 (“Pulse Amplitude Modulation with four levels”) signaling. PAM4 combines two bits per clock cycle for four amplitude levels (00, 01, 10, 11) vs. PCIe 5.0, and earlier generations, which used NRZ modulation with one bit per clock cycle and two amplitude levels (0, 1). There are always tradeoffs, however, and the transition to PAM4 signal encoding introduces a significantly higher Bit Error Rate (BER) vs. NRZ. This prompted the adoption of a Forward Error Correction (FEC) mechanism to mitigate the higher error rate inherent in PAM4. PCIe 6.0 adopts an FEC that is sufficiently lightweight to have minimal impact on latency.

But an FEC requires fixed-size packets, so PCIe 6.0 introduces FLIT mode, where packets are organized in Flow Control Units of fixed sizes, as opposed to variable sizes in past PCIe generations. FLIT mode also simplifies data management at the controller level and results in higher bandwidth efficiency, lower latency, and smaller controller footprint.

Higher speeds, of course, mean higher power, so PCIe 6.0 introduces a new low-power state of operation called L0p Mode. L0p enables traffic to run on a reduced number of lanes to save power. L0p always maintains at least one active lane to ensure uninterrupted traffic flow. The link always trains at the highest possible width and can modulate down (and back up again) as needed by the traffic.

For SoC designers, the number of signal integrity and power integrity (SI/PI) issues compound as data rates rise. Designing for 64 GT/s operation can be exceedingly tricky. Rambus has over 30 years of leadership in SI/PI, as well as experience in PAM4 signaling, to support this next generation of PCIe designs.

Rambus offers a complete PCIe 6.0 interface subsystem comprised of PHY and controller IP. The PCIe controller features an Integrity and Data Encryption (IDE) engine dedicated to protecting the PCIe links and the valuable data transferred over them. On the PHY side, full support for the latest version of the Compute Express Link (CXL) specification, version 3.0, is available to enable chip-level solutions for cache-coherent memory sharing, expansion, and pooling in the data center.

Additional resources

• Rambus PCIe 6.0 Controller IP
• Rambus PCIe 6.0 PHY IP
• Data Center Evolution: The Leap to 64 GT/s Signaling with PCI Express 6.0