Advancing Signaling Rates To 64 GT/s With PCI Express 6.0

From the introduction of PCI Express 3.0 (PCIe 3.0) in 2010 onward, each new generation of the standard has offered double the signaling rate of its predecessor. PCIe 3.0 saw a significant change to the protocol with the move from 8b/10b to highly efficient 128b/130b encoding. The PCIe 6.0 specification, now officially released, doubles the signaling rate to 64 gigatransfers per second (GT/s) and does so with some of the most fundamental changes yet seen by the standard.

PCIe has proliferated widely beyond servers and PCs, with its economies of scale making it attractive for data-centric applications in IoT, automotive, medical, and elsewhere. That being said, the initial designs using PCIe 6.0 will target applications requiring the highest bandwidth possible and those can be found in the heart of the data center: AI/ML, HPC, networking, and cloud graphics, to name a few.

To achieve 64 GT/s, PCIe 6.0 introduces new features and innovations with the first and foremost being the adoption of PAM4 signaling. PAM4 signaling (“Pulse Amplitude Modulation with four levels”) combines two bits per clock cycle for four amplitude levels (00, 01, 10, 11) vs. PCIe 5.0, and earlier generations, which used NRZ modulation with one bit per clock cycle and two amplitude levels (0, 1).

There are always tradeoffs, however, and the transition to PAM4 signal encoding introduces a significantly higher Bit Error Rate (BER) vs. NRZ. This prompted the adoption of a Forward Error Correction (FEC) mechanism to mitigate the higher error rate. PCIe 6.0 adopts an FEC that is sufficiently lightweight to have minimal impact on latency. It works in conjunction with strong CRC (Cyclic Redundancy Check) to minimize the Link Retry probability. This new FEC feature targets an added latency of under 2ns.

PCIe 6.0 also introduces FLIT mode, where packets are organized in Flow Control Units of fixed sizes, as opposed to variable sizes in past PCIe generations. The initial reason for introducing FLIT mode was that error correction requires working with fixed size packets; however, FLIT mode also simplifies data management at the controller level and results in higher bandwidth efficiency, lower latency, and smaller controller footprint.

With fixed-size packets, the framing of packets at the Physical Layer is no longer needed yielding a 4-byte savings for every packet. FLIT encoding also does away with the 128B/130B encoding and DLLP (Data Link Layer Packets) overhead from previous PCIe specifications. This results in a significantly higher TLP (Transaction Layer Packet) efficiency, especially for smaller packets.

A final innovation to highlight in this blog is the introduction of a new Low Power State or L0p mode. L0p enables traffic to run on a reduced number of lanes to save power. L0p maintains at least one active lane at all times to ensure uninterrupted traffic flow. The link always trains in the highest possible width and can modulate down (and back up again) as needed by the traffic. One of the upshots as a tradeoff between power savings and complexity is that PCIe 6.0 only supports x1, x2, x4, x8 and x16 links. Support for x3, x5, x12, etc. and the huge (and rarely if ever implemented) x32 width have been dropped.

To wrap up, PCIe is everywhere in modern computing architectures, and we expect PCIe 6.0 will gain quick adoption in performance-critical applications in AI/ML, HPC, cloud computing and networking. For designs for both the upcoming generation of PCIe as well as previous generations, Rambus offers a broad family of PHYs, controllers, and complete interface subsystems for easy integration into SoCs and ASICs.

Additional resources:

• Website: Rambus PCIe 6.0 Controller
• Solution Brief: Rambus PCIe 5.0 Interface Subsystem
• Website: Rambus PCIe 5.0 PHY
• Website: Rambus PCIe 5.0 Controller