32GT/s PCI Express Design Considerations

Today’s networking and rapidly emerging artificial intelligence (AI) applications are requiring more bandwidth in accelerators and GPUs, as well as faster interconnects to transmit and receive greater amounts of data.

Towards the middle of 2017 the PCI-SIG industry consortium announced its latest specification, PCIe 5.0, which raised the data rate from 16GT/s to 32GT/s and doubled the link bandwidth from 64GB/s to 128GB/s.

Figure 1: PCI-SIG Bandwidth Growth

However, moving to 32GT/s designs comes with several challenges that both system and PHY designers must consider. This article describes such challenges and how designers can successfully design systems with the new PCIe 5.0 interface.

Challenges for system designers
Printed circuit board (PCB) traces, connectors, cables, and even IC packages are system-level bandwidth limiters that are making designs at high data rates challenging. The high signal frequencies increase copper loss and power loss, which causes transmission distances to decrease. In addition, the channel loss in higher signal frequencies cause signal integrity (SI) issues.

To meet the requirements of a wide range of applications, there are many types of PCIe channels, from chip-to-chip topology without connectors to complex server topology with backplane interfaces that include multiple PCB cards and two or more connectors. As shown in Figures 2 through 5, most PCIe channels consist of an IC package on each end with multiple PCBs including: the processor board, an add-in card, and a riser card; all connected with one or more mezzanine or PCIe card electromechanical (CEM) connectors.

Figure 2: Chip-to-chip interface, the simplest channel with no connector

Figure 3 (a): Channel with one mezzanine connector

Figure 3 (b): Channel with an edge connector (add-in card)

Figure 4 (a): Channel with two connectors using a riser card and an add-in card

Figure 4 (b): Standard backplane channel with two line cards and two connectors

Figure 5: Complex backplane channel with more than two connectors

Historically, PCIe system designers use of generic low-cost FR4 PCB materials and wirebond packaging for most applications up to 8GT/s data rates (Gen3) has proven to be successful. However, the use of such materials and packaging is not feasible at the 32GT/s data rate.

Due to the increase of channel loss, even at the 16GT/s maximum speed of PCIe 4.0, needed to maintain existing channel lengths for next-generation board designs, most designers are moving from an FR4 PCB to lower-loss materials such as MEGTRON. PCBs may also be designed with wider spaces between traces to further improve system-level SI performance. Similarly, in regards to SI, many designs will use enhanced CEM connectors or custom mezzanine connectors, and will back-drill the PCB vias to minimize stub length. In a few cases with very long channels, retimers may also be used.

However, all of these enhancements come at a cost. MEGTRON materials can cost 1.2x to 2.5x more than the standard FR4 materials, and PCB traces may need to be spaced-out further for better jitter performance, resulting in a larger and more expensive PCB. Via stub back-drilling, an incremental step in the board manufacturing process, also adds to the total PCB cost. An additional factor to consider is how enhanced and custom surface mount connectors are more expensive than the standard through-hole CEM connectors. Also, the use of retimers increase the bill of materials (BOM) cost, datapath latency, and system power; they also occupy additional area on the PCB, which increases the board and assembly costs.

To validate their designs, system designers must closely collaborate with signal integrity engineers, package designers, SoC designers, and board layout designers to model each component in their channel and verify their entire end-to-end performance.

Challenges for PHY designers
Making incremental improvements to 16GT/s PHY designs is insufficient for meeting the PCIe 5.0 channel requirements in most applications. Due to the significant increase in channel loss at 32GT/s, equalization circuits in the transmitter (TX) and receiver (RX) need significant improvements. In addition, tighter jitter parameters and jitter limits, as well as return loss specifications require many sub-circuit redesigns in both the TX and RX.

The PHY for PCIe 5.0 is expected to support features such as lane margining via the controller and the Separate Reference Clock Independent Spread Spectrum Clocking (SRIS), while meeting tighter timing and jitter requirements across process, voltage, and temperature corners.

Such enhancements and additional constraints make designing a PCIe 5.0 32GT/s PHY extremely complex, requiring competencies in many areas to achieve a low-power, small-area, and low-latency PHY with optimal signal and power integrity (PI) performance.

A silicon-proven PHY with accurate models enables designers to model, design, and simulate an end-to-end channel for system design validation and optimization.

Summary
The requirement for higher bandwidth in data-intensive applications, such as networking, storage, and emerging artificial intelligence, forces the need for faster interconnects such as the new PCIe 5.0 technology at 32GT/s. However, designers must understand and consider the numerous challenges of moving to 32GT/s PCIe designs. Addressing issues like signal integrity, packaging, and channel performance at higher data rates require competencies in several areas. This is why more System-on-Chip (SoC) designers are using proven third-party IP for successful IC integration.

Companies are utilizing the signal and power integrity services of their reliable and proven third-party IP providers, like Synopsys. SoC designers can get an early start on their 32GT/s designs by leveraging Synopsys’ IP for PCIe 5.0, which is built on decades of PCIe expertise. SoC designers can collaborate with Synopsys to discuss PCIe channel performance needs at higher data rates, while addressing IP integration, timing closure, signal integrity, packaging, and manufacturing needs. We will elaborate on each challenge in subsequent documents.

Watch the industry’s first PCIe 5.0 IP demonstration video to learn more. For more information visit the DesignWare IP Solutions for PCI Express web page or contact us now.