Benefits Of A Silicon-Proven 800G Ethernet Solution For High-Performance Computing

The evolution of high-speed Ethernet began in 2014 when Arista, Broadcom, Microsoft, Mellanox and Google formed the Ethernet Consortium, now called the “Ethernet Technology Consortium.” Since then, the technology has been adopted by more than 45 members. The push for 200G, then 400G, and now 800G Ethernet is driven by the insatiable need to process and transmit high-performance workloads in diverse applications such as high-performance computing (HPC), 5G, and deep learning applications – not to mention the aggregate bandwidth needed to deliver the sum of these applications to and from the cloud. In 2020, the consortium rolled out the 800G specification, which has since been ratified by the IEEE 802.3 standards committee and working group. With the use of high-speed interfaces, hyperscale data centers can support the increasing Ethernet data rates in SoCs for compute rack, switches, retimers, NICs, optical modules, and more. The Ethernet standards provide the reference frame for interoperable interfaces required for design and manufacturing of such SoCs and the devices they support.

There are multitude of ways that an 800G Ethernet controller and PHY solution can meet the scalable and high-data-rate connectivity requirements of data-intensive applications. This article describes a silicon-proven and robust 800G Ethernet solution using the Synopsys MAC, PCS and PHY IP that companies can use as a reference guide to converge on their own Ethernet SoC design faster.

The need for 800G Ethernet

Switches have evolved in the last 12 years, going from 640G to 102.4T and supporting Ethernet speeds from 10G to 100G/lane and beyond, switch radices from 64 to 512 lanes and pluggable connectors from QSFP+ to QSFPDD800/OSFP, as illustrated in figure 1. 800G Ethernet is at the heart of the latest 51.2T switch as well as 800G pluggables it supports.

Fig. 1: Switches have evolved from 640G to 102.4T in the last 12 years.

800G Ethernet chips with DACs can be used for in-rack communication and as the interface to standard pluggable optics. The standard for 400G pluggables, capable of delivering up to 400G Ethernet when plugged into the faceplate of a line card, is now widely available and adopted in optical architectures in the HPC world. It is very important that the aspect ratio of 800G Ethernet designs conform to industry-standard form factors and connector interfaces. It must also be interoperable when swapped with another pluggable. Comprehensive interoperability will provide a credible proof-of-concept to get a head start on the design. Once the standard has been approved, you’ll be that much ahead. Next generation 800G modules offer a doubling of the bandwidth per port. Figure 2 shows the statistical data and predicts that 800G pluggables will surpass 400G by 2025 as presented by Dell’Oro group at OCP 2022. With twice the lanes, data centers will not need to totally change their port configuration to handle 800G Ethernet, which will make the transition easier.

Fig. 2: 800G pluggables are expected to surpass 400G by 2025.

Anatomy of a robust 800G Ethernet solution

To enable adopters to deploy advanced, high-bandwidth, interoperable Ethernet technologies today, they need a proven 800G Ethernet solution to reference. The Synopsys 800G Ethernet solution is based on 8 lanes of 100Gb/s technology. The 800G architecture consists of a new media access control (MAC) and physical coding sublayer (PCS), basically repurposing two sets of the existing 400G Ethernet logic and distributing the data across eight 106.25 Gbps physical lanes. As the 400G Ethernet PCS logic is reused, the forward error correction block is retained for simple compatibility with existing physical layer specifications. However, SoC designers will need to develop chips to support 800G Ethernet to meet the requirement of all the transmit and receive data path elements, allowing the lowest latencies and higher bandwidth while minimizing excessive power or cost penalties. It is also important to retain backwards compatibility with slower speeds, which will ensure the seamless adoption and integration of 800G Ethernet and beyond into existing data centers.

To deliver this performance simultaneously in 200G/400G/800G networking, multiple challenges in the logical and physical domains have emerged. Meeting lowest latency, power, and area with the faster clock speeds, parallel paths, and complex signaling requirements are difficult to achieve. What is needed is a highly efficient FEC to ensure minimum latency with low retransmission rates to compensate for the higher error rates inherent to faster communication speeds.

Figure 3 depicts a reference schematic for an 800G Chip with 8 lanes of 100G SerDes, 800G PCS, MAC, test-logic and application interface. Having robust testability and debuggability features with an application SW interface is the key to silicon success.

Fig. 3: An example of a block diagram for an 800G chip with 8 lanes of 100G SerDes.

The Synopsys 800G Ethernet solution is designed to function as the network interface for applications such as next-generation data center networks. To keep up with increasing CPU, bus and storage bandwidth, rack or blade servers must support aggregate 800Gb/s throughputs from their Network Interface Card (NIC) requiring100G per lane SerDes and cabling technology. Such technology needs to support 100 Gb/s per physical lane via copper twin-ax cable or fiber. Given the increased bandwidth to endpoints, uplinks from Top-of-Rack (TOR) switches will need to transition from 8 lanes of 400 Gb/s SerDes 8 lanes of to 800 Gb/s SerDes while ideally maintaining the same per-lane breakout capability and reach. To maintain the overall system performance with many lanes switching simultaneously and to compensate for increasing crosstalk, designers are building a lot of margins into their designs. Carefully optimized bump-maps in the design help to mitigate any IR drop issues, package stacks and route escapes on die and through the package are more important than ever. A 51.2T switch on-die implementation requires both north-south and east-west 800G Ethernet tiles. These re-usable tiles can be constructed and replicated multiple times in all edges of the die to minimize the beachfront. Package escape, signal integrity and power integrity analysis provide the groundwork for the basic floorplanning of the 800G Ethernet tile. Hardening the 800G Ethernet test chip involved expert knowledge of block partitioning for optimizing beachfront, SerDes, PCS and MAC design. Full visibility into the end-to-end datapaths is needed for retiming, pipelining or optimizing latency versus power and area. Lastly, it involves SoC-level full RTL-to-GDS flow for front-end and back-end integration and close collaboration with EDA tools for signoff.

Figure 4 illustrates the various use-cases of an end-to-end 800G implementation from 51.2T switches, 800G pluggables, extended-reach DACs or active copper cables for in-rack communications to 800G pluggable optics for extended reaches.

Fig. 4: End-to-end 800G implementation use cases.

Conclusion

In addition to silicon IP supporting the target area, latency, performance, power and signal integrity, SoC designers also need all the required documentation and deliverables to achieve fast integration. Synopsys has been a developer of Ethernet IP for many generations, playing an integral role in defining the 800G Ethernet and beyond specification. Now, Synopsys has a silicon-proven 800G Ethernet solution that customers can reference to achieve their own silicon success. Synopsys provides integration-friendly deliverables for 112G Ethernet PHY, PCS, and MAC with expert-level support which can make customers’ life easier by reducing the overall design cycle and helping them bring their products to market faster.