PCIe 6.0, NVMe, And Emerging Form Factors For Storage Applications

PCIe 6.0 implementations are expandable and hierarchical with embedded switches or switch chips, allowing one root port to interface with multiple endpoints (such as storage devices, Ethernet cards, and display drivers). While the introduction of PCIe 6.0 at 64GT/s helped to increase the bandwidth available for storage applications with minimal or no increase in latency, the lack of coherency still limits PCIe to applications like solid state drives (SSDs), which are block storage devices. For these storage applications, NVMe, which uses PCIe as the transport interface, has become the dominant SSD technology.

Apacer announced the first PCIe 5.0 SSDs at Computex in Taipei in May 2022. Even though the PCIe 5.0 SSDs have not yet rolled out broadly, at Synopsys we are already working with designers on the development of new SSD devices that will take advantage of the PCIe 6.0’s 64GT/s data rate.

With PCIe 5.0 SSDs just getting started, what is driving the need for even faster SSDs? The market demand comes down to trying to keep up with the increasing capacity of SSDs by raising the bandwidth available to connect to the drives. Since most SSD form factors, including the pervasive M.2 and U.2, support a maximum lane configuration of x4 (four lanes), the only way to increase the bandwidth is for each lane to go faster, since going wider with the interface is not possible. This has driven the PCIe speed from 4.0 to 5.0, and next to PCIe 6.0.

Non-Volatile Memory Express & PCIe for SSDs

NVM Express (NVMe) is a logical device interface specification for accessing non-volatile storage media attached via the PCIe bus with performance targeting a broad range of enterprise and client systems, and it now dominates the SSD landscape. NVM stands for non-volatile memory, which is often NAND flash memory, and it comes in several physical form factors, including 2.5″ SSDs (U.2), PCIe add-in cards (AICs), M.2 cards, and others. NVMe technology is not a transport protocol, but rather a storage-oriented command set that uses PCIe transport.

NVMe was designed specifically for SSDs and communicates between the storage interface and the system CPUs using high-speed PCIe connections, independent of the storage form factor. NVMe is faster and transfers more data more quickly than older SATA-based SSDs. Because it was designed specifically for SSDs, NVMe has become the new industry standard for both data center servers and client devices like laptops, desktop PCs, and advanced gaming consoles.

Since the release of the NVMe specification in 2011, NVMe has grown to dominate the SSD market (figure 1). NVMe over Fabrics, or NVMe-oF, was added to the specification in 2016, and expanded the use of NVMe to other protocols, like Ethernet, Infiniband, or FibreChannel, allowing Host systems to access storage devices over a network or fabric, well beyond a single backplane. NVMe-oF devices typically utilize arrays of PCIe/NVMe SSDs together with a controller to interface to the network transport, and PCIe remains a critical part of these systems.

Today, most Host platforms are based on PCIe 4.0 with PCIe 5.0 just beginning deployment. As a result, most SSDs are based on PCIe 4.0, with some new ones based on PCIe 5.0, and some leading-edge SSD developments already targeting PCIe 6.0.

Fig. 1: Enterprise SSD Capacity Shipment Forecast by Interface Type and Year. Source: IDC Worldwide Solid State Drive Forecast 2020-2024, Doc #US4590920, December 2020

PCIe (NVMe) SSD Form Factors

While the M.2 and U.2 form factors limit the bandwidth to a x4 PCI Express link, newer form factors like E1 and E3, as well as the Add-in Card form factor, enable wider links of x8 and even x16 with increased bandwidth. The form factors do not determine the PCIe data rates supported; data rates are primarily driven by the current PCIe specification version and its deployment in the industry.

U.2

U.2 is a 2.5-inch form factor that is commonly used for SSDs and is offered with PCIe (with NVMe), SAS, or SATA interfaces. It is typically used in desktops, servers, and storage systems built around HDDs, and some motherboards have U.2 interfaces built into them. Strictly speaking, U.2 means compliance with the PCI Express SFF-8639 Module specification, and there are both 7mm and 15mm modules that use U.2 which can support PCIe x4 with NVMe.

M.2

M.2 is a form factor specification for internally mounted SSDs targeting the mobile, laptop, and PC markets, and is significantly smaller than U.2. M.2 supports PCIe, SATA, and USB interfaces. M.2 SSDs come in a variety of widths and lengths, but most versions share the same style connector that plugs directly into available M.2 slots on the motherboard. The M.2 form factor supports x2 and/or x4 lanes of PCIe running the NVMe command set.

EDSFF form factors

EDSFF stands for Enterprise and Data Center Standard Form Factor, and it includes a family of specifications maintained by SNIA as part of the SFF Technology Affiliate Technical Work. EDSFF includes a wide range of form factors that all share the same PCIe + NVMe protocol and the same SFF-TA-1002 edge connector. EDSFF has advantages over some of the other SSD form factors in terms of maximum drive capacity, scalability, performance, and thermal and power management. Some of the EDSFF form factors support x8 and even x16 PCIe links, thereby offering the highest possible bandwidth for future PCIe 6.0 base SSDs.

E1.S

E1.S, an abbreviation for EDSFF 1U Short, is a flexible, power-efficient building block for hyperscale and enterprise storage applications such as cloud compute servers and OEM 1U performance servers. E1.S is a small form factor, being just a bit longer than M.2, but wider to accommodate more NAND for increased capacity per drive. E1.S offers advantages over the typical 110mm M.2 form factor popular in hyperscale data centers, such as hotplug capability, improved thermal performance, and scaling to higher capacities due to the support for x8 PCIe links, up from the maximum of x4 seen in M.2.

E1.L

E1.L, an abbreviation for EDSFF 1U Long, is a form factor developed to maximize available capacity per drive and per rack unit in a 1U server or storage array. E1.L has options supporting x4 or x8 lanes of PCIe while fitting vertically in a 1U chassis to allow for scalable bandwidth per drive. It is hot pluggable and front access serviceable with LEDs built into an integrated enclosure, improving data center serviceability and allowing for more power efficient storage (TB/W). E1.L, like E1.S, fits vertically in a standard 1U chassis.

E3

E3 is a family of form factors, including E3.S and E3.L, intended to replace the traditional U.2 2.5-inch form factor in many servers and storage systems. The different versions all share the same height (76mm) and are hot-pluggable drives. They are optimized for flash density of the SSD and system chassis. The E3 family connector is designed for x4 to x16 PCIe lanes and power envelopes up to 70W, making it potentially the highest bandwidth SSD form factor along with AIC. All versions are slot/connector compatible and should be front accessible. Designed for future servers and storage systems, EDSFF E3 will accommodate future generations of PCI Express, including PCIe 6.0.

Add-In Card

An Add-in Card (AIC) form factor for SSDs utilizes a standard PCIe card form factor and can plug in to an available PCIe slot. Given the larger physical size of AIC over the form factors previously discussed, the AIC could potentially have larger capacity and potentially higher performance, since it can also support any available PCIe link widths, including x16. In addition, the larger size allows for the potential to add additional computational functionality to a storage device. The AIC form factor is versatile and will likely be able to evolve for solid-state storage applications.

Summary

PCI Express is the dominant interface for today’s SSDs, driven by the broad adoption of the NVMe standard. Available in multiple form factors, NVMe SSDs are hitting the market supporting connection to Hosts via PCIe 5.0 and are poised to move to PCIe 6.0 within the next 2-3 years. The introduction of the E3 form factor, together with AICs, can potentially push the aggregate bandwidth available for SSDs as high as ~ 100GB per second for a PCIe 6.0 x16 SSD.

Synopsys has delivered PCIe IP solutions across the current five generations with proven silicon in customer products and successful third-party interoperability. Synopsys has delivered PCIe 6.0 solutions to multiple customers, including for next generation SSD applications. For more information, visit https://www.synopsys.com/pcie.