GDDR6 And HBM2: Signal Integrity Challenges For AI

In a nutshell, Artificial Intelligence (AI) and its growing list of applications demand a considerably large amount of bandwidth to push bits in and out of memory at the highest speeds possible. AI has been getting a lot of industry attention, and certainly it’s not a new phenomenon because it’s been gaining even greater traction in the last year or two.

This is especially true since a number of chipmakers have steadily minted new AI silicon to comply with the demands of the industry. There are two reasons for this growing technology trend. One is AI is becoming more complicated; it has to run sophisticated algorithms on a larger scale, thus the need for new processor architectures.

The other is the fact that the sheer volume of information is growing far faster than any other technology curve. According to the experts, the world’s digital data is doubling every two years.

So, with significantly more data to sift through, process and make decisions, specialized silicon is needed to crunch the growing data sets. Looking specifically at the memory subsystem, there are two types vying for a variety of different AI applications including high bandwidth memory (HBM) and GDDR6, both providing the highest memory bandwidth available in the industry today.

Which memory will AI system designers choose? It depends on the type of application. It’s not a case where one memory system is better than the other. It’s really what’s fitting your application. As the industry starts to mature, HBM is being used to support those highest-end AI applications providing over 300GBytes/s bandwidth per device (HBM2E). This high bandwidth does come with a higher cost since both 3D manufacturing technology is needed for the DRAM, and 2.5D manufacturing for the system (see Figure 1 below).

For applications that are more cost sensitive, GDDR6 is an efficient solution providing 64GBytes/s per device. GDDR6 is being used in mid-range AI applications and is also being considered for more traditional network applications like mid-range routers, or even in wireless networking.

Regardless, of the memory technology selected, each of these systems bring unique design challenges. In my last blog, the signal integrity (SI) challenges for GDDR6-based systems were explored. Now, let’s look at the challenges for HBM.

In order to achieve very high bandwidth, HBM has 1024 signal pins that need to be routed to the DRAM (many more when you include control and ground signals). To route this large number of pins from the SoC to the DRAM, a silicon interposer is used (see Figure 1).

Figure 1: HBM2 DRAM

When routing signals through the interposer, all the usual signaling effects associated with SI are making their presence known, again. That includes insertion loss, reflections, noise, distortion, crosstalk, and power supply induced jitter (PSIJ).

The silicon interposer is a resistive channel, so the SI effects will vary from a traditional PCB design. The channel considerations include channel length, trace width and thickness, signal spacing, number of routing and ground layers.

As expected, the channel length has a significant impact on the signal integrity. The channel length from the SoC to the DRAM can vary between 4.5 to 8.5mm. As the channel length increases, so does the insertion loss and far-end cross talk. The use of patterned ground layers, needed to meet metal density rules, also introduces additional insertion loss and far-end cross talk.

In addition, the number of signal routing layers in the interposer is important to consider. More layers will help with routing and signal spacing to minimize cross talk but adds cost. Given the expense of HBM, it is important to balance these parameters to minimize cost at the maximum performance.

In summary, SI will be at the forefront of these HBM2 and GDDR6-based AI system designs. AI SoC and system designers should closely collaborate with a vendor that has extensive SI experience with both PCB and 2.5D designs in order to mitigating signal integrity effects and design the optimal system.