GPUs Dominate (Again) The Green500 List

The Green500 has released its latest list of the top 500 most energy-efficient Supercomputers. The top 17 are heterogeneous systems (systems that use more than one type of processor), with the top 15 systems all using NVIDIA Kepler K20 GPUs paired with Intel Xeon CPUs.

Still at the top of the list is the Tokyo Institute of Technology GSIC Center’s TSUBAME-KFC, an oil-cooled Kepler powered system. From a pure efficiency standpoint of MFLOPS/W, it is better than 20% higher than the next system on the list (Cambridge University’s Wilkes).

Figure 1. Energy Efficiency vs. Total Compute Capability

While the highest efficiency systems are still cranking out well over 100 TFLOPS, you’ll note that they’re still nearly two orders of magnitude down in terms of total compute power. Another list called the Top500, is based solely on the total compute capabilities of the systems. The machine at the top of the Top500 list is the National Super Computer Center in Guangzhou’s Tianhe-2 with an efficiency of 1901 MFLOPS/W (and ranking 49th on the Green500).

Clearly, it would seem to be much harder to build a machine to top the Top500 and still do well on the Green500. In order to take a closer look at how efficiency drops with larger systems, I’ve created the plot in Figure 1. The green line marks the “frontier” for energy efficient machines given their total compute capabilities. Machines near this line are beating their peers in that they are delivering more efficiency (MFLOPS/W) than other systems delivering as much compute capability. From this perspective, Tianhe-2 looks pretty good.

Other systems along this frontier line are the larger (than TSUBAME-KFC) TSUBAME 2.5, Eni S.p.A.’s HPC2 and the Swiss National Supercomputer Centre’s (CSCS) Piz Daint. Piz Daint deserves special mention here because it appears to be ahead of the frontier in terms of the efficiency it delivers for its compute capabilities. Of the remaining three, Tianhe-2 has already been mentioned as the overall compute leader. DOE/SC/Oak Ridge National Laboratory’s Titan and DOE/NNSA/LLNL’s Sequoia are also #2 and #3 on the Top500 respectively (and #43 and #38 on the Green500). Sequoia deserves special mention because of all of the seven systems called out near the frontier line, Sequoia is the only homogeneous system made up entirely of IBM Power BQC 16C 1.6GHz processors.

If the Exascale Challenge is going to be met (1 ExaFLOP in 20-40MW by the year 2020) then the efficiency at the large system end is going to need to reach 25,000 MFLOPS/W or still roughly 6x the efficiency of TSUBAME-KFC (or 13x the efficiency of Tianhe-2). There’s still plenty of work to do in the next 6 years, but a 10x improvement in 6 years is within the realm of possibility based on past history in the industry. It should make for a very interesting race to the 2020 goal line.