Why HBM reminiscence and AI processors are glad collectively

Excessive bandwidth reminiscence (HBM) chips have turn out to be a recreation changer in synthetic intelligence (AI) functions by effectively dealing with advanced algorithms with excessive reminiscence necessities. They turned a significant constructing block in AI functions by addressing a crucial bottleneck: reminiscence bandwidth.

Determine 1 HBM contains a stack of DRAM chips linked vertically by interconnects known as TSVs. The stack of reminiscence chips sits on high of a logic chip that acts because the interface to the processor. Supply: Gen AI Consultants

Jinhyun Kim, principal engineer at Samsung Electronics’ reminiscence product planning crew, acknowledges that the mainstreaming of AI and machine studying (ML) inference has led to the mainstreaming of HBM. However how did this lover affair between AI and HBM start within the first place?

05.01.2024

04.18.2024

04.18.2024

As Jim Helpful, principal analyst with Goal Evaluation, put it, GPUs and AI accelerators have an unbelievable starvation for bandwidth, and HBM will get them the place they wish to go. “Should you tried doing it with DDR, you’d find yourself having to have a number of processors as an alternative of only one to do the identical job, and the processor price would find yourself greater than offsetting what you saved within the DRAM.”

DRAM chips wrestle to maintain tempo with the ever-increasing calls for of advanced AI fashions, which require huge quantities of information to be processed concurrently. However, HBM chips, which supply considerably greater bandwidth than conventional DRAM by using a 3D stacking structure, facilitate shorter knowledge paths and sooner communication between the processor and reminiscence.

That enables AI functions to coach on bigger and extra advanced datasets, which in flip, results in extra correct and highly effective fashions. Furthermore, as a reminiscence interface for 3D-stacked DRAM, HBM makes use of much less energy in a type issue that’s considerably smaller than DDR4 or GDDR5 by stacking as many as eight DRAM dies with an non-obligatory base die that may embrace buffer circuitry and check logic.

Subsequent, every new era of HBM incorporates enhancements that coincide with launches of the newest GPUs, CPUs, and FPGAs. As an example, with HBM3, bandwidth jumped to 819 GB/s and most density per HBM stack elevated to 24 GB to handle bigger datasets.

Determine 2 Host gadgets like GPUs and FPGAs in AI designs have embraced HBM attributable to their greater bandwidth wants. Supply: Micron

The neural networks in AI functions require a major quantity of information each for processing and coaching, and coaching units alone are rising about 10 instances yearly. Which means the necessity for HBM is prone to develop additional.

It’s vital to notice that the marketplace for HBM chips continues to be evolving and that HBM chips aren’t restricted to AI functions. These reminiscence chips are more and more discovering sockets in functions serving high-performance computing (HPC) and knowledge facilities.

Associated Content material