High-bandwidth memory (HBM) is a kind of DRAM made by stacking several memory chips on top of each other and placing the stack right beside an AI processor, inside the same package. Thousands of tiny connections join the stack to the processor. So data reaches the processor fast enough to keep it busy.
DRAM is the fast working memory that a processor uses, and it loses everything when the power goes off. In an ordinary server, DRAM sits on separate memory sticks a few centimetres from the processor, joined to it by a few hundred wires. An AI processor needs far more than that. When a GPU runs a large language model, it must read all of the model's numbers, called weights, again for every word of the answer. The processor can compute faster than ordinary memory can supply numbers. This gap is called the memory wall.
HBM narrows the gap in three ways. First, it is close: the stack sits a few millimetres from the processor. Second, it is wide: each stack has 1,024 data connections, compared with 64 for an ordinary memory channel. Third, it is stacked. A die is a chip-sized piece of silicon cut from a wafer, a thin round disc of silicon on which many chips are made at once. In an HBM stack, 8 or 12 DRAM dies, each thinner than a sheet of paper, lie on top of one another. Tiny holes through each die, filled with copper, carry signals up and down the stack. These are called through-silicon vias. The stacks and the processor sit side by side on a thin silicon base, called an interposer, which carries the thousands of wires between them. All of this is sealed in one package, the case that joins the chips to the circuit board.
Consider a 12-high HBM stack made in South Korea. Each of its 12 DRAM dies holds 3 GB, so the stack holds 36 GB. It can supply data at about 1.2 terabytes per second. A recent AI GPU carries eight such stacks around its processor: 288 GB of memory, supplying about 8 terabytes per second. By comparison, the ordinary DRAM of a server CPU supplies a few hundred gigabytes per second.
HBM is hard to make. Only three companies make DRAM, two based in South Korea and one in the United States, and they make nearly all of the world's HBM. A single faulty die spoils the whole stack, so every die must be tested before it is stacked. The share of good units, the yield, is lower than for ordinary DRAM. As a result, HBM costs several times as much per gigabyte, and it has often been sold out a year ahead. Its demand has changed the memory chip cycle, the industry's swing between shortage and surplus.
India's part is so far in assembly and testing. Micron is building a chip assembly and test plant at Sanand in Gujarat, which packages and tests memory chips made elsewhere. Every AI accelerator and custom AI chip also needs HBM. In-memory computing tries a different route: it does the arithmetic inside the memory, so that the weights do not have to travel at all.