Memory wall

The memory wall is the gap between how fast a processor can do arithmetic and how fast its memory can deliver the numbers it needs. Today's AI processors can compute far faster than memory can feed them. So, for much AI work, the time and the energy go on moving data, not on doing sums. Engineers named the problem in the 1990s, when they noticed that processors were getting faster each year much more quickly than memory was.

Here is how it works in general. A processor keeps only a small amount of data on the chip itself. Everything else sits in DRAM, the main working memory, on separate chips nearby. Data travels between the DRAM and the processor along a link with a fixed bandwidth: the amount of data the link can carry each second. The processor fetches some numbers, does its sums on them, and fetches the next numbers. If the sums take less time than the next fetch, the processor stops and waits. Fetching a number from DRAM also uses far more energy than adding or multiplying it, often a hundred times more.

AI models make the gap especially wide. A large language model is stored as billions of numbers, called weights. To write each word of an answer, the processor must read nearly all of those weights from memory once. Say a model has 70 billion weights, each stored in 2 bytes. That is 140 gigabytes to read for every word. A memory that delivers 3,000 gigabytes a second needs about 47 milliseconds just for the reading, however fast the arithmetic is. Here the memory, not the processor, sets the speed.

Chip makers narrow the gap in a few ways. Take, for example, high-bandwidth memory. It stacks DRAM chips in layers right beside an AI processor, joined by thousands of connections, so that far more data arrives each second than from ordinary DRAM.

Consider in-memory computing. It does the arithmetic inside the memory itself, so that most of the data never has to travel to a processor.

For instance, a wafer-scale chip keeps a very large amount of memory on one giant chip, so that the data never has to leave it.

How is the memory wall different from the ideas in bandwidth and latency? Bandwidth and latency describe any link, such as a network cable. The memory wall is one particular gap: between the speed of a processor and the speed of its own memory. A program that spends most of its time waiting for memory is called memory-bound. A program that spends most of its time on arithmetic is called compute-bound. Training an AI model on GPUs is often compute-bound, because each number fetched is used in many sums. Answering a single user's request is usually memory-bound.