In-memory computing means doing the arithmetic inside the memory that stores the data, instead of first moving the data to a separate processor. It is also called compute-in-memory. Its aim is to save energy, because in an ordinary chip, moving data costs far more energy than the arithmetic itself.
In an ordinary computer, the memory and the processor are separate. To multiply two numbers, the processor must read them from memory, carry them along wires to its cores, multiply them and write the result back. Measurements on chips show how unequal these steps are. Reading one 32-bit number from external memory chips (DRAM) takes roughly 640 picojoules of energy. Multiplying two 32-bit numbers takes about 4 picojoules. A picojoule is a millionth of a millionth of a joule. So the trip costs about 150 times as much as the sum. AI makes this worse. A model's numbers, called weights, run into billions, and all of them must be read again for every word of an answer. This is the memory wall.
How does in-memory computing avoid the trip? The main operation in AI is multiply-and-add: multiply each input by a weight, and add up all the results. Memory is laid out as a grid of cells, in rows and columns. In-memory computing stores each weight in one cell of the grid. The inputs are applied to all the rows at once, as voltages. Each cell then lets through an electric current that depends on both its input and its stored weight. That is the multiplication. The currents from all the cells in a column meet on one wire and add up. That is the addition. So each column produces one multiply-and-add result, and all the columns work at the same time. This kind of calculation is analog: the result is a current, not a number. A converter at the foot of each column turns the current back into a digital number.
Take, for example, one layer of a language model with 512 inputs and a 512 × 512 table of weights, which means 2,62,144 multiplications. The weights are written into a 512 × 512 grid of memory cells once, when the model is loaded. Each time new inputs arrive, all 2,62,144 multiplications and 512 additions happen in one pass through the grid. No weight leaves the memory, so no energy is spent carrying it to a processor.
There are trade-offs. Analog currents are slightly imprecise and pick up electrical noise, so results are less exact than digital arithmetic. That is acceptable for running trained models, which work well with lower precision, but less suited to training. Models must also be adjusted to fit the grid. Some designs therefore use digital circuits placed inside the memory, which are exact but save less energy. Start-ups building AI accelerators, some with engineers in Bengaluru, use both approaches.
Compare high-bandwidth memory (HBM). HBM shortens the trip between memory and processor by stacking memory right beside the processor. In-memory computing removes most of the trip. The page on training vs inference explains why this matters most for answering requests.