AI accelerators

AI accelerators are chips built to do the arithmetic of AI models much faster than an ordinary processor can. The best known is the GPU, the graphics processing unit, which was first built to draw video-game graphics. Others are designed only for AI, such as Google's TPU and the custom chips that large cloud companies design for their own work. An accelerator does not replace the main processor, the CPU. It sits beside the CPU and takes over the heavy arithmetic.

Here is how it works in general. An AI model is mostly a very large set of numbers. Training or running it means multiplying huge grids of numbers together, billions of times. A CPU has a limited number of powerful cores: from a handful in a laptop to a few hundred in the largest server chips. These cores do different tasks one after another. An accelerator has thousands of small cores, which do the same multiplication on different numbers at the same moment. It also has high-bandwidth memory stacked beside it, so that the numbers reach the cores quickly. To train a large model, thousands of accelerators work on one job together, joined by very fast links into an AI cluster.

These chips have changed the buildings around them. A rack of AI accelerators uses many times the electricity of an ordinary rack of servers. The newest racks give off so much heat that air cannot cool them, so cold liquid is piped right up to the chips. That is why data centre power and cooling is being redesigned, and why data-centre operators now look first for sites with plenty of electricity. The links between the accelerators matter as much as the chips themselves, which is the subject of AI cluster networking.

Take, for example, a small start-up that wants to train a model in Indian languages. It cannot afford to buy a cluster, so it rents accelerators by the hour from a GPU cloud. The government-backed IndiaAI Mission also offers GPU time to start-ups and researchers at lower prices.

Consider wireless earphones that cancel noise or recognise a spoken command. A TinyML chip inside them is a very small accelerator that runs a tiny model on battery power.

How are AI accelerators different from a CPU? The CPU is the general-purpose processor that runs the operating system and decides what happens next. An accelerator is very fast at one kind of work, the same arithmetic on many numbers at once, and poor at most other work. The kinds of processor are compared in CPU, GPU, SoC and FPGA. Training a model and answering users with it place different demands on these chips, as training vs inference explains.