A GPU, or graphics processing unit, is a processor with thousands of small cores that all do the same arithmetic at the same time. It was first built to draw images for games. A screen has millions of pixels, and each pixel needs the same kind of sum. Today, GPUs in data centres are used mainly to train and run AI models, because AI needs exactly that kind of arithmetic.
A processor is the chip that computes. A CPU has a few dozen large cores, and each core is built to do many different tasks, one after another. A data centre GPU has more than ten thousand small cores. Each core is simple. But together they can do hundreds of trillions of multiplications every second. The page on CPU, GPU, SoC and FPGA compares the kinds of processor.
Why does AI need this? An AI model is a very large set of numbers, called weights. Training a model means showing it examples, measuring how wrong its guesses are, and adjusting the weights a little each time. At every step, the GPU multiplies large tables of numbers, called matrices. Each small multiplication inside does not depend on the others. So thousands of cores can do them at the same moment.
Consider a speech start-up in Bengaluru that is training a model to turn spoken Kannada and Tamil into text. It has 20,000 hours of recorded speech. It rents 64 GPUs from a GPU cloud for three days. Each GPU has 80 GB of high-bandwidth memory (HBM) stacked beside it. This memory holds the model's weights and the batch of audio clips that the GPU is working on. In each training step, every GPU takes a batch of clips, checks the model's guesses against the correct text, and works out how each weight should change. The 64 GPUs then share these changes through chip interconnect, so that they all keep the same copy of the model. At about ₹250 per GPU per hour, the three days cost about ₹11.5 lakh.
A GPU cannot work alone. A CPU in the same server starts each job and feeds the GPU its data. Usually eight GPUs sit in one server, and many servers are joined into an AI cluster. Memory is often the real limit, because a GPU can compute faster than its memory can supply numbers. This is called the memory wall, and it is why GPU makers care so much about HBM.
One company makes most of the GPUs on which large AI models run. A large reason is its software, the CUDA and GPU software stack, which programmers use to run their work on its GPUs. Moving years of code to another chip is costly. This is a form of vendor lock-in. Newer AI accelerators compete mainly on the cost of running trained models, which the page on training vs inference explains. GPUs are also used outside AI. For instance, a scanner in a hospital uses one to turn its raw readings into pictures, which is called image reconstruction.