Training vs inference

Training vs inference is the difference between the two jobs that every AI model needs. Training teaches the model, once, using a huge amount of data and thousands of chips working together. Inference is running the trained model to answer each request, which happens billions of times a day across all its users. Training needs raw computing power. Inference needs each answer to be fast, and to use as little energy and money as possible.

Here is how training works. An AI model is a very large set of numbers, called weights or parameters. At the start, the numbers are random, and the model's answers are useless. Engineers feed it examples. For a large language model, the examples are trillions of words of text. For each example, the model makes a guess, such as the next word in a sentence. The guess is compared with the right answer, and every number is adjusted very slightly to make the next guess better. This repeats an enormous number of times. Training a large model takes weeks or months on thousands of linked GPUs, the chips built for this kind of arithmetic. It can cost hundreds of crores of rupees. The result is a file of numbers: the trained model. Fine-tuning is a shorter, cheaper round of training that adapts a trained model to one task.

Here is how inference works. The trained model's file is loaded onto a chip. A request arrives, such as a question typed into a chatbot. The chip runs the request through the model's numbers and produces the answer. The numbers do not change. Each answer costs very little, but there are billions of answers. So, across a year, inference becomes the larger bill. What matters most here is latency, the time the user waits for the answer, and the energy each answer uses. Many companies shrink a model before running it, so that each answer needs fewer calculations.

The two jobs suit different chips. Training needs the most powerful chips, linked so that thousands can work as one machine. Inference can run on cheaper chips built only for running models, such as many AI accelerators. It can even run on small chips inside phones and devices, which is called edge AI.

Take, for example, a chatbot that answers in Hindi. Its model was trained once, over months. Each reply to each user is a separate inference that takes a second or two.

Consider a pair of wireless earphones that react when the wearer says a wake word. The small model was trained in a data centre. Inference runs on a tiny chip inside the earphone, with no internet connection.

Say a bank uses a model to catch fraud. The bank retrains the model every week on the newest fraud cases, and the model scores each card payment, by inference, in a few milliseconds, as part of payment fraud detection.

How is training different from giving a model information at the moment of the question? Training and fine-tuning change the model's numbers. Retrieval-augmented generation (RAG) does not. It looks up passages in a company's documents and hands them to the model with each question, during inference. So a company can keep a model's answers up to date without training it again.