An AI accelerator is a processor designed from the start only for the arithmetic that AI models need. Most AI accelerators are built to run trained models cheaply, that is, to answer each question with less time and less electricity than a GPU. Several start-ups build them, and each is betting that a chip with nothing but AI circuits can beat the GPU on the cost of each answer. The page on AI accelerators explains the general idea.
Why could a new chip beat the GPU? A GPU was first designed for graphics. It still carries circuits for many kinds of work, because it is sold to everyone who trains and runs models. An accelerator can leave out everything except AI arithmetic. It can also arrange its parts differently. The hardest part of running a model is moving its numbers, called weights, from memory to the cores for every word of an answer. This is the memory wall. So the start-ups try different layouts. One keeps a whole silicon wafer as a single, very large wafer-scale chip, so that data never has to leave it. Others use in-memory computing, which does the arithmetic inside the memory itself. Some build their chips from many small cores based on RISC-V, the open instruction set that anyone may use without paying a fee.
Most of them aim at inference, not training. Training teaches a model once, using thousands of chips for weeks. Inference runs the trained model for every request, billions of times a day, and it is now the larger bill. The page on training vs inference explains the two jobs.
Here is a typical case. An insurance company in Ahmedabad runs a chatbot that answers 20 lakh customer questions a month about premiums and claims. It owns no AI chips. It sends each question to a cloud service, which charges for each answer by its length. On a service that uses GPUs, an average answer costs it about 5 paise. An accelerator start-up's cloud service offers the same open model, with faster replies, for about 3 paise. Over a month, that is ₹1 lakh against ₹60,000. Before switching, the company tests whether the answers are exactly as good, and whether the service can handle its busiest hour.
The hardest problem is software. A new chip is useless if programmers cannot run their models on it, and most AI software is written for the GPU maker's tools. So each start-up builds an AI compiler that runs models written for GPUs on its own chip. This is why these companies hire many software engineers, several of them in Bengaluru.
The start-ups earn in different ways. Some sell chips, or whole systems that fit into an AI cluster. Some run their chips in their own cloud and charge for each answer. Some license their designs to other chip companies, earning a fee and a royalty, as explained in semiconductor IP licensing. Governments that build a sovereign AI cloud, AI computing kept inside their own country, are also possible buyers.