AI and data centre chips are the chips inside the servers that train AI models and answer every question sent to an AI chatbot. A chip is a small, flat piece of silicon that carries billions of transistors. A transistor is a microscopic electrical switch, and a chip computes by turning its transistors on and off. In a data centre, these chips sit on boards inside servers, and thousands of servers are joined so that they can work on one model together.
Take, for example, a legal-research chatbot built by a start-up in Gurugram. A lawyer types a question: "Can a landlord evict a tenant without notice?" The start-up owns no servers. It rents GPU servers from a GPU cloud provider in a data centre in Chennai, and it pays for every hour it uses them. Here is what happens to the question in the next few seconds.
First, the question arrives at the data centre over the internet. A data centre switch passes it to the right server. Inside the switch, a networking chip reads the address on each piece of data and sends it out through the right port.
Second, the server's CPU, its general-purpose processor, receives the question. The CPU runs the operating system and the start-up's software. It cuts the question into tokens, the small pieces of words that a large language model reads, and sends them to the GPUs.
Third, the GPUs work out the answer. A GPU is a processor with thousands of small cores that do the same arithmetic at the same time. The model itself is a very large set of numbers, called weights. Say the model has 70 billion weights. Stored at 2 bytes each, they take about 140 GB, so they are split across two or more GPUs. Each GPU keeps its share in high-bandwidth memory (HBM), memory chips stacked right beside it. The model writes the answer one token at a time, and for every token each GPU must read all of its weights again. So the speed of the memory decides the speed of the answer. This limit is called the memory wall.
Fourth, the GPUs exchange their partial results through chip interconnect, the fast links that join one processor to the next. This exchange happens for every token, so slow links leave the GPUs waiting.
Fifth, the finished answer goes back to the lawyer, a few seconds after she asked. The conversation is saved on hard disk drives, the cheapest storage for large amounts of data.
The same job can run on other chips. An AI accelerator is a newer chip designed only for AI arithmetic, and several start-ups hope it will cost less per answer. The largest cloud companies, most of them based in the United States, also design their own custom AI chips. Answering questions with a trained model is called inference. Teaching the model in the first place is called training, and the page on training vs inference explains the difference.
Almost none of these chips are made in India. They are designed by fabless companies, which design chips but do not make them, and they are made mostly in Taiwan. But much of the design work is done by engineers in Bengaluru, Hyderabad and Pune.