An AI compiler is the software that takes an AI model written for GPUs and turns it into instructions that a different chip can run. Every AI chip start-up must build one. Without it, nobody could run their models on the new chip. Building it is the main reason these start-ups hire so many software engineers.
Why is it needed? Data scientists write models in frameworks such as PyTorch. On a GPU, the framework calls ready-made, hand-tuned programs from the GPU maker, which are part of the CUDA and GPU software stack. A new chip has none of these programs. Its cores, its memory and the links between them are arranged differently from a GPU. So the start-up's compiler must work out automatically what the GPU maker's engineers worked out by hand.
A compiler of this kind works in stages. First, it captures the model as a graph: a map of every operation, such as "multiply these two tables of numbers", and of which operation feeds which. Second, it combines small operations into larger ones, so that data stays on the chip between them instead of going back to memory. This is called fusion. Third, it decides which operation runs on which cores and where each weight is stored, so that data travels as short a distance as possible. Fourth, it plans the timing, so that each piece of data arrives just before the cores need it. Finally, it produces a binary, the file of instructions that the chip runs.
Consider a compiler engineer in Bengaluru at an AI accelerator start-up. A customer wants to run an open language model with 8 billion weights on the start-up's chips. The model arrives as standard PyTorch code. The compiler captures it as a graph of about 1,000 operations, and fusion cuts this to about 300. At 2 bytes each, the weights take 16 GB, so the compiler spreads them across 8 chips, 2 GB on each. On the first run, the model reaches only 40% of the chips' top speed. The engineer studies the timing and finds that one operation, used often by this particular model, was not fused. She adds a new fusion rule to the compiler. The model now reaches 70%, and so will every later model that uses the same operation.
That last point is why the compiler matters so much. Each new model that customers want brings new operations. A start-up whose compiler handles them quickly can offer new models within days. One whose compiler lags behind loses customers to GPUs. The compiler is the start-ups' main answer to vendor lock-in.
Once the model is compiled, it still has to be served to users. A model deployment platform loads the compiled model, sends it requests and scales it up and down. The start-up has to make its chips work with such platforms too, because customers already use them with GPUs.