The CUDA and GPU software stack is the set of drivers, compilers and libraries that lets programmers run their work on a GPU. CUDA is the best-known example. It is the software platform that the leading GPU maker provides for its own GPUs. A stack means layers of software, each built on the layer below.
From the bottom up, the layers are these. The driver is software inside the operating system that sends work to the GPU and manages its memory. The compiler turns a programmer's code into instructions that the GPU's cores can run. The libraries are ready-made, carefully tuned programs for common jobs, such as multiplying two large tables of numbers. On top sit AI frameworks such as PyTorch. A data scientist writes a model in the framework, and the framework calls the libraries underneath. The data scientist may never see CUDA at all.
A small program that runs on the GPU is called a kernel. The GPU maker writes and tunes the library kernels again for each new GPU. Many companies also write kernels of their own, for steps that the libraries do not cover or do too slowly. The stack is free to download, but it runs only on that maker's GPUs. Much of it is written by the maker's engineers in Bengaluru, Pune and Hyderabad.
Say a start-up in Kochi reads handwritten insurance forms with an AI model that runs on GPUs. Over three years, its engineers have written 30 kernels of their own in CUDA, for the slowest steps of the model. Now another company offers it a cheaper AI chip, at about 30% less per hour. The model's PyTorch code would move with little change, if the new chip's maker supports PyTorch. But the 30 kernels would not. They would have to be rewritten for the new chip and tested again, which the team estimates at four engineers for six months. The team also worries that the new chip's libraries might be slower for its model. So it stays on GPUs for now.
This is vendor lock-in: the cost of leaving grows with every piece of code built on one maker's tools. It is a large part of why one company leads the market for AI chips. Rivals answer in two ways. Some offer their own open stack that accepts code written in a similar style. AI chip start-ups build an AI compiler that takes models written for GPUs and runs them on their chips.
The stack also supports the layers above it. A distributed computing framework uses it to split one job across many GPUs. A GPU-as-a-service platform installs it on the servers it rents out, so that customers can start work at once.