A GPU-as-a-service platform is software that turns a room full of GPU servers into a cloud that its owner can rent out. It lets many customers share the same GPUs, decides whose job runs on which GPU, and measures each customer's use for the bill. One company in this market calls its product "a neocloud in a box". A neocloud is a new cloud provider that rents out GPUs for AI work, rather than general computing.
A GPU, or graphics processing unit, is a chip that does thousands of simple calculations at the same time. That makes it the main chip for training and running AI models, as the page on AI accelerators explains. A GPU server usually holds eight GPUs and costs crores of rupees. So an owner who buys hundreds of GPUs must keep them busy, and must charge many customers for them.
The software does three jobs, which the industry calls multi-tenancy, scheduling and metering. Multi-tenancy means that many customers use the same machines, but none of them can see or touch another customer's data or jobs. Scheduling means deciding which job runs on which GPUs, and when. A training job may need 64 GPUs connected by fast links, while a small test may need only part of one GPU. The scheduler fits the jobs together so that few GPUs sit idle. Metering means counting each customer's use, usually in GPU-hours. One GPU-hour is one GPU used for one hour.
Say a data centre company in Chennai installs 32 GPU servers, which is 256 GPUs, and rents them out. In one week, three customers use them. A speech AI startup trains a model on 64 GPUs for 36 hours: 2,304 GPU-hours. A bank runs its document-reading model on 4 GPUs all week, for 168 hours: 672 GPU-hours. A university lab runs experiments on 8 GPUs for 50 hours: 400 GPU-hours. The scheduler places the startup's job on 8 servers that sit next to each other, so that their GPUs can exchange data quickly. It fits the bank's and the lab's work on other servers. At ₹200 per GPU-hour, the startup's bill for the week is ₹4,60,800. The platform produces all three bills automatically.
Below the platform sit the AI clusters and rack-scale systems, the hardware itself, and the CUDA and GPU software stack that programs use to reach the GPUs. Customers then use the result as a GPU cloud, or run their trained models through a model deployment platform. A country that wants its AI work done on machines inside its own borders builds a sovereign AI cloud, and this kind of software is one way to run one.
The platform is a part of private cloud software. What sets it apart is the scheduling. GPUs are scarce and costly, and an idle GPU loses its owner money every hour.