An LLM gateway is software that places one API in front of many large language model providers. An app sends every request to the gateway, and the gateway decides which model receives it. It also moves requests to another provider when one is down, and records how much each request costs. An API here is the agreed format in which one program sends a request to another program and gets an answer back.
Why is this needed? A company that builds with AI rarely uses just one model. A large model gives better answers, but it costs more and responds more slowly. A small model is cheap and fast, but it makes more mistakes. Each provider also has its own API, its own prices and its own outages. Without a gateway, every team in the company would write separate code for each provider, and nobody would know the total bill.
The gateway does four jobs. It routes each request by rules, for example sending short, simple questions to a cheap model and long documents to a larger one. It retries and falls back: if a provider returns errors or is too slow, the gateway sends the same request to another provider within a second or two. It can store the answers to repeated questions, so that the same answer is not paid for twice. And it records every request, with the model used, the time taken and the cost, team by team.
Consider an edtech company in Noida whose app explains maths problems to school students. The app sends about 3 lakh requests a day. The gateway sends simple 'explain this step' requests to a small model, which costs a fraction of a paisa each. It sends full solutions of board-exam problems to a larger model. One evening in exam season, the larger model's provider has an outage at 9 p.m. The gateway sees the failed requests and moves them to a second provider within two seconds. Students see a short delay, not an error. At the end of the month, the finance team sees that the app spent ₹6.2 lakh on models, and which feature spent most of it.
The gateway also sends specialist requests to domain-specific language models, and an AI agent framework sends each of its model calls through it. Because every request passes through the gateway, many companies place LLM guardrails there. The security team can also use it for shadow AI control, to see which outside AI tools are being used.
How is it different from an ordinary API gateway? An API gateway sends requests to a company's own services. An LLM gateway sends them to outside large language model (LLM) providers. It also counts tokens, the small pieces of text by which those providers charge.