An AI gateway is an API gateway placed in front of AI language models. Every request that a company's apps and AI agents send to a model passes through it. It sends each request to the right model, checks that the caller is allowed, limits how much each team may spend, and records every call. It is the same idea as an API gateway, applied to a new kind of service.
Why do companies need one? A large company may have a dozen apps that use large language models: a support chatbot, a tool that summarises contracts, a coding assistant. The models come from several providers, and each provider charges by the token. A token is a piece of a word; on average, an English word is a little more than one token. Without a gateway, each app holds its own key to each provider, and nobody knows the total bill until the month ends.
Take, for example, an IT services company in Kolkata whose 12 internal apps use three different models. Its HR chatbot answers employees' questions about leave and company policies. It has a budget of ₹50,000 a month. An employee asks: "How many casual leaves do I have left?" The chatbot sends the question to the gateway, not directly to a model provider. The gateway checks the chatbot's key. It sees that this is a short, simple question, and sends it to a smaller, cheaper model, as the company's rules say. The question, with the employee's leave record, and the answer together use 1,200 tokens going in and 300 coming out. The gateway records the cost, about 5 paise, against the HR chatbot's budget. On the 24th of the month, the chatbot reaches 90% of its budget, and the gateway alerts the employee who owns the chatbot.
Say one provider stops answering for ten minutes. The gateway sends the requests to a second provider instead, so the chatbot keeps working.
The gateway also handles AI agents, software that uses a language model to take actions on its own. When an agent calls the company's own APIs, the gateway applies the same checks and limits to it as to any other caller. The page on AI agent guardrails explains why an agent should have only the access its job needs. Before a prompt reaches a model, the gateway can also pass it through LLM guardrails, which block attacks such as prompt injection.
How is an AI gateway different from an LLM gateway? Both sit in front of the models. An LLM gateway is usually a separate product, built only for model traffic. An AI gateway is often an extension of the API gateway the company already runs. So one system holds the rules for both the company's APIs and its model calls. Most of the models behind it are foundation models from a few large providers.