Retrieval-augmented generation (RAG) is a method in which software first finds the passages in a company's own documents that are relevant to a question, and then hands those passages to a language model together with the question. The model writes its answer from those passages. So the answer rests on the company's own facts, not only on what the model learned during training.
Why is this needed? A large language model (LLM) knows only what was in its training data, and that data ends on a certain date. It has never seen a company's internal rules, its product manuals or last week's circular. When asked about them, it may make up an answer that sounds right. This is called AI hallucination. RAG reduces the problem by giving the model the right text to read, every time.
RAG works in two stages. The first stage happens in advance. The software splits every document into short pieces, called chunks, often a few hundred words each. It turns each chunk into a list of numbers that captures its meaning, called a vector embedding, and stores the vectors in a vector database. The second stage happens with every question. The question is also turned into a vector. The database returns the chunks whose vectors are closest to it, which are the chunks with the closest meaning. The software places those chunks in the prompt, the text sent to the model, with an instruction such as 'answer only from these passages, and name the source'.
Imagine a customer of a health insurer, writing from Coimbatore, who asks the insurer's chatbot: 'Is my knee replacement covered, and is there a waiting period?' The RAG system searches 3,000 pages of policy wordings. It returns the five chunks that match the question most closely, including the clause on joint replacement and the clause on a two-year waiting period. The model reads those five chunks. It answers that the surgery is covered once two years have passed from the policy start date, and it gives the clause numbers, so that the customer can check.
Before documents are stored, data masking can hide personal details, such as policy-holders' names. Teams then use LLM evaluation to check that each answer really comes from the passages it was given. An AI agent framework often uses RAG as its long-term memory. The same method sits inside a legal AI assistant and clinical decision support.
How is RAG different from fine-tuning? Fine-tuning trains the model further, so that it remembers new material. RAG leaves the model unchanged and gives it the material at the moment of each question. RAG is also easier to keep up to date: when a policy changes, only the document has to be replaced.