LLM evaluation is software that measures whether a large language model, or an app built on one, answers correctly. A security team uses the same tools to measure whether the model answers safely. Evaluation done while the model is serving real users is usually called observability or monitoring. Evaluation done before release is usually called testing.
Why is a special tool needed? Ordinary software gives the same output for the same input, so a test can check for one exact result. A language model can word its answer differently each time, and there may be many correct answers. So evaluation tools score answers in other ways. Some compare the answer with an approved answer written by a person. Some use a second model as a judge, with written instructions such as 'score 1 if the answer is supported by the passage, and 0 if it is not'. Some run simple checks, such as whether the reply is in the right language or contains a phone number.
Before release, a team builds a test set: a fixed list of questions with approved answers, often written by data labelling experts. Every new version of the model or of its instructions is scored on the same set, so the team can see whether a change helped. After release, the tool scores a sample of live answers every day and raises an alert when a score falls.
Take a telecom operator in Kolkata whose support bot answers questions about prepaid plans in Bengali, Hindi and English. Its test set has 600 questions. A new version of the bot scores 93% for correct answers, against 90% for the old one. But the safety tests include 50 prompts that try to make the bot read out another customer's recharge history. The new version refuses 47 of them and reveals data in 3. The team does not release it until it refuses all 50. After release, the tool checks 2% of live chats each day. Two weeks later, it reports that more answers are no longer supported by the documents the bot was given. The cause is an outdated price list, which the team replaces.
Safety test prompts often come from an automated red team, software that writes attacks on the model. When evaluation finds a weakness, the team may add a rule to LLM guardrails. The results also go into AI governance records. Voice bots need their own kind of test, described on the page on voice agent testing.
How is evaluation different from guardrails? Evaluation measures the model's behaviour, on test questions and on samples of live answers. Guardrails act on each live request, and block a harmful input or answer at that moment.