AI hallucination

An AI hallucination is an answer from an AI model that states something false or invented as if it were a fact. Examples are a court judgment that does not exist, a wrong dose of a medicine, or a quotation that nobody said. The false answer sounds as confident and fluent as a correct one, so the reader cannot tell the difference just by reading it. That is what makes hallucination dangerous.

Here is why it happens. A large language model does not look facts up. It was trained on a huge amount of text, and it writes its answer one word at a time, choosing each time a word that is likely to follow the words before it. Most of the time, the likely words are also the true ones. But when the model has seen little about a topic, or when the question asks for an exact detail such as a case name, a date or a number, the most likely-sounding answer can be false. The model has no built-in step that checks its answer against a source. So it can produce a case name in the right format, with a believable year and court, for a case that was never decided.

Hallucination cannot be removed completely, but it can be reduced and caught. The most common method is to give the model the right source text with each question, and to tell it to answer only from that text. This method is called retrieval-augmented generation (RAG). The product can also show the source of each statement, so that a person can open it and check. For important work, a person reviews the answer before it is used, an approach described in human in the loop. An answer is only as reliable as the source behind it.

Take, for example, a lawyer who asks an AI assistant for judgments that support an argument. Courts abroad have fined lawyers who filed such made-up cases. A legal AI assistant that answers from a real legal database, together with a citator that shows whether each judgment exists and is still good law, guards against this.

Consider a farmer who asks a chatbot how much pesticide to spray. An invented dose can damage the crop or harm the farmer, so an AI farming assistant has to answer from checked advice for that crop.

Say a student asks for help with a physics problem. AI doubt solving can give a wrong step in a confident tone, and the student may learn the wrong method.

How is a hallucination different from an ordinary error in data? If a database holds a wrong figure, every answer drawn from it is wrong in the same way, and correcting the database fixes all of them. A hallucination is invented by the model itself, and the same question can get a different invented answer the next time it is asked. That is why AI products are tested in a different way, as described in testing non-deterministic software.