What is AI hallucination?
Also called: Hallucination · Confabulation · AI fabricationDefinition
An AI hallucination is a response from a generative AI model that is presented fluently and confidently but is factually wrong, unsupported by the given sources, or invented, such as a fake citation, an incorrect figure or a policy that does not exist. Hallucinations happen because language models generate statistically likely text rather than looking up verified facts. They can be reduced with grounding techniques such as retrieval-augmented generation, clear instructions and evaluation, but not eliminated, so human verification remains necessary.
A language model produces each token based on patterns learnt in training and the context it is given. When the prompt asks about something the model has little reliable information on, or the retrieved context is missing or ambiguous, it may still produce a fluent answer. Fluency is not evidence of accuracy, which is why hallucinations are easy to miss.
At work, hallucinations create real risk: invented case law in a legal draft, a wrong tax rate in a finance note, or a misquoted HR policy sent to an employee. The risk is highest where output is published, sent to customers or used for decisions without review.
Practical safeguards include giving the model the source material and asking it to answer only from that, requesting citations and checking them, allowing the model to say it does not know, using RAG for factual systems, and evaluating outputs on test sets. Training people to verify specifics such as names, numbers, dates and references is the most important control.
Key points
- Confident, fluent output that is false or unsupported.
- Caused by generating likely text rather than retrieving facts.
- Reduced by grounding, citations and evaluation.
- Cannot be fully eliminated today.
- Verify names, numbers, dates and references before use.
An example at work
A finance associate in Bengaluru asks an AI tool for the GST rate on a service; it answers confidently with the wrong rate, which she catches by checking the official notification before the invoice goes out.
Where this is used at Bodhih
Related terms
Large language model
A large language model (LLM) is an AI model trained on vast amounts of text to understand and generate language by predicting the next token.
Retrieval-augmented generation
Retrieval-augmented generation (RAG) is a technique where an AI system retrieves relevant documents first and gives them to an LLM to ground its answer.
LLM evaluation
LLM evaluation is the systematic testing of a language model or LLM application against defined criteria to measure quality, accuracy and safety.
Responsible AI
Responsible AI is the practice of designing, deploying and using AI so that it is fair, safe, transparent, accountable and respects privacy.
AI literacy
AI literacy is the ability to understand what AI can and cannot do, use it effectively and safely, and judge the quality of its output.
More about AI hallucination
Why do AI models hallucinate?
Language models are trained to produce plausible continuations of text, not to check facts. When they lack reliable information, or the prompt pushes them to answer anyway, they generate text that fits the pattern of a correct answer without being one. Missing or poor-quality context in a RAG system has the same effect.
How can you reduce AI hallucinations?
Provide source material and instruct the model to answer only from it, ask for citations, allow it to say it does not know, break complex tasks into steps, and use retrieval-augmented generation for factual applications. Then verify key facts yourself and, for production systems, run regular evaluations on real questions.