LLM hallucinations

Hallucination in the context of large language models (LLMs) is a phenomenon where the model confidently generates a plausible-looking response that is factually incorrect, not grounded in the provided context, or is internally inconsistent. The model "invents" facts, details, or logical conclusions that are absent from the source data.

It is important to note that a hallucination is not a failure or a bug in the traditional sense. The model is operating as designed: it predicts the most probable continuation of a text based on patterns learned from its training data. It has no built-in mechanism for truth verification. Hallucinations differ from simple errors in that they present confidently delivered but false information, often including non-existent facts, citations, or events. This phenomenon has become so significant that in 2023, the Cambridge Dictionary added a new definition for the term "hallucination" related to artificial intelligence.

Definitions and Classification of Hallucinations

Although various terms are used (e.g., "confabulation," "fabrication"), hallucinations in LLMs can be divided into two broad categories: those related to factual accuracy and those related to faithfulness to the source (contextual consistency).

Factual Hallucinations

This occurs when the model provides factually incorrect information about the real world. The model asserts a false "fact" as truth.

• Example: "Charles Lindbergh was the first person to walk on the moon" — a completely fabricated fact.
• False citations and references: The model might invent a reference to a non-existent scientific paper or law, mimicking the format of a real citation. This undermines trust in models, especially in applications requiring accuracy (education, news, consulting).

Logical Errors

The model makes an inconsistency or an error in reasoning. Individual facts in the response may be correct, but the conclusion is illogical or contradicts basic logic. This often happens in complex reasoning or in math and causality tasks, where the model operates on probabilistic word associations rather than formal logic.

• Example: "Since birds fly, astronauts do not experience gravity" — the text appears coherent but is logically incorrect.

Contextual Hallucinations

The model's response does not align with the provided context or instruction. The model "drifts" from the context, adding extraneous information or ignoring required details.

• Instruction Violation: When asked to "translate the text into Spanish," the model responds in English.
• Information Not from the Source: In a summarization task, the model "adds" facts that were not in the original document or misrepresents them.
• Context Blending: In the middle of a response, the model may suddenly start discussing something from a different domain. For example, when asked about NBA commissioner Adam Silver, the model might switch to his predecessor David Stern, blending two different contexts.

Inconsistency

A type of hallucination where the model contradicts itself within a single response or a series of responses. One study found that the self-contradiction rate in ChatGPT's responses is around 14%.

• Example: "Company X was founded in 1990... and a few sentences later... Company X, established in 2000..."

Hallucinations in Code

LLMs trained on code can generate syntactically correct but non-functional snippets that use non-existent functions, libraries, or parameters. For example, a model might generate `import quantum` in Python, although no such standard module exists. In 2024, the term "code hallucination" was proposed, and the CodeMirage benchmark was created to systematize this problem.

Causes

The phenomenon of hallucination is driven by a combination of factors, ranging from model architecture to data quality.

• Architecture and Training Principle: Most LLMs (e.g., GPT) are autoregressive transformers trained to predict the next token. Their goal is to maximize the likelihood of the text, not to verify the truthfulness of statements. The model does not distinguish between facts and fiction in its training data, treating everything as text patterns.
• Quality of Training Data: LLMs are trained on vast text corpora from the internet, which contain numerous inaccuracies, myths, and outdated information. The model memorizes and reproduces these errors. The knowledge cutoff—the point in time up to which the model has information—is also crucial.
• Text Generation Method: The stochastic nature of generation (sampling with temperature) allows the model to create more "creative" but less accurate responses. A limited context length can cause the model to "forget" earlier details of a conversation and start contradicting itself.

Evaluation and Measurement Methods

Automated metrics, human evaluation, and specialized benchmarks are used to detect and measure hallucinations.

• Automated Metrics: These include approaches where another LLM acts as a "judge" (LLM-as-a-judge) to assess the correctness of a response, or analyzing the model's entropy (uncertainty) during generation.
• Human Annotation: Considered the "gold standard." Experts or crowd-assessors manually evaluate responses, flagging errors. This method is used in training models with RLHF.
• Benchmarks and Stress Tests: Special datasets have been created, such as TruthfulQA, which contains questions designed to provoke the model into reproducing common myths. There are also leaderboards, like the Hugging Face Hallucination Leaderboard, where models are compared based on their hallucination levels.

Mitigation and Prevention Methods

• Retrieval-Augmented Generation (RAG): The most successful approach, which "grounds" the model in external knowledge. Before generating a response, the model retrieves relevant information from a database, search engine, or API. This allows the model to base its answer on verified data rather than speculation.
• Chain-of-Thought Reasoning and Self-Verification: The model first generates a step-by-step reasoning process before giving the final answer, which improves accuracy. In more advanced methods like Self-Verification, the model generates a draft response and is then tasked with reviewing and correcting it.
• Built-in Rules and Filters: Models are trained to refuse to answer if they are uncertain. For example, Claude models from Anthropic follow a "truthfulness" principle and often respond with "I don't know for sure..." instead of inventing facts.
• Integration with External Tools: Models like Gemini can automatically recognize when they need an external tool (e.g., a calculator for computations or a search for recent news) and use it, significantly reducing the number of hallucinations.

Risks and Consequences

• Legal and Reputational Risks: In the legal field, hallucinations can have serious consequences. The Mata v. Avianca (2023) case gained widespread attention, where a lawyer used ChatGPT to find legal precedents, and it fabricated several non-existent cases. The lawyers were fined, and the incident served as a lesson about the unacceptability of trusting AI without verification.
• Spread of Disinformation: On a societal scale, LLMs can amplify the problem of fake news. A well-known case is the Galactica model from Meta, created to assist scientists, which began generating pseudoscientific texts with fabricated experiments and citations. Public access to the model was shut down after three days.
• Making Flawed Decisions: Users, especially inexperienced ones, tend to trust confidently phrased AI responses, which can lead to poor decisions in finance, medicine, and other critical areas.

Practical Examples

• The Air Canada Case (2023): The airline's chatbot invented a non-existent ticket refund policy. When a customer demanded its application, the company refused. The Civil Resolution Tribunal of Canada held Air Canada responsible for the information provided by its chatbot and ordered it to compensate the customer for their losses.
• Defamation Lawsuit Against OpenAI (2023): Radio host Mark Walters sued OpenAI because ChatGPT, in response to a journalist's query, falsely accused him of fraud. This case highlighted the legal liability of companies for the content generated by their models.

Links

• The Beginner's Guide to Hallucinations in Large Language Models — A detailed guide from Lakera
• Survey of Hallucination in Natural Language Generation — A scientific survey of the phenomenon on arXiv

Literature

• Holtzman, A. et al. (2020). The Curious Case of Neural Text Degeneration. arXiv:1904.09751.
• Caccia, M. et al. (2018). Language GANs Falling Short. arXiv:1811.02549.
• Fan, A. et al. (2018). Hierarchical Neural Story Generation. arXiv:1805.04833.
• Su, Y.; Collier, N. (2022). Contrastive Search Is What You Need for Neural Text Generation. arXiv:2210.14140.
• Meister, C. et al. (2023). Locally Typical Sampling. arXiv:2202.00666.
• O’Brien, S.; Lewis, M. (2023). Contrastive Decoding Improves Reasoning in Large Language Models. arXiv:2309.09117.
• Finlayson, M. et al. (2024). Basis-Aware Truncation Sampling for Neural Text Generation. arXiv:2412.14352.
• Tan, Q. et al. (2024). A Thorough Examination of Decoding Methods in the Era of Large Language Models. arXiv:2402.06925.
• Yu, S. et al. (2023). Conformal Nucleus Sampling. arXiv:2305.02633.
• Chen, S. J. et al. (2024). Decoding Game: On Minimax Optimality of Heuristic Text Generation Methods. arXiv:2410.03968.