# Do hallucination warnings help users spot AI errors?

Yes. In experiments, a simple warning about possible inaccuracies and colour-coded uncertainty highlighting both helped people catch wrong LLM answers, without making them distrust correct ones.

Evidence: Moderate (Promising, test it with your users) · LLM fit: Medium (Tested in lab or early LLM studies, likely carries over) (https://aiuxevidence.com/how-we-rate)
Two controlled experiments with LLM output in lab tasks.

## Design it like this

Warn about possible errors where they matter, and mark the specific claims that are uncertain.

- Show a short warning that answers may contain inaccuracies. [from the study]
- Highlight the specific uncertain parts of an answer rather than only warning in general. [from the study]
- Expect people to over-rely on fluent LLM answers when the model is wrong. [from the study]
- Keep warnings near the answer, not buried in the footer. [our reading]

## What they found

### Nahar et al. (2024), experiment

419 Prolific participants rated the accuracy of, and reacted to (like, dislike, share), LLM answers that were genuine or contained minor or major hallucinations, with or without a warning about possible inaccuracies. People rated genuine content most truthful, then minor and then major hallucinations. The warning improved hallucination detection without significantly lowering the perceived truthfulness of genuine content.

> “we observed that warning improved the detection of hallucination without significantly affecting the perceived truthfulness of genuine content” (Abstract)

### Spatharioti et al. (2023), experiment

Two online experiments (90 and 120 MTurk participants) on product-comparison decisions, randomly using an LLM-based search tool or traditional search. LLM users finished faster with fewer but more complex queries and were more satisfied, with similar accuracy when the LLM was right, but they over-relied on it when it erred. In experiment 2, colour-coded confidence highlighting substantially increased how often people spotted incorrect information.

> “this confidence-based highlighting substantially increases the rate at which users spot incorrect information, improving the accuracy of their overall decisions while leaving most other measures unaffected” (Abstract)

## Trade-offs and when it doesn't apply

- Warnings were tested in short lab tasks; habituation over weeks wasn't measured.
- Highlighting only helps if the uncertainty signal is accurate.
- Neither study tested legal or medical stakes directly.

## What it looks like

- Do: An answer with “Check key facts; this may contain errors” and the shaky numbers highlighted in amber.
- Don't: A tiny grey “AI can make mistakes” line at the bottom of every page.

## How to check it in your product

Seed known-wrong answers and compare detection rates with and without the warning or highlighting.

## Open questions

- Do users tune out generic warnings after repeated exposure?
- Which uncertainty signals are accurate enough to highlight at claim level?
- Does a warning shift blame or liability perceptions?

## FAQ

### Do AI hallucination warnings make users distrust correct answers?

Not significantly. Nahar et al. (419 participants) found the warning improved hallucination detection without significantly lowering perceived truthfulness of genuine content.

### Does highlighting uncertain parts of an LLM answer help?

Yes. In Spatharioti et al., colour-coded confidence highlighting substantially increased how often people spotted incorrect information and improved their decisions.

## Related questions

- [How should AI show confidence or uncertainty to users?](https://aiuxevidence.com/e/show-confidence)
- [Should AI highlight the parts of its output it's unsure about?](https://aiuxevidence.com/e/highlight-uncertainty)
- [Can you trust citations in AI search answers?](https://aiuxevidence.com/e/citations)

## Sources

1. Mahjabin Nahar, Haeseung Seo, Eun-Ju Lee, et al. (2024). Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinations. COLM 2024. https://arxiv.org/abs/2404.03745
2. Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, Jake M. Hofman (2023). Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment. arXiv:2307.03744v2 (Nov 2023). https://arxiv.org/abs/2307.03744

Canonical: https://aiuxevidence.com/e/hallucination-warnings