Should AI show its reasoning or explanations to users?
Only with care: LLM explanations make checking faster, but people over-rely on them when they're convincingly wrong. Showing both sides reduces that.
Promising, tested in lab or early LLM studies
Test it with your users.
One controlled LLM experiment (80 people) plus large pre-LLM experiments.
2 studies · 1 with LLMs · 2021–2024 · How we rate
Show reasoning that helps people check, and present both sides on uncertain claims.
- Pair explanations with the source passages, especially for high-stakes claims.From the study
- Offer contrastive explanations: why it might be true and why it might be false.From the study
- Don't treat a fluent explanation as evidence; it raised acceptance even when wrong.From the study
- Let users expand reasoning on demand instead of showing it by default.Our reading
What they found
Si et al. (2024)experiment
80 crowdworkers fact-checked claims using ChatGPT explanations, search results, both, or nothing. On hard claims, accuracy was similar with explanations (74%) and retrieved passages (73%), vs. 59% with no AI, and explanations were much faster (1.01 vs. 2.53 minutes per claim). But people over-relied on wrong explanations. Contrastive explanations (why true and why false) reduced over-reliance but didn't beat search.
“they over-rely on the LLMs when the explanation is wrong”
Si et al., 2024 · Abstract
Bansal et al. (2021)experiment
1,626 crowd participants across three tasks (sentiment of beer and book reviews, and LSAT logic questions), working with an AI about as accurate as people alone. Human+AI teams beat either alone, but explanations didn't add to that; they raised acceptance of the AI's answer whether it was right or wrong.
“Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness.”
Bansal et al., 2021 · Abstract
Trade-offs and when it doesn't apply
- Contrastive explanations reduced over-reliance but didn't beat plain search.
- Showing search results plus explanations added no benefit over search alone.
- Tested on claim verification with crowdworkers.
What it looks like
How to check it in your product
Seed wrong answers with convincing explanations; measure how often users accept them versus when sources are shown.
Open questions
- Do chain-of-thought style traces behave like these explanations?
- Can explanations flag their own weak steps?
- Which tasks justify the slower source-reading route?
FAQ
Are LLM explanations faster than search for fact-checking?
Yes. In Si et al. (80 participants), explanations took 1.01 minutes per claim vs. 2.53 with retrieved passages, at similar accuracy (74% vs. 73%).
What reduces over-reliance on LLM explanations?
Contrastive explanations that argue both why a claim is true and why it is false, though they didn't outperform search.
Sources
- Large Language Models Help Humans Verify Truthfulness—Except When They Are Convincingly WrongChenglei Si, Navita Goyal, Sherry Tongshuang Wu, et al. (2024). arXiv:2310.12558v2 (Apr 2024).
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team PerformanceGagan Bansal, Tongshuang Wu, Joyce Zhou, et al. (2021). CHI 2021 (arXiv 2020).