# Is an AI chatbot better than traditional search for decisions?

Faster and more satisfying, with similar accuracy when the AI is right, but people follow it when it's wrong. Add uncertainty cues or keep sources one click away.

Evidence: Moderate (Promising, test it with your users) · LLM fit: Medium (Tested in lab or early LLM studies, likely carries over) (https://aiuxevidence.com/how-we-rate)
Three controlled experiments with LLM tools in lab tasks.

## Design it like this

Use chat for speed, and pair it with sources and uncertainty cues for decisions.

- Expect faster task completion and fewer queries with LLM search. [from the study]
- Counter over-reliance with confidence highlighting on uncertain facts. [from the study]
- Keep source passages available; explanations alone can mislead. [from the study]
- Encourage follow-up queries on surprising answers. [our reading]

## What they found

### Spatharioti et al. (2023), experiment

Two online experiments (90 and 120 MTurk participants) on product-comparison decisions, randomly using an LLM-based search tool or traditional search. LLM users finished faster with fewer but more complex queries and were more satisfied, with similar accuracy when the LLM was right, but they over-relied on it when it erred. In experiment 2, colour-coded confidence highlighting substantially increased how often people spotted incorrect information.

> “this confidence-based highlighting substantially increases the rate at which users spot incorrect information, improving the accuracy of their overall decisions while leaving most other measures unaffected” (Abstract)

### Si et al. (2024), experiment

80 crowdworkers fact-checked claims using ChatGPT explanations, search results, both, or nothing. On hard claims, accuracy was similar with explanations (74%) and retrieved passages (73%), vs. 59% with no AI, and explanations were much faster (1.01 vs. 2.53 minutes per claim). But people over-relied on wrong explanations. Contrastive explanations (why true and why false) reduced over-reliance but didn't beat search.

> “they over-rely on the LLMs when the explanation is wrong” (Abstract)

## Trade-offs and when it doesn't apply

- Lab tasks with crowdworkers.
- Accuracy was similar only when the LLM was correct.
- Combining search results with explanations added nothing in Si et al.

## What it looks like

- Do: A chat answer with source links and highlighted uncertain specs.
- Don't: A single confident chat answer with no way to check.

## How to check it in your product

Compare decision accuracy on items where the AI is known to be wrong, chat vs. search.

## Open questions

- Does the speed advantage hold for complex research tasks?
- How do people decide when to switch from chat to search?
- Do highlights work in voice interfaces?

## FAQ

### Is LLM search faster than Google-style search?

In Spatharioti et al., LLM search users completed tasks more quickly with fewer but more complex queries.

### Is LLM search as accurate?

When the LLM was right, yes; when it erred, users over-relied on it.

## Related questions

- [Can you trust citations in AI search answers?](https://aiuxevidence.com/e/citations)
- [Do hallucination warnings help users spot AI errors?](https://aiuxevidence.com/e/hallucination-warnings)
- [Should AI show its reasoning or explanations to users?](https://aiuxevidence.com/e/llm-explanations)

## Sources

1. Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, Jake M. Hofman (2023). Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment. arXiv:2307.03744v2 (Nov 2023). https://arxiv.org/abs/2307.03744
2. Chenglei Si, Navita Goyal, Sherry Tongshuang Wu, et al. (2024). Large Language Models Help Humans Verify Truthfulness—Except When They Are Convincingly Wrong. arXiv:2310.12558v2 (Apr 2024). https://arxiv.org/abs/2310.12558

Canonical: https://aiuxevidence.com/e/chat-vs-search