How should a chatbot recover when it misunderstands?
Offer options and explain what went wrong: people preferred repair strategies that showed initiative and gave a clear way forward.
One scenario-based preference study with pre-LLM chatbots.
experiment0 of 1 tested with LLMsEvidence from 2019How we rateWhen a chatbot misunderstands, offer concrete options and briefly explain why.
- Offer a few likely interpretations as options.From the study
- Explain briefly why the bot didn't understand.From the study
- Show initiative rather than asking the user to rephrase.From the study
- Keep an easy route to a human or a manual path.Our reading
What they found
Ashktorab et al. (2019)experiment
Scenario-based study with 203 Mechanical Turk workers comparing eight chatbot repair strategies after a misunderstanding, including confirmation, offering options and explaining the system. Providing options and explanations were generally favoured, because they showed initiative and gave an actionable way to recover.
“We found that providing options and explanations were generally favored, as they manifest initiative from the chatbot and are actionable to recover from breakdowns.”
Ashktorab et al., 2019 · Abstract
Trade-offs and when it doesn't apply
- Scenario-based preferences, not real behaviour.
- Pre-LLM chatbots.
- Options can be wrong too.
What it looks like
How to check it in your product
Track recovery rate after breakdowns: share of conversations that continue successfully after a misunderstanding.
Open questions
- Do these preferences hold for LLM chatbots that rarely fail to parse but often fail to understand intent?
- How should repair work in voice?
- When should the bot hand off to a human?
FAQ
What is the best chatbot repair strategy?
In Ashktorab et al. (203 participants), providing options and explanations were generally favoured.
Why do users prefer options?
They show initiative and give an actionable way to recover.
Sources
- Resilient Chatbots: Repair Strategy Preferences for Conversational BreakdownsZahra Ashktorab, Mohit Jain, Q. Vera Liao, Justin D. Weisz (2019). CHI 2019.