Do AI reply suggestions change how people communicate?
Yes. Smart replies made chats faster and more positive and improved how partners rated each other, but suspected AI use was judged negatively.
Two randomized experiments with a pre-LLM smart-reply system.
experiment0 of 1 tested with LLMsEvidence from 2021How we rateOffer suggestions that save time, and let users personalise them before sending.
- Expect suggestions to shift tone toward positive language.From the study
- Make suggestions easy to edit so messages stay personal.Our reading
- Be careful with disclosure in personal contexts; suspected AI use carried a penalty.From the study
- Measure partner perception, not just reply speed.From the study
What they found
Hohenstein et al. (2021)experiment
Two randomized experiments (1,036 participants; Study 1: 438 MTurk workers) where pairs chatted with or without a commercially deployed smart-reply system. Using suggested replies made communication more efficient, with more positive language and more positive evaluations by partners. But people were evaluated more negatively when partners suspected them of using AI replies.
“people are evaluated more negatively if they are suspected to be using algorithmic responses”
Hohenstein et al., 2021 · Abstract
Trade-offs and when it doesn't apply
- Pre-LLM smart replies in lab chats.
- Effects on long-term relationships not measured.
- Positive tone may not suit every message.
What it looks like
How to check it in your product
Survey message recipients on perceived warmth and authenticity with and without suggestions.
Open questions
- Do LLM-written messages carry a bigger suspicion penalty?
- Should AI-assisted messages be disclosed?
- Does constant positive phrasing flatten communication?
FAQ
Do smart replies improve conversations?
Hohenstein et al. (1,036 participants) found more efficiency, more positive language and better partner evaluations.
Is there a downside?
People were evaluated more negatively when suspected of using AI replies.
Sources
- Artificial intelligence in communication impacts language and social relationshipsJess Hohenstein, Dominic DiFranzo, Rene F. Kizilcec, et al. (2021). arXiv:2102.05756v1 (2021); later in Scientific Reports.