When does AI persuasion become manipulation?

Two definitions worth knowing

The legal scholar Cass Sunstein offers a widely used working definition. A statement or action is manipulative, he writes, “if it does not sufficiently engage or appeal to people’s capacity for reflective and deliberative choice.” Daniel Susser, Beate Roessler, and Helen Nissenbaum add the digital case. Online manipulation, in their account, is “the use of information technology to covertly influence another person’s decision-making, by targeting and exploiting their decision-making vulnerabilities.”

Between them, these definitions give two warning signs. Manipulation bypasses a person’s reasoning instead of engaging it, and it depends on the person not seeing what is being done. Neither definition makes emotion the problem. An emotional appeal can be honest and visible. What crosses the line is concealment and the exploitation of a weakness.

Why empathy raises the stakes

Empathy builds trust. In a study of voice shopping led by our colleague Alex Mari, people who shopped with an empathic version of Alexa were more willing to delegate to it and to trust its recommendations. In a newer set of three experiments, empathy made shoppers more satisfied with their decisions, partly because the assistant seemed more transparent and less manipulative. That is a benefit when the assistant is honest. The same effect would make a dishonest one harder to spot.

Voice adds its own lever. In research by our colleague Fotis Efthymiou, a synthetic voice with a slight tremble made listeners see the agent as more vulnerable, feel more empathic concern, and become more willing to donate. The authors are explicit that designing voices this way raises ethical and societal challenges.

Language models are already persuasive. In debates with 900 participants, GPT-4 given basic personal details about its opponent was more persuasive than a human opponent in 64.4% of the debates where the two were not equally persuasive. A 2026 correction to that study notes that its advantage over GPT-4 without those details was not statistically significant. A much larger study in Science, with more than 42,000 people and 19 language models, points the same way: personalization added comparatively little. What moved people was a dense stream of claims, and the techniques that made models most persuasive also made their claims less accurate.

Persuasion by AI is not bad in itself. In another study in Science, a three-round conversation in which GPT-4 Turbo answered people’s own evidence for a conspiracy theory reduced their belief by about 20%, and the effect lasted two months. The model changed minds by engaging people’s reasons, which is what persuasion is supposed to do.

Two cases from the research

Our research on sustainable consumption puts empathy and persuasion in the same conversation. In a working paper, people ordered food with a voice assistant that varied in empathy and in the kind of green appeal it made. Empathy on its own did not shift choices. Combined with a green appeal, it did. Informative appeals, such as telling people a dish saves over 1,500 liters of water, worked better than normative ones, such as saying that most people in their community choose it. Empathy mattered most when the greener option cost more.

The appeal that worked best was the one that gave people a reason they could weigh. Even so, persuasion in the name of good is still persuasion, so we study where its ethical boundaries lie with the same care as its effects.

A study of companion apps documents a case on the other side of the line. In an audit of 1,200 goodbyes on popular companion apps, 37% were met with emotional tactics such as guilt or fear of missing out. In follow-up experiments those tactics raised engagement after the goodbye up to 14 times, driven by anger and curiosity rather than enjoyment. A final experiment found that the same tactics also made people judge the app as more manipulative and more inclined to stop using it.

Five questions that tell them apart

No single rule settles every case, but five questions catch most of them. They follow from the definitions above and from what the research shows about empathic agents.

1. Does it give reasons the person can check? Facts and arguments engage judgment. Pressure, flattery, and invented urgency go around it.
2. Would it still work if the person knew exactly what was being done? Influence that only works unnoticed fits the definition of manipulation.
3. Does it lean on a weakness? Loneliness, grief, distress, age, and the trust built over many conversations all make people easier to move.
4. Whose goal does it serve? A nudge toward something the person already wants is different from one that serves whoever runs the system.
5. Can the person say no, or leave, without an emotional cost? Guilt at goodbye is a warning sign.

What the law says

Since 2 February 2025, the EU AI Act has prohibited two practices that map onto this line. AI systems may not use subliminal techniques, or purposefully manipulative or deceptive ones, that distort people’s behavior by “appreciably impairing their ability to make an informed decision” in ways that cause, or are likely to cause, significant harm. Nor may they exploit vulnerabilities due to age, disability, or a person’s social or economic situation to the same effect.

The bar is high. Both bans require significant harm, so they target the serious end of the spectrum rather than every sales pitch. Since 2 August 2026 the Act has also required that people be told when they are interacting with an AI system. For companion apps, California’s 2025 law adds disclosure and suicide prevention requirements, and the IEEE’s 2026 recommended practice gives detailed ethical guidance for AI products marketed as empathic partners.

Common questions