On the Conversational Persuasiveness of GPT-4

This research article presents a controlled, preregistered study examining the conversational persuasiveness of the large language model GPT-4 compared to human opponents in structured online debates involving 900 participants. The core experiment tested the efficacy of personalization, where the opponent was provided with sociodemographic data to tailor their arguments.

The central finding demonstrated that a personalized GPT-4 opponent was significantly more persuasive than a human counterpart, resulting in an 81.2% relative increase in the odds of shifting a participant's opinion toward the opponent's view. Specifically, the personalized AI was found to be more persuasive 64.4% of the time compared to human debaters, indicating that GPT-4 was able to leverage the personal information more effectively than humans themselves.

These results highlight the potent persuasive capabilities of LLMs when utilizing microtargeting and raise serious concerns about the potential for AI-driven manipulation on a massive scale.

On the conversational persuasiveness of large language models: A randomized controlled trial

Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti, Robert West