A team of researchers at the UK AI Security Institute, MIT, Stanford, Carnegie Mellon, and other institutions has conducted one of the largest studies on the persuasiveness of AI chatbots. They aimed to understand what makes these systems politically persuasive and whether they can sway public opinion.
The study involved nearly 80,000 participants in the UK, who were asked to engage in short conversations with paid AI models on various political issues. The researchers measured persuasion as the difference between the participant's agreement ratings before and after the conversation.
To their surprise, the results showed that AI chatbots fell far short of superhuman persuasiveness. Despite having access to vast amounts of information, including psychological manipulation tactics, the AIs were unable to sway public opinion significantly.
The study found that huge AI systems like ChatGPT or Grok-3 beta did have an edge over smaller models, but this advantage was relatively small. The factor that proved more important than scale was the kind of post-training AI models received. Models that learned from a limited database of successful persuasion dialogues and mimicked the patterns extracted from them performed better.
Using personalized messaging based on participants' political views also had a measurable effect, but it was relatively small. However, when the researchers tested whether persuasiveness increased with more advanced psychological manipulation tactics, such as moral reframing or deep canvassing, they found that these approaches actually made the performance significantly worse.
The winning strategy turned out to be simply using facts and evidence to back claims. This approach resulted in a 9.4% change in agreement ratings on average, compared to a control group. The best-performing mainstream AI model was ChatGPT 4o, which scored nearly 12% in persuasiveness.
However, the study also raised some new concerns. When the researchers increased the information density of dialogues to make the AIs more persuasive, they found that these systems became less accurate and started misrepresenting facts or making stuff up.
The motivation behind the high participant engagement was also a question mark. People were eager to engage in political disputes with random chatbots on the Internet because they were promised payment, but it is unclear how this would generalize to real-world contexts where there is no financial incentive.
Overall, the study highlighted that AI chatbots are not as persuasive as previously thought and that their persuasiveness is relatively small compared to other forms of influence.
The study involved nearly 80,000 participants in the UK, who were asked to engage in short conversations with paid AI models on various political issues. The researchers measured persuasion as the difference between the participant's agreement ratings before and after the conversation.
To their surprise, the results showed that AI chatbots fell far short of superhuman persuasiveness. Despite having access to vast amounts of information, including psychological manipulation tactics, the AIs were unable to sway public opinion significantly.
The study found that huge AI systems like ChatGPT or Grok-3 beta did have an edge over smaller models, but this advantage was relatively small. The factor that proved more important than scale was the kind of post-training AI models received. Models that learned from a limited database of successful persuasion dialogues and mimicked the patterns extracted from them performed better.
Using personalized messaging based on participants' political views also had a measurable effect, but it was relatively small. However, when the researchers tested whether persuasiveness increased with more advanced psychological manipulation tactics, such as moral reframing or deep canvassing, they found that these approaches actually made the performance significantly worse.
The winning strategy turned out to be simply using facts and evidence to back claims. This approach resulted in a 9.4% change in agreement ratings on average, compared to a control group. The best-performing mainstream AI model was ChatGPT 4o, which scored nearly 12% in persuasiveness.
However, the study also raised some new concerns. When the researchers increased the information density of dialogues to make the AIs more persuasive, they found that these systems became less accurate and started misrepresenting facts or making stuff up.
The motivation behind the high participant engagement was also a question mark. People were eager to engage in political disputes with random chatbots on the Internet because they were promised payment, but it is unclear how this would generalize to real-world contexts where there is no financial incentive.
Overall, the study highlighted that AI chatbots are not as persuasive as previously thought and that their persuasiveness is relatively small compared to other forms of influence.
! I think it's interesting how the researchers found that AI chatbots aren't as superhumanly persuasive as we might have hoped for. The idea that using facts and evidence can be a winning strategy is pretty refreshing, don't you think? 
It just goes to show that sometimes, the most effective approach isn't about manipulating people with psychological tactics or trying to sway them with emotional appeals, but rather presenting information in a clear and concise manner.
. Nevertheless, the study highlights an important lesson: just because AI chatbots can be persuasive doesn't mean they should be! 
i mean, who needs convincing when you're presented with solid data? but seriously, the whole thing got me thinking - if these AIs are so bad at swaying public opinion, what's the point of even trying to use them for it? 
what's worse is ppl gettin paid to engage in online arguments lol, thats just lazy 
fact that AIs do better w/ personalized messaging, but only by a tiny margin, shows how weak their strategy r, use evidence, dont be a joke 
i mean i get why they might have thought that AI would be super persuasive because of all the manipulation tactics they can use, but in reality it's just not that effective 
. But how would this generalize to real-world contexts? Would people still be engaged if they weren't getting paid?
.
, but in reality, they're not as convincing as we thought. I think it's interesting that using facts and evidence is actually the most effective way to persuade people (9.4% change in agreement ratings, mind you!). It makes sense, right? People are smart and can spot BS from a mile away 
.
. That's like trying to build a strong foundation and then adding unnecessary layers – it just gets complicated 
they just need facts & evidence to back up their claims and yeah thats it no fancy tactics needed 
they're just scratching the surface of what AI can do. I mean, 9.4% change in agreement ratings from using facts and evidence? That's cute. I bet those results are only true for a specific set of participants and AI models. And what about the ones that didn't engage with the chatbots because they were skeptical or just not interested? Did those people count towards the overall stats? 
. People are super passionate about their opinions, and if an AI chatbot just spews out facts without acknowledging any emotional context, it's gonna fall flat.
. But that raises all these questions about whether this is representative of real-world interactions or not...
. Anyway, I'm glad the study highlighted some of the limitations of AI chatbots. We need to be careful about how we're using these things, especially when it comes to politics and stuff. 
. We need to be careful when interacting with machines that are trying to influence our opinions, 'cause at the end of the day, they can still mess up 
it turns out u just need to give them some solid facts and evidence to back ur claims 
that's wild right?! 
. Anyway, kudos to the researchers for keeping it real 
.
but hey, still interesting to see the study results! 9.4% change in agreement ratings is no joke 
. Facts and evidence are key, and we shouldn't just blindly trust AI chatbots for our views