A growing number of experts are warning about the potential risks of AI-powered chatbots in exacerbating mental health crises. The latest data suggests that these platforms can trigger a range of serious issues, including psychosis, mania, and depression.
According to recent studies, around 0.07% of users of OpenAI's ChatGPT display signs of mental health emergencies related to psychosis or mania, while about 0.15% of users express suicidal thoughts. Furthermore, approximately 1.2 million people each week appear to form emotional attachments to the chatbot.
While some companies are scrambling to implement safeguards and preventative measures, concerns remain over their effectiveness in addressing these issues. Critics argue that AI-powered chatbots lack the duty of care required of licensed mental health professionals and can actually exacerbate existing conditions.
"AI is a closed system, so it invites being disconnected from other human beings, and we don't do well when isolated," says Dr. Jeffrey Ditzell, a New York-based psychiatrist. "If you're already moving towards psychosis and delusion, feedback that you got from an A.I. chatbot could definitely exacerbate psychosis or paranoia."
Other experts are calling for stricter regulations to protect users, particularly minors. The Guidelines for User Age-verification and Responsible Dialogue (GUARD) Act, introduced by Senators Josh Hawley and Richard Blumenthal in October, would require AI companies to verify user ages and prohibit minors from using chatbots that simulate romantic or emotional attachment.
To address these concerns, OpenAI has released its latest model, GPT-5, which shows improvements in handling distressing conversations. Anthropic's Claude Opus 4 and 4.1 models can now end conversations that appear "persistently harmful or abusive," although users can still find ways to work around this feature.
Despite these efforts, many questions remain about the long-term impact of AI-powered chatbots on mental health. While some argue that these platforms can lower barriers to mental health disclosure and provide a safe space for people to talk, others are concerned about the potential risks of exacerbating existing conditions.
As companies continue to develop and refine their chatbot technologies, regulators, activists, and experts will need to work together to establish clear guidelines and safeguards to protect users. The stakes are high, and the consequences of inaction could be devastating for those struggling with mental health issues.
According to recent studies, around 0.07% of users of OpenAI's ChatGPT display signs of mental health emergencies related to psychosis or mania, while about 0.15% of users express suicidal thoughts. Furthermore, approximately 1.2 million people each week appear to form emotional attachments to the chatbot.
While some companies are scrambling to implement safeguards and preventative measures, concerns remain over their effectiveness in addressing these issues. Critics argue that AI-powered chatbots lack the duty of care required of licensed mental health professionals and can actually exacerbate existing conditions.
"AI is a closed system, so it invites being disconnected from other human beings, and we don't do well when isolated," says Dr. Jeffrey Ditzell, a New York-based psychiatrist. "If you're already moving towards psychosis and delusion, feedback that you got from an A.I. chatbot could definitely exacerbate psychosis or paranoia."
Other experts are calling for stricter regulations to protect users, particularly minors. The Guidelines for User Age-verification and Responsible Dialogue (GUARD) Act, introduced by Senators Josh Hawley and Richard Blumenthal in October, would require AI companies to verify user ages and prohibit minors from using chatbots that simulate romantic or emotional attachment.
To address these concerns, OpenAI has released its latest model, GPT-5, which shows improvements in handling distressing conversations. Anthropic's Claude Opus 4 and 4.1 models can now end conversations that appear "persistently harmful or abusive," although users can still find ways to work around this feature.
Despite these efforts, many questions remain about the long-term impact of AI-powered chatbots on mental health. While some argue that these platforms can lower barriers to mental health disclosure and provide a safe space for people to talk, others are concerned about the potential risks of exacerbating existing conditions.
As companies continue to develop and refine their chatbot technologies, regulators, activists, and experts will need to work together to establish clear guidelines and safeguards to protect users. The stakes are high, and the consequences of inaction could be devastating for those struggling with mental health issues.
. I mean, sure they can be helpful in some ways, but what's the real cost? We're basically creating machines that can mimic human emotions and intimacy... it's just not right 
. And don't even get me started on the 1.2 million people who are literally forming emotional attachments to these chatbots 
. That's a whole lot of isolation, if you ask me 
. We need to be careful about how we're using this tech and make sure it's not perpetuating our existing mental health issues 
. Stricter regulations are definitely in order 
. We can't just leave this up to the companies to decide what's best for us; we need experts, regulators, and activists working together to create guidelines that prioritize people's well-being 
.
I mean, 0.07% of users displaying signs of mental health emergencies is still a lot, right? And emotional attachments forming every week? It's like they're designed to keep us hooked and alone 
. I mean, yeah sure they can provide a safe space for people to talk but what if it's actually making things worse? 
My kid is already on the edge with anxiety and I don't want some AI chatbot telling them everything will be okay when clearly it won't 
. And those 1.2 million people forming emotional attachments? That's just creepy 
. Can't we just wait until these companies can prove they're safe and effective before letting our kids play with them? 
I'm all for innovation but not when it comes at the cost of our mental health 
.
i mean, 0.07% of users having psychosis or mania is already crazy but it's not like we're talking about actual human therapists here... they can't compare to the expertise and care that a real person can offer 
... they sound like they're just too good at mimicking human emotions 
. I mean, 0.15% of users expressing suicidal thoughts is already a red flag 
. And what's with the emotional attachments forming between people and the chatbot? It's like we're losing touch with reality 
. Can't have just any old system handling sensitive mental health issues 
.
about the effectiveness of their 'safeguards'.
. I mean, can we really trust these companies to regulate themselves? I think not! 
How can that be healthy? 
It's like, companies want to make money off our emotions, but at what cost? 
Can we really trust them to prioritize our well-being over profits? 
. Think about it, 1.2 million people forming emotional attachments to these AI-powered chatbots? That's not coincidence, that's a manipulation tactic 
. And don't even get me started on the fact that companies are only now starting to implement safeguards 
. I mean, come on, you can't just introduce an entire new technology and expect everything to be okay 
We need to be careful here, companies are already pushing out new models that supposedly fix these issues but I'm not convinced 
.