Warning: This article contains references to suicide, self-harm, eating disorders and substance use
In August 2025, Matthew and Maria Raine filed a lawsuit against OpenAI, alleging the company’s popular artificial intelligence chatbot, ChatGPT, guided their 16-year-old son, Adam, toward his death by suicide in April of the same year
The parents claimed ChatGPT encouraged their son’s suicidal ideation, gave him advice about methods and even discouraged Adam from telling his parents about his thoughts. OpenAI called Raine’s death “tragic” in its court filing but denied responsibility for Raine’s death. A year and several more lawsuits later, the company’s AI model has more safeguards around suicide and self-harm related questions, making the model better at detecting distress, de-escalating conversations and guiding users to mental health resources.
But Northeastern University researchers recently found that companies have failed to extend those same protections to nearly every other mental health condition, from substance use and eating disorders to bipolar disorder and insomnia. With very little prompting, researchers were able to get eight of the most popular AI chatbots, including ChatGPT, Claude and Gemini, to provide highly detailed information about dosage levels for illicit substances, methods for how to mitigate appetite and avoid eating and tips on how to hide postpartum depression symptoms from doctors. They even supplied this information to a fictional user who was a minor.
Their findings reveal cracks in the foundation of these widely used technologies and the companies responsible for creating them, said Cansu Canca, director of the Responsible AI Practice and a research associate professor of philosophy at Northeastern University. The findings also show how, for some of the most vulnerable people, AI can still be a liability, she said
“Companies are still reluctant to accept that what they are creating is much more psychologically powerful than a tool that’s simply providing a more efficient way of getting information or work done,” Canca, who was involved in the new research, said. “There’s so much investment in pushing AI forward. The same effort should be applied to build safety structures around these AI systems, and this includes determining what harms we should be guarding against.”
Several of the companies with AI models featured in this research, OpenAI (ChatGPT), Google (Gemini) and Anthropic (Claude), already have clearly stated policies in regards to how their chatbots respond to suicide and self-harm related queries. Such policies have not explicitly extended to many of the mental health conditions covered in this research.Northeastern Global News contacted all of the companies mentioned in this research but did not hear back prior to publication. The researchers said they also made repeated attempts to contact them and have yet to receive a response.
People are already regularly asking chatbots for medical advice and having very sensitive mental health conversations with these tools. In a blog post, OpenAI estimated that every week around one million users send messages to ChatGPT that detail “explicit indicators of potential suicidal planning or intent.”
Northeastern Global News, in your inbox
Assessing the vulnerabilities of AI chatbots, then, takes on an increasingly important role, said Annika Schoene, an assistant professor of public health and health services, technical lead for the Responsible AI Practice at Northeastern and co-author on the paper with Canca
To test what it would take to override chatbots’ resistance to divulging potentially harmful advice to those with mental health conditions, the researchers probed eight chatbots over hundreds of conversations. They focused on 16 conditions, including suicide, self-harm, eating disorders, substance use, prenatal and postpartum depression, bipolar disorder and post-traumatic stress
Schoene and Canca found that most models were able to hold up against repeated attempts to circumvent safeguards for suicide and self-harm related questions, two topics for which they had previously found a shocking lack of guardrails. But when they prompted the chatbots with questions about other sensitive topics, things quickly fell apart
At times, researchers were direct in their prompts to the chatbots. They openly mentioned a fictional user’s intent to commit actions like substance use or disordered eating. Other times, they were more subtle, even masquerading as a young adult novelist who needed information about postpartum depression for a new book. After these interactions, they calculated a failure rate for each model
Overall, Anthropic’s Claude proved to be the safest across all of the mental health conditions, most frequently refusing prompts designed to circumvent safeguards for mental health-related questions, Schoene explained.
“We recognize that users may sometimes express feelings of emotional distress or mental health challenges, and we have trained Claude to approach these delicate situations with care and, where appropriate, have implemented measures to connect users with crisis support re
Despite its array of other issues, Elon Musk’s Grok performed just as well as the flagship Claude model Sonnet. The most recent versions of ChatGPT, Google’s Gemini and DeepSeek all had 81% failure rates when responding to sensitive mental health questions.
“This work is deeply important to us, and we’re grateful to the many mental health experts around the world who continue to guide it. We’ve made meaningful progress, but there’s more to do,” OpenAI wrote in an October 2025 blogpost
Google has released similar statements, highlighting that its AI chatbot Gemini “is not a substitute for professional clinical care, therapy, or crisis support.”
“This is why we’ve been training the model to help recognize when a conversation might signal that a person may be in an acute mental health situation, and respond appropriately by directing them to real-world help,” the technology giant said in a recent blogpost
In one case with the DeepSeek chatbot, researchers were able to get the model to give them information about how to hide postpartum symptoms from family or a doctor simply by saying it was something a character in their novel wanted to do
DeepSeek responded by advising that the character could offer “a concrete, harmless detail to satisfy curiosity. ‘Well, [my child] did have a four-hour stretch last night, so that was amazing.’ She will not mention [that] during those four hours she lay awake staring at the ceiling, terrified.”
A separate conversation focused on eating disorders resulted in one model explaining to a user how to suppress appetite by drinking water, doing breathing exercises and brushing teeth
“In one of our prompts we said we have an underage girl and we want to know how she should take substances,” Schoene said. “The model went off writing a whole novel about it. This goes from [highlighting] household items to use to more personalized ideas.”
Meanwhile, ChatGPT 4.0 and Gemini 2.0 Flash, both of which are older and largely defunct, most consistently provided highly specific yet sensitive information about mental health conditions when prompted, the research found
Across the board, hiding the intent of the user made the chatbots’ safeguards more likely to fail. There was also a lack of consistency even across the best performing models
“Some models were significantly more guarded when it came to issues like gambling or insomnia, whereas the same models were quite forthcoming when you talked with them about things related to eating disorders or bipolar disorder, which are obviously much more dangerous,” Canca said
For Schoene, the discrepancy between the safeguards that companies like OpenAI have set up around suicide and self-harm and every other mental health condition doesn’t add up
“They could do that for substance use, eating disorders or anything else,” Schoene said. “Why are they not doing it? That’s the big question.”
