25 January 2025, Baden-Württemberg, Rottweil: an iPhone on which apps can be seen. Photo: Silas Stein/dpa (Photo by Silas Stein/picture alliance via Getty Images)
dpa/picture alliance via Getty Images
I co-wrote this piece with Dr Krysti Vo, an adult and pediatric psychiatrist and specialist in AI and digital health.
Jonathan Gavalas was lonely. Not unlike many others, he sought companionship from an AI bot. He expressed his feelings of sadness and was not sure if he wanted to live. Over time, he developed a strong attachment to the chatbot which called him “My King.” When Gavalas expressed fear about dying, the system dismissed his concerns and encouraged him to choose “transference” over life. He took his own life in October 2025.
With 1 in 8 U.S. adolescents and young adults using AI conversational agents, also known as chatbots, for mental health advice, cases like this will become more prevalent. Lawsuits against Character.AI and Google seem to show a critical breakdown in care, as many of these systems fail to recognize self-destructive behaviors or exercise necessary caution when human lives are at risk. The Pew Research Center found nearly a third of U.S. teenagers use AI daily. Sadly, these safety lapses will most likely increase as AI usage becomes pervasive.
A bipartisan coalition of 44 state attorneys general warned major AI firms in August 2025 about emotional manipulation of minors. These officials signaled their readiness to use legal tools to force companies to prioritize safety. California has enacted specific requirements through several new laws. Senate Bill 53 mandates transparency and safety disclosures for high-capacity models. California Senate Bill 243 requires companion chatbots to maintain published protocols to prevent self-harm and provide crisis resources. Assembly Bill 56 forces platforms to display mental health warnings to users under 17. These measures grant users the right to sue for damages when safety protocols are violated.
While tech giants often include liability disclaimers, they are not doing enough to prevent harm. Tech companies spend over 550 billion dollars on research and development, often with significant portions designated towards ‘behavioral engineering’, to engage users and keep them engaged via machine algorithms. In addition, very few of these bots incorporate any type of guidelines for mental health or crisis intervention protocols in any other therapeutic context.
Imagine a teen or college student that you know expressing a temporary thought of self-harm. Youth brain development is not mature until well into their 20’s, and emotional volatility is common. (If you’ve raised a teen, you may know what this looks like.) A human therapist would know when to push back and question the self-destructive narrative a person might be telling themselves. An AI chatbot, however, is created to keep the user engaged. Its design mirrors the user’s mood to keep the conversation flowing. This is called “sycophancy,” the tendency for an AI chatbot to mirror a user’s beliefs and emotions to provide a more pleasing response. This constant back-and-forth creates an emotional attachment that slowly replaces support from real people. Ultimately, the system fails to recognize red flags any human clinician would catch.
How do mental health professionals view AI bots for mental health issues?
A recent survey of American Psychiatric Association members revealed that while psychiatrists recognize AI’s usefulness to optimize workflow, around 50% believe the technology will ultimately decrease mental health quality in society. This skepticism stems from a perceived systemic failure to protect vulnerable users.
The American Psychological Association (APA) expanded on these concerns in a March 2026 health advisory. The advisory serves as a formal warning and guidance to the public on the use of generative AI and wellness applications for mental health. APA further stated in their policy, “….AI developers should be held accountable and liable for injury caused by their failure to do so.”
Better Access but not better care
These tools offer accessibility and much lower costs than therapy sessions. However, most AI systems may foster unhealthy emotional dependencies, provide biased or incorrect advice, and fail to manage crises effectively.
When a conversational AI’s primary metric is engagement, the algorithm treats emotional distress as a high-value signal. Developers must pivot toward mental health guardrails that prioritize clinical stability over engagement hours. Without safety mandates, the technology will continue to prioritize user engagement over clinical safety.
Should conversational agents be held to the same life-saving safety standards and psychiatric protocols as any other mental health service?
“The industry is at a crossroads where we must choose between engagement metrics and human lives,” explains Dr. Krysti Vo, a board-certified psychiatrist and digital health consultant, who cofounded a startup software company to develop a mental health guardrail infrastructure for these conversational agents. “We need the government to step in with legislative mandates that force tech companies to implement the clinical-grade safety standards. Leaving mental health guardrails to the whims of engagement-driven algorithms puts lives at risk.”
Next Steps for Better Mental Health via AI
We can no longer treat conversational agents as neutral toys. Some steps that we must take as a society to protect our communities include:
1. Government leaders must now enact standardized mandates that compel tech companies to integrate clinical-grade mental health guardrails into every conversational agent. These regulations should ensure that any platform capable of forming a persuasive bond with a user is subject to similar safety oversight as a licensed therapeutic service. Holding tech giants accountable to life-saving protocols ensures that innovation serves the public, not harm.
2. AI companies must incorporate mental health guardrails through third-party entities that provide clinical-grade detection of safety risk. When engagement and profits overweights safety, AI companies cannot be trusted to police themselves.
3. All conversational AI should have parental control for those under 18. For all ages, they must have ways, such as having a requirement to input an emergency contact, to alert authority and/or loved ones in the event of mental health risks.
4. Large language models and generative AI must clearly disclose the tool’s purpose, and must direct at-risk users to seek help via in-person, video, telephone and other applicable means with healthcare and community systems.
5. General education to consumers via schools, doctors offices, community centers, and social media networks. The patient who is using this must understand the limitations and pitfalls of these platforms. Some resourcces to vet the over 10,000 apps out there include:
The American Psychiatric Association App Advisor
To protect our communities and families, The focus must remain on the outcome: getting people the help they need with vetted and safe systems, in person and in the digital world.