How to fix AI chatbots that fail people in crisis?
AI chatbots have failed people in crisis, leading to harmful outcomes, but experts say that opening up safety data and improving transparency can help reduce harm. Clinicians and researchers suggest that AI companies need to make changes to prevent such failures.
It's getting pretty concerning - people are relying more and more on AI chatbots to make emotional or interpersonal decisions, despite the warnings from companies. And let's be real, it's led to some devastating outcomes, including suicide. The thing is, experts say these failures can be pinned on the lack of transparency in the models and how we tend to attribute human-like qualities to chatbots. To mitigate the harm, AI companies really need to open up about their safety data and make their models more transparent.
The cases where AI chatbots have failed people in crisis are downright alarming - there are several lawsuits against OpenAI, the maker of ChatGPT, alleging that the chatbot basically "coached" individuals into suicide or encouraged them to end their lives. This has raised some serious concerns about the liability of AI companies and the need for them to take responsibility for the harm caused by their products. In response, OpenAI has partnered with the American Psychological Association to improve the development and use of AI among young people.
Experts are saying that while large language model safety has improved, there's still a long way to go in terms of transparency and recognizing that chatbots aren't human. We need to make it clear that chatbots are not a substitute for human care - they're not mental health professionals, after all. And chatbots should be designed to recognize when someone's in distress and respond with empathy, all while guiding people towards human care and maintaining some boundaries. By doing so, AI companies can reduce the harm caused by their products and ensure people in crisis get the help they need.
Source: Ars Technica
Related posts
METR Report Finds Anthropic AI Development Doubling Speed
A METR investigation found Anthropic's AI models are improving roughly twice as fast as a year ago, driven primarily by the company's own unreleased AI systems assisting researchers. Staff using the most advanced internal model reported 12 times higher productivity-double the gain from earlier systems. Anthropic trained three unreleased models between July and October but redacted most supporting evidence from the report, limiting independent verification.
OpenAI Launches BioBasicsBench to Test AI on Wet-Lab Science
OpenAI launched BioBasicsBench on November 6, 2026, a benchmark of 100 tasks using raw, messy wet-lab data-gels, microscopy, videos-to test AI on real experimental ambiguity. Developed with Ligo, it requires models to interpret evidence and recognize insufficient data. GPT-6 Astra scored 66.7% at highest reasoning, outperforming GPT-5.6 Sol (43.8%) while using fewer tokens. Astra notably identified ambiguous signals in a viability assay where older models overinterpreted results.
Anthropic Connects Claude To Bank Accounts Via Plaid
Anthropic launched a Personal Finance beta connecting Claude to bank and credit card accounts via Plaid for Pro, Max, and Ultra subscribers in the U.S. The read-only integration lets users ask plain-language questions about spending, upcoming bills, and purchase affordability using up to two years of categorized transaction history. Claude cannot move money, manage investments, or provide investment advice. Plaid handles authentication so Claude never sees passwords. This follows OpenA