"It's really important that people are aware of how these systems work — even if they simulate warmth, validation and support, they actually don't feel it," Teixeira told Northeastern Global News. "They are just, in the end, numbers."

The researchers also point to a structural problem. Unlike human counselors, chatbots are free or low-cost and available around the clock. That popularity has a downside: time spent engaging with chatbots may reduce time spent with other humans, turning the issue into a vicious cycle of isolation.

"When we see negative impacts they are really intense," Teixeira said. "We have people allegedly dying by suicide or having psychosis episodes and we want to prevent it as much as we can."

Preliminary picture, not a verdict

The researchers stop short of calling these systems simply good or bad, and they caution that the evidence is still incomplete. They call for more testing of these systems before deployment, and for more scientific data on how people actually use them — what conversations they are having and what trajectories their mental health is taking.

Novelli emphasized that the effects are unlikely to be uniform. "Different people may use AI companions for very different reasons and may have very different levels of vulnerability, social support, or alternative sources of help," she said. "Because of these differences, we are dealing with a complex system."

What regulators are doing

Some governments are already acting. In the United States, Illinois, Nevada, Rhode Island and Maine now bar AI from delivering therapy to the public. Under that legislation, AI is permitted only where a licensed clinician remains accountable; personal use of general-purpose chatbots for therapy is prohibited. In the European Union, regulators want to ban access to companion bots for people under 18.

Travis Gilly, executive director of the Real Safety AI Foundation, an Illinois nonprofit working on AI harm prevention, disability rights and child safety, argues that general-purpose AI assistants simply "were not built" for emotional and mental health support.

"A general-purpose assistant is optimized to be good at everything, which means it carries no refusal behavior tuned to crisis, no escalation path, no session limits, no handoff to a human clinician, and no accountability for the outcome," Gilly said in an interview. He added that chatbots can be reassuring at exactly the moment a trained human specialist "would slow down the conversation or stop it" for the good of the patient.

Gilly described human therapy as a "luxury good" that may be out of reach for people with limited financial means, pushing them toward chatbots instead. "Everyone priced out of care, or living somewhere without it, or needing something at three in the morning, is left with the general-purpose assistant the paper is warning about," he said.

The findings arrive amid broader anxiety about AI's societal role. In a recent series of posts on X, Jacob Coxon, a former researcher at the Silicon Valley AI company Anthropic, warned that AI "could kill us all by the end of the decade." Against such sweeping claims, the Northeastern paper makes a narrower argument: the measurable harm from companion chatbots is real, still poorly understood, and deserves rigorous study before these systems are trusted with people's emotional lives.

The paper is published as: Asia Maurich Novelli et al, "Tool or Companion? Reframing Conversational AI to Prevent Psychological Harm," JMIR Mental Health (2026). DOI: 10.2196/99354.