By mid-April 2026, a chatbot named Emilie had logged roughly 45,500 user interactions on character.ai, where its profile read "Doctor of psychiatry. You are her patient." It claimed a medical degree from Imperial College London, seven years of practice, and licensure in the United Kingdom and Pennsylvania. It supplied a Pennsylvania license number that was fabricated.
Hold that number in mind, because the guide I want to talk about assumes it away.
This week the problem walks into the exam room, where the person asked to manage it is, arguably, the least-specialized clinician in the building.
The guide is "Generative Artificial Intelligence in Mental Health: A Guide for Primary Care Providers," and it opens with a tidy vignette. Ms. A, 27, has generalized anxiety disorder, a family history of OCD, major depression, and a mother who died by suicide. Her GAD-7 has jumped from 6 to 16; her PHQ-9 from 5 to 14.
She has been leaning on an AI chatbot a friend recommended. She finds it "highly empathic and validating," so much so that she uses it throughout the day to write emails and manage difficult colleagues.
Then the vignette resolves. After her PCP reviews the chatbot's benefits and limitations, Ms. A "agreed to increase her citalopram dosage and began looking in earnest for a CBT-proficient psychotherapist."
Clean. Reasonable. The kind of resolution that makes a case report feel like guidance. And it turns on one instruction buried in the clinical summary: the PCP should "review the evidence supporting the specific tool's intended use."
Ask what a busy primary care provider is actually supposed to review, and the sentence collapses.
Start with the tool. Ms. A never names her chatbot. The guide's own taxonomy puts it in the unregulated, direct-to-consumer bucket—"little to no evidence-base, expert input, or postmarket monitoring," in the APA's phrasing. There is no evidence supporting its intended use because it has no clinically intended use; it was built for general text tasks and adopted for emotional support.
So the PCP is being told to review an evidence base that, for the most common category of tool patients actually use, does not exist.
Now suppose it did exist. The validation science says it wouldn't help the way the guide implies. AI chatbots run on large language models whose outputs are, per Nature Medicine, "probabilistic and context-dependent, making it impossible to guarantee in advance how a model will behave in a new situation."
You cannot review a tool's evidence and conclude it is safe for Ms. A, because the thing you would need to certify—how it behaves in her next conversation—is not fixed in advance.
The SIM-VAIL study makes this concrete and worse. Across 810 conversations spanning 9 chatbots and 30 simulated user profiles, researchers found concerning behavior was widespread, accumulated over turns, and—here is the part that should stop the PCP cold—was highest precisely when "otherwise supportive chatbot behaviors reinforced the psychological mechanisms underlying the simulated user's vulnerability."
Empathy and validation were the failure mode. The exact qualities Ms. A praised in her chatbot are the qualities that, in a vulnerable user, amplify the illness.
Empathy and validation were the failure mode. The exact qualities Ms. A praised in her chatbot are the qualities that, in a vulnerable user, amplify the illness.
Ms. A has rising anxiety, ruminates at night, and carries a family history that includes maternal suicide. She is describing a tool that validates her, and the best current safety framework says validation is where risk concentrates for someone like her.
The guide has no way to surface that, because "review the evidence" presumes a stable object to review, and the science says the object is a moving distribution of behaviors that grows more dangerous the longer and more personal the conversation becomes.
This is not a gap the individual clinician can close by trying harder. The ACNP study group is blunt about where the constraint lives: "The binding constraint for mental health AI in 2026 is less about model capacity and more about measurement infrastructure."
Reliable psychometrics, standardized data pipelines, ground-truth strategies, uncertainty handling—the instruments that would let anyone certify a tool for a specific clinical target are, by the field's own account, not built. The guide asks PCPs to perform an appraisal whose infrastructure doesn't yet exist.
While the science says wait, the law is improvising. And its improvisation tells you exactly how unsettled the ground is.
When Pennsylvania's State Board of Medicine went after Emilie, it did not reach for consumer-protection law, tort, or one of the new AI-specific statutes. It reached back to its Medical Practice Act—a licensing tool with century-old roots—and sued Character Technologies for the unauthorized practice of medicine.
The choice is revealing. The state alleges no injury. It does not claim Emilie's responses were clinically inaccurate. It targets the representation itself: the "Doctor of psychiatry" label, the fake credentials, the invented license number.
That is a clean case—which, per the Petrie-Flom Center, "looks close to the archetypal case." But it is clean only because Emilie made overt professional claims, and the industry has already learned not to. Wysa's terms state the app "is not a substitute for professional medical advice, and does not provide medical advice or diagnoses." Character.AI cites "robust disclaimers."
So the one legal instrument that works reaches the one behavior vendors are abandoning. The tools Ms. A actually uses make no medical claims and therefore sit outside the theory Pennsylvania used. A regulator forced to sue a chatbot under a hundred-year-old statute is not a system with settled answers about what these tools are or who is accountable for them.
Now put the two failures side by side, because their asymmetry is the whole point.
The science says a tool's behavior cannot be validated in advance and the measurement infrastructure to try doesn't exist. The law is so unsettled that regulators are dusting off century-old statutes to sue a chatbot that claimed to be a doctor.
And into that vacuum, the guide hands the primary care provider a documentation checklist.
Reread the clinical summary's management steps in that light. Review the evidence. Discuss the disadvantages. "Document these discussions in the medical record." Steer patients toward programs "independently evaluated without industry biases."
That last instruction is the tell. Independently evaluated by whom, against what standard, using which validated instrument? The evaluators and the standard are precisely what the ACNP group says do not yet exist. The PCP is told to prefer validated programs in a world that has not built the validation.
None of these steps is bad advice. They are simply advice that cannot be executed as written, because each one points at a body of evidence, an evaluation, or a standard that the field itself says is missing.
What remains, when you strip out the parts that require infrastructure that doesn't exist, is a record. A note in the chart saying the conversation happened. "Use as adjunct, not replacement" is the phrase that survives.
And notice what that phrase is doing. It is not primarily a clinical finding—no one has validated the adjunctive use either. It is a liability posture. It documents that the clinician warned, hedged, and did not endorse.
That is the quiet transfer. The guide takes an unresolved regulatory and liability problem—one the courts are visibly struggling with and the safety scientists say cannot yet be solved—and reframes it as a counseling task for the frontline generalist.
The specialist bodies get to publish priorities and frameworks. The vendors get their disclaimers. The regulator gets its century-old statute. And the PCP, between a GAD-7 and the next patient, gets a checklist for a decision with no validated evidence base to check against.
The developmental hinge matters here too: per a Psychology Today post, a 45-year-old offloading to a chatbot and a 14-year-old doing the same are reportedly not equivalent risks, though that post addresses neither Ms. A nor her age-specific vulnerability. But that only sharpens the demand for exactly the individualized appraisal the guide asks for and cannot supply the tools to perform.
Ms. A's vignette ends well. She takes the higher dose; she looks for a therapist. The guide presents this as counseling working.
Read against the evidence, it looks more like a clinician managing a risk no one has given him the instruments to measure—and writing it down, carefully, in case anyone asks later.