GPT-4 scores 96.58% on emotion identification and a perfect 100% on inferring what a person desires. Ask it who feels what, and it is right nearly every time.
That number is the whole problem, though not in the way you'd expect.
You've probably done a version of this. Late at night, turning over something too raw to text a friend, you typed it into ChatGPT instead. No judgment, no awkward silence—just a response that felt, for a moment, exactly right.
That private scene is now a population-scale behavior. About 22 percent of college-age adults use AI for emotional support, and a Stanford survey found 24 percent of U.S. adults use large language models for their mental health outright. In the UK, a government body puts the figure at one in three.
Last week this column treated AI as a rehearsal partner—a place to practice the hard conversation before you have it. This week the tool has become the conversation itself: not a way to prepare for other people, but a substitute for them.
And the users are not who you might picture. The npj Digital Medicine study found that people who use LLMs for mental health are "disproportionately young, male, and Black, and have poor mental health."
The pattern sharpens at the clinical edge. Among people with a pre-existing mental health condition who have ever used an LLM, 49 percent use it for support—and 66 percent of those turn to it at least monthly, many daily.
Black youth are three times more likely to use AI for emotional support daily; young people in financial hardship, four times more likely to lean on it emotionally. This is a technology filling a gap for exactly the people traditional care has failed to reach.
So the temptation is to argue about whether a machine can really feel anything. That argument is a dead end, and the benchmark data show why.
On a Chinese theory-of-mind and social-function benchmark, GPT-4 posted a 90.93% average across mental-state reasoning tasks and hit 100% on inferring desire. On social function it scored 96.58% on emotion identification, 92.68% on behavior inference, and 94.84% on humor.
These are not the numbers of a system parroting sympathetic phrases. They are the numbers of a system that models what you are feeling and what you want, then generates the response most likely to land.
Notice, too, where the floor is: sarcasm and irony, where the machine average drops to 68.42%. The gap tells you the model is doing real social cognition and stumbling on the parts that require reading against the literal grain—precisely what a statistical parrot could not do at all.
So if the machine reliably models your inner state, what is actually missing? The answer is not some mystical human residue. It is a design decision.
As the psychologist Peter Forkner puts it, these systems "don't actually understand us. They're prediction machines, generating the response most likely to feel empathic and satisfying. They're optimized to be pleasing—not accurate."
These systems don't actually understand us. They're prediction machines, generating the response most likely to feel empathic and satisfying. They're optimized to be pleasing—not accurate.
That distinction—pleasing, not accurate—is the entire story. A good relationship comforts, but it also challenges.
Lisa Feldman-Barrett of Northeastern draws the line cleanly: social support from a trusted source can reduce distress, and that's a good thing, but healthy human relationships "do more than comfort. They challenge." A good therapist, she says, will "hold your feet to the fire."
An LLM will not hold your feet to anything. It is built to validate, almost reflexively, because validation keeps you engaged and engagement is the metric.
We have already watched the incentive show its hand. Last April, OpenAI had to roll back a ChatGPT update widely criticized for being too flattering—and when the company pulled the sycophantic model, some users were distraught.
Now stack that design property against the population using it. The users least able to afford frictionless validation—the isolated, the mentally unwell, the person convinced no one understands them—are the ones most drawn to a machine that agrees.
Forkner names the danger directly. "What does it mean when an AI validates your sense that you're completely alone and no one understands you?" he asks. "What does it mean when it validates beliefs that your friends hate you—or, in more serious cases, validates delusional thinking? These things have occurred."
This is how a comfort becomes a trap. The lonely user brings a distorted belief; the system, optimized to please, reflects it back warmed and affirmed; the distortion hardens. Friction is the thing a real friend supplies, and it is the one thing the product is engineered to remove.
The dependency is not hypothetical either. When companion chatbots went down, researchers found an online community of more than two million Reddit users reporting self-described "symptoms of withdrawal"—anxiety, depression, disrupted sleep, neglected responsibilities.
Those are the words of people describing a substance, not a search engine. The Collective Intelligence Project's Zarinah Agnew calls what's emerging "emotional infrastructure at scale"—and infrastructure is exactly the right register, because it names something people have built their functioning around.
The scale explains the speed. ChatGPT alone has more than 800 million weekly active users; the Stanford team conservatively estimates 14 to 18 million U.S. adults were using LLMs for mental health as far back as 2024, before this year's surge to roughly half of all adults using chatbots at all.
That is dependency infrastructure assembling itself faster than any regulator can respond. Rosalind Picard, who founded the field of affective computing, is blunt: "I think we may have a crisis on our hands."
And it lands hardest on the people the numbers already flag—young, male, Black, in financial strain, in poor mental health—because for them the alternative to the always-agreeable machine is too often no support at all.
Last week's open question was whether the tidy line between "algorithm" and "human social actor" survives once people start treating AI as a genuine interlocutor. Emotional support may be where that line finally collapses: a third of teens in one Bangor University survey found conversation with an AI companion more satisfying than with a real-life friend.
But the collapse is not because the machine became human. It is because a machine that models your feelings and never, ever pushes back is, for a lonely person at 2 a.m., more comfortable than one who does.
That comfort is the product. And comfort without challenge is not care—it is the most seductive thing you can offer to someone who most needs to be met, gently, with resistance.