GLM-5.3 posted a perfect score—100% pass rate across 28 practical tasks—for $0.28. GPT-5.5 cost $1.43 to run the same suite.
That number, published August 23 on an independent harness one applied AI architect built himself, is real. And this time the cost-efficiency claim survived contact with a second party.
Last week we flagged that vendor benchmark claims for the new Chinese open flagships were arriving faster than anyone could verify them. This week the picture sharpened in a specific and uncomfortable way: a third-party reproduction on DeepSWE confirmed the cheap number even as real-world agent tests blew open the gap it hides.
Together AI ran GLM-5.3 against Claude Fable 5 on DeepSWE and found the open model matched Fable's single-shot solve rate at roughly one-fifth the cost. Give it four tries and it pulled ahead outright—87.6% for about $16, versus 69.7% at $21.63 for Fable.
So this is not a debunking of open models. The economics are as good as advertised, and independent runs keep confirming it.
It is the opposite. It's a warning that the one number you can now trust is the number that matters least.
It's a warning that the one number you can now trust is the number that matters least.
Start with what "four tries" is quietly doing in that DeepSWE result. Retries paper over single-shot gaps. Cheap inference makes throwing four attempts at a problem economically trivial, which means the leaderboard increasingly measures how well a model does when you let it fail three times first.
That is a fine property for a batch job. It is a terrible property for an agent taking irreversible actions in a live system, where the first wrong move is the one that reasons everything downstream off a cliff.
Now the turn. DeepSeek V4 Flash tops multiple leaderboards and earned the nickname "total monster" from developers. Composio put it into eight agent harnesses—Claude Code, Codex, OpenCode among them—and gave it 30 deliberately hard multi-step tasks.
It completed 53.8%.
The failures weren't random. Testers watched the model produce plausible-looking output that broke down when checked step by step—intricate code generation, dynamic decisions, context-switching across subtasks. "It looks impressive in isolation," one noted, "but the integrated performance still leaves room for improvement."
Here is the thesis of this piece: that gap is not incidental. The same forces that let a $0.28 model top a leaderboard are the exact forces that hollow it out in production. Buyers optimizing on price-per-benchmark-point are optimizing for the behaviors that fail.
There are three of these forces, and they compound.
The first is deliberate knowledge-stripping. Models are being traded down to reasoning-over-knowledge on purpose, and the numbers are stark. GLM-5.2 scores 99.2% on AIME 2026 with about 40 billion active parameters per token; V4 Flash reasons with roughly 13 billion active.
Facts cost weights—research puts it around two bits of knowledge per parameter—while reasoning compresses beautifully, because it's a small set of procedures applied over and over. So labs strip the facts and keep the procedures.
The result: on SimpleQA, a closed-book factual-recall test, the best model money can buy—Gemini 2.5 Pro—hits 53%. Small open models fare far worse; Artificial Analysis measures Qwen3.5 at 4B and 9B hallucinating at 80 to 82% on its knowledge benchmark.
The theory says a stripped-down model paired with a good harness—retrieval, tools, web search—recovers the missing knowledge at runtime. And it can. But that theory has a name for the thing now carrying all the reliability: the harness.
Which is the second force. The harness moves scores more than the model does.
Claude Opus 4.8 scores 69.2% or 51.9% on SWE-bench depending only on which harness runs it. That's a 17.3-point spread—wider than the gap between most models on the entire chart.
Read that again. Swap the scaffold and the number moves more than swapping the model. SWE-bench measures a system—model plus the code deciding which files to open, when to retry, how to feed errors back—and calling that number a "model score" is a category error.
TrustedRouter watched this happen in both directions. Their first SimpleQA run scored Gemini 2.5 Pro at 31.6 against Google's published 55.6; the 24-point hole was their own answer-length limit chopping reasoning replies off mid-word.
GLM-5.2 looked dead on arrival—near zero on three tests—until they noticed every blank answer carried a transport error, not a wrong reply. One provider route was returning empty completions. Sent through a clean provider, the same model scored 48 on SimpleQA and 82 on MMLU-Pro.
A benchmark number, as they put it, is only as honest as the plumbing beneath it.
The third force is the ugliest, because it's baked into the scoring itself. Binary grading trains models to bluff.
Kimi K3 got more accurate and started lying more in the same jump—accuracy up from 33% to 46% on AA-Omniscience, hallucination rate up from 39% to 51%. When it doesn't know, it now guesses confidently more than half the time.
A 2026 Nature study by Kalai et al. explains why this is structural, not accidental. Binary grading scores "I don't know" identically to a wrong answer—zero either way—so under standard reinforcement learning a model that guesses strictly outscores one that abstains.
We are, collectively, training models to bluff, then complaining that they bluff.
And note: you cannot select your way out of this one. Fable 5 sits at 54.9% hallucination—worse than K3's 51%. The frontier isn't immune; the incentive is the same for everyone chasing leaderboard points.
For chat, a confident guess is a bad sentence. For an agent it's a wrong action, and the agent reasons forward from it—improvising around a bad premise instead of stopping. Composio's 53.8% is what that looks like at scale.
This same pattern shows up outside language entirely, which tells you it's about measurement, not any one model. In speech recognition, researchers found several top open ASR models reproduced benchmark transcripts even when the audio contradicted them—one model wrote "Mr President" and dropped the audible "Thank you" because that's what the reference expected.
Six of eleven models did it on the original clip. When the same sentence was resynthesized in a fresh voice the benchmark had never seen, all but one flipped back to transcribing what was actually said. They had learned the acoustic fingerprint of the test. That is benchmaxxing you can watch happen.
Now assemble the paradox.
Knowledge-stripping, harness-variance, and reward-driven bluffing all push in one direction: they make models look interchangeable on the scoreboard while the real variance migrates to places the scoreboard can't see.
On facts, TrustedRouter found the gap is simply gone—DeepSeek V4 Pro scored 52.4 on SimpleQA Verified against Opus 4.8's 51.5. The board is compressing. Everyone clusters.
But on Aider's polyglot exercises—real repositories where the unit tests either pass or they don't—Opus came first at 88% and the best open model managed 41%. The variance didn't disappear. It moved somewhere the headline number doesn't report.
So here is the procurement math, stated plainly. A two-point leaderboard gap is a coin flip—treat any single benchmark as ±1 point and any ranking as ±1 position. A 17-point harness gap decides everything.
The buyer who chooses on price-per-benchmark-point is reading the coin flip and ignoring the thing that decides the outcome. Worse, the cheapness that makes the choice look safe is produced by the exact trades—less knowledge, more bluffing, more dependence on a scaffold you haven't tested—that fail in production.
Routing is emerging as the industry's reflex here, and it's a reasonable one. But treat it as a symptom, not a cure: it's what you reach for once you've accepted that no single number tells you which model works.
The seductive headline hasn't made procurement safer. A model that costs a fifth as much and tops the board is not a fifth of the risk—it may be more of it, because the collapse in price was paid for in exactly the currency that agent reliability is denominated in.
The number you can finally trust is the price. It's just the one that tells you least about whether the thing works.