Here is a contradiction worth sitting with. Anthropic pitched Sonnet 5 as Opus-level quality at Sonnet prices, and the benchmark data mostly backs that story—it lands within a few points of Opus 4.8 on the coding tests that vendors love to cite. Then Claire Vo built a bench in 45 minutes to check it against her own work, ran it blind against four other models, and Sonnet 5 lost.
"The results were not what I expected," she said. That gap—between the vendor's story and what happens when you measure your own workload—is the entire argument for building your own eval harness.
Last week, the frame here was that model releases have become weekly per-task decisions—you don't upgrade, you re-decide which model does which job. This week, the practice has hardened. Practitioners aren't running vibe checks anymore. They're publishing repeatable, taste-encoded harnesses. Claire's is called the How I AI Bench, and the way she built it is the whole point.
She did it live in Claude Code, in under 45 minutes, starting from her own stored session history. Every Claude Code session on your machine is saved locally, so she pointed the model at her actual past work and asked it to brainstorm an eval set relevant to her workflows. Frozen inputs. Blind scoring. A rubric. Then a deliberate weighting choice: 70% human vibe, 30% LLM-as-judge.
That split is not a technicality. It's a statement about what a leaderboard can't hold. Claire's phrase for it: she didn't want to "lose that Clairvo taste" by handing scoring entirely to an AI judge. The bench exists to encode her point of view about what good looks like—and to do it repeatably, so she can compare across models and over time instead of re-litigating a gut feeling every release.
The bench exists to encode her point of view about what good looks like—repeatably—so she can compare across models instead of re-litigating a gut feeling every release.
Now here is why the vendor number can't do this job for you, structurally, no matter how honest the vendor is trying to be.
Public benchmarks are contaminated. MMLU, HumanEval, GSM8K—these are published, discussed on Reddit, folded into training data. Models have almost certainly seen them. When a vendor reports 86.8% on MMLU, you can't tell whether the model learned the concept or memorized the test.
And the incentive on top of contamination is worse. Analysis of the Chatbot Arena leaderboard found that selective disclosure could inflate reported scores by up to 112%—companies privately test many variants and publish only the best. Meta reportedly tested 27 Llama-4 variants before release and published only the winners. As one benchmark researcher put it, headline scores often "measure how well a model gamed the test harness rather than how well it solved the underlying tasks."
Read that carefully, because it defines the ceiling on what a leaderboard can tell you. A frontier vendor is not lying when they report a high SWE-bench number. But the number answers a question you didn't ask. It tells you how the model performs on a public, gameable target. It cannot tell you the one thing that authorizes a substitution: is this good enough for my workload to fire Opus?
Which brings us to GLM-5.2, and the cleanest demonstration of why that question is workload-specific.
On paper, GLM-5.2 is a frontier substitute. It's the top open-source model, MIT-licensed, with a 1M-token context window. On FrontierSWE it reportedly trails Opus 4.8 by only 1%, per Z.ai. On SWE-bench Pro it posts 62.1 against 58.4. Z.ai positions its coding capability roughly between Opus 4.7 and Opus 4.8. If you were routing purely by leaderboard, this is a swap you'd make today.
Claire ran it inside her actual ChatPRD codebase instead. She gave it one prompt in Claude Code: pull the last 72 hours of Sentry errors and Vercel logs, then build a prioritized bug-fix plan.
Over 45 minutes, autonomously, it ran MCP tool calls, authenticated into external services, and produced an engineering canvas with 20 Sentry errors, five Vercel signals, and 14 planned fixes—including, per Claire, two P0s she had missed through her normal monitoring. It surfaced signal-to-noise problems in her error pipeline that weren't showing up anywhere else. The whole run cost $3.36 across 6 million tokens, helped by a 72% cache rate.
That's the ceiling. Here's the floor. During that same long-running task, GLM-5.2 choked on TypeScript compilation errors before eventually producing clean React. Claire's read: HTML and CSS generation is reliable; React under agentic, multi-step pressure is shakier.
React is roughly 98% of her codebase.
Sit with what just happened. The leaderboard number—1% behind Opus—is real, and it is useless for her decision. The thing that would actually block a substitution is the model's inconsistency on the exact language stack that constitutes nearly all of her work. No public benchmark surfaces that. Her own bench did, immediately, on the first serious task.
So her verdict wasn't "adopt" or "reject." It was "in rotation, not in the spotlight"—per Claire, keeping it in Cursor for frontend and design work, keeping it in Claude Code for long-running agentic tasks, alongside the frontier models rather than replacing them. The constraint she's watching is explicit: can it handle her React-heavy workload at the same consistency she gets from Composer? If it can, she says, the cost-and-control argument gets much harder to ignore.
Notice that this is not a verdict a leaderboard could have produced. "In rotation, not in the spotlight" is a routing decision with a named condition attached to it. It requires knowing what 98% of your codebase is, and having an instrument sensitive enough to catch the model failing on that specific thing. A number designed to be gamed can't get you there.
This is the part practitioners keep getting backwards. The reason to build your own harness is not rigor as a virtue. Rigor is the cost, not the point. The point is leverage: a repeatable, taste-weighted bench is the only instrument that actually authorizes you to pull the trigger on a cheap substitution.
Everyone else is trusting a number. And the number is drifting in a direction that makes owning the instrument more valuable, not less. Frontier labs are increasingly gating access to their best models. Open-weight substitutes are catching up—GLM-5.2 is not a hobbyist toy, it's production-grade, and the gap to closed frontier is now measured in single-digit percentage points on the benchmarks the vendors themselves chose.
When the swap becomes economically obvious, the teams positioned to make it will be the ones who can answer "good enough for what, exactly?" in their own terms. Open-weight means you're not locked in: run inference on your own hardware, switch providers when pricing or policy changes, without touching application code. That optionality is only worth something if you can tell when the substitute is actually good enough. The eval harness is what converts optionality into a decision.
The mechanics are not hard, which is what makes the neglect so strange. Claire's rig is a local HTML page that scores outputs on gut feel and exports the ratings as JSON. Other practitioners describe the minimum viable version as 50 rows in a spreadsheet—real problems from your own codebase, a rubric, blind scoring, run it more than once and average. The load-bearing decisions are all product decisions: what "good" means, which cases represent your real distribution, where the pass/fail line sits.
None of that requires you to write eval code. It requires you to own the eval strategy—the same way you own the PRD. The failure mode is letting the eval set drift to engineering, or to no one, until it reflects what's easy to measure instead of what users care about.
The through-line from Sonnet 5's blind loss to GLM-5.2's split verdict is the same lesson twice. In both cases the vendor's story was directionally true and locally wrong, and in both cases only a taste-encoded, workload-specific bench could tell the difference.
Build the instrument. It's the difference between routing your own model stack and trusting a score that was engineered to beat scores.