Here is the contradiction most builders haven't noticed yet: the same person who tells you "it's not the model, it's the harness" is also the person who just discovered her own benchmark disagrees with her taste almost completely.
That person is Claire Vo, and both of those things happened in the same week.
Last week the thread was PMs building their own model evals—the realization that a 45-minute HTML scoring page beats waiting for a lab leaderboard. This week the thread moves up a layer, into harness architecture and multi-agent orchestration, and the eval problem doesn't get solved. It gets worse.
Start with what a harness actually is, because almost nobody who repeats the slogan bothers to say.
Vo built one live: a Sentry bug-debugging harness for ChatPRD, using the Claude Agent SDK, a custom terminal UI in the Ink library, and opinionated adapters for Sentry, Linear, GitHub, and Vercel. It gathers evidence, runs root-cause analysis, and creates the follow-up artifacts—all without her ever typing "dear agent, please fix this bug."
That's the real shape of orchestration. Runs, tasks, tools, artifacts. Permissions encoded so the agent can touch Sentry but not nuke production. It's buildable engineering, not a vibe.
Alessio Fanelli's version is the same idea wearing different clothes: Linear as a state machine, OpenAI Symphony managing agents through the whole dev lifecycle, run from his phone with zero babysitting. His mental model is "agent manager," not "agent prompter." The harness is what turns a prompter into a manager.
So the harness is real. Here's the turn.
Once you've built a harness, you have to decide which model runs inside it. That's a real decision with real money attached, so Vo did the disciplined thing: she built a second thing to answer the question. A benchmark. The How I AI Bench.
She built it live in Claude Code, then blind-tested five models—Sonnet 5, Sonnet 4.6, Opus 4.8, GPT-5.5, Gemini 3 Pro—across PRDs, prototypes, agentic tasks, and personality. Frozen inputs, fixed rubric, same tasks every release. Repeatable, which is the whole point; a one-off vibe check can't be compared across time.
Then the benchmark told her the opposite of what she believed.
She hand-scored 64 generations across the five models on a 1-to-5 gut scale, using a scoring page she'd built in about 45 minutes. She also had two LLMs—GPT-5.5 and Opus 4.8—judge the same outputs.
The LLM judges ranked Gemini 3 Pro highest and Sonnet 4.6 lowest. Claire's ranking was almost exactly the opposite. Not "somewhat different." Inverted.
When she built a 70/30 Claire-to-LLM weighted index, Sonnet 4.6 vaulted from last place to first.
Sit with that. The single decision the whole harness hangs on—which model to run—flips entirely depending on whose judgment you trust. And the two judges point in opposite directions.
The single decision the whole harness hangs on—which model to run—flips entirely depending on whose judgment you trust. And the two judges point in opposite directions.
Worse, the LLM judges weren't just differently calibrated. They were dull. They clustered toward the middle of the scale, too generous, never spiky. They missed things Vo flagged instantly on a visual pass: broken prototypes, ignored wireframe constraints. As she put it, models can't yet see what the human eye catches in the first screenshot.
That's the load-bearing failure. The judge missed a broken prototype—not a subtle taste call, a broken thing—that a human caught in one glance.
This is what the four-models piece calls the "re-evaluation tax." Here's what that tax literally is: an operational verification loop where the judge you built to save yourself judgment cannot be trusted, so you're back to doing the judging by hand, on 64 outputs, at 45 minutes of setup plus however long the scoring takes.
And notice the direction of the whole exercise reverses on you. You build the harness to stop typing "dear agent." You build the benchmark to stop hand-grading. Then the benchmark fails, and you're hand-grading again—except now you're also maintaining a benchmark.
So the durable skill isn't the harness. Everyone can watch Vo's episode and build a Sentry harness. The durable skill is the verifier that tells you whether your harness works—and right now the most sophisticated builders are discovering their homemade verifiers are wrong.
Until you fix that, the harness advantage everyone's chasing is unfalsifiable. You cannot claim your orchestration is better if the thing measuring "better" ranks models backwards.
The harness and the eval are one system. You don't have an orchestration advantage; you have an orchestration-plus-verifier advantage, and the verifier is the part almost nobody is building carefully.
Now make it worse, which the field is already doing.
Jared Z, Builder in Residence at Devin, has a master agent that spins up 10 sub-agents on the same problem, each with its own context, then judges which one produced the best solution. The parallel-spawn pattern. It's genuinely powerful.
It's also the exact failure mode from Vo's benchmark, promoted to production and running on a loop. The master agent is an LLM judge. And we just watched LLM judges rank things backwards and miss broken prototypes on the first screenshot.
If your master agent has the taste of Vo's LLM judges, it will confidently pick the mediocre sub-agent's answer nine times out of ten and you'll never know, because the whole appeal of the pattern is that you stop looking.
And this one has a meter running. Ten sub-agents, ten contexts, ten times the tokens—to produce one answer selected by an adjudicator you have no reason to trust yet.
The cost is not abstract. Alessio tracks token spend per task and talks about what 221 million tokens actually buys. Vo, running a bootstrapped AI company, describes feeling a prompt-cache bug "in your credit card." When your judge is spending real money to pick among ten answers, "is the judge right" stops being a philosophical question.
So here's the operational reframe.
DIY benchmarks are not a one-time setup you do before a model launch and file away. Vo's bench is version one and she says so: the agentic bug-hunting task turned out too easy—every model aced it, so it can't tell good from great, and she's retiring it. She plans to keep encoding more of her taste into the rubric and keep running it blind on every release.
That's not maintenance. That's the core loop. The rubric is a living document that has to keep converging on your actual taste, because the day it stops converging is the day your automated scores quietly start lying to you.
The question isn't "did I build a harness." Plenty of people have. The question is "who verifies the judge"—and right now the honest answer, even for the best builders, is: a human, by hand, one screenshot at a time.
The builders who win won't be the ones with the cleverest orchestration. They'll be the ones who treated that question as the main event instead of a footnote—who built the verifier alongside the harness, kept sharpening it against their own eye, and refused to trust an automated judge until it could catch the broken prototype they caught in the first glance.
Everyone else is optimizing a system they can't measure. That's not an advantage. That's a bet you can't check.