Two people who both build evals for a living can't agree on how hard it is.
Daniel McKinnon, who reportedly wrote evals for Llama at Meta, tells partner teams that the first eval set "takes an afternoon" — so trivial that when a team wants Llama to do something, he asks them to skip the spec entirely. "Please not send me a product spec or PRD or description of the problem," he wrote in his viral piece "Show, Don't Tell." "Cut the middleman and send me an eval directly."
Rasty Turek, who spent the past year building with coding agents, now spends around 90% of his time on evals and calls the difference "indescribable."
Both are right. That contradiction is the whole story, and most people copying the artifact are on the wrong side of it.
Last week and the week before, this newsletter argued that verification — not plumbing — is what actually decides whether autonomous engineering works: the outer loop, the thing that tells an agent when it's done. This week we go one layer inward, to the artifact that encodes that loop: the eval set. Because a slogan has hardened into doctrine, and the doctrine is half-true in a way that will quietly wreck products.
The slogan is "evals are the new PRD," and it's everywhere. Kevin Weil, OpenAI's CPO, calls writing evals "the most important thing a PM can do in the AI era." Expedia's first chief AI officer, Xavi Amatriain, told a VB Transform audience that "all your thinking is gonna go into the evals." Braintrust ran a whole essay under the title. The logic is clean: a PRD that says "be helpful and concise" is too vague to verify against a non-deterministic system, so you replace prose with a runnable spec — inputs, expected outputs, a score. Make the number go up.
And the swap really does work at the artifact level. Falkster's handbook lays it out: 30 to 200 real input/output pairs, spiked with adversarial cases, scored daily in CI. Feature reviews shrink from "40 minutes of opinions" to "20 minutes of score diffs." The eval becomes the spec, the changelog, and the definition of done. First set takes an afternoon; by the third you have a template and wonder why you ever wrote PRDs.
Here's the gap the slogan hides. If the artifact is this easy to make, why do only 5% of enterprises trust the evals they've built?
That number is from VentureBeat's VB Pulse survey of 157 enterprises. Sixty-six percent already permit some production deployment without human review or are building toward it inside twelve months. Only 5% fully trust the automated evals that would justify that. And half have already shipped an agent that passed internal evals and then failed with a real customer.
Read those three numbers together. Most teams are removing the human from the loop, almost none trust the machine they're replacing the human with, and half have already been burned. The scoreboard says green while the product bleeds.
The reason is that the artifact and the discipline are two different things, and only one of them is copyable.
The artifact is the eval set. The discipline is the flywheel around it — and, more importantly, the human judgment that authors what "good" and "bad" mean in the first place. Braintrust names four maturity stages: vibes, on-demand test sets, CI-gated evals, and finally the flywheel where production failures feed back into the suite every week and compound into a durable advantage. Stage 3 is the whole point. Stage 1 is where most teams land and stop.
But notice what even the flywheel takes for granted. Someone has to look at a production failure and decide it was a failure. Someone has to label the correct output. Falkster is blunt that this labeling — step two of five — "is the actual PM work. If I can't write it, I don't understand the problem yet." McKinnon is blunter: "You will not write a good eval for a space you don't know."
That is the load-bearing sentence in the entire eval discourse, and it's the one the slogan drops. The eval is only as good as the domain knowledge of the person writing the labels. McKinnon reportedly picked cystic fibrosis for a floor test because he knew there was a canonical gene behind it. He knew the right answer before the model did. Strip that out and you have a beautifully automated system for measuring your own confusion.
This is why the two-person contradiction resolves the way it does. McKinnon's "afternoon" and Turek's "90% of my time" are describing the same activity at different depths. Building the file is an afternoon. Knowing whether the file is measuring the right thing — re-labeling as the product shifts, hunting the adversarial cases, catching the offline score that no longer predicts online satisfaction — is a permanent job that eats most of your week if you take it seriously. Turek takes it seriously. Most copiers build the afternoon version and call it done.
And here's the part that should stop the "PRDs are dead" crowd cold: the frontier labs haven't actually killed the PRD.
When Pragmatic Engineer went inside Anthropic, the single most complex project the Claude Platform team shipped in the past year — Claude Managed Agents — took roughly six months. The biggest matter, according to head of engineering Katelyn Lesse, was planning that "looked more like a typical pre-AI planning process." Idea docs dated back two years. On the hard, long-lived infrastructure work, the PRD stayed relevant.
The distinction Lesse draws is exactly the right one. A trivial TypeScript CLI, you jump straight to prototyping. Claude Managed Agents, you "first figure out what we are doing." Evals replace the spec where the spec was mostly encoding output behavior you can label. They don't replace the spec where the hard part is deciding what to build in the first place.
So the honest version of the doctrine is narrower than the slogan. Evals are the new PRD for well-scoped, behavior-heavy AI features owned by people with deep domain knowledge and a flywheel to feed them. That's a real, important discipline. It's also a specific set of preconditions most companies don't have.
Mind the Product's account of building a zero-to-one AI product gets the failure modes right and names them out loud: overfitting to a fixed dataset until the system aces your test and generalizes to nothing. The remedy — constantly replacing stale examples with fresh production failures — is precisely the flywheel labor that talent-dense teams do reflexively and everyone else skips.
There's a second, quieter tell in that same piece: the golden dataset works best when it isn't owned by product alone. Engineering sets it up; product curates the criteria and edge cases. "A dataset gatekept by Product recreates the problem PRDs had." The eval isn't a document you hand off. It's a shared judgment surface, and it degrades the moment one function stops arguing about what belongs in it.
Which is the whole point about talent density. The flywheel isn't a tool you install; it's a standing argument between people who each know enough to say "that output is wrong, and here's why" — every week, forever. Buy the tool without the argument and you get a scoreboard nobody has standing to challenge.
The artifact is copyable in an afternoon. The discipline is not, because the discipline is talent density wearing a dataset as a costume.
So here is the replicability verdict. The artifact is copyable in an afternoon. The discipline is not, because the discipline is talent density wearing a dataset as a costume.
When a non-lab company "adopts evals," what it usually imports is the scoreboard: a number that goes up. What it does not import is the judgment that makes the number mean anything — the domain expert who knows the right answer, the flywheel that keeps the test honest, the cross-functional argument over what "good" is.
That's why 66% are shipping without human review while 5% trust the thing they're trusting. They swapped the artifact and assumed the discipline came with it. It didn't. And the specific