The verification layer becomes the bottleneck: Blacksmith's 10x valuation, AI review outrunning humans, and eval tooling": Two engineers are looking at the same problem and drawing opposite conclusions about what your money buys.
Aditya Jayaprakash, CEO of Blacksmith, frames it as a throughput crisis: "Validating code is still a bottleneck, and it's an even bigger bottleneck because people are writing even more." His pitch is compute — hundreds of thousands of cores today, an order of magnitude more soon, all to run tests faster.
Tommy Tran, a Meta engineer working on AI infrastructure, is worried about something Blacksmith's compute can't fix: whether the tests are measuring the right thing at all. When AI writes both the code and the tests, he warns, "you risk testing whether the AI agrees with itself."
When AI writes both the code and the tests, you risk testing whether the AI agrees with itself — a very fast machine for checking self-agreement.
Both of them can be right. And that's the trap.
This week the question follows the money: capital is consolidating hard around the verification layer, and most of it is being spent on the wrong axis.
Start with the raw numbers, because they're extraordinary. Blacksmith's Series B valued the company at $550 million, up from $60 million on its Series A less than a year earlier — nearly 10x. Customers went from 700+ to more than 5,000 in the same window, reportedly per TechCrunch. Around the same time, per InformationWeek, CodeRabbit — an AI code review company — hit a $1.5 billion valuation.
The line every investor is underwriting is the throughput line. AI generates more code, code needs validating, therefore validation is the new bottleneck, therefore whoever runs the validation fastest and cheapest wins. Blacksmith says as much: the money is for compute, to "stay ahead of demand"; per Blacksmith, CI jobs have grown 5–10% week over week since the start of the year.
Here's the doubt worth planting. If the code is generated by a probabilistic system, and the tests are generated by a probabilistic system, and the review score is produced by a probabilistic system — what exactly is all that compute verifying? You've built a very fast machine for checking whether the AI agrees with itself.
This is the failure mode Tran names precisely. An automated test can show that an implementation behaves consistently with a particular expectation — without ever demonstrating that the expectation was correct. Pass rates and coverage numbers start measuring internal consistency, not correctness.
His fix is not more tests. It's an anchor: tie tests to requirements and system contracts, not to the implementation the AI produced. Otherwise you've closed the loop, and the loop will happily confirm itself all day at whatever throughput you can afford.
Balaji Srinivasan at LinkedIn puts the same warning in metric terms. Conventional coverage — the percentage of code exercised by tests — can obscure whether the tests validate anything meaningful. "Useful coverage" becomes the number that matters, and useful is precisely the part a faster CI cloud cannot manufacture for you.
Now the counter-evidence, because this is where you can actually tell real eval tooling from theater.
LangChain built ReviewBench for one reason: they didn't trust existing benchmarks, because those benchmarks didn't encode their own review standards. So they built the ground truth by hand — comments from trusted reviewers on real merged PRs in their LangSmith monorepo, curated down into concrete, verifiable findings.
The curation is the whole point. Raw reviewer comments were "too noisy to use directly" — full of nits and questions. They ran them through an LLM gate, then manually reviewed every remaining comment. The human judgment is what makes the benchmark mean something.
The result is the number every buyer at a $550 million valuation should sit with. Across 59 tasks covering 64 baseline issues, the strongest agent runs recovered only about 30% of the issues real reviewers caught, per LangChain.
Not 90%. Not 70%. Thirty percent — and that's the best runs. Many of the misses were exactly the issues that require reconstructing an implicit system contract: a SQL query that deleted a resource by ID without checking the tenant, an endpoint migration that silently dropped a filter. The kind of thing a fast test suite passes cleanly.
That gap is the actual state of the art. Any tool telling you AI review has closed it — through throughput, through more coverage, through more attempts — is selling you the closed loop, not a measurement of it.
So what does the discipline look like when someone gets it right? Intercom's Evals product is the clearest example on the market.
The mechanics matter. You build an Eval around a theme — refund requests, escalation rules, tone of voice — from simulations mirroring real conversations. Each one is scored against criteria you set, and if any single criterion fails, the whole simulation fails.
Two things separate this from throughput theater. First, per Intercom, you set the criteria — you decide what "right" means, though the scoring itself combines deterministic checks with an LLM judge. Second, the discipline is re-running the entire set on every change.
Listen to how AutoDS actually operates it: "We group our Evals around different topics, so every time we make a change, we rerun the whole set and see immediately whether anything regressed." That is the ground-truth anchor Tran is asking for, applied to support agents instead of code.
Intercom applies the same rigor to its internal code-review agent, reportedly per a walkthrough of the pattern: it logs every PR review its agent produces and has an engineer assess whether the score and recommendation were correct. A human keeps grading the grader. The AI's self-agreement never becomes the source of truth.
What's new is that there's now a capital thesis stapled to it. And the thesis is being spent, overwhelmingly, on the wrong half.
Because the two halves are genuinely different products. One is a compute business: run more tests, faster, cheaper — this is what Blacksmith raised $45 million to scale, and it's a real and valuable business. The other is a judgment business: encode human ground truth, expose how far agents fall short of it, and refuse to let the loop close. ReviewBench and Intercom's Evals live here.
Only the second one actually solves Tran's problem. The first one, run without the second, industrializes it.
Here's the uncomfortable prediction. Most organizations will buy the throughput and skip the discipline — and not because they're foolish.
Claire Vo has the reason. Asked why AI stalls inside companies, she landed on this: "the blocker is never tools or intelligence. Human systems, human problems." The ground-truth discipline is human work — curating findings, authoring criteria, having an engineer grade the grader, holding the line when the loop wants to close.
That work is unglamorous, slow, and never done. Buying more compute is a purchase order. Building a maintained corpus of human-curated ground truth is a standing commitment nobody gets promoted for.
So the money flows to the thing you can put on a slide — CI cores, test throughput, pass rates trending green — and quietly skips the thing that makes any of it mean something.
The tell, when you're evaluating a verification tool or your own setup, is one question: where does the ground truth come from? If a human wrote the criteria and keeps regrading the machine, you're buying correctness. If the AI is grading against assumptions the AI produced, you're paying — at $550 million, at $1.5 billion, at whatever multiple comes next — for a very fast, very expensive way to watch the model nod at itself.
Thirty percent. Until that number moves on a benchmark built from human judgment, everything above it is throughput.