Read Mustafa Suleyman's announcement thread twice and you find two claims that do not sit comfortably together.
The headline is a triumph: MAI-Cyber-1-Flash, running inside Microsoft's MDASH harness, hit 96% on the CyberGym benchmark, twelve points above Mythos, at half the cost. A clean win, marketed as a model win.
Then, three posts down, Suleyman quietly undercuts his own headline: "The hard part isn't routing 90% of tasks to a smaller model. It's knowing which 90% are safe to route." And the kicker: "The real asset here is the evaluation and routing layer."
That is the contradiction most readers will skim past. The benchmark score is a claim about the model. Suleyman's own caveat is a confession that the model is not where the value lives. The number sells; the footnote is the thesis.
The benchmark score is a claim about the model. Suleyman's own caveat is a confession that the model is not where the value lives. The number sells; the footnote is the thesis.
To understand why that footnote matters, start where the pain actually lands.
It is 11:47 PM. A senior SOC analyst is pulled into a high-severity alert—a domain controller flagging an anomalous administrative command sequence from a mid-level employee's workstation. She triggers the platform's agents. They map the account's authentication timeline, cross-reference network logs, scan threat feeds, build lateral-movement queries. The investigation moves at machine speed.
Then the screen changes: "You have reached your monthly AI limit. Upgrade to Enterprise Plus to continue. Your limit resets at 3:30 AM."
That scene, from SecurityWeek, dramatizes a shift that matters more than any capability chart. For defenders, the binding constraint is no longer whether the model is smart enough. It is whether the meter runs out before the job finishes.
The economics are brutal and specific. Alert triage might cost a thousand tokens. A guided investigation, twenty to fifty thousand. But a fully autonomous agentic loop—reading hundreds of thousands of lines of raw logs, formatting API calls, feeding context back to plan the next move—can burn millions of tokens in minutes. Multiply that by a day's alert volume.
The concrete number that should frighten every CISO comes from Palo Alto Networks. When it ran Claude Mythos against its own source code, the model found more than two dozen critical vulnerabilities. It also burned through more than a million dollars in tokens doing it.
That is the real story behind Suleyman's benchmark. Agentic AI removed the human who used to pace the machine. With no human bottleneck, the meter simply runs until the job is done—and in security, the jobs never stop.
So when Microsoft says its specialist model handles 90% of CyberGym's detection and patching tasks, escalating only the hardest 10% to GPT-5.4, the framing is exactly right for the moment. Cost, not capability, is now the axis of competition. Suleyman is explicit: token efficiency is "the next big focus across the industry," and to build a frontier firm you must "optimize frontier performance against cost."
But here is why the model is the commodity and the router is the moat.
A model that solves 90% of tasks cheaply is valuable only if you can identify, in advance and without seeing the answer, which 90% are the safe ones. Get that call wrong and you have handed a critical vulnerability to the cheap model, gotten a confident miss, and skipped the escalation that would have caught it.
That failure mode is not hypothetical. Endor Labs ran a controlled benchmark—same model, same 34 prompts, same step budget—comparing an agent equipped with precomputed security evidence against one left to do its own reconnaissance.
The equipped agent reportedly used 91.7% fewer tokens: 6.6 million versus 79.5 million. It made 77.6% fewer tool calls. It finished 2.8 times faster, per Endor Labs.
The unequipped agent, meanwhile, reportedly spent 22 times the tokens on a single reachability question and produced what Endor Labs' report described as "a well-cited, completely incorrect answer." That is the nightmare in one sentence: maximum confidence, maximum spend, wrong result.
The lesson generalizes directly to the routing problem. An agent is good at synthesis once the facts are in front of it. It is expensive and unreliable when asked to excavate the facts itself. A routing layer that decides what to hand to the cheap model is making exactly that excavation call—and if it decides wrong, the savings invert into retries, missed vulns, and escalation failures.
This is why a benchmark score cannot prove the thing that matters. CyberGym rewards the average case. Production cost, as Suleyman himself concedes, "has to include retries, human review, missed vulnerabilities, and escalation failures." The router's quality shows up only in the tail.
Which reframes the whole market. Cybersecurity agents are hardening into a distinct deployment category, and the competitive axis is cost-per-task—not raw accuracy. The moat Microsoft is describing is co-optimization: your own model, tuned in your own harness, trained in your own reinforcement-learning environments, on your own data.
Suleyman is packaging exactly this as a product, "Frontier Tuning," and the pitch is a flywheel that lets firms "swap models, build your own IP, and control costs." Notice what is load-bearing there—not the model weights, but the harness and control layer that wraps them.
Perplexity's move the same week confirms the layer is maturing into its own tier. It open-sourced Numbat, an agent-detection and response layer designed to work across agent harnesses—desktop, CLI, IDE, gateway—shipping as a single Go binary under Apache 2.0.
Numbat's job is visibility and control: live monitoring, pre-action blocking, forensic reconstruction. It feeds telemetry to a system that escalates suspicious behavior and proposes new detections—a "continuously improving detection flywheel." The vocabulary is telling. Both Microsoft and Perplexity are describing flywheels built around the control layer, not the model.
That is the structural shift. The model is becoming interchangeable; the routing, evaluation, and control scaffolding around it is where firms are planting their flags.
So end on the skeptic's question, because it is the one production will answer and CyberGym cannot.
A 90/10 split is not a natural constant. It is a distribution—a claim that 90% of tasks are easy enough for the small model and 10% are not. That ratio was measured against CyberGym's particular mix of difficulty.
Real-world traffic does not owe CyberGym anything. When an actual incident arrives—a novel exploit chain, a codebase larger than any in the benchmark, an insider campaign that spawns thousands of simultaneous investigations—the true "safe to route" fraction may not be 90%. It may be 60%. And every task the router misjudges downward costs a retry, a missed vuln, or a blown escalation.
The genuine efficiency case and the overfit case look identical on a leaderboard. They diverge only when the traffic stops matching the test.
Suleyman's twelve-point lead at half the cost is real, and it points at the right frontier. But the achievement he is selling and the asset he actually built are different things. The model is the commodity. The router is the moat.
And the router's worth is precisely the number no benchmark can print.