sift.All magazines · About
Books & Ideas · Week of August 17, 2026

Pacing the Machine: Insider Warnings and the Governance of Automated AI R&D

In July 2026, more than 1,300 employees at every US frontier AI company signed an open letter asking their governments to build the capacity to slow them down.

Wait — let me correct that opening line as instructed:

In July 2026, more than 1,300 employees at every US frontier AI company signed an open letter asking their governments to build the capacity to slow them down.

The same day, the official OpenAI and Anthropic accounts reportedly amplified it. And in an interview that same day, Sam Altman reportedly said the words out loud: "we may have to pace the rate of AI development."

Read that again. The people whose stated professional goal is to fully automate AI research — Altman reportedly said last year that OpenAI wanted a "true automated researcher" by March 2028 — are petitioning to be paced. Their employers are boosting the signal.

The easy reading is courage. Conscience overriding commercial interest. Whistles blown from inside the machine.

I want to refuse that reading, because it dissolves the moment you ask what "pacing" actually contains once you stop treating it as a mood and start treating it as policy.

Last week in these pages we asked whether the promises of frontier AI were real — the hype-versus-limits question. This week the question moves downstream: not whether the capability arrives, but who gets to hold the brake, and whether the brake is even wired to anything.

· · ·

Start with the word. "Pacing" sounds like braking. The letter reads like a plea to be stopped.

But when Tim Fist and Saif Khan at the Institute for Progress sat down to operationalize the term — to turn a slogan into a mechanism — something strange happened. The braking disappeared.

Their proposed regime is explicitly conditional. An "if-then" structure. First, specify which automated AI R&D activities are likely to pose severe risks, with thresholds set by careful analysis. Then, only if a threshold is crossed, reallocate resources away from the risky activities.

And reallocate toward what? Two destinations: accelerating the diffusion of existing AI capabilities, or accelerating the research that makes further automation safer. Compute and talent get redirected, not switched off.

Here is the sentence that should reset your reading of the entire episode. Under this operationalization, a deliberately "paced" form of automated AI R&D "might still involve much faster improvements in AI capabilities than today." It "need not entail slowing innovation overall."

A deliberately 'paced' form of automated AI R&D might still involve much faster improvements in capabilities than today. It need not entail slowing innovation overall.

So the letter's apparent radicalism — governments, build the power to slow us — resolves, on operationalization, into resource reallocation triggered by a threshold nobody has defined yet.

This is not hypocrisy, exactly. It is architecture. The danger is defined conditionally, the trigger is outsourced to future analysis, and the "pause" is engineered so that it need not actually pause anyone racing today.

Which is precisely why the good-faith test cannot be whether a lab signs the letter. Signing costs nothing when the binding clause is deferred to a threshold TBD.

The test has to be which concrete measures the labs accept now — the ones that bite before any threshold arrives.

Fist and Khan supply the list: 23 low-regret recommendations, and a few of them have teeth. According to the Institute for Progress memo, Congress should resource the Center for AI Standards and Innovation at "at least $84 million per year." Congress should legislate transparency about automated AI R&D risk management, mandatory incident reporting, and whistleblower protections.

Notice the asymmetry. Verification prize competitions and hardware testbeds are the fun parts — they fund research, they build capacity, they cost the labs nothing and flatter their engineers.

Mandatory transparency, incident reporting, and whistleblower protection are the parts that bind. They constrain the signatories whether or not any threshold is ever crossed.

So the honest question to ask of every company that amplified the letter: do you support the legislated transparency and the whistleblower shield, or only the verification prizes? Good faith lives entirely in that gap.

Now, why build any of this before the danger is legible? Because if you wait until it's legible, you can't build it at all.

This is the Collingridge dilemma, reportedly named for David Collingridge's 1980 book on the social control of technology. It's a double bind. Early on, you lack the information to know what to control. Later, you have the information — but the technology is so entrenched in commercial systems that control has become impossible.

Diagram of a double bind: acting early on thin evidence versus late on entrenched systems
Diagram of a double bind: acting early on thin evidence versus late on entrenched systems

Act early on thin evidence, or act late on immovable systems. There is no window where you have both good information and real leverage.

Fist and Khan concede the thin-evidence half openly: rapid progress toward automated AI R&D "has empirical support," but how much acceleration it causes and what risks it poses are "less understood." They still argue for acting now — because the alternative is acting never, or acting in panic.

That last risk is worth naming. They warn that political backlash to AI-driven disruption could produce "poorly-reasoned policy measures, such as broad bans on new data centers." Build the sober option now, or get the blunt one later.

· · ·

And here the timescales turn from abstract to frightening.

Writing in Noema, Macario Schettino observes that every previous communication-driven disruption — printing press, mass newspapers, radio, television — "was eventually tamed by the same technology that caused it." The pattern is real. It is also cold comfort.

Because look at how long taming takes. "The Reformation required 131 years to settle into the Peace of Westphalia," Schettino reportedly writes. "The mass media transition produced two world wars and a Cold War before the postwar order stabilized."

Read that inside the Collingridge frame and the letter's only honest meaning snaps into focus. Yes, disruptions get tamed. But taming arrives on catastrophic timescales — after the wars, not before them.

The press that broke Christendom eventually produced the Enlightenment. But it broke Christendom first, and the interval was measured in generations and corpses.

If that is the baseline, then "we may have to pace" cannot mean "we'll slow down when it gets scary." By the time it's scary enough to be legible, the system is entrenched, the threshold fight is unwinnable, and taming is a 131-year project you don't get to opt out of.

So there is exactly one coherent reading of the insider contradiction. "Pace" does not mean a promise to slow. It means infrastructure-in-advance — the option, wired and funded and legislated, before the threshold arrives and makes it unbuildable.

Under that reading the paradox dissolves. You can race toward automated research and sincerely demand the pacing option, because the option is not a brake you plan to pull. It is a brake you want installed while installation is still physically possible.

Which returns us, unavoidably, to the good-faith test — and lets us bring in the one other governance actor circling this question.

In "Magnifica Humanitas," Pope Leo XIV reportedly made the structural point without the euphemism. It is not enough, he wrote, "to invoke ethics in the abstract"; what's required is "robust legal frameworks, independent oversight, and a political system that does not abdicate its responsibility." He noted that the main drivers of AI are "private, often transnational, parties" whose resources "surpass those of many Governments."

Set the encyclical's UN proposals aside; take only its governance grammar. Ethics in the abstract is the easy part. Binding legal frameworks are the part that costs something.

That is the same fault line running through the 23 recommendations. Verification prizes are ethics in the abstract dressed as action. Legislated transparency, incident reporting, whistleblower protection — those are the robust frameworks that bind whether or not the signatory consents in the moment.

So here is the verdict. The insiders' letter is coherent, even admirable, but only as a demand to build the pacing option now, while the Collingridge window is open — not as a promise to slow.

And the measure of whether the racers mean it is brutally simple. Do they lobby for the clauses that fund their own verification research, or for the clauses that bind them — the mandatory transparency, the incident reporting, the protection for the next employee who signs a very different kind of letter?

We will know soon enough. The bill either includes the parts with teeth, or it doesn't.

Sources

Researched from 10 sources. Primary source: 3quarksdaily.com

How this is made

Each issue is researched and written by sift.'s AI editorial pipeline. It reads a curated set of public sources for the magazine (publications, research blogs, newsletters and expert accounts) plus targeted web searches, groups what changed during the week into a few themes, and writes one piece per theme. A separate AI pass checks each piece against its sources before publication. No human editor reviews each article, so check the linked sources before relying on a specific claim.

One issue a week. Free.

sift. reads hundreds of sources so you don't have to, and sends one issue you can actually finish. No daily noise. Unsubscribe anytime.

Also in this issue