- Lead. Anthropic CEO Dario Amodei published a long essay on September 12 calling on frontier AI labs to deliberately slow capability development until alignment, security and third-party evaluation can catch up—on the same day two senior safety researchers publicly resigned to join the independent evaluator METR.
- Fact. Joe Benton, who led Anthropic’s Scalable Oversight team, and Josh Engels, a safety researcher at Google DeepMind, both announced their departures on September 12; Benton stated: “AI companies are racing to build machines that are much smarter than any human, and we may not survive this.”
- Stake. The convergence of an insider CEO warning and two simultaneous researcher defections to an external oversight body signals a deepening credibility gap between what labs say publicly about safety and what their own staff believe they are practising.
Amodei’s essay, titled “We Must Pace the Frontier,” published on his personal Substack on September 12, argues that leading AI laboratories have reached a juncture at which further capability improvements outpace the governance and evaluation infrastructure designed to constrain them. He writes that without a deliberate slowdown, “the window for meaningful human oversight could close faster than regulators or the public realise,” and warns that unchecked recursive self-improvement could enable an agent swarm to “take over the entire internet with a persistent botnet” within as few as six to twelve months, causing hundreds of billions of dollars in damage. The essay does not endorse government-mandated pauses but calls for a voluntary cross-lab pact to pause at specified capability thresholds until agreed safety benchmarks are met.
Two Safety Researchers Leave for METR
Hours after Amodei published, Joe Benton announced his resignation as lead of Anthropic’s Scalable Oversight team to join METR—the Model Evaluation and Threat Research organisation that provides independent safety assessments. Josh Engels, a safety researcher at Google DeepMind, posted a similar departure notice the same day. Both cited a desire to work externally where their evaluations could be free of competitive and commercial pressures. Benton’s departure follows that of Jacob Coxon, a former safety researcher at both Anthropic and OpenAI who resigned on September 9 with a public warning that “race pressure creates incentives to cut corners.” The clustering of departures within days of each other is unusual even by the standards of a field that has seen significant researcher churn in 2026.
Industry and Regulatory Context
The Amodei essay and the departures land inside a rapidly changing regulatory environment. California recently became the first US state to mandate third-party audits of high-risk AI systems, a law that Anthropic publicly backed. At the same time, Sam Altman told Fortune this week that OpenAI would not pursue its mooted IPO in 2026, saying “given everything happening with safety, right now would be an ill-advised moment to go public.” Amodei and Altman have also, according to reporting from AI Weekly, been driving a quiet effort since July to create an industry-led standards body as an alternative to waiting on government regulation. Whether that initiative can coexist with the departures of researchers who concluded internal safety culture was insufficient remains a central open question.
What “Pacing” Would Mean in Practice
Amodei’s essay is short on operational detail but long on diagnosis. It identifies three bottlenecks that need to close before frontier capability should advance: interpretability tools that can reliably explain model behaviour at scale, red-teaming infrastructure capable of evaluating autonomous agents in realistic environments, and external evaluators—bodies like METR—that are independent enough to publish adverse findings without commercial risk. The implicit argument is that moving Benton to METR, rather than retaining him internally, may actually accelerate one of those bottlenecks. Whether that reading is shared by Anthropic’s board, or represents the personal view of a CEO watching his own safety talent depart for the organisation he is publicly calling to be strengthened, is a distinction the essay does not address.