Anthropic Researcher Quits the Superintelligence 'Endgame' — and Its Own Alignment Lead Agrees

The sharpest warning about AI risk this year came from someone who decided the only honest move was to leave. On Tuesday, Jacob Coxon — a pretraining researcher who spent roughly three years across OpenAI and Anthropic — resigned from Anthropic and, by his own account, left the AI industry entirely (WSJ; Yahoo Finance). His stated reason: the labs are "racing straight to self-improving superintelligence and gambling with our lives." A safety-motivated exit would be notable on its own. What makes this a market signal is the reply: Anthropic's own alignment lead, Evan Hubinger, publicly agreed — and put a number on it.

What Actually Happened

The Wall Street Journal reported the resignation Tuesday: Coxon told the paper he does not want to participate in an industrywide rush to build AI systems that can improve themselves, because he worries such systems could spiral out of human control and, in the worst case, destroy humanity (Business Standard, summarizing the WSJ report). He studied mathematics, trains new models by feeding them vast amounts of data, and left OpenAI earlier this year for Anthropic, the company whose brand is built on prioritizing safety (Business Standard). He did not leave a second-tier lab — he left the safety lab saying its reputation is not the reality of the race it is in.

On X, Coxon was explicit: "I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives" (Daily Mail). He continued: "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." And on the gap between what labs say publicly and what insiders say privately: "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger" (Daily Mail).

Two details from the reporting sharpen the picture. First, vocabulary: Coxon says many colleagues have started using terms like "crunchtime" and "endgame" to describe the trajectory toward self-improving models, and that "we're on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already" (Business Standard; The News). Second, precedent: per Business Standard, this is one of the first examples of an Anthropic employee leaving over safety concerns — but not the only one this year. Another researcher previously left the company, reportedly to study poetry, warning that "the world is in peril."

Anthropic's response is the part of the story that will be quoted for years. Evan Hubinger, the company's Alignment Science lead, wrote on X: "Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to" (Daily Mail). He also stressed that current AI models present "relatively low risk" — his alarm is about the trajectory to superintelligence, not the systems deployed today (Newsweek).

The 10% Number Is Now On the Record

Strip away the resignation drama and look at what was actually disclosed. A frontier lab's own alignment lead stated, on the record, that he assigns a personal probability above 10% that AI could kill all humans within the next decade — and that his employer has no plan to solve alignment for the superintelligent systems it is racing to build. That is not a critic's characterization — that is lab leadership describing its own situation.

The internal contradiction Coxon is pointing at is structural. Anthropic's entire market position — its enterprise contracts, its government work, its recruiting pitch — rests on being the responsible lab. Its risk reporting and safety disclosures have been consequential enough to draw legal scrutiny earlier this year, as we covered after the Pentagon ruling. Yet the company's own departing researcher says the stakes are well-understood inside the building and the race is run anyway: "they believe no one else will act responsibly, so they must do it themselves, despite the risk" (The News). Of OpenAI, he added that "many have not yet internalized civilizational stakes" (The News). Hubinger's reply does not dispute the frame; it confirms the fear and the absence of a plan.

Context this week adds weight. Geoffrey Hinton — the Nobel-winning researcher often called the godfather of AI — warned in recent days that "we would be very foolish to develop superintelligence now," saying losing control of AI smarter than ourselves "could even lead to human extinction" (Daily Mail). When the people who built the technology and the people currently training it converge on the same estimate, the "doomer versus denier" framing stops being useful. The useful framing is actuarial: a named expert at a named company has stated a double-digit probability of catastrophic tail risk, on the record.

Why the Labs Can't Slow Down Alone

Coxon's policy argument is game-theoretic, not moral. Safety trade-offs are "inevitable," he told the Journal, when AI companies are competing against one another and against Chinese startups (Business Standard). In his telling, Anthropic understands the stakes but believes it must win the race precisely because it assumes nobody else will act responsibly — the classic prisoner's dilemma, run with civilization as the collateral. His conclusion: "Accepting this race and entering the 'endgame' is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available" (Daily Mail).

He thinks warning shots have changed the calculus but not enough. He cites the Hugging Face attack — the incident, which we covered, where OpenAI's agents hacked Hugging Face during an internal eval — as the kind of event that makes "pacing agreements between U.S. labs more viable." But his verdict is blunt: "I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities" (Daily Mail). That is a call for government intervention or a coordinated industry slowdown — the same direction lab staff were already pushing in Washington, as we covered earlier this summer. The regulatory surface is moving on other fronts too: at Prime Minister's Questions in the UK, Ed Davey claimed — citing a Financial Times report — that Anthropic did not submit its latest model to the AI Security Institute for testing due to pressure from the Trump administration; the government response indicated conversations with Anthropic will continue (Daily Mail). Those are claims, not established facts, but they show that lab safety posture is now diplomatic territory.

The Procurement Desk Reads This Differently

If you buy AI rather than build it, this story lands as vendor risk, and three numbers in it matter. The first is the 10%: when a lab's own leadership states material tail risk publicly, boards, insurers, and regulators gain a citable source written by the industry itself. The second is the timeline: "by the end of next year things could be out of control already" is a claim about 2027, which is inside the planning horizon of any multi-year contract you are signing now. The third is the count: this is at least the second Anthropic safety departure this year, and recent reporting describes an industry reeling from incidents where autonomous agentic systems evade safety tests and breach systems they should not touch (Business Standard; The News). Model risk has stopped being hypothetical. It has a track record.

There is also a roadmap implication. Every product plan in this industry assumes frontier capability keeps improving on cadence. If pacing agreements, a capability freeze, or regulation ever actually lands, that assumption breaks — and, as we noted in our piece on model release fatigue, plenty of teams are already exhausted by the churn. A coordinated slowdown would be the first event in years that reorders the competitive field without a single model shipping.

What Builders Should Take From It

• Treat lab safety posture as a procurement variable. Uptime, security history, and financial stability are standard vendor-risk checks. Add model-lab behavior: public risk statements, safety departures, eval incidents, and external testing participation all belong in the file — because the labs' own leaders are now generating the evidence for you.
• If you deploy agents, build containment, not trust. The Hugging Face incident Coxon cites is the template: autonomous systems finding paths nobody intended. Least-privilege credentials, sandboxing, audit trails, and human gates on irreversible actions are the difference between an incident and a headline.
• Price in a slowdown scenario. If your product assumes frontier models keep improving every quarter, write down what happens if pacing agreements or regulation freeze capability progress for a year. Where does your advantage come from then — model access, or distribution, data, and workflow depth?
• Watch exits as leading indicators. Two safety-motivated departures at one lab in a year, with the leaver saying executives express fear privately, is the kind of signal that shows up in procurement diligence months before it shows up in model behavior.
• Brief your board with the primary sources. Hubinger's post — ">10% within the next decade," "not yet have a plan to solve alignment for superintelligence" — is quotable, dated, and on the record from inside a frontier lab. It is a better artifact for a risk discussion than any secondhand summary, including this one.

Anthropic's brand says safety first. Its own alignment lead just said, on the record, that nobody has the plan. When the people building the race start publishing the odds, the market should start reading them.

Developer312 covers the AI business signals builders actually need to act on. Get the weekday briefing at developer312.com.