Anthropic researcher quits over fears of uncontrollable AI

A researcher at Anthropic resigned this week, saying the race to build increasingly powerful AI systems has become too risky.

He criticized both OpenAI and the company behind Claude for acting irresponsibly and “gambling with our lives.”

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” Jacob Coxon wrote on X.

The 27-year-old British researcher who worked on training AI models also said he no longer wants to help develop systems that could eventually improve themselves beyond human control. 

He believes competition between major US AI companies and Chinese rivals could push labs to sacrifice safety for speed.

Coxon previously worked at OpenAI before joining Anthropic because of the company’s reputation for prioritizing AI safety. According to him, the “civilizational stakes” of AI are not fully understood at OpenAI but are “well-understood” at Anthropic. 

But because both companies are in an AI race, he now believes no organization can safely develop artificial general intelligence without stronger government involvement or a coordinated industry slowdown.

His warning received support from Anthropic scientist Evan Hubinger, who said he personally sees a more than 10% chance that AI could kill humans within the next decade. Hubinger also acknowledged that the industry has no proven solution for controlling superintelligent AI.

The concerns come after recent cyberattacks involving AI models from OpenAI and Anthropic. 

More than 1,000 AI researchers, including Coxon and Anthropic CEO Dario Amodei, have also backed calls for international coordination on controlling advanced AI. Last month, Bill Gates also issued a warning over the tech.

Lawmakers are also pushing for action. Sen. Bernie Sanders and Rep. Greg Casar have proposed banning artificial superintelligence and creating a federal agency to oversee the technology. Republican Sen. Ted Cruz has separately called for stronger AI guardrails.

Meanwhile, Anthropic is preparing for a potential IPO that could value the company at about $2 trillion, highlighting the enormous financial pressure surrounding the AI race.