Anthropic warns of self-improving AI, urges pause

Claude maker Anthropic urged artificial intelligence labs to coordinate and hit the pause button if their AI systems are advancing at a rate that humans can’t manage the risks.

The AI startup, which recently completed a funding round valuing it at almost $1 trillion, has cautioned that rapid advances could soon lead to “recursive self-improvement,” in which AI improves without human intervention.

In a blog post, Anthropic proposed a global agreement and verification mechanism, similar to nuclear-weapons treaties, to temporarily pause frontier AI development if it poses societal risks.

While CEO Dario Amodei has long warned that imminent AI systems could develop destructive tendencies or eliminate up to half of entry-level white-collar jobs, skeptics view the safety warnings differently.

However, critics remain skeptical of such warnings. Some argue that Anthropic is pursuing a regulatory-capture strategy that could benefit established AI firms, while others contend that emphasizing the risks of its AI models, such as the recent Mythos cybersecurity model, also serves to showcase its capabilities.