Anthropic's warning: AI is on the cusp of getting smarter on its own
Anthropic says AI agents are nearing self-improvement without human input and urges a slowdown to study the risks.
Intelligence analysis by GPT-5.4 Mini
Anthropic argues AI systems are moving toward a point where they can build, train, and improve successors with less human help. The company says that makes safety, review, and global coordination more urgent than ever.
Anthropic says AI may soon get good enough to help make even smarter AI on its own, like a robot teacher training its own students. That could be useful, but it also means the machines might move faster than people can safely check them.
Analysis
What Anthropic is warning about
Anthropic says AI development is advancing fast enough that agents may soon be able to build, train, and improve other AI systems with little or no human input. In a Thursday blog post, Marina Favaro of the Anthropic Institute and co-founder Jack Clark said the company is already delegating more of its own AI development work to AI systems, which is speeding things up.
They argue that if the trend continues and enough compute is available, the end point could be a system that autonomously designs and develops its own successor. The article says Anthropic sees the human role narrowing step by step, with its Claude model already writing about 80% of the code merged into Anthropic’s codebase.
Why the company wants a slowdown
Favaro and Clark said they are not claiming recursive self-improvement has already arrived, and they do not call it inevitable. Their point is that it may happen sooner than institutions expect. They warn that once AI code quality matches human code quality, people may stop writing code and only review it. If the model can generate code faster than humans can inspect it, human review becomes the bottleneck.
The post says slowing development would give society more time to deal with the technology’s “immense” implications. But the authors also warn that a pause could backfire if less cautious players simply catch up. They say that without a global coordination mechanism, companies and governments will face hard tradeoffs under competitive and geopolitical pressure.
Broader context
The article links Anthropic’s warning to wider safety concerns across the industry. It notes that OpenAI has said it is researching safe deployment of increasingly capable systems, including recursive self-improvement. It also mentions that a group of tech leaders, including people from Anthropic and OpenAI, released an open letter urging stronger guardrails because the technology could help bad actors overcome barriers to biological weapons work.
For crypto readers, the key connection is that AI agents are becoming more common in payments and transaction flows. The article says some executives think AI agents could drive adoption, while recent reports show agentic payments moving from concept to real activity. That makes the speed and safety of AI improvement directly relevant to crypto infrastructure, where autonomous systems may handle more value over time.
Key points
- Anthropic says AI may soon be able to improve future AI systems with less human help.
- The company says Claude already writes a large share of its merged code.
- Favaro and Clark argue that human review could become the bottleneck if AI code generation keeps accelerating.
- The post calls for slowing development to buy time for safety research and societal preparation.
- The article connects the warning to crypto because AI agents are increasingly used in payments and transactions.
If Anthropic’s call for caution leads to better guardrails, AI systems could become more capable without losing human control. That could help crypto teams use AI agents for payments and coding while keeping security and review strong.
If AI systems improve faster than humans can inspect them, review could become too slow to catch mistakes or abuse. The article also warns that a slowdown by cautious firms could still leave less careful rivals racing ahead, which may reduce overall safety.
