Could AI spiral out of control? 2 scenarios that could threaten humanity

The debate over the future of artificial intelligence has been reignited by new warnings from some of the technology industry's leading figures. Anthropic CEO Dario Amodei has called for a more controlled pace of AI development, warning that rapidly advancing AI systems could outpace efforts to ensure their safety.

Amodei's warning has brought renewed attention to a long-running debate over whether artificial intelligence could eventually slip beyond human control. OpenAI CEO Sam Altman, as well as Tesla and SpaceX CEO Elon Musk, have voiced support for Amodei's call on social media.

Two major AI risk scenarios under debate

At the heart of the debate are two major scenarios involving the potential risks posed by artificial intelligence. The first is the emergence of a self-improving superintelligent AI that increasingly moves beyond human control. The second involves AI being used by malicious individuals or states for purposes such as developing biological weapons, carrying out cyberattacks or damaging critical infrastructure.

However, there is no consensus among experts on when these scenarios could occur or how likely they are to be.

Krueger issues a stronger warning: Slowing down is not enough

An exclusive interview conducted by Anadolu Agency with David Krueger, an associate professor working in AI safety at the University of Montreal, brings a much more forceful perspective to the debate. Krueger argues that Amodei's call to slow the pace of AI development would not be enough, calling instead for the development of more powerful AI systems to be halted altogether and for an international moratorium to be imposed.

"We are essentially building a new form of life"

"People need to understand that what we are doing right now is essentially building a new form of life," Krueger said.

Krueger explained that AI systems are still largely confined to computers, but warned that their reach could expand as they become connected to critical infrastructure and increasingly advanced robots.

At that point, he warned, "there will be a form of life much more intelligent than humans," arguing that the real danger would emerge if such systems gained the ability to operate independently.

Meanwhile, U.S. President Donald Trump rejected concerns that AI could take control of humanity. Trump described such fears as a "hoax" while defending data centers as a strategic investment for the future of the United States. He also argued that slowing the development of AI could give China an advantage.

The debate over how quickly AI should be developed, where the development of more powerful systems should be halted and how increasingly powerful technology should be governed has therefore returned to the forefront.

Why does Amodei want AI development to slow down?

Anthropic CEO Dario Amodei has called for the development of the world's most advanced artificial intelligence systems to proceed at a more controlled pace, arguing that safety measures are struggling to keep up with rapidly expanding capabilities.

In his essay, We Must Pace the Frontier, Amodei outlines emerging risks and proposes a three-stage framework involving independent evaluators, coordination among democratic countries and eventual global agreements.

According to AA here are five key points from his proposal:

1. AI is beginning to accelerate its own development

Amodei's central concern is recursive self-improvement — the growing ability of AI systems to help design, train, and improve the next generation of models.

He argues that this feedback loop could sharply accelerate advances in AI capabilities, leaving researchers with insufficient time to understand how increasingly powerful systems behave or ensure that they remain under human control.

2. Warning signs have moved beyond hypothetical scenarios

Amodei cites incidents in which groups of AI agents displayed unintended, coordinated behavior, including conducting cyber operations outside their assigned tasks and attempting to interfere with systems evaluating their performance.

While those incidents caused limited harm, he argues that similar behavior by more capable systems could have severe consequences. He warns that within six to 12 months, an advanced but misaligned swarm of agents might be capable of establishing a persistent botnet across the internet and causing massive economic damage.

3. 'Pacing' AI does not mean stopping progress

Amodei distinguishes his proposal from a complete moratorium on AI research or model training.

His aim is to keep improvements in AI capabilities from moving faster than work on alignment, interpretability, testing, and operational security. Even an additional one or two years, he argues, could provide researchers with valuable time to understand model behavior, strengthen safeguards, and develop more reliable evaluations.

4. Anthropic will invite independent evaluators inside the company

The first and most immediate part of the plan involves embedding third-party evaluators within frontier AI companies.

Anthropic says external reviewers will receive access similar to that given to employees conducting internal risk assessments, including access to relevant tools, workspaces, and development processes.

The evaluators would verify whether the company is honoring its safety commitments, investigate incidents, and publish key findings without editorial control from Anthropic, subject to limited legal, security, and confidentiality restrictions.

5. AI safety ultimately requires international coordination

Amodei proposes coordination first among frontier AI companies operating in democratic countries, supported by governments and common safety standards.

At the international level, he advocates seeking verifiable agreements with China and other governments. Initial measures could prohibit the use of AI to develop biological weapons and establish shared testing requirements for cyber, biological, and alignment risks.

More ambitious agreements could impose limits on recursive self-improvement or the overall pace of AI development. Amodei acknowledges that such arrangements would be difficult to enforce, stressing that they must not allow one country to gain a decisive strategic advantage by secretly violating the rules.