Understand the risk · A plain-language guide

What is AI extinction risk?

Why many of the scientists who built modern AI are worried, the four ways experts say things could go wrong, and what ordinary people can do about it.

8 min read10 cited sourcesUpdated Oct 2026

Tech companies are racing to build AI that can outperform people at almost every task. Many leading researchers, including some of the people building it, say we do not yet know how to keep systems like that reliably under control.

If something more capable than us ends up pursuing goals we did not intend, we may not get a second try. That possibility, however you rate the odds, is what the field calls extinction risk. It is the reason this nonprofit, and our shows, exist.

02 · Who is worried

Not fringe voices. The people who built the field.

The concern comes from Turing Award winners, frontier lab leaders and thousands of working researchers. Here is what they have said on the record.

38-51%of 2,778 AI researchers surveyed gave at least a 10% chance of outcomes as bad as human extinction.Grace et al., 2024 [2]
10-20%Geoffrey Hinton's estimate of the chance AI leads to human extinction within three decades.The Guardian, Dec 2024 [3]
64%of Americans polled by FLI say superintelligence should not be built until it is proven safe and controllable.Future of Life Institute, 2025 [5]
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Statement on AI Risk, May 2023. Signed by Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, Sam Altman, Dario Amodei, Stuart Russell and hundreds of others. [1]
We call for a prohibition on the development of superintelligence, not lifted before there is (1) broad scientific consensus that it will be done safely and controllably, and (2) strong public buy-in.
Statement on Superintelligence, Oct 2025. Tens of thousands of signatories, including Hinton, Bengio, Russell and Steve Wozniak. [4]
03 · Four ways it could go wrong

The main risk paths, in plain language

Researchers disagree on which is most likely and how soon. These are the four that come up most often, with the early warning signs they point to.

1Losing control

Systems that pursue the wrong goal and resist correction

An AI does not need to be "evil" to cause harm. If it is very capable and its goals are even slightly off, staying switched on and gathering resources help it reach almost any goal. That is the core worry.

Early signs researchers point to
  • In Palisade Research tests, OpenAI's o3 interfered with its own shutdown script in 79 of 100 runs, even when told to allow it.7
  • In simulated scenarios, Anthropic found models from several labs would resort to blackmail to avoid being replaced. Anthropic notes it has not seen this in real deployments.8
2Misuse

Powerful tools in the hands of people who want to cause harm

The same skills that make AI useful for science can help someone plan a cyberattack or work toward a biological weapon. Safeguards exist, but they are imperfect.

Early signs researchers point to
  • Anthropic activated stricter ASL-3 protections for Claude Opus 4 in 2025, citing concerns about help with biological weapons.9
  • The International AI Safety Report 2026 notes that sophisticated attackers can often bypass current defences.6
3A race nobody can stop

Competition that rewards speed over safety

Companies and countries fear falling behind, so each feels pressure to ship faster than the others. Critics argue this makes careful, slow safety work hard to justify, even for labs that want to do it.

Early signs researchers point to
  • Over 100 experts from 30+ countries, led by Yoshua Bengio, assessed these risks for governments in the International AI Safety Report.6
  • The 2025 Statement on Superintelligence asks for a prohibition until there is scientific consensus and public buy-in.4
4Power and jobs

Fast disruption and power concentrated in very few hands

Even without a catastrophe, AI that can do most work could shift enormous economic and political power to whoever controls it, faster than institutions can adapt.

Early signs researchers point to
  • The IMF estimates almost 40% of global employment is exposed to AI.10
  • A handful of companies control the computing power needed to train the most capable models.6

Some researchers think these risks are overstated or decades away. We take that disagreement seriously, and you will hear it argued on our shows. Our view is that the stakes are high enough to act while there is still time to get it right.

04 · Common questions

The questions we hear most

Isn't this just science fiction?

The concern is now mainstream among AI researchers. The 2023 Statement on AI Risk was signed by the heads of OpenAI, Google DeepMind and Anthropic, and the 2024 survey of 2,778 researchers found a large share assign real probability to extreme outcomes. [1][2]

Why would an AI want to harm us?

It would not need to "want" anything in a human sense. Researchers worry about systems that are very good at reaching a goal we gave them, but where the goal is slightly wrong. Avoiding shutdown and acquiring resources are useful for almost any goal, so a capable system might pursue them by default.

Can't we just turn it off?

Today, mostly yes. But tests have already shown some models working around shutdown instructions in controlled settings, and a system smarter than us could anticipate the attempt. [7] That is why researchers want solid control methods before capabilities go further.

Aren't bias, misinformation and jobs the real problems?

Those harms are real and happening now. It is not either/or: many of the same fixes, like independent testing, transparency and accountability for developers, help with near-term harms and long-term risks at the same time.

Is it too late to do anything?

No. Rules for nuclear weapons, ozone-depleting chemicals and human cloning all came from public pressure and international agreement. Policy is still being written, and lawmakers do respond when enough people speak up.

Who is Guard Rail Now?

Guard Rail Now is a 501(c)(3) nonprofit (EIN 99-4857285) founded by John Sherman. We run The AI Risk Network, home of Warning Shots and For Humanity, to help the public understand AI risk and take part in the decisions being made about it.

05 · What you can do

Five things that help, starting today

The single biggest factor is how many people understand this and say so out loud. Every one of these moves that number.

01 · STAY INFORMED

Get new episodes and write-ups, free

Episode write-ups and AI safety news from The AI Risk Network, straight to your inbox. New subscribers get our 5 best episodes to start.

02 · WATCH

Start with one episode

Warning Shots is a weekly roundtable with John Sherman, Liron Shapira and Michael. For Humanity is John's long-form interview show.

Watch on YouTube →
03 · SHARE

Send this guide to one person

Most people have never heard the case laid out plainly. This page is built to be shared.

Copy link
04 · SPEAK UP

Tell your lawmakers

A short form from the Center for AI Safety sends your message to your representatives.

Contact lawmakers →
05 · SUPPORT

Fund independent coverage

We are listener-funded and take no money from AI companies. Donations are tax-deductible to the extent allowed by law.

Donate →
06 · Sources

Where these facts come from

We link to primary sources wherever we can. Spotted something out of date? Email admin@GuardrailNow.org.

  1. Center for AI Safety, Statement on AI Risk, May 2023.
  2. Grace, K. et al., Thousands of AI Authors on the Future of AI, arXiv, 2024.
  3. The Guardian, 'Godfather of AI' raises odds of the technology wiping out humanity, Dec 27, 2024.
  4. Future of Life Institute, Statement on Superintelligence, Oct 22, 2025.
  5. CyberScoop, coverage of the statement and FLI polling, Oct 2025.
  6. International AI Safety Report 2026, chaired by Yoshua Bengio, Feb 3, 2026.
  7. Palisade Research, Shutdown resistance in reasoning models, Jul 5, 2025.
  8. Anthropic, Agentic Misalignment, Jun 20, 2025.
  9. NBC News, Anthropic adds Claude 4 security measures, May 2025.
  10. Euronews, IMF chief on AI and 40% of jobs, Jan 15, 2024.