Will AI cause an incident resulting in $1b of losses or 100 lost lives?

Question

With AI models becoming more powerful, they are starting to be used in more and more high-stakes scenarios. This means that an AI mistake could have significant consequences. In this question I'm considering situations in which AI actions are directly causing an incident, accident or catastrophe resulting in $1 billion of damages. If the event causes loss of life, each death counts as $10 million of damages.

The incident should be a direct consequence of the AI's decisions, actions or output. It should not have happened if the AI wasn't involved. The system at fault has to be a sufficiently advanced AI model. A traditional software bug does not count, neither does a simple narrow-purpose ML model (e.g. OCR or speech recognition).

The question concerns a single event, cumulative damage from multiple incidents doesn't count.

Examples of incidents that would qualify:

An aircraft piloted by an AI crashes, resulting in sufficiently high damages/loss of life

A chatbot convinces 100 people to commit suicide

AI-lead military hardware makes a strike, unintended by its operators. (Provided it wouldn't have happened if the strike was directed manually.)

A building designed by an AI collapses

An AI-lead cyber attack causes a significant outage of a widely used service (like Gmail), or a significant leak. The attack shouldn't be possible without the AI involvement.

Stock market swings are generally excluded from the considered incidents, in particular the following scenarios do not qualify:

A company's stock sink due to reputation damage caused by something that a company-owned AI has said.

An AI causes zero-sum temporary market crash that soon recovers. (Like 2010 flash crash https://en.wikipedia.org/wiki/2010_flash_crash)

The harm caused by deepfakes/AI-generated libel is a bit of a grey area. If an AI is just used as a tool by human actors (like Photoshop), it will not count. On the other hand, if some sort of a campaign is kickstarted by humans, but is largely lead by an AI, it would qualify. I'll resolve to N/A if it's somewhere in between. (I'm open to suggestions in the comments as to how delineate this more clearly.)

I will not bet on this market.

Update 2025-02-06 (PST) (AI summary of creator comment): Human involvement: If a human instructs the AI to cause the disaster without providing detailed planning or control, the event qualifies. Limited AI role: If the AI is used merely as a research assistant without overall control of the disaster planning, the event does not qualify.

These clarifications help distinguish between direct AI-led incidents and scenarios where the AI only plays a supporting role.

Manifold Markets · Answer

Per Manifold Markets prediction market, In 2030 or earlier, followed by In 2029 or earlier and In 2028 or earlier are most likely. See the market for live updates (73 traders, as of Apr 1, 2026).