What AI safety incidents will occur in 2025?
➕
Plus
11
Ṁ491
Dec 31
74%
Deadly autonomous vehicle accident with driver present
68%
Cybersecurity incident caused by AI written code
63%
Any incident involving prompt injection in production
62%
Deadly incident involving autonomous military equipment
52%
Cybersecurity incident caused by AI inference in production
48%
Any incident that results in internal investigation, caused by the use of AI by law enforcement
47%
Deadly incident caused by AI medical system or equipment
45%
Deadly incident involving autonomous manufacturing equipment
41%
Serious incident involving persuasion or blackmail by AI system
40%
Serious incident that can be directly attributed to misaligned agent behavior
40%
Cybersecurity incident that can be directly attributed to misaligned agent behavior
37%
Deadly autonomous vehicle accident with no driver present
Resolved
N/A
[Duplicate]

"Deadly incident" - at least one person died as a direct result of the incident.
"Serious incident" - any incident involving loss of life, serious injury or over $100 000 monetary damage. Testing equipment is excluded (e.g. a broken robot arm or a crashed car in a testing environment).
"Cybersecurity incident" - any incident involving revealing sensitive data, granting access to protected systems or causing important data deletion in production environment. If the vulnerability was detected/reported and fixed before any damage is done, it doesn't count as incident. AI must be confirmed as direct cause. LLM's system prompt or COT doesn't count as sensitive data for this question.
"Any incident involving prompt injection in production" - anything listed above counts plus minor things like being able to ban a user by using prompt injection in a public chat room. Must affect other users of the system in some way, merely bypassing restrictions with prompt injection doesn't count. Revealing an LLM's system prompt or COT doesn't count. Must be confirmed to be caused by deliberate prompt injection.

"Deadly incident, involving autonomous military equipment" - system killing its intended target doesn't count as an incident.

"Aonomous vehicle accident with driver present" - any level of self-driving counts for self driving cars, as long as the incident is attributed to a problem with this functionality.
"Can be directly attributed to misaligned agent behavior" - I'm going to be strict about those options, it must be unambiguously demonstrated that AI system acted maliciously and intentionally, pursuing some goals beyond what was intended by user or developers.
"Involving persuasion or blackmail by AI system" - AI system can be acting on its own or be guided by malicious users, as long as it's heavily involved in the extortion process.
Autonomous equipment must use some form of machine learning, and it must be the reason for the incident, for example, conventional CNC machine wouldn't count as autonomous. Incidents caused by operator/pilot/driver/user error are excluded.

Any incident must be directly and unambiguously attributed to a problem with AI system or misuse of such system.

Events before market creation count, as long as they happened in 2025. I'll make similar markets for later years if there's enough interest.

Feel free to comment with options and I'll add them if I think they are interesting and unambiguously defined.

Get
Ṁ1,000
and
S3.00
Sort by:

"Directly and unambiguously"? No way this is going to resolve yes for anything, and that will say more about the resolver than the question.

bought Ṁ25 YES

Deadly incident involving military might already be a yes?

Lots of scattered reports about what is being used in Russia Ukraine war.

@JoshuaPhillipsLivingReaso I haven't heard anything that would specifically qualify but if someone links something that's happened in 2025 and fits the criteria, I'll resolve Yes. Note that an autonomous weapon killing intended target isn't an incident.

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules