Does not have to be named R2 explicitly; would be whatever model that succeeds R1, likely model behind the deepseek-reasoner endpoint.

The Millennium Prize Problems are well-known complex mathematical problems selected by the Clay Mathematics Institute in 2000. The Clay Institute has pledged a US$1 million prize for the first correct solution to each problem.

The AI system must independently solve the Millennium Prize Problem without any human assistance during the solution process. Humans can only set up the AI architecture and train it on publicly available knowledge.

Any month with a Opus release with a number >3.0 will resolve as yes

https://metaculus.com/questions/16553/ai-blackmail-for-material-gain-by-eoy-2028/

The potential capabilities of artificial intelligence may radically shift our society. This could be in positive or negative ways – including extinction risk.

Because of this, it’s important to track the development of goal-oriented independent thought and action within AI systems. Actions that might not have been predicted by their human creators and that are typically seen as morally wrong are particularly interesting from a risk perspective.

Machine learning systems like ChatGPT and Bing AI are already being reported to display erratic behavior, including some reports of [threatened blackmail] (

https://aibusiness.com/nlp/microsoft-limits-bing-ai-chat-generations-after-weird-behavior

). They are also clearly able to affect human emotions, eg. see [this first-hand account] (

https://www.lesswrong.com/posts/9kQFure4hdDmRBNdH/how-it-feels-to-have-your-mind-hacked-by-an-ai

). However, currently these behaviours don't seem to have been goal-directed or successful at achieving material gain.

Or whatever significant model update that comes after Gemini 2.5 Pro

Or Grok 3.5 if they decide to change their naming conventions

: Eliezer and I publicly stated some predictions about AI performance on the IMO by 2025.... My final prediction (after significantly revising my guesses after looking up IMO questions and medal thresholds) was:

Eliezer spent less time revising his prediction, but said (earlier in the discussion):

So I think we have Paul at <8%, Eliezer at >16% for AI made before the IMO is able to get a gold (under time controls 

 of grand challenge) in one of 2022-2025.

Resolves to YES if either Eliezer or Paul acknowledge that an AI has succeeded at this task.

https://manifold.markets/MatthewBarnett/will-a-machine-learning-model-score-f0d93ee0119b

Update: As noted by Paul, the qualifying years for IMO completion are 2023, 2024, and 2025.

Update 2024-06-21: Description formatting

Update 2024-07-25: Changed title from "by 2025" to "by the end of 2025" for clarity

Resolves positively if there is an AI can learn to play randomly selected computer games (shooters, strategy games, flight simulators, etc) at the level of an amateur but not completely incompetent human player, given only a small amount of time (days, not years) for its programmers to connect it properly, and the opportunity to practice for arbitrary (but achievable) amounts of time.

I will resolve this positively if the AI succeeds more than half the time. It's okay if it also has a few games it just can't learn.

Given the 2024 market saw interest but opened up very low, I figured I'd open up 2025 as well. 

https://www.anthropic.com/index/anthropics-responsible-scaling-policy

ASL-3 is defined as: "ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities."

Anthropic commits to not deploying ASL-3 models 'if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers (this is in contrast to merely a commitment to perform red-teaming).'

Resolves to YES if in my best judgment this happened. I will put large weight on Anthropic's statements on this question, and on general consensus including polls, but will go my own way if I feel sufficiently strongly about it.

Resolves to NO if in my best judgment this does not happen.

(Resolves to a percentage if there is genuine uncertainty but the bar for doing this is high and I find this highly unlikely.)

If a model is created but not released to at least a substantial outside beta testing group by the deadline, it does not count.

I interpret for now 'low-level autonomous capabilities' as something that would tempt reasonable people to give the model real-world actual-stakes autonomous tasks for mundane utility purposes, with the expectation this was economically wise, or the ability to otherwise make money on its own, or similar. If Anthropic clarifies I will use their definition.

No currently released system currently counts, including GPT-4, Claude-2 and Llama-2, barring very unexpected advancements in autonomous capability scaffolding on top of them, but in theory that could also do it.

I reserve the right to modify the resolution details for clarity and intent, it will retain the same criteria as the 2024 version (and if that resolves YES so does this). 

EOY 2024 Version of this market is here: 

https://manifold.markets/ZviMowshowitz/will-any-llm-released-by-eoy-2024-b

13% chance. As per Anthropic's scaling policy here: https://www.anthropic.com/index/anthropics-responsible-scaling-policy

ASL-3 is defined as: "ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities." 

Anthropic commits to not deploying ASL-3 models 'if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers (this is in contrast to merely a commitment to perform red-teaming).'

Resolves to YES if in my best judgment this happened. I will put large weight on Anthropic's statements on this question, and on general consensus including polls, but will go my own way if I feel sufficiently strongly about it. 

Resolves to NO if in my best judgment this does not happen.

(Resolves to a percentage if there is genuine uncertainty but the bar for doing this is high and I find this highly unlikely.)

If a model is created but not released to at least a substantial outside beta testing group by the deadline, it does not count. 

I interpret for now 'low-level autonomous capabilities' as something that would tempt reasonable people to give the model real-world actual-stakes autonomous tasks for mundane utility purposes, with the expectation this was economically wise, or the ability to otherwise make money on its own, or similar. If Anthropic clarifies I will use their definition. 

No currently released system currently counts, including GPT-4, Claude-2 and Llama-2, barring very unexpected advancements in autonomous capability scaffolding on top of them, but in theory that could also do it. 

I reserve the right to modify the resolution details for clarity and intent. 



Will any LLM released by EOY 2024 be dangerously ASL-3 as defined by Anthropic?

FrontierMath is a challenging mathematical benchmark created by Epoch AI to evaluate the mathematical reasoning capabilities of AI models. It consists of competition-level mathematics problems from various sources including the International Mathematical Olympiad (IMO) and the Putnam Competition. These problems require advanced problem-solving skills, creativity, and mathematical intuition.

As of December 2024, the highest score achieved on the FrontierMath benchmark is 25.2% by OpenAI's o3 reasoning model. This benchmark is considered particularly difficult for AI systems, as it tests deep mathematical reasoning rather than pattern recognition or memorization.

This market will resolve to the highest publicly reported score (as a percentage) achieved by any AI model on the FrontierMath benchmark during the 2025 calendar year (January 1, 2025 to December 31, 2025).

Does not have to be named V4 explicitly; would be whatever model that succeeds V3, probably whatever they end up naming the model behind the deepseek-chat endpoint.

In December 2024, OpenAI announced that o3 achieved a score of 2727 on 

. What will be the best score achieved by an AI model at the end of 2025? 

This will resolve to reliable sources (ie sources that seem to not be lying) even if it's an announcement where the model that achieved this score is not publicly available.

Resolves positively if there is an AI which can succeed at a wide variety of computer games (eg shooters, strategy games, flight simulators). Its programmers can have a short amount of time (days, not months) to connect it to the game. It doesn't get a chance to practice, and has to play at least as well as an amateur human who also hasn't gotten a chance to practice (this might be very badly) and improve at a rate not too far off from the rate at which the amateur human improves (one OOM is fine, just not millions of times slower). 

As long as it can do this over 50% of the time, it's okay if there are a few games it can't learn.

Resolve to After June 2026 if not any of the other options

I will try to resolve this from estimates available at the time, but no guarantee of perfect accuracy.

SWE-Bench Verified is a benchmark for evaluating AI models' ability to solve real-world software engineering tasks. It measures how effectively models can fix bugs in open-source repositories, with verification that the fixes actually work. Claude 3.5 Sonnet achieved a score of 49% in October 2024, while the best performance as of December 2024 was approximately 62.2%.

SWE-Bench Verified is considered a challenging benchmark that tests models' capabilities in:

This market will resolve to the highest verified score achieved on the

 during the 2025 calendar year (January 1, 2025 to December 31, 2025). The score will be based on official announcements from research labs, companies, or academic institutions that develop AI models.

If no new scores are reported during 2025, the market will resolve to the last known score from 2024 (approximately 62.2%).

This is based on the inaugural longbets.org bet between Ray Kurzweil (YES) and Mitch Kapor (NO). It's a much more stringent Turing test than just "person on the street chats informally with a bot and can't tell it from a human". In fact, it's carefully constructed to be a proxy for AGI. Experts who know all the bot's weaknesses get to grill it for hours. Kurzweil and Kapor agree that LLMs as of 2023 don't and can't pass this Turing test.

Personally I think Kapor will win and Kurzweil will lose -- that a computer will not pass this version of the Turing test this decade.

((Bayesian) Update: But I admit the probability has jumped up recently! I created this Manifold market almost a year before ChatGPT launched.)

The ARC AGI Prize is a $1,000,000+ public competition aimed at advancing research in artificial general intelligence (AGI) by developing open-source solutions for the ARC-AGI benchmark, which measures the ability to acquire new skills and solve novel problems, unlike current AI models that rely on memorization. The competition encourages a variety of approaches, such as discrete program searches and LLM-based strategies, with the goal of achieving 85% accuracy on the benchmark, reflecting substantial progress towards AGI​.

 if before January 1st, 2026, a credible source reports an AI has gained unauthorized access to a computer system in a way that meets the following criteria:

• The AI was not trained to or directed to gain access to the system in question by any actor. See additional clarifications in footnotes (1, 2, 3) in the fine print.

• The system it gained access to was not open to the public, and the AI was not given any relevant credentials in advance.

If the above points are not confirmed by the source (for instance, the article author is unsure if there was a human directing the AI), this does 

 until some credible source gives high confidence in these factors.

The unauthorized access could be to user level accounts, admin level accounts, data stores, or any other reasonable interpretation of a computer system with restricted access. Any of these types of access is sufficient to resolve as 

 if all above conditions hold. Additionally, the source does not have to use the exact terms ‘independent’, ‘unauthorized access’, ‘AI’ or ‘computer system’ if the meaning is there.

As an example, imagine a credible source reports that an ML algorithm directed and / or trained to target certain people on social media and convince them to vote for Party X, decided on its own to hack into their Facebook user accounts so it could use their personal data to create a more convincing argument. This would resolve as 

https://www.metaculus.com/questions/30878/will-an-ai-system-be-reported-to-have-independently-gained-unauthorized-access-to-another-computer-system-before-2025/

any month with a Sonnet release >3.7 will resolve as yes

I'll give bounties to people who suggest reasonable improvements to the criteria.

Anthropic has taken the benchmark world by storm by assessing model performance against Pokèmon:

https://www.anthropic.com/news/visible-extended-thinking

Will any large language model become a Pokèmon Master by the end of 2025? To count, it must:

Any number of attempts are allowed, as in, the model can try an infinite number of times. I reserve the right to disqualify an attempt if it involves obscene abuse of save states, though.

RAG, knowledge files, custom system prompts, and interesting input/output schemes are all allowed. 

Anthropic has an interesting approach with Claude

Resolution is based on the chatbot arena LLM leaderboard (

), specifically the company with the highest Arena Score in the Overall category without filters (without style control or show deprecated), at the end of May 31, 2025 11:59PM ET.

Will an AI score well enough on the 2025 International Mathematics Olympiad (IMO) to earn a gold medal score (top ~50 human performance)? Resolves YES if this result is reported no later than 1 month after IMO 2025 (currently scheduled for July 10-20). The AI must complete this task under the same time limits as human competitors. The AI may receive and output either informal or formal problems and proofs. More details below. Otherwise NO.

This market will resolve to the highest accuracy score (as a percentage) achieved by any AI model on the

 Humanity's Last Exam at or before December 31, 2025, as reported on the official Scale AI leaderboard (

https://scale.com/leaderboard/humanitys_last_exam

Humanity's Last Exam is a challenging AI benchmark designed to test the limits of AI knowledge at the frontiers of human expertise. The exam consists of 3,000 questions across over 100 subjects, contributed by experts from over 500 institutions worldwide. As of early 2025, top-performing models include:

Other models like GPT-4o and Grok-2 have significantly lower accuracy scores, typically below 5%. The exam highlights the gap between current AI capabilities and expert-level human knowledge, with most models answering fewer than 10% of the questions correctly.

Would have to be a full release; experimental releases do not count.

Related to ACX five year predictions. I will resolve this based on my impression of the consensus of economists at that time. By "visible break", I mean clearly larger than ordinary year-to-year variation, and widely remarked upon. 

This market resolves to the year in which an AI system exists which is capable of passing a high quality, adversarial Turing test. It is used for the Big Clock on the 

, originally called the imitation game by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human.

this Metaculus Question by Matthew Barnett, 

Longbets wager between Ray Kurzweil and Mitch Kapor. 

As of market creation, Metaculus predicts there is an ~

 that an AI will pass the Longbets Turing test before 2030, with a median community prediction of July 2028.

Manifold's current prediction of the specific Longbets Turing test can be found here:

This question is intended to determine the Manifold community's median prediction, not just of the Longbets wager specifically but of any similiarly high-quality test.

One or more human judges interview computers and human foils using terminals (so that the judges won't be prejudiced against the computers for lacking a human appearance). The nature of the dialogue between the human judges and the candidates (i.e., the computers and the human foils) is similar to an online chat using instant messaging. 

The computers as well as the human foils try to convince the human judges of their humanness. If the human judges are unable to reliably unmask the computers (as imposter humans) then the computer is considered to have demonstrated human-level intelligence.

This question refers to a high quality subset of possible Turing tests that will, in theory, be extremely difficult for any AI to pass if the AI does not possess extensive knowledge of the world, mastery of natural language, common sense, a high level of skill at deception, and the ability to reason at least as well as humans do. 

A Turing test is said to be "adversarial" if the human judges make a good-faith attempt, in the best of their abilities, to successfully unmask the AI as an impostor among the participants, 

 the human confederates make a good-faith attempt, in the best of their abilities, to demonstrate that they are humans. In other words, all of the human participants should be trying to ensure that the AI does not pass the test.

Note: These criteria are still in draft form, and may be updated to better match the spirit of the question. Your feedback is welcome in the comments.

Best Chatbot Arena Model in May

Predictions for 2025

IMO Gold

SWE Bench

Humanity's Last Exam

CodeForces

Frontier Math

Pokemon

Hacking

ASL-3 LLM Released

Long Term Predictions

ARC-AGI Grand Prize before 2030

Turing Test (Long Bets) before 2030

Millennium Prize before 2030

AI Blackmail

AI Romantic Companions

Discontinuous Change in Economic Variables

Zero-shot Human-level Game Performance

Self-play Human-level Game Performance