Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?

14

220Ṁ2546

resolved Jan 1

Resolved

NO

1H

6H

1D

1W

1M

ALL

Resolves "Yes" if, at time of closure, there is an entry on the SWE-bench leaderboard (https://www.swebench.com/) with score greater or equal to 90%.

Linked Questions:

Market context

Technical AI Timelines

Get

1,000

to start trading!

🏅 Top traders

#	Name	Total profit
1		Ṁ54
2		Ṁ32
3		Ṁ21
4		Ṁ16
5		Ṁ9

Sort by:

What if there's evidence that the training data is contaminated with the SWE-Bench tasks somehow?

@DavidFWatson That's an excellent question. Let's explore possibilities:

This could be included in the question, i.e. what matters is only the number on the benchmark, regardless of whether it was gamed
I could wait a certain amount of time to check if no controversy emerges. Feels like one month would be safe. The question then resolves yes if one month after the deadline, I judge that there is no consensus that the number was gamed. This makes the question more informative.

People are also trading

Will any AI solve more than four of AI 2027 Marcus-Brundage tasks in 2025?

By 2026 will there be autonomous AI good enough that I use it?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

What will AI score on TheAgentCompany benchmark in early 2026?

Will an AI system capable of doing 50% of knowledge job arrive by 2027?

Will I believe my prediction about AI enabling more SWEs to solve less lucrative problems by shrinking team sizes to have been fulfilled by EoY 2030

Will AGI be created before the beginning of 2050? (Definition: Autonomous system surpassing majority of economic tasks)

Related questions

Will any AI solve more than four of AI 2027 Marcus-Brundage tasks in 2025?

By 2026 will there be autonomous AI good enough that I use it?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

What will AI score on TheAgentCompany benchmark in early 2026?

Will an AI system capable of doing 50% of knowledge job arrive by 2027?

Will I believe my prediction about AI enabling more SWEs to solve less lucrative problems by shrinking team sizes to have been fulfilled by EoY 2030

Will AGI be created before the beginning of 2050? (Definition: Autonomous system surpassing majority of economic tasks)

© Manifold Markets, Inc.•Terms•Privacy