Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?
14
1kṀ560
2051
73%
chance

"Entry level" is deliberately fuzzy: in 2022 terms this would look like an AI (or AIs) that is assigned an issue, checks out code, makes edits, and submits a PR (that is accepted). Rough criteria: the AI acts with little oversight, performs similar (coding) work to entry-level coders at the time, the issue/task assignment is not *significantly* specialized for an AI (e.g. no full technical specs if the same wouldn't be given to a human coder) AI being used in this way in significant open source projects counts as "industry use". If there are technical demos of such AIs but none of them are actually being used question resolves as no. No requirement that it be a single model. A group of specialized models working together counts. If superhuman performance is not achieved by market end, resolves N/A.

Dec 20, 12:15am: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use? → Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Get
Ṁ1,000
to start trading!
Sort by:

Are there any specific existing benchmarks you had in mind?

Mana is worth less if this is true due to forthcoming end of world.

Related questions

Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 5 years before there are "entry level" AI programmers in industry use?
92% chance
When will an AI model be better than me at competitive programming?
Benchmark Gap #1: Once we have a language model that achieves expert human performance on all *current* major NLP benchmarks, how many years will it be before we have an AI with human-level language skills?
4.3
Benchmark Gap #2: Once we have an algorithm with human level sample efficiency for major RL benchmarks, how many years will it be before there is an algorithm with human level sample efficiency on essentially all AAA video game tasks?
1.6
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
63% chance
© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules