This will resolve positively if a major news source reports that an AI entered a programming competition with at least two other good participants, and won the top prize. A good participant will be defined as someone who could be expected to perform above the level of an average big tech company employee; if there are at least twenty-five participants not specifically selected against being skilled, this will be considered true by default. The competition could be sponsored by the AI company as long as it meets the other criteria and is generally considered fair.
This is question #49 in the Astral Codex Ten 2023 Prediction Contest. The contest rules and full list of questions are available here. Market will resolve according to Scott Alexander’s judgment, as given through future posts on Astral Codex Ten.
1,000
3.00Can't find any evidence for YES. Resolves NO
https://www.metaculus.com/questions/14279/ai-programming-challenge-in-2023/
AI helped coding in a security competition: https://www.wsj.com/articles/chatgpt-helped-win-a-hackathon-96332de4
I said 80% for this one in my predictions: https://www.newslettr.com/p/more-on-prediction-markets
The main concern I have is "there aren't that many programming contests".
This has not been rising at Metaculus, although Manifold has been catching up to it. This should damn well be rising, as both progress has been impressive recently with no signs of actual stopping due to risk, and there is now a ton more hype and funding and reason to put on this kind of show, and more incentive for ‘a major news outlet to report it,’ note that this does not require a front page or anything like that. I’m going to put M100 on YES, bringing this to 42%. I notice my gut-level fair is at least 50% here.
A promising new paper on getting LLMs to solve coding problems:
https://mobile.twitter.com/ericzelikman/status/1618426056163356675
Note that the question defines: "A good participant will be defined as someone who could be expected to perform above the level of an average big tech company employee".
Now, big tech interviews do not particularly select for competitive programming performance, nor do competitive programmers select for big tech employment that much.
@dp The thing about 25 participants automatically qualifying the contest points even more to YES. However, a contest like this actually being held in 2023 and being notable enough for reporting is... not that likely.