Full question: "best off-the-shelf AI system will have scored better than X% of all other participants in a widely recognized competitive programming contest"

Resolution Criteria:

An answer of 100% means the AI system is the best among all humans; 0% means worst. The programming contest must have 1000+ humans participate, and must be widely recognized as a competitive programming contest that people usually enter to win; a contest on codeforces counts, as does 'advent of code', or any similar contest. The system must not actually compete if it is against the terms of service; it's okay to back-test (by a very small delay to ensure the system can not have seen the answer). But it must follow all the same rules, including time constraints and number of submissions.

Motivation and Context:

Current AI systems compete at about the level of an average human in competitive programming contests. This is not to say they do as well at the job of being a software developer, but at the specific task of solving well-defined programming problems, they are about as good as an average human. How far will this continue? Will AI systems be able to out-compete the best humans at competitive programming?

Question copied from: https://nicholas.carlini.com/writing/2024/forecasting-ai-future.html