Short-term AI 3.4: By June 2024 will SOTA on APPS be = 25%?

Question

APPS is the more challenging code benchmark (compared to HumanEval). SOTA at market creation is 15.7 by CodeRL. I will use Competition Pass@any.

Notable that the current SOTA is using a very old LLM as the base model, and yet it still beats davinci-002.

Other short-term AI 3 markets:

(https://manifold.markets/embed/vluzko/short-term-ai-31-by-june-2024-will)(https://manifold.markets/embed/vluzko/short-term-ai-32-by-january-2024-wi)(https://manifold.markets/embed/vluzko/shortterm-ai-33-by-june-2024-will-s)

Manifold Markets · Accepted Answer

No — resolved on Jun 8, 2024 by Manifold Markets prediction market.