Will the ARC-AGI Grand Prize be claimed in 2024?
Dec 4

>=85% performance on Chollet's abstraction and reasoning corpus, private set. As judged by Chollet et al.

2025 version: https://manifold.markets/JacobPfau/will-the-arcagi-grand-prize-be-clai-srb6t2awj1

Get Ṁ600 play money
Sort by:
bought Ṁ250 NO

Betting no based on the difficulty of YES resolution, in particular requiring models to work offline.

Can't use Gemini in the challenge. It has to run offline and with a maximum runtime of 12 hours on kaggle.

They make it sound a lot more interesting than it really is. They used 1000x the compute of prior sota to achieve the same results. The real ARC challenge is limited in compute to 12 hours runtime on kaggle and has no internet access (so no access to large LLMs).

Note that this prize doesn't allow for close source models to be used in doing the actual task.

Of course, distillation is possible etc.

bought Ṁ100 YES

Ah, I should have read this comment before I made a bet on yes 😅

"No ARC human baseline exists! http://arcprize.org/arc: "most humans can solve on average 85% of ARC-AGI tasks." But this study used the train set http://arcprize.org/guide: "The public training set is significantly easier than the...public evaluation and private evaluation set""

I tried solving about 20 public test set problems and they were all pretty easy as well. I don't know what the average human would get, but I doubt it would be much lower than 85%.

bought Ṁ250 NO

Chollet believes we’ll see an improvement from 35% to 50% in 2024. A score of 85% or better is required to win the Grand Prize.


Why does it close on Dec 4 rather than Dec 31?

@JacobPfau Ahh, thanks, I missed that.