What will be the human baseline for the Abstraction and Reasoning Corpus (ARC-AGI)?
What will be the human baseline for the Abstraction and Reasoning Corpus (ARC-AGI)?
18
1kṀ3065
resolved Sep 10
Resolved
NO
90-100%
Resolved
NO
75-90%
Resolved
YES
60-75%
Resolved
NO
35-60%
Resolved
NO
15-35%
Resolved
NO
0-15%

There is not currently an established human baseline for François Chollet's Abstraction and Reasoning Corpus (ARC-AGI).

On the public training set humans solve 84% of the tasks.

It it known that the public training set is easier than the public evaluation set and the private evaluation set. The public and private evaluation sets are apparently the roughly the same level of difficulty.

For the first credible human baseline study, what fraction of evaluation set tasks will humans successfully solve?

Note that this can include tasks from either the public or private evaluation sets.

In the extremely unlikely case that the number would fit in two intervals, the lowest will be chosen.

Get
Ṁ1,000
to start trading!

🏅 Top traders

#NameTotal profit
1Ṁ240
2Ṁ106
3Ṁ106
4Ṁ80
5Ṁ35


Sort by:
6mo

We have a new preprint estimating human performance on the full training and evaluation sets of ARC: https://arxiv.org/abs/2409.01374. The empirical average performance on all tasks from the training set is 76.2%, while for the evaluation set it is 64.2%.

6mo

@WaiKeenVong Amazing!! Thanks for doing this!

6mo

@WaiKeenVong Resolving based on this :)

bought Ṁ100 NO9mo

Only one interval resolves YES, right? Shouldn't this have been a linked market?

9mo

Oh yeah… hmm I think I messed up

What is this?

What is Manifold?
Manifold is the world's largest social prediction market.
Get accurate real-time odds on politics, tech, sports, and more.
Or create your own play-money betting market on any question you care about.
Are our predictions accurate?
Yes! Manifold is very well calibrated, with forecasts on average within 4 percentage points of the true probability. Our probabilities are created by users buying and selling shares of a market.
In the 2022 US midterm elections, we outperformed all other prediction market platforms and were in line with FiveThirtyEight’s performance. Many people who don't like betting still use Manifold to get reliable news.
ṀWhy use play money?
Mana (Ṁ) is the play-money currency used to bet on Manifold. It cannot be converted to cash. All users start with Ṁ1,000 for free.
Play money means it's much easier for anyone anywhere in the world to get started and try out forecasting without any risk. It also means there's more freedom to create and bet on any type of question.
© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules