There is not currently an established human baseline for François Chollet's Abstraction and Reasoning Corpus (ARC-AGI).
On the public training set humans solve 84% of the tasks.
It it known that the public training set is easier than the public evaluation set and the private evaluation set. The public and private evaluation sets are apparently the roughly the same level of difficulty.
For the first credible human baseline study, what fraction of evaluation set tasks will humans successfully solve?
Note that this can include tasks from either the public or private evaluation sets.
In the extremely unlikely case that the number would fit in two intervals, the lowest will be chosen.
1,000
3.00We have a new preprint estimating human performance on the full training and evaluation sets of ARC: https://arxiv.org/abs/2409.01374. The empirical average performance on all tasks from the training set is 76.2%, while for the evaluation set it is 64.2%.