Preface:

Please read the preface for this type of market and other similar third-party validated AI markets here.

Third-Party Validated, Predictive Markets: AI Theme

Market Description

ARC

The AI2 Reasoning Challenge (ARC) aims to promote research in advanced question-answering, in particular questions that require reasoning, use of commonsense knowledge, and other methods for deeper text comprehension. In particular, the ARC Challenge questions are those that are hard to answer with simple baselines.

Example ARC Question

Which property of a mineral can be determined just by looking at it?

• (A) luster

• (B) mass

• (C) weight

• (D) hardness

https://leaderboard.allenai.org/arc/submissions/public

https://paperswithcode.com/sota/common-sense-reasoning-on-arc-challenge

Papers with Code - ARC (Challenge) Benchmark (Common Sense Reasoning)

The current state-of-the-art on ARC (Challenge) is GPT-4 (few-shot, k=25). See a full comparison of 34 papers with code.

Market Resolution

As of the time of market creation in July 2023, the top submission is GPT-4 with 96.3:

Resolution Criteria

• We will define Superintelligence for the purposes of this question as, "achieving 99% accuracy on the test in question."

• Will any entry from the above two links result in a 99% Accuracy Rating? If so, resolves YES, otherwise NO.

20230727 - Changed title, "Superintelligence" to "Higher than Human Level"