Preface:

Please read the preface for this type of market and other similar third-party validated AI markets here.

Third-Party Validated, Predictive Markets: AI Theme

Market Description

MMLU

Paper: https://arxiv.org/abs/2009.03300

Resolution Criteria:

https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu

Papers with Code - MMLU Benchmark (Multi-task Language Understanding)

The current state-of-the-art on MMLU is GPT-4 (few-shot, k=5). See a full comparison of 74 papers with code.

• GPT4 tops the list at 86.4 currently.

• We seem to have been gaining around 10 points per year from Jan 2022 until early 2023.

• Scaling out about 1/4th of the year hypothetically we should be at 93.9 by the end of 2023 assuming continued linear growth.

• However, the test might not happen by then, so assuming we get a test by end of October, that would be (0.583 years)*(10 points/year) = 5.83 additional points, + 86.4 = 92.2

• If > 92.2 by end of year, this resolves as YES.