Will the gap between open-weights and frontier models on GPQA be at most 7%? | Manifold

Will the gap between open-weights and frontier models on GPQA be at most 7%?

Basic

5

Ṁ150

2026

52%

chance

1D

1W

1M

ALL

At the end of 2026, there will be a model that performs best on GPQA. There will also be an open-weights model that performs best on GPQA.

Question resolves positively if and only if the score of the best open-weights model on 0-shot CoT GPQA is at most 7% less than the score of the best-performing model on 0-shot CoT GPQA.

As of the time of writing, the model that performs best on GPQA is Claude Sonnet 3.5, with a score of 59.4. The best performing open-weights model is Llama 3.1-405B, with a score of 51.1. This would not be sufficient for a positive resolution, as the gap is 8.3%. If the gap is exactly 7%, the question still resolves positively, but if it is 7.1%, it resolves negatively. The question also resolves positively if open-weights models are at the frontier on GPQA (i.e. if they beat closed-weights models).

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

bought Ṁ25 YES

Interesting question! I won't be surprised if the benchmark saturates by the end of 2026.

Maybe worth a question as well 🤔

If GPQA saturates by 2026, then this question would almost always resolve yes. It might be worth asking the question in a more general way, like "will an open weights model get within 7% of performance average on leading benchmarks in 2026"

Makes sense, I guess I wanted to have something concrete, still being influenced by the Metaculus question formulation.

I don't like saying "leading benchmark" because that's quite ambiguous. SWE-bench seems difficult for anything at the moment, but it's not limited to single models.

Related questions

What will be the best score on the GPQA benchmark before 2025?

Will model weights obtaining >90% on the Winogrande benchmark become publicly available in 2024?

Will a LLM beat human experts on GPQA by Jan 1, 2025?

Will OpenAI allow near full access to the weights of their best-trained model to an external auditor by the end of 2030?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Will there be a model that has a 75% win rate against the latest iteration of GPT-4 as of January 1st, 2025?

Will OpenAI's next major LLM (after GPT-4) surpass 70% accuracy on the GPQA benchmark?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will OpenAI make GPT3.5 or 4 model weights open before 2025?

The gap will close between the quality of open source language models and Google's internal language models in 2 years

Related questions

What will be the best score on the GPQA benchmark before 2025?

Will there be a model that has a 75% win rate against the latest iteration of GPT-4 as of January 1st, 2025?

Will model weights obtaining >90% on the Winogrande benchmark become publicly available in 2024?

Will OpenAI's next major LLM (after GPT-4) surpass 70% accuracy on the GPQA benchmark?

Will a LLM beat human experts on GPQA by Jan 1, 2025?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will OpenAI allow near full access to the weights of their best-trained model to an external auditor by the end of 2030?

Will OpenAI make GPT3.5 or 4 model weights open before 2025?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

The gap will close between the quality of open source language models and Google's internal language models in 2 years

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules