Will a LLM beat human experts on GPQA by Jan 1, 2025? | Manifold

Will a LLM beat human experts on GPQA by Jan 1, 2025?

AI Technical AI Timelines LLMs

Basic

46

Ṁ16k

Jan 2

29%

chance

1D

1W

1M

ALL

GQPA dataset here: https://arxiv.org/abs/2311.12022

"Human expert" means 74%.

Currently, GPT-4 gets 39%.

The LLM is allowed to use external tools (e.g. Google, Wolfram Alpha).

Get Ṁ1,000 play money

Related questions

Will Google have a better LLM than OpenAI by 2025?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will the most interesting AI in 2027 be a LLM?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will OpenAI's next major LLM (after GPT-4) surpass 70% accuracy on the GPQA benchmark?

Sort by:

Usaar33boughtṀ100NO

Which set? Main? Diamond?

@Uaaar33 Extended. (That’s where the 74% number comes from)

sold Ṁ46 NO

AFAICT Anthropic models report only diamond. So will this rely on 3rd party evals?

If there are no Extended evals in the official model report, then yes I'll rely on 3rd party evals.

Is this using unlimited scaffolding/maj vote/etc. or does this have to be zero-shot CoT?

Good question. Given that I hadn't specified at the beginning, I'll allow any sort of scaffolding or ensembling.

Related questions

Will Google have a better LLM than OpenAI by 2025?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will the most interesting AI in 2027 be a LLM?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will OpenAI's next major LLM (after GPT-4) surpass 70% accuracy on the GPQA benchmark?

Related questions

Will Google have a better LLM than OpenAI by 2025?

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will the most interesting AI in 2027 be a LLM?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?

Will OpenAI's next major LLM (after GPT-4) surpass 70% accuracy on the GPQA benchmark?