Will a LLM beat human experts on GPQA by Jan 1, 2025? | Manifold

Will a LLM beat human experts on GPQA by Jan 1, 2025?

AI Technical AI Timelines LLMs

Basic

45

Ṁ12k

2025

53%

chance

1D

1W

1M

ALL

GQPA dataset here: https://arxiv.org/abs/2311.12022

"Human expert" means 74%.

Currently, GPT-4 gets 39%.

The LLM is allowed to use external tools (e.g. Google, Wolfram Alpha).

Get Ṁ600 play money

Related questions

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will the most interesting AI in 2027 be a LLM?

Will there be any simple text-based task that most humans can solve, but top LLMs can't? By the end of 2026

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will Google have a better LLM than OpenAI by 2025?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?

Sort by:

Usaar33boughtṀ100NO

Which set? Main? Diamond?

@Uaaar33 Extended. (That’s where the 74% number comes from)

sold Ṁ46 NO

AFAICT Anthropic models report only diamond. So will this rely on 3rd party evals?

If there are no Extended evals in the official model report, then yes I'll rely on 3rd party evals.

Related questions

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will the most interesting AI in 2027 be a LLM?

Will there be any simple text-based task that most humans can solve, but top LLMs can't? By the end of 2026

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will Google have a better LLM than OpenAI by 2025?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?

Related questions

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

Will the most interesting AI in 2027 be a LLM?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

Will there be any simple text-based task that most humans can solve, but top LLMs can't? By the end of 2026

Will AIs beat human experts in question-answering on the GPQA benchmark before January 1st, 2027?

Will OpenAI's next major LLM (after GPT-4) surpass 74% accuracy on the GPQA benchmark?

Will an LLM (a GPT-like text AI) defeat the World Champion at Chess before 2035?

Will Google have a better LLM than OpenAI by 2025?

Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?