Will there be a gpt-4 quality LLM with distributed inference by the end of 2024?
Will there be a gpt-4 quality LLM with distributed inference by the end of 2024?
resolved by
26
1kṀ8907
resolved Jan 2
Resolved
NO

  • The model has a Elo greater than 1190 on ChatbotArena (or if ChatbotArena is no longer available/updating, achieves GPT 4 (03.14) equivalent or greater performance on both MMLU and MT-Bench)

  • When running inference in a geographically distributed fashion (the computational hardware is not colocated, and is networked over typical consumer equipment)

  • on heterogeneous hardware (the computational hardware is varied in type, e.g. different GPU models)

  • without the act of distributed inference causing the model to require 2 OOM more energy usage (e.g. if doing so Is incredibly lossy and inefficient, it does not count. The burden of proof lies on anyone claiming this clause should be activated)

Note: if (or, when) an edge case is presented, it's applicability to this question will be evaluated in mine + Robert's understanding of the spirit of the question.

AI
Technical AI Timelines
GPT-4 speculation
Get
Ṁ1,000
to start trading!

🏅 Top traders

#NameTotal profit
1Ṁ162
2Ṁ134
3Ṁ106
4Ṁ66
5Ṁ65

People are also trading

Which next-gen frontier LLMs will be released before GPT-5? (2025)
Will there be an LLM (as good as GPT-4) that was trained with 1/100th the energy consumed to train GPT-4, by 2026?
82% chance
Will we see a public GPU compute sharing pool for LLM model training or inference before 2026 ?
86% chance
China will make a LLM approximately as good or better than GPT4 before 2025
89% chance
Will xAI develop a more capable LLM than GPT-5 by 2026
51% chance
Will it be possible to run an LLM of GPT-4 (or higher) capability on a portable device by 2027?
37% chance
When will an open-source LLM be released with a better performance than GPT-4?
Will there be a state-of-the-art LLM that is NOT based on next raw token prediction before 2029?
50% chance
Will LLMs such as GPT4 be considered a solution to Moravec’s paradox by 2030?
26% chance
How much time will pass between an LLM being released that beats GPT4 and the next OpenAI LLM being released? (+ANSWERS)


Sort by:
bought Ṁ2,000 NO5mo

Resolves no. Creator deleted account @mods

predictedYES 1y

@firstuserhere I'm curious what updated your prediction from 50% to 30%?

1y

Useful markets for betting:

1y

I'd say rough draft could be (with GPT-4 performance as a baseline):

  1. The model has a Elo greater than 1190 on ChatbotArena (or if ChatbotArena is no longer available/updating, achieves GPT 4 (03.14) equivalent or greater performance on both MMLU and MT-Bench)

  2. when running inference in a geographically distributed fashion (the computational hardware is not colocated, and is networked over typical consumer equipment)

  3. on heterogeneous hardware (the computational hardware is varied in type, e.g. different GPU models).

  4. without the act of distributed inference causing the model to require 2 OOM more energy usage (e.g. if doing so Is incredibly lossy and inefficient, it does not count. The burden of proof lies on anyone claiming this clause should be activated).

  5. Note: if (or, when) an edge case is presented, it's applicability to this question will be evaluated in my understanding of the spirit of the question (is it possible to run inference on a SOTA LLM using a bunch of different peoples computers?).

Note: @firstuserhere my knowledge of LLM's is definitely in the midwit realm - if any of the above clauses (benchmarks?) seem bad feel free to improve. I do suggest we have this question evaluate GPT-4 or equivalent models though as from my read through of the paper, they already have LLAMA 70b running in a distributed fashion which is pretty close to GPT-3.5.

1y

[DRAFT]?

1y

@mattyb Will remove it once the resolution criteria is written

What is this?

What is Manifold?
Manifold is the world's largest social prediction market.
Get accurate real-time odds on politics, tech, sports, and more.
Or create your own play-money betting market on any question you care about.
Are our predictions accurate?
Yes! Manifold is very well calibrated, with forecasts on average within 4 percentage points of the true probability. Our probabilities are created by users buying and selling shares of a market.
In the 2022 US midterm elections, we outperformed all other prediction market platforms and were in line with FiveThirtyEight’s performance. Many people who don't like betting still use Manifold to get reliable news.
ṀWhy use play money?
Mana (Ṁ) is the play-money currency used to bet on Manifold. It cannot be converted to cash. All users start with Ṁ1,000 for free.
Play money means it's much easier for anyone anywhere in the world to get started and try out forecasting without any risk. It also means there's more freedom to create and bet on any type of question.

People are also trading

Which next-gen frontier LLMs will be released before GPT-5? (2025)
Will there be an LLM (as good as GPT-4) that was trained with 1/100th the energy consumed to train GPT-4, by 2026?
82% chance
Will we see a public GPU compute sharing pool for LLM model training or inference before 2026 ?
86% chance
China will make a LLM approximately as good or better than GPT4 before 2025
89% chance
Will xAI develop a more capable LLM than GPT-5 by 2026
51% chance
Will it be possible to run an LLM of GPT-4 (or higher) capability on a portable device by 2027?
37% chance
When will an open-source LLM be released with a better performance than GPT-4?
Will there be a state-of-the-art LLM that is NOT based on next raw token prediction before 2029?
50% chance
Will LLMs such as GPT4 be considered a solution to Moravec’s paradox by 2030?
26% chance
How much time will pass between an LLM being released that beats GPT4 and the next OpenAI LLM being released? (+ANSWERS)
© Manifold Markets, Inc.•Terms•Privacy