In yottaFLOPs (10^24), how much compute will GPT-4 be trained with?

AI Technical AI Timelines GPT-4 speculation

Mini

Ṁ791

2027

expected

ALL

Resolves N/A if the number is not public/calculateable from public information, or if GPT-4 is not released by market close.

Get Ṁ600 play money

Related questions

How much compute will be used to train GPT-5?

Will GPT-4 be trained (roughly) compute-optimally using the best-known scaling laws at the time?

30% chance

What will be true about GPT-4.5?

Will there be an LLM (as good as GPT-4) that was trained with 1/100th the energy consumed to train GPT-4, by 2026?

52% chance

Was GPT-4 trained in 4 months or less?

59% chance

Will GPT-4 be trained on more than 10T text tokens?

36% chance

What hardware will GPT-5 be trained on?

When will GPT-5 finish training?

GPT-5 trained with >=24k GPUs?

81% chance

Will GPT-5 have over 10 trillion parameters?

Sort by:

It seems very unlikely that the release of GPT-4 will resolve this. However the market does not close until 2027, and I will leave it open until then in case the information is either (credibly) leaked or OpenAI decides to release it after the fact. If neither of those occurs the market will resolve N/A

opened a Ṁ1,000 HIGHER at 22% order

Hey, I think this market should be resolved. From what I can tell OpenAI people treated the semianalysis leak (https://www.semianalysis.com/p/gpt-4-architecture-infrastructure) seriously and it reports 280B active parameters for 13 trillion tokens, so 13T tokens * 0.28T params x 6 flops/token/param = 21.84 x 10^24 flops.

I think the semianalysis leak is probably accurate, but I'm being cautious about resolving from leaks. I don't think there's any rush to resolve this now when I can just wait and see if more information comes out?

hmm, I find this leak highly credible because ~every AI person I know takes it seriously. Seems fine to wait though.

(Epistemic status: I’m doing all this calculation from memory so might be wrong) GPT-3 was trained on 3.2e23 FLOPs (general calculation is C=6ND, N number of params and D datasize), I think we should expect around twice the size and 50x the dataset size, in which case 3.2e25 FLOPs should be reasonable for GPT-3. This would imply a resolution of 32 to this question. 150 is way too high imo

pretty close! 22