This cost should not include the salaries of researchers that worked on developing it, but rather only the cost of electricity + hardware. I will resolve this as best I can, based on potentially given estimates and other pieces of evidence.
Related questions
https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over
> At the MIT event, Altman was asked if training GPT-4 cost $100 million; he replied, “It’s more than that.” […]
@JavierPrieto Thanks for sharing. I’ll take that as confirmation and resolve accordingly unless someone has a valid counterargument or puts doubt to this before the end of the day.
@BionicD0LPH1N Oh wait actually, the resolution criterion from this market is not the cost of training. Forget what I said above, sorry. I guess, I wont resolve until it’s confirmed. Hmm if anyone has recommendations as to what should be done about this, please mention.
@JeffreyLadish I decided (maybe mistakenly) to operationalize it this way, and can’t really change it now. A reason why your method might give unintuitive results is if, say, Microsoft made a deal with OpenAI to pay fully for the training but to be then free to use the model as they wish; this would mean a resolution of $0 of training cost, which doesn’t seem right either.
@JacobPfau I'm guessing the cost of electricity trumps the cost of hardware most of the time, so I'm hoping this isn't crucial to resolving this market.
But to answer your question, the way I was thinking about it, the hardware cost is the cost that OpenAI spends on buying GPUs and other hardware components. But you make a good point, they may not be training GPT-4 on their GPUs. In that case, I don't think the cost of these GPUs should be added to the training cost if they haven't bought them, but I'm open to counterarguments. I could edit this market to remove the hardware costs entirely, but I'm always reluctant to edit markets after they are created.
@BionicD0LPH1N If this, or some other aspect of operationalization, ends up being the deciding factor, then I recommend resolving probabilistically. Alternatively you could resolve to the geometric mean of costs under various definitions.
@RealityQuotient My understanding is that the training tests and experiments made before training the final iteration of a LLM is never counted on that LLM’s compute cost estimate. So this question is looking at the “relevant” run, as in the one that was used to train the model they end up calling GPT-4.
If others have other opinions about how these things are counted in practice I’d be curious to hear more.
@ValeryCherepanov What percentage would you put on that rumor being correct? Was it from someone reliable?
As I understand it, the possibilities for it to be around $10M would be the following:
They stupidly trained a LM with not enough data. (unlikely? Could be explained by the cost of data acquisition in multi-T-sized datasets)
They figured out ways to make the training super cheap and trained with Chinchilla-law-abiding numbers. (unlikely)
They somehow found ways to improve Chinchilla laws. (plausible)
GPT-4 has much less than 175B parameters. (not unlikely?)
Using GPT-3-proportional compute cost, training a Chinchilla-abiding GPT-4 with 175B parameters would cost hundreds of millions of dollars.
My guess is that it's somewhat a combination of all of the above or something like that. If you're relatively confident that your rumor is correct, it lets us update on other related markets.
@BionicD0LPH1N I would put maybe 70% on it being mostly correct but with not very much confidence.
It mentioned a few things, including that GPT-4 is significantly larger than GPT-3.
I think 10M$ is not a small amount. It's plausible that somebody like OpenAI can train 175B+ LLM with Chinchilla laws for 10M. MosaicML claim they can do GPT-30B with original GPT3-like quality for 450k; they probably started a couple of months ago and may have worse methods and/or more expensive hardware https://www.mosaicml.com/blog/gpt-3-quality-for-500k
Scaling laws can be somewhat different too. Especially for multimodal data.
Another possibility would be (this is my idea) to train using maybe 50% of optimal data and then just wait until they got H100 or better datasets etc., and then just continue training and release a new checkpoint.