There's a news report, paper, or blog post that I consider trustworthy reporting on a 1T parameter dense language model that was trained before January 1st, 2023.
Close date updated to 2022-12-31 11:59 pm Jun 16, 10:15pm: Since there's been some confusion in the comments, I want to clarify that "dense" here is in contrast to sparse models like mixture-of-experts. I've clarified more in a comment below, from which I've copied the relevant snippet:
> It's not "transformer" vs. not either though, it's specifically focused on the distinction between mixture of experts model (ex: https://arxiv.org/abs/1701.06538) which are sparsely activated vs. dense models which use all their nodes for each forward pass.
EDIT: In the comments, Lauro Langosco di Langosco pointed out that it's possible we won't know as of the end of the year whether someone trained a model with 1T params in 2022 and just hadn't announced it yet. He also noted that the current phrasing of the question is very focused on trained vs. announced. Given that, my compromise here is to wait 3 months after the start of 2023 to resolve this rather than doing it right away. If noone announces something they trained with 1T parameters by then, I'll resolve "No".
EDIT 2: Lauro pushed back on three months as being too short, so I'll wait another year to resolve.
EDIT 3: After further deliberation, I've decided to stick with 3 months after 01/01/2023 for resolution. Another clarification I wanted to make is I won't definitely resolve positively unless the 1T model seems at least competitive with existing SoTA models. As mentioned in the comments, if someone trains a 1T parameter dense model as good as say davinci-002 or something, I'll have to decide what I'm going to do. I don't think that's likely enough to pre-plan for though.
Dec 12, 10:40am: Will someone train a 1T parameter dense language model model this year? → Will someone train a 1T parameter dense (non-sparsely activated mixture of experts) language model this year?
Does this count? https://arxiv.org/abs/2110.03888
"We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days"
@DanElton I don't think so.
1) These seem at least partially like mixture-of-expert models:
> M6 is built with stacking transformer layers, which includes self attention and feed-forward neural nets (FFN). For the transformation from dense models to sparse expert models, we should only replace FFN layers with the Mixture-of-Expert (MoE) layers. MoE consists of multiple experts, which are usually FFNs distributed on different devices. A gating network decides the dispatching and combining behaviors of each token and thus tokens can be processed in diverse devices. Such mechanism is a combination of data parallelism and expert parallelism, and thus it is highly efficient though with large model capacity. For the training, to realize the learning of both understanding and generation, the model is trained with text denoising and language modeling on plain text data and with image-based text denoising and image captioning on multimodal data. The model is compatible with different types of downstream tasks and can process information of multiple modalities.
2) I specified in this comment that the model had to be competitive and this model is seemingly not at all.
@rocketsan I'm curious about your models-- why do you think someone will train a 1T parameter transformer?
My earlier joke aside, what is the cutoff for quality? Does it need to be near SOTA? If someone trains a 1T parameter model to, say, GPT-2 quality would that resolve YES?
@vluzko this isn't in the original description but the spirit of the question was a model that's at least trying to be somewhat better than existing ones.
@vluzko the biggest edge case is if someone trains something that's original or davinci-002 level quality with 1T dense parameters. I honestly don't know what I'd do then, kind of just hoping it doesn't happen. If it does, I'll solicit feedback from market participants.
Suppose that GPT-3 -> GPT-4 is a 2 OOM compute increase. If costs scale linearly, that would be $250~500M, which would be an ambitious but not implausible percentage of the OpenAI and DeepMind budget [DM annual budget ~1.7B, OA budget .2~.4B? less certain about OA budget than DM budget]. Probably budget isn't the right way of thinking about it, given that DM and OA have their own hardware, but it still gives a sense of the scale. That kind of spending doesn't look like it would quite reach 1T if the model is Chinchilla trained (eyeballing these graphs: https://arxiv.org/pdf/2203.15556.pdf). It looks like it might be another OOM to get to 1T, which would be out of OA's budget for GPT-4.
I'm not sure how to think about Google Brain, but I'm imagining that they have within a factor of 2 the budget of DeepMind.
@NoaNabeshima I don't think that's right: GPT3 was 175b params on 500b tokens. Scaling to 1T params and 20T tokens (chinchilla-optimal) means roughly 6x params and 40x compute, which gives 240x FLOP/s. That seems not crazy, especially given that compute costs probably scale sublinearly
@L I thought GPT-3 would be considered dense for the purposes of this market
> It's not "transformer" vs. not either though, it's specifically focused on the distinction between mixture of experts model (ex: https://arxiv.org/abs/1701.06538) which are sparsely activated vs. dense models which use all their nodes for each forward pass.
@Lauro GPT-3 was trained on 300B tokens
https://arxiv.org/pdf/2005.14165.pdf
@Lauro Metaculus predicts GPT-4 will use 14.4X FLOPs of GPT-3
https://lambdalabs.com/blog/demystifying-gpt-3
https://www.metaculus.com/questions/9519/flops-used-for-gpt-4-if-released/
@L GPT3 isn't dense in a different sense - MoE vs banded attention.
@vluzko I agree that it isn't dense in that sense, but my guess is that GPT-3 is dense in the spirit of the market. @StephenMalina ?
@Lauro Think my work might be wrong, but I think Metaculus community predicts ~3% chance of GPT-4 having that many FLOPs
@NoaNabeshima yes that's correct, GPT3 is dense by this definition!
@LostFutures It is dense in the sense of this market - it isn't a mixture of experts model where only a fraction of the parameters are actually used in any forward/backward pass.
But it "isn't dense" - or rather it is sparse - in that some of the attention layers are locally banded.
@NoaNabeshima oh good point! So if GPT3 is 300b tokens, that would make it 6x params and 66x tokens, which makes 400x FLOP/s.
Re sublinear: I expect compute costs to have gone down somewhat since GPT3 was trained, and I also expect standard economies of scale to help a little. I don't have a good sense for how much costs go down - I guess I'd expect compute costs to be something like 30%-80% of the GPT3 per FLOP cost? Don't take this estimate seriously though.
FWIW I think the metaculus market is miscalibrated. I'd put the probability at ~30% maybe
@Lauro hmmm, I don't have models of compute costs over time, so I'm just throwing thoughts around but do you think the chip shortage would affect things?
I'm surprised you think the metaculus market is miscalibrated in a particular direction. I do notice you've been profiting on Manifold. I'm curious about when you expect to beat Metaculus.
I can't resolve this myself by training a 1T parameter model really really slowly and then posting about it on my blog, right?
@vluzko You could train it on a single token maybe :b
You would have trouble fitting it in memory but there's ways to use a hard disk for that.
@VictorLevoso I know I can train a 1T parameter model at... checks disk rates maybe one pass per hour, the question is whether doing it would resolve this market.
@StephenMalina hm... how good does a model have to be in order to qualify?
I'm surprised by how much weight people are still putting on YES. PaLM was 540B parameters, but that was with previous scaling laws. With current best estimates for scaling laws 1T parameters would be a massive jump in compute over the current largest models. We're only going to see this in 2022 if someone burns millions of dollars training a suboptimal model as a publicity stunt, discovers a breakthrough architecture that scales differently, or decides it's time to go all in on a particular model (and they started training a few months ago)
Really surprised this market hasn't moved at all as end of 2022 approaches. Maybe people are either not seeing it or they are holding out for GPT4?
@StephenMalina presumably many models trained in 2022 will only be known to the public much later. Can take >1y from training to release / publication
@LauroLangoscodiLangosco I admittedly hadn't considered that. I was implicitly thinking of models we'd know about this year, but I can see how it's ambiguous. That said, do you really think that's why the price was high for so long? Given it eventually dropped, I tend to think it's just that people thought that one might get announced.
@StephenMalina I imagine it may have been different groups of people interpreting the question different ways, or possibly just not thinking of publishing delays. I do think the probability is low even given that we wouldn't have heard of it if someone had trained a 1T model.
IMO the current description seems pretty unambiguous in that it asks for a model trained before Jan 1st 2023, not published. If you want to resolve differently you probably should update the description (though I'd encourage you to not do that, ideally).
@LauroLangoscodiLangosco yeah I think the more fair option is for me to update the description to say that I'll wait to resolve for 6 months after and if we haven't heard by then, I'll resolve to "No". Seem fair to you?
@StephenMalina actually 6 months is a long time, thinking 3 now.
@StephenMalina Personally I don't mind much either way. IMO: if your goal is to resolve markets accurately as per the description, then I think you should wait for at least a year. If you're fine with your markets implicitly being about "what I mean by the description, not what it says", then it seems fine to resolve sooner. If you plan on doing so regularly, it's probably good to include a notice to that effect in your market descriptions.
(fyi regarding your note in the description: my name is Lauro and my pronouns are he/him 🙂 )
@Lauro oops, so sorry about that (fixed)!
(fyi regarding your note in the description: my name is Lauro and my pronouns are he/him 🙂 )
then I think you should wait for at least a year.
I want to be considered as explicit as possible, so I'll wait a year, although it pains me to have to track the market that long. Thank you for pushing back on my unprincipledness.
@StephenMalina I don't think you should wait a year, trained models are rarely if ever kept quiet and it seems kind of pointlessly annoying to drag this one out
@vluzko thanks, I wish there was a way to create a poll of people who have currently invested in this market on different resolution times...
@vluzko @Lauro thinking about it more, I will stick with 3 months. I have learned a lot from this and other comment threads on this market. Going forward, I will clarify that my markets should be assumed to resolve based on description + title not just title in the future. While I'd like to promise I'll come up with less ambiguous markets in the future and will try, I also think part of the value of Manifold is being able to bet on things that aren't perfectly specifiable so I want to be open that I will continue to create markets where there's some discretion involved. I'll try to be as explicit about this as possible when I do it though. Thank you both for your comments on this market, I really appreciate them.
