@L Against my own self-interest as a NO bettor, I made a few calculations and I'm afraid it could be >1T (f*** OpenAI!!!! Couldn't you just go with a big Chinchilla?! No, you had to go & kick PaLM's ass....) However, I could still hope the exact number doesn't get revealed until next year, hehe. I'll think about whether to show my calculations or not. I'll try tobdouble check with a few acquaintances of mine before (they're researchers at Ivy League unis)

@R2D2 what uni's ? Doesn't ivy league suck at ai?

@R2D2 you think it's MoE?

@kenakofer yes

@VictorLi oh yea just to clarify this is the new acc of the market maker lmfao (the markets on the old acc will still be resolved dont worry)

@LeoSpitz No they did not, it was from this Tweet, directly from a VP at Microsoft. https://twitter.com/yusuf_i_mehdi/status/1635733309631389696

@PatrickDelaney They reported that Bing will eventually include GPT-4 without certifying any knowledge that it, "is" GPT-4, which came from the above tweet.

https://www.semafor.com/article/02/01/2023/chatgpt-is-about-to-get-even-better-and-microsofts-bing-could-win-big

@NexVeridian Right, I know, saw that, thank you...but... "poised to incorporate" is not the same as, "is."

@Mira https://youtube.com/clip/UgkxVV3PsvO3CeOQTiElSTjwaAXL4Ik0cRgw

@Mira Probability within that range is nor evenly distributed. For example, the age of a human can be anywhere between 0 and 122 doesn't mean the expected age of a human would be 61.5 years

@Mira this couldn't be more wrong. 1) interval constraints are not the same as uniform probabilities https://www.stat.berkeley.edu/~stark/Preprints/constraintsPriors13.pdf 2) even if they were (and they aren't), there's no reason to assume a uniform pdf over said interval 3) using non-informative priors when you actually have prior knowledge is a big mistake. And we do have prior knowledge here: we know that Sam Altman debunked the 100 trillion fake news in the StrictlyVC interview. Also, there's no way they could train a 100 trillion attention-based model in Q1-Q2 '22, just on elementary cost considerations alone. So nope, the expected value is not what you claim it to be.

Don’t feed the trolls :)

@JimHays you sure she's trolling? She did buy >10000 YES shares

@JimHays if Mira is a troll, they're throwing away a very large amount of mana.

I just don't get their actions though. If I had more mana I'd bid this down further, but I don't.

@Mira this is... a joke, right?

Mira is trading very reasonably (buying a lot of shares at a reasonable price), and posting a comment that is an obvious troll :)

👆🏼

@jack if she sells them (enough time) before market close, then that's reasonable trading, otherwise...:-)

@R2D2 GPT-5 is being trained on 25k GPU so it's probably >1T, but I don't think GPT-4 is as big. At least, not the 8k version

@R2D2 lmao agreed on "paper" barf. It is a 98 page report, half of which is an ad.

@firstuserhere yep, i don't know what info I'm not aware of has pushed all these markets so high but my estimate remains that gpt4 used probably 300-500 billion parameters, no early stopping, same cut off date as 3.5 has but trained for much much longer.

@firstuserhere Yep, if 1) it ever gets submitted somewhere and 2) I happen to be the reviewer, I’m gonna hard-reject it sooner than I can say “ClosedAI”. I’m gonna be like, (Reviewer 2)^512

@firstuserhere hahahaha

@firstuserhere I agree, I don’t think it wa bigger than the biggest PaLM

@R2D2 yep yep, using that as my upper bound as well. I've heard multiple OpenAI people's and even saltman's comments here and there which fit in with the estimate

@NexVeridian wait, why would anybody think 3.5 turbo is 20b parameters? That would make it 4x smaller than chinchilla and 9x smaller than gpt3. I know the scaling laws have swing more towards data but that's ridiculous.

@ErickBall why else do you think it's so much cheaper?

@ErickBall Chinchilla optimized for training flops, but if you're spending more money on inference than training (which presumably OpenAI is) you'll want to go even smaller. The llama paper made this point. Basically if you look at the empirical data from Chinchilla etc, we haven't saturated the performance of the smaller models, so if you're willing to spend extra training flops you can get more juice out of a smaller model.

@dayoshi well when you put it that way, yeah, that actually makes a ton of sense. Now I don't really understand why this market is so high.

@NoaNabeshima A little more than 2.5 OOMs

@NoaNabeshima https://www.metaculus.com/questions/9519/flops-used-for-gpt-4-if-released/

metaculus community prediction assigns prob that it's >2 OOM greater at ~10%

@NoaNabeshima so looks like this is mostly a mixture of experts market?

@NoaNabeshima oh but also they could have separate params for images

@NoaNabeshima this seems surreal

@NoaNabeshima does this figure imply that it was trained with >1000x the compute of any previous openai LLM?

@ErickBall The grey dots probably are newly trained models so probably this doesn't imply that

@NoaNabeshima No, the "paper" explicitly states that this extrapolation was performed before end of training, abs training completed in 2022, so they're not newly trained models

@rockenots https://twitter.com/apples_jimmy/status/1612558970169982976

@rockenots Tweet that leaked the release date and multimodality: https://twitter.com/apples_jimmy/status/1629939273469394945

@rockenots 125 trillion would be a ridiculous increase over 1-2 trillion, even bigger than the 100 trillion that Sam Altman confirmed was bs.

@wadimiusz GPT-4 is a Transformer-style model [33] pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF) [34]. Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

Arguably this market should be taken down. I don't have a strong take on this though.

@Nikola Perhaps we need to wait, not resolve as ambiguous right away?

@wadimiusz I don't just mean "OpenAI didn't say therefore this market is ambiguous", i also mean something like "OpenAI doesn't want this public, and this market is a mechanism for making it public, so this market should be taken down because infohazards"

@Nikola this feels like a ridiculous take, to me; why should we care if some company would prefer if something stayed a trade secret?

@Nikola That sounds like their problem, not mine.

@Nikola Even if this market is taken down, someone will inevitably post it here and it will be revealed.