Conditional on being a transformer.
Nov 25, 11:11pm: Will GPT-4 be a dense model? → GPT-4 #5: Will GPT-4 be a dense model?
Related questions
Do these count as yes/no:
An appropriate dense expert is sampled per prompt, so that it's almost dense per-prompt.
ChatGPT-4 is distilled from 8 expert dense models.
GPT-4 base model was trained with 8 models and then only one model is used for ChatGPT-4 with no switching between models.
By GPT-4 do you mean the base model or the RLHF'd chat model called "GPT-4" in the API?
@NoaNabeshima
1. Depends on the sampling procedure. I lean towards resolving a uniform sample with no selection/filtering as yes, any sampling more complex/intelligent than that will probably be no
2. I assume you mean "and ChatGPT-4 is itself dense"? In that case see my answer to 4.
3. See 4, but would resolve YES for ChatGPT-4
4. This question was asked before the distinction between different GPT-4 variants existed. I'm inclined to resolve with based on an OR of all the models since that most closely captures my original intent with the question, but before I decide one way or the other I want to give people a chance to make arguments.
"i might have heard the same -- I guess info like this is passed around but no one wants to say it out loud. GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. Glad that Geohot said it out loud.:"
(https://twitter.com/soumithchintala/status/1671267150101721090)
Apparently none of the people who knew this in advance use Manifold.
I think this means we should slightly downgrade our opinion of the efficiency of prediction markets?
@LoganZoellner lol, the info's been out there for months now, if you know the right twitter users to follow 😉
@rockenots That's my point. If Manifold was an efficient market, even a small number of users who knew the correct answer would quickly push the market to the correct answer due to arbitrage.
But, of course, Manifold is not an efficient market.
Soumith Chintala confirms
https://twitter.com/soumithchintala/status/1671267150101721090
@vluzko Sure, but where there’s smoke there’s fire :) And both of these folks have status in the community and aren’t random folk.
And the beauty of prediction markets is that they don’t need to resolve in order for the market to be convinced that it’s gonna resolve a certain way.
Btw, another Twitter account agreeing with the stated info (https://twitter.com/main_horse/status/1671279346307440640?s=46&t=yqOem5ktaowo8FyJ-ilbzQ)
@chilli I'm not saying you shouldn't bet, I just want it to be clear that I'm not going to resolve the market based on this.
@chilli You may also want to arbitrage with this one: /Gigacasting/is-gpt4-a-mixture-of-experts
I'm too lazy, I just click buy
I think there is a 90% chance it is dense, but I also think there's a high chance it will never resolve because OpenAI will never tell us.
Edit: So the leaks say it is a mixture, OpenAI being smarter than I expected I guess. That is a bit of a deviation from their previous strategy.
@ShadowyZephyr Agreed, I think it would be better if the end date of the market was continually pushed back until the information was made available. I think the most likely medium term outcome is that the architecture details get leaked through unofficial channels and there is a big controversy around whether it's sufficient to resolve the market.
@jonsimon Right now the close date is 2027. I'm confident we will know before then.
My expectation is that after a year or so information leaks will become more prevalent but right now people working at openAI are probably pretty well taken care of and are happy to see the product they work on explode. After a bit of staff turnover, particularly with them hiring new people en masse, there are bound to be a lot of leaks.
@Dreamingpast Who knows. I don't think this pair of markets represents the beacon of rational behavior.
@PeterWildeford Yeah these numbers are so bizarre. Just glad I'm getting in heavily now when it's so delusionally low. I looked at your history on this market... been a rough ride.
@jonsimon Every major language model from every company and large research lab to date has been a dense model. There is no reason to believe that this isn't yet another.
@Dreamingpast There's no reason to think that it being multimodal implies that it's non-dense. The best multimodal models prior to this were models like gato and flamingo which were still dense.
@Dreamingpast I'm definitely 90%+ that GPT-4 is dense. That's why I put so much money into this market earlier. But the main concern of mine is that this market will never resolve YES or NO, but rather that all money will be stranded until a N/A resolution in 2027.
@PeterWildeford True that that's a risk for all of the markets related to architectural details of GPT-4. Same reason that I didn't dump a lot of money in until it got really low. Even if it's a Keynesian beauty contest for the foreseeable future, I expect it to equilibriate at a point much higher than where it is now.
@PeterWildeford Curious why it matters, given the loan system? I imagine you'd probably already gotten most of the mana you'd invested in this market back in loans.
@chilli Yeah but that's a far slower way to make profits than investing in markets that might actually resolve in the next 1-2 years.
Also, the bigger risk I see is that a lot of these markets might never resolve at all, or the resolutions will be so contentious that they'll have to be marked NA. OpenAI is keeping the model architecture very under wraps, so if it does eventually get released, I either expect it to be many years in the future, or else via some indirect channel which will lead to a lot of controversy around the veracity.
@chilli I hadn't thought too much about it and maybe I made a mistake but I thought the opportunity cost of the money was still too high even with the loan system
@jonsimon @PeterWildeford Another downside to long-term Manifold bets in addition to opportunity cost can be that your Manifold "portfolio" and "profits" fall if you expect betting to continue to be irrational (e.g., due to AI hype) between now and resolution. I think this is one reason calibration measures are often uninformative or even negatively predictive of Manifold profit maximization in many cases.
@JacyAnthis If you expect the probability to decrease later but the bet to be resolved as YES, maybe don't buy YES at once? E. g. buy NO, wait for it to go up, then sell, or just wait the probability to drop, and then buy YES more cheaply?
@wadimiusz That could be set up at one time to handle some amount of change in the wrong direction, but it's brittle, entangles with real-world rational updating, etc.