GPT-4 #5: Will GPT-4 be a dense model?

Conditional on being a transformer.

Nov 25, 11:11pm: Will GPT-4 be a dense model? → GPT-4 #5: Will GPT-4 be a dense model?

Get Ṁ600 play money
Sort by:
predicts YES

Do these count as yes/no:

An appropriate dense expert is sampled per prompt, so that it's almost dense per-prompt.

ChatGPT-4 is distilled from 8 expert dense models.

GPT-4 base model was trained with 8 models and then only one model is used for ChatGPT-4 with no switching between models.

By GPT-4 do you mean the base model or the RLHF'd chat model called "GPT-4" in the API?

1. Depends on the sampling procedure. I lean towards resolving a uniform sample with no selection/filtering as yes, any sampling more complex/intelligent than that will probably be no
2. I assume you mean "and ChatGPT-4 is itself dense"? In that case see my answer to 4.
3. See 4, but would resolve YES for ChatGPT-4
4. This question was asked before the distinction between different GPT-4 variants existed. I'm inclined to resolve with based on an OR of all the models since that most closely captures my original intent with the question, but before I decide one way or the other I want to give people a chance to make arguments.

So it's clear: the claims from George Hotz and Soumith Chintala are not sufficient to resolve this market.

predicts NO

"i might have heard the same -- I guess info like this is passed around but no one wants to say it out loud. GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. Glad that Geohot said it out loud.:"


Apparently none of the people who knew this in advance use Manifold.

I think this means we should slightly downgrade our opinion of the efficiency of prediction markets?

predicts NO

@LoganZoellner lol, the info's been out there for months now, if you know the right twitter users to follow 😉

predicts NO

@rockenots That's my point. If Manifold was an efficient market, even a small number of users who knew the correct answer would quickly push the market to the correct answer due to arbitrage.

But, of course, Manifold is not an efficient market.

bought Ṁ2,000 of NO
bought Ṁ2,000 of NO

@chilli Neither of these people is affiliated with OpenAI

predicts NO

@vluzko Sure, but where there’s smoke there’s fire :) And both of these folks have status in the community and aren’t random folk.

And the beauty of prediction markets is that they don’t need to resolve in order for the market to be convinced that it’s gonna resolve a certain way.

Btw, another Twitter account agreeing with the stated info (

@chilli I'm not saying you shouldn't bet, I just want it to be clear that I'm not going to resolve the market based on this.

bought Ṁ465 of YES


Just 500 more so I can reach 50k even

bought Ṁ400 of NO

@Gen I’ll buy as much no as you’re willing to stake on yes :)

predicts NO

@Gen Only constraint is my mana balance >:(

bought Ṁ500 of NO

@chilli i bet 500 in ur place this time

bought Ṁ11 of YES

@chilli You may also want to arbitrage with this one: /Gigacasting/is-gpt4-a-mixture-of-experts

I'm too lazy, I just click buy

predicts YES

Well it's about time some sanity is returning to this market.

I think there is a 90% chance it is dense, but I also think there's a high chance it will never resolve because OpenAI will never tell us.

Edit: So the leaks say it is a mixture, OpenAI being smarter than I expected I guess. That is a bit of a deviation from their previous strategy.

predicts YES

@ShadowyZephyr Agreed, I think it would be better if the end date of the market was continually pushed back until the information was made available. I think the most likely medium term outcome is that the architecture details get leaked through unofficial channels and there is a big controversy around whether it's sufficient to resolve the market.

predicts YES

@jonsimon Right now the close date is 2027. I'm confident we will know before then.

My expectation is that after a year or so information leaks will become more prevalent but right now people working at openAI are probably pretty well taken care of and are happy to see the product they work on explode. After a bit of staff turnover, particularly with them hiring new people en masse, there are bound to be a lot of leaks.

bought Ṁ100 of YES

why did this market stop arbing with MoE market?

predicts YES

@Dreamingpast Who knows. I don't think this pair of markets represents the beacon of rational behavior.

sold Ṁ63 of YES

I hate this market so much

bought Ṁ0 of YES

@PeterWildeford Yeah these numbers are so bizarre. Just glad I'm getting in heavily now when it's so delusionally low. I looked at your history on this market... been a rough ride.

@jonsimon I wish I could bet on my own markets because 30% is completely bonkers.

sold Ṁ603 of YES

@vluzko I'm just excited to have paid M2995 to get out of this dumpster fire market

@PeterWildeford Bonkers as in too low or too high @30

predicts YES

@Dreamingpast Way too low. Anything below 50% is nuts

predicts YES

@jonsimon Every major language model from every company and large research lab to date has been a dense model. There is no reason to believe that this isn't yet another.

@jonsimon well this one is multimodal, none of the others were

predicts YES

@Dreamingpast There's no reason to think that it being multimodal implies that it's non-dense. The best multimodal models prior to this were models like gato and flamingo which were still dense.

@Dreamingpast I'm definitely 90%+ that GPT-4 is dense. That's why I put so much money into this market earlier. But the main concern of mine is that this market will never resolve YES or NO, but rather that all money will be stranded until a N/A resolution in 2027.

predicts YES

@PeterWildeford True that that's a risk for all of the markets related to architectural details of GPT-4. Same reason that I didn't dump a lot of money in until it got really low. Even if it's a Keynesian beauty contest for the foreseeable future, I expect it to equilibriate at a point much higher than where it is now.

bought Ṁ0 of NO

@PeterWildeford Curious why it matters, given the loan system? I imagine you'd probably already gotten most of the mana you'd invested in this market back in loans.

predicts YES

@chilli Yeah but that's a far slower way to make profits than investing in markets that might actually resolve in the next 1-2 years.

Also, the bigger risk I see is that a lot of these markets might never resolve at all, or the resolutions will be so contentious that they'll have to be marked NA. OpenAI is keeping the model architecture very under wraps, so if it does eventually get released, I either expect it to be many years in the future, or else via some indirect channel which will lead to a lot of controversy around the veracity.

predicts YES

@chilli Unrelated, but why are you betting so heavily on the No side of this market?

@chilli I hadn't thought too much about it and maybe I made a mistake but I thought the opportunity cost of the money was still too high even with the loan system

predicts NO

@jonsimon I enjoy losing mana.

predicts YES

@jonsimon @PeterWildeford Another downside to long-term Manifold bets in addition to opportunity cost can be that your Manifold "portfolio" and "profits" fall if you expect betting to continue to be irrational (e.g., due to AI hype) between now and resolution. I think this is one reason calibration measures are often uninformative or even negatively predictive of Manifold profit maximization in many cases.

predicts YES

@JacyAnthis If you expect the probability to decrease later but the bet to be resolved as YES, maybe don't buy YES at once? E. g. buy NO, wait for it to go up, then sell, or just wait the probability to drop, and then buy YES more cheaply?

predicts YES

@wadimiusz That could be set up at one time to handle some amount of change in the wrong direction, but it's brittle, entangles with real-world rational updating, etc.