Resolves YES if OpenAI's GPT-4 has 500 billion or more parameters at the time of initial release. Otherwise NO.
This is a mirror of this Polymarket question and will resolve the same way: https://polymarket.com/event/will-gpt-4-have-500b-parameters/will-gpt-4-have-500b-parameters, except if Polymarket resolves 50-50, their version of ambiguous, this question resolves N/A. (This exception means that the two questions are not equivalent, because there is a significant chance the number of parameters will not be publicly reported by the close date.)
Details:
This market covers how many parameters GPT-4 has at the time of its initial release, and will resolve based on that number regardless of whether that figure is later increased or decreased.
The primary resolution source for this market will be official information from OpenAI, however a consensus of credible reporting (including corroborating statements from employees of OpenAI where available) may also be used.
Even in the unlikely case the Polymarket question is obviously misresolved, this question will still resolve to the same.
Related questions
Why is this market, at 67%, so much higher than the Polymarket probability of 51% that this market is designed to match? Is it just trying to strike a balance between that and the Manifold "over 1 trillion parameters" market that is so out of sync with Polymarket, assuming resolution criteria are similar enough? That's an interesting cross-market tension.
@JacyAnthis Polymarket predictors also don't get 2% back per day of their investment. We may never know the params.
Polymarket also converges to 50% as it resolves 50/50 in ambiguous rulings
@JacyAnthis maybe someone on Manifold has inside info. Both this and the trillion parameter market have been consistently high with no good public reasoning. After the letter market, I'm taking Jack's advice and limiting my position in these two even though they seem mispriced.
@JacyAnthis Yes, the Polymarket question resolves to 50% if it's ambiguous, and that substantially changes their pricing compared to this one's.
@ErickBall There's been plenty of reasoning discussed, no hard evidence but we can make educated guesses. What makes you think they are mispriced?
@jack For one thing, the 500 billion and 1 trillion are priced almost identically, implying about a 1% chance that it's somewhere in between. Also, we know that PaLM 2 is significantly smaller than PaLM (which at 540 billion is the largest LLM I've heard of so far), and in general the trend has been toward squeezing more capabilities out of smaller models with additional training. For OpenAI to go the opposite direction would be very strange, especially since their large customer base makes inference costs very important to them. We already know they reduced GPT-3.5 inference costs dramatically, presumably by switching to a smaller but more heavily trained model. What motivation would they have had to do the opposite with GPT-4?
I don't read too much into them being priced almost identically: These probabilities aren't accurate to within +/- 1%, so we shouldn't read this as saying there's a 1% chance of between 500b and 1t params. I generally take forecasts like these as more like +/-10%.
Also, this one resolves N/A if it isn't known by end of year, while the other could stay open indefinitely, so that is another reason it's tricky to directly compare them.
Also, OpenAI is pricing GPT4 far more than 3.5, which is evidence that its inference costs are correspondingly higher and it has a lot more params. I'm having a hard time finding the exact numbers and math here, and pricing is about more than just costs, but this is probably a good line of evidence to look at.
@jack To me, the clearest way to explain why @jonsimon, me, and others think these markets are way too high is that is that if you surveyed NLP researchers or NLP researchers working on scaling-related topics, they would say estimates of over a trillion are absurd and over 500 billion are unlikely but plausible—especially with Chinchilla scaling laws. There's just a huge gulf between expert and public discourse on this particular topic. (But I agree that we shouldn't read much into the price difference between the two Manifold markets.)
@JacyAnthis Agreed. Personally I think it would be fun to do a survey of individual user's opinions on the topic as a function of experience in the field. I have a feeling it would look pretty bimodal.
@jack I suspect the high pricing for GPT-4 is not reflective of costs, but rather a "state of the art" premium that they can charge because no other model matches GPT-4's performance, and that we will see a big price drop once there is something else equivalent or better on the market.
@Gen sure but those are sparse models... It's certainly possible that GPT-4 is sparse too, and if so that would go a long way to explaining what's going on here.
@ErickBall /vluzko/will-gpt4-be-a-dense-model
My positions on the >500B param market(s) are to offset my position on the dense market. I think people are way overhyping the potential for GPT4 being sparse, but yes, that seems to be the reasoning as you pointed out