On December 6th, 2023, the day Google's Gemini was announced, Quanquan Gu (Director at ByteDance Research) said:
"Uncertain about GPT-5, but a super-strong model (more powerful than Gemini) is expected to arrive anytime now."
"Open source?"
"Open model weights."
https://twitter.com/QuanquanGu/status/1732484036160012798
On the day this model is released, I (Alyssa) will run a Twitter poll asking if the new ByteDance model is more powerful than Google's Gemini Ultra (options Yes/No/Results). This question resolves YES if more people respond Yes than No, otherwise it resolves NO.
If no LLM with open weights is released by ByteDance within three months (by March 7th, 2024), the question resolves NO. If fewer than 50 people answer the poll either Yes or No, the question resolves NO.
Related questions
Why is this market so high???
1. Creating a model more powerful than gemini ultra is incredibly difficult.
In theory gemini ultra is (ever so slightly) better than GPT 4, something that very few companies have been able to compete with (just anthropic as I am aware). Most models struggle to get past GPT3.5:
https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH
2. Why would ByteDance open source it?
Very few companies have been able to create a model of that quality, they are extremely valuable right now. Why throw all that value away?? What incentive does ByteDance have to open source it?
Am I missing some key bits of information? Do we have skepticism about how the market is resolving to a twitter poll? Is the weird binary-but-multiple-choice market throwing people off? Someone explain to me why this isn't at 1%!
@Nikos on point #2 – I think there can be some legitimate arguments for a company to open source their own model. One is if you can't guarantee that the techniques to build the model can't be outdone by others, it could be worth having your model open simply to have the community find ways to improve it. Really simplifying the argument here, but it's similar to the reasons for companies open sourcing their own code.
And a bit of plug, but in case you're interested, here's a market I just made about whether any "big tech" companies will open source their own (foundation) model/LLM in 2024: https://manifold.markets/VictorsOtherVector/which-if-any-big-tech-companies-wil
EDIT: Fixed some basic grammatical typos for clarity. (I'm bad at using a mobile keyboard, especially when I'm tired.)
@VictorsOtherVector Thanks! That adds some clarity for me. I like that market you linked! I also found this one to be useful for this question (which seems to just be limited to my first point above)
https://manifold.markets/Ledger/will-china-be-competitive-in-the-ll?r=Tmlrb3M
@Nikos I agree that this should be at 1/1000 or less by a reasonable interpretation:
I don't think there's more than a 1% chance that ByteDance will somehow be better than Anthropic, Facebook, AND google at LLMs
If they are, I don't think there'd be more than a 10% chance they'd desire to release the weights (maybe also less than 1%? seems like a slightly insane thing to do).
Even if all of the above is true, I'd bet there's less than a 10% chance that CCP would allow it to be released.
Even if all that is true, March 7th is an aggressive timeline
Furthermore, the resolution criteria is relatively favorable towards Gemini ultra. It doesn't matter if it hasn't been released and default is NO if fewer than 50 people respond.
Nevertheless, there's a lot of mana keeping this at 15%. My best guess is that maybe there's a hope that the twitter poll can be gamed?
I wonder to what degree the probability of this was adversely affected by the bonkers market structure here, being a market for YES and a market for NO where you have to by YES on the NO market (or NO on the YES market) to actually express "NO".
Frankly, I really wish the market creator would N/A this market and make one that uses a normal market.
@josh I don't think the market structure is to blame as much as the fact that there's one guy who owns the majority of YES and keeps buying it up
@dominic Yeah, the structure is insane but no point in fixing it now and I doubt it is impacting the price much.
What will the resolution be if Gemini Ultra is not released by March 7? Polymarket currently says 32% that Gemini Ultra will not be released by 2024-01-31, so this seems like there's a nontrivial chance of it also not being released by 2024-03-07.
@FaulSname Doesn't matter, I think we have a pretty good estimate of its capabilities from the paper
@AlyssaVance Are there markets for that? I'd bet against the levels of capability claimed in the paper actually translating to the implied levels of practical usefulness, if that's operationalizable.
I see this one and have bet NO, but I don't love bigbench as the operationalization.
@jacksonpolack Ah, I was about to ask if that was intentional. It works fine but it's definitely confusing with the double negatives!
@Jai I think you're running into the same wording issue I did? If you want to buy NO (as in you think ByteDance will NOT release a more powerful model), then you want to "Buy YES" on the "NO" option" (or alternatively "buy NO" on the YES option).
1. Click "Bet" on option 2 labeled NO
2. In the betting window you will see NO in the top left, that is the stock you are buying
3. You will want to buy YES shares (on NO) (ie please pay out if it resolves NO)
If you buy No on NO, it is sort of double negative where you end up buying yes.
(Sorry, I feel like I'm stuck inside one of those "who's on first" jokes here).