On December 6th, 2023, the day Google's Gemini was announced, Quanquan Gu (Director at ByteDance Research) said:
"Uncertain about GPT-5, but a super-strong model (more powerful than Gemini) is expected to arrive anytime now."
"Open source?"
"Open model weights."
https://twitter.com/QuanquanGu/status/1732484036160012798
On the day this model is released, I (Alyssa) will run a Twitter poll asking if the new ByteDance model is more powerful than Google's Gemini Ultra (options Yes/No/Results). This question resolves YES if more people respond Yes than No, otherwise it resolves NO.
If no LLM with open weights is released by ByteDance within three months (by March 7th, 2024), the question resolves NO. If fewer than 50 people answer the poll either Yes or No, the question resolves NO.
1,000
3.00
@tr Unfortunately that is not a bug, but just very bad UX. For each option in the market you could buy yes or no. It looks like you bought NO shares on the no answer.
Things that would move the probability up towards (and pay out if) yes:
Buying YES on the yes answer
Buying NO on the no answer
Things that would move the probability down (and pay out if) no:
Buying NO on the yes answer
Buying YES on the no answer
So if you only bought on the yes answer it would be clear, but if you bought on the no answer it would be inverted to what you probably wanted.
You were not the only person to be confused by this, I made the same mistake at first.
I believe when the market was created binary markets could not be created so the creator of this market did they best they could at the time.
@jacksonpolack @Nikos Argh - thanks for the explanation and sending the mana back. I thought I was benefiting from a bug when I kept betting so I guess no one to blame but myself 🤷♂️
@jacksonpolack Thank you for compensating people who ran into this. I don't suppose you might be able to do the same for me? I had M350 bet on NO-on-NO.
EDIT: Thank you! Much appreciated.
Why is this market so high???
1. Creating a model more powerful than gemini ultra is incredibly difficult.
In theory gemini ultra is (ever so slightly) better than GPT 4, something that very few companies have been able to compete with (just anthropic as I am aware). Most models struggle to get past GPT3.5:
https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH
2. Why would ByteDance open source it?
Very few companies have been able to create a model of that quality, they are extremely valuable right now. Why throw all that value away?? What incentive does ByteDance have to open source it?
Am I missing some key bits of information? Do we have skepticism about how the market is resolving to a twitter poll? Is the weird binary-but-multiple-choice market throwing people off? Someone explain to me why this isn't at 1%!
@Nikos on point #2 – I think there can be some legitimate arguments for a company to open source their own model. One is if you can't guarantee that the techniques to build the model can't be outdone by others, it could be worth having your model open simply to have the community find ways to improve it. Really simplifying the argument here, but it's similar to the reasons for companies open sourcing their own code.
And a bit of plug, but in case you're interested, here's a market I just made about whether any "big tech" companies will open source their own (foundation) model/LLM in 2024: https://manifold.markets/VictorsOtherVector/which-if-any-big-tech-companies-wil
EDIT: Fixed some basic grammatical typos for clarity. (I'm bad at using a mobile keyboard, especially when I'm tired.)
@VictorsOtherVector Thanks! That adds some clarity for me. I like that market you linked! I also found this one to be useful for this question (which seems to just be limited to my first point above)
https://manifold.markets/Ledger/will-china-be-competitive-in-the-ll?r=Tmlrb3M
@Nikos I agree that this should be at 1/1000 or less by a reasonable interpretation:
I don't think there's more than a 1% chance that ByteDance will somehow be better than Anthropic, Facebook, AND google at LLMs
If they are, I don't think there'd be more than a 10% chance they'd desire to release the weights (maybe also less than 1%? seems like a slightly insane thing to do).
Even if all of the above is true, I'd bet there's less than a 10% chance that CCP would allow it to be released.
Even if all that is true, March 7th is an aggressive timeline
Furthermore, the resolution criteria is relatively favorable towards Gemini ultra. It doesn't matter if it hasn't been released and default is NO if fewer than 50 people respond.
Nevertheless, there's a lot of mana keeping this at 15%. My best guess is that maybe there's a hope that the twitter poll can be gamed?
I wonder to what degree the probability of this was adversely affected by the bonkers market structure here, being a market for YES and a market for NO where you have to by YES on the NO market (or NO on the YES market) to actually express "NO".
Frankly, I really wish the market creator would N/A this market and make one that uses a normal market.
@josh I don't think the market structure is to blame as much as the fact that there's one guy who owns the majority of YES and keeps buying it up
@dominic Yeah, the structure is insane but no point in fixing it now and I doubt it is impacting the price much.