Microsoft's blog says:
We’re excited to announce the new Bing is running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search. It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.
Bing itself claims to be running on GPT-4:
I use a ChatGPT interface that is powered by OpenAI's GPT-4.0 technology, which is a large language model that can generate natural and conversational text.
This market resolves once more information about it has been made public and we can know for sure. If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system. Resolves based on what Bing was running on at the time of market creation.
🏅 Top traders
This was very obvious if you did any evaluation at all of the relative capabilities of ChatGPT and Bing tbh, I have no idea why people thought OpenAI wouldn't do this with their biggest partner. Good market though :)
@hyperion For once, someone's claims of "this was super obvious" are actually backed up by their bets before the information was publicly known! Well done. :)
@IsaacKing Only reason I pulled out earlier was because another GPT-4 market was at 30% and I didn't have enough liquidity for both. Easy $$$$
Any concerns before I resolve this to YES?
@IsaacKing plz do it, need liquidity
Market should be resolved to YES
"We are happy to confirm that the new Bing is running on GPT-4, customized for search. If you’ve used the new Bing in preview at any time in the last six weeks, you’ve already had an early look at the power of OpenAI’s latest model."
The new Bing launched Feb 7, which is more than 6 full weeks prior to this statement, or 5 full weeks and two partial weeks. This implies that it not only is GPT-4 now, but has been since launch.
@rkn Wanna actually bet on that?
@IsaacKing not really, I already did lol. In hindsight it's perfectly reasonable for Microsoft to pay (or already $10B) as Bing desperately needs market share vs Google ^^ Good thing OpenAI "... rolled out the ability to purchase ChatGPT Plus internationally." in yesterdays release notes for chatgpt, you know, since they're only letting Plus-users use GPT4 (free use not yet known) 😂
@rkn So you think this market should resolve NO, because Bing was not using GPT-4 at the time of release, and is now switching it over and lying about having always been using it?
I'm willing to seriously consider this possibility if you seriously believe it, but the fact that you have not bought a bunch of NO at such incredibly good odds for you makes me think you don't seriously hold this belief and are just trolling.
@IsaacKing while I think it's *possible*, I really don't think should resolve to no as long as MS states that Bing has been running gpt4 all along. Unless they walk it back somehow
not trolling got my shares earlier than this (~60-70% iirc) so imma realize this loss
@IsaacKing I think this is implausible because Bing was clearly more advanced than ChatGPT. There was little doubt if it uses some other better model, it was just not clear if it was GPT-4 or GPT-3.6 or something like this.
@ValeryCherepanov also, it is plausible that Bing used GPT-4 from the start because GPT-4 existed for months already (this is not rumors, it is what the official OpenAI post says).
@rkn Ok, I'm going to resolve this YES without further research unless you or someone else wants to provide a serious counterargument within the next few minutes.
From the GPT-4 blog: "We’ve been working on each aspect of the plan outlined in our post about defining the behavior of AIs, including steerability. Rather than the classic ChatGPT personality with a fixed verbosity, tone, and style, developers (and soon ChatGPT users) can now prescribe their AI’s style and task by describing those directions in the “system” message. System messages allow API users to significantly customize their users’ experience within bounds. " doesn't that remind you the precise / creative / balanced behavior of Bing?
You really think BING got GPT-4 before ChatGPT?
@grahambo I don’t know about that. Since Microsoft paid very good money, they might demand to be the first to release it. We don’t know the details of their commercial agreement. Anyway, that doesn’t matter for what it concerns this market. “Resolves based on what Bing was running on at the time of market creation” and there’s no way GPT-4 was out already when this market was created.
@R2D2 Bing could have been running on a prerelease of GPT-4. Scott Alexander thinks this is the case and thinks they just fumbled it because they were less technically competent than OpenAI expected them to be. But he's not an AI expert.
@grahambo OAI already has access to GPT-4 and plan to publicly release it soon. It is a smaller model as well
I think it would be bad strategy from OpenAI's perspective to retroactively label the first iterations of Bing AI as GPT-4 because of all the issues people had with it—better to start discussions of GPT-4 with a clean slate by just labeling the better model they have now.
@JacyAnthis Exactly -- whether some OpenAI product release gets touted as "GPT-4" or something else is a marketing question, not a technical one, and as you say it doesn't seem to be to OpenAI's advantage to claim early Bing as GPT-4 regardless of where Sydney's latest weights came from. Though, had things played out differently, without so much high-profile unhinged-itude, maybe it would have been.
I heard from a friend who works at a different company (a very big unicorn that all of us have heard of) that they are using GPT-4 on a limited license that only 5 companies have access to. Their AI writing is kinda shit as well so it tracks with Bing being worse than 3.5, so I don't think it's that far of a jump to assume Msft/Bing are one of the other 4 considering Msft owns OAI who plan to release GPT4 next week publicly -- it should have been beta tested before public release and since there were no waitlists I know of I assume this was GPT-4.
It had better not be, because it's worse than 3.5
“It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.”
If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system
I'd suggest resolving to a percentage instead of yes/no then. E.g. the so called GPT-3.5 could have been 50% for GPT-4 (if GPT-3 is the baseline).
"Apparently Mikhail has spoken to Gwern personally and told him bingchat does not use gpt4. I'll take his word on it."
@LeoSpitz I believe that person on Twitter has misread Gwern's comment on LW. It doesn't say that Gwern talked to Mikhail, just that someone told Gwern it's not GPT-4, and that Mikhail is in charge of deployment.
That's the same comment this market updated on on February 20th, so no new information there.
@IsaacKing You're right, just seeing that he has linked to that LW post as his source.
@mkualquiera this market is about model at the time of market creation, so this is (i guess) evidence that it was probably not gpt4 before.
Suppose Bing sometimes uses GPT-4 to generate tokens and sometimes uses a model that isn't GPT-4. How would this resolve?
@NoaNabeshima I think if it uses GPT-4 for more complicated or challenging queries, that should count.
@IsaacKing That should not count. The question is "Is Bing's chatbot GPT-4?" and not "Does Bing's chatbot use GPT-4 in any way shape or form". You should make a new question if you have a different intent as these 2 questions have differing answers.
@RahulShah I don't see a meaningful difference between a chatbot "using" vs. "being" a specific language model.
Bing seems confused about whether it is or isn't GPT-4.
@IsaacKing It is being a smart-alec. It clearly thinks it is GPT-4 plus some layer on top, and if that's true the market should resolve Yes.
But I don't understand how it does reasoning about itself, so this answer might be just hallucination.
@RobinGreen Oh I'm sure it is hallucination. There's no reason its training data or prompt would include such information about itself.
@IsaacKing you confirm that this refers to Bing AI at market creation (February 16 2023), right? Bing AI keeps getting new releases (we’re at v96 right now), so at some point in time it will feature GPT-4…but I’m very confident it didn’t at the time of market creation
@R2D2 Yes, at the time of market creation. I'll edit the description.
@IsaacKing Potential for a bounty if this ends up mattering
@JimHays Someone else already mentioned it below and I provided a clarification, so I don't think this one qualifies.
If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system.
This seems so vague to me. It's either going to be GPT-4 or not.
Why the recent change?
@StrayClimb no clue, the person deleted their account as well so this smells lowkey fishy lmao
@VictorLi "the person deleted their account" - which person are you referring to?
@RobinGreen Mira, the current highest YES shareholder who was responsible for the temporary spike to 75%. when i click onto their profile it just says deleted
@VictorLi weird. i saw this in my feed just now. when i click though, it also says deleted.
@higherLEVELING My guess is that they deleted their account (after that legendary market https://manifold.markets/levifinkelstein/will-this-market-be-in-the-interval).
This would mean that their user profile is deleted. However, it's possible that manifold did not delete the user's API key, only their profile/personal data, which is why the user can bet in a programatic fashion.
They don't delete the (for example) comment data (or trading history on markets that the user bet on) of that user either:
Probably to avoid a lot of [deleted]-[deleted] comments from a lot of different markets that the user had commented on?
@firstuserhere I don't believe the account is deleted though. They liked the comment i just wrote above.
@higherLEVELING yea they just liked my comment as well lmao wtf
@VictorLi Mira is the ghost of Manifold. Dead but not gone...
@IsaacKing so much for taking a break from manifold 😂
This feels like a market that is careening toward a highly contested resolution even with @IsaacKing‘s best efforts to resolve it fairly.
Greedily bets more NO.
@Charlie I think the ambiguous "closer" criterion is the cause of most contestation. I would personally be more excited about a market that just resolves YES if Microsoft or OpenAI explicitly say that ChatGPT, as it was in February 2023, is based on GPT-4, and resolves NO otherwise. This has downsides, such as that F23-ChatGPT could be based on GPT-4 but Microsoft and OpenAI never get around to mentioning that. Personally, I think the unambiguous resolution would outweigh those downsides.
@JacyAnthis perhaps you can create that market. I’m sure @IsaacKing would welcome it, and I’d be interested in seeing any discrepancies.
@JacyAnthis I think that would end up being pretty similar to this one
More direct confirmation from Microsoft that Sydney is based on a "next generation GPT model" "more powerful than GPT-3.5"
Last Summer, OpenAI shared their next generation GPT model with us, and it was game-changing. The new model was much more powerful than GPT-3.5
@hyperion Note that "much more powerful than GPT-3.5" is probably a lower bar than question resolution. Bing's chatbot needs to be closer to GPT-4 than "GPT-3 and its derivatives such as Codex and ChatGPT," and, presumably, ChatGPT is already (much) more powerful than GPT-3.5.
@JacyAnthis nope. 1) GPT-3.5 is not a single model but a series of models https://platform.openai.com/docs/model-index-for-researchers/models-referred-to-as-gpt-3-5 2) the last one of this models (text-davinci-003) was trained with PPO, the same way ChatGPT was trained https://openai.com/blog/chatgpt/ so there’s really no evidence at all that ChatGPT is more powerful than GPT 3.5
@JacyAnthis Really? "much more powerful than 3.5" is exactly what we're expecting to get from GPT-4, expecting anything more is a bit silly. ChatGPT is GPT-3.5.
@StrayClimb 1) Davinci is (one of the iterations of) GPT-3, not GPT-4 https://platform.openai.com/docs/models/overview 2) ChatGPT already has a context of 8k, so that’s nothing to be impressed about 3) 32k, now this sounds a bit more peculiar. However if GPT-4 is going to be a multi-trillion parameters model, as rumored, with many optimizations over GPT-3.5, then this isn’t it at all.
Apparently Sydney sometimes says GPT4 unprompted. Hard to be sure though give that by now there's enough speculation online about it that she might have found
@Mqrius It could easily claim to be the Pope, but that’s not evidence it’s the Pope. These models lie like there’s no tomorrow. A non-technical intro: https://noahpinion.substack.com/p/why-does-chatgpt-constantly-lie
@PippoPluto do you mean evidence in the yes/no sense, or evidence in the Bayesian update sense? Because it's definitely the latterz even if it's weak. In the world where Sydney is based on GPT-4, she's more likely to say so than in the world where she's not. But I agree that in the world where she's not based on GPT-4, it's still plausible for her to say she is. Just a bit less plausible.
@Mqrius I meant in the yes/no sense, but even in the Bayesian update sense, P(claims to be GPT-4| it’s GPT-4) is not necessarily higher than P(claims to be GPT-4| it’s not GPT-4). Because of the LM objective (“predict the next token”), these models tell you what you want to hear. They’re trained to sound plausible, not to tell the truth. Have a look at the link I shared.
Yes, yes, it's the next-generation model. But what will OpenAI call it? GPT-4 or something completely else? We know that Bing is running on something more powerful than GPT-3.5, yet OpenAI could release a completely different kind of model under the GPT-4 name.
I heard Sam Altman mention in an interview that they "need to figure out this naming thing."
OpenAI models are not publicly released (meaning it's impossible to have an independent verification of the claim) and the names are there for marketing purposes only. For example, there are many models that, at some point in time, were called GPT-3 (Davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003...). Thus it seems difficult to determine what is the model behind Bing. Almost surely, it won't be a fixed model, but different models will be served through the same Bing AI service at different points in time. So "when" should Bing AI be GPT-4, for this to resolve as YES?
@PippoPluto As of market creation.
Apparently Sydney has been running in several markets for a year, making it seem less likely it's based on GPT-4. https://twitter.com/MParakhin/status/1627330287276261381?t=eYUlpATUkoUzvX8WOLPjkA&s=19
@Stefan i couldn't confirm if this is a official account and to be honest it screams a fake account
@ShortStick It's real, see the other comments below.
They have changed the underlying model a number of times.