resolved Mar 14
Is Bing's chatbot GPT-4?
Resolved
YES

Microsoft's blog says:

We’re excited to announce the new Bing is running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search. It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.

Bing itself claims to be running on GPT-4:

I use a ChatGPT interface that is powered by OpenAI's GPT-4.0 technology, which is a large language model that can generate natural and conversational text.

This market resolves once more information about it has been made public and we can know for sure. If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system. Resolves based on what Bing was running on at the time of market creation.

🏅 Top traders

#NameTotal profit
1Ṁ1,983
2Ṁ1,475
3Ṁ1,133
4Ṁ813
5Ṁ761
Sort by:
GabeMukobi avatar
Gabe Mukobi
hyperion avatar
hyperionbought Ṁ100 of YES

This was very obvious if you did any evaluation at all of the relative capabilities of ChatGPT and Bing tbh, I have no idea why people thought OpenAI wouldn't do this with their biggest partner. Good market though :)

IsaacKing avatar
Isaac King

@hyperion For once, someone's claims of "this was super obvious" are actually backed up by their bets before the information was publicly known! Well done. :)

Mira avatar
Mirapredicted YES at 99%

@IsaacKing Only reason I pulled out earlier was because another GPT-4 market was at 30% and I didn't have enough liquidity for both. Easy $$$$

Butanium avatar
Butaniumpredicted YES at 99%

@Mira your account is still deleted though 🥴

IsaacKing avatar
Isaac King

Any concerns before I resolve this to YES?

LeoSpitz avatar
Leo Spitzsold Ṁ7 of NO

@IsaacKing plz do it, need liquidity

MichaelKlyachman avatar
Michael Klyachmanbought Ṁ100 of YES
JimHays avatar
Jim Hayssold Ṁ124 of NO

@MichaelKlyachman
"We are happy to confirm that the new Bing is running on GPT-4, customized for search. If you’ve used the new Bing in preview at any time in the last six weeks, you’ve already had an early look at the power of OpenAI’s latest model."

The new Bing launched Feb 7, which is more than 6 full weeks prior to this statement, or 5 full weeks and two partial weeks. This implies that it not only is GPT-4 now, but has been since launch.

rkn avatar
rknpredicted NO at 99%

@JimHays bet this is straight up an attention grabbing lie, but you never know. feels weird that OpenAI clearly states that the free usage tier of GPT-4 will be more limited than GPT3 was. But then again, it's no longer a for-profit org, is it... 😢

IsaacKing avatar
Isaac King

@rkn Wanna actually bet on that?

rkn avatar
rknpredicted NO at 99%

@IsaacKing not really, I already did lol. In hindsight it's perfectly reasonable for Microsoft to pay (or already $10B) as Bing desperately needs market share vs Google ^^ Good thing OpenAI "... rolled out the ability to purchase ChatGPT Plus internationally." in yesterdays release notes for chatgpt, you know, since they're only letting Plus-users use GPT4 (free use not yet known) 😂

IsaacKing avatar
Isaac King

@rkn So you think this market should resolve NO, because Bing was not using GPT-4 at the time of release, and is now switching it over and lying about having always been using it?

I'm willing to seriously consider this possibility if you seriously believe it, but the fact that you have not bought a bunch of NO at such incredibly good odds for you makes me think you don't seriously hold this belief and are just trolling.

rkn avatar
rknpredicted NO at 99%

@IsaacKing while I think it's *possible*, I really don't think should resolve to no as long as MS states that Bing has been running gpt4 all along. Unless they walk it back somehow

not trolling got my shares earlier than this (~60-70% iirc) so imma realize this loss

ValeryCherepanov avatar
Valery Cherepanovpredicted YES at 99%

@IsaacKing I think this is implausible because Bing was clearly more advanced than ChatGPT. There was little doubt if it uses some other better model, it was just not clear if it was GPT-4 or GPT-3.6 or something like this.

ValeryCherepanov avatar
Valery Cherepanovpredicted YES at 99%

@ValeryCherepanov also, it is plausible that Bing used GPT-4 from the start because GPT-4 existed for months already (this is not rumors, it is what the official OpenAI post says).

IsaacKing avatar
Isaac King

@rkn Ok, I'm going to resolve this YES without further research unless you or someone else wants to provide a serious counterargument within the next few minutes.

Butanium avatar
Butaniumpredicted YES at 55%

From the GPT-4 blog: "We’ve been working on each aspect of the plan outlined in our post about defining the behavior of AIs, including steerability. Rather than the classic ChatGPT personality with a fixed verbosity, tone, and style, developers (and soon ChatGPT users) can now prescribe their AI’s style and task by describing those directions in the “system” message. System messages allow API users to significantly customize their users’ experience within bounds. " doesn't that remind you the precise / creative / balanced behavior of Bing?

rkn avatar
rknpredicted NO at 57%

@Butanium I've used the gpt3.5 API and these "system" messages, seeing as they just announced a waitlist for GPT-4... I don't think Bing has been running anything more than a "custom" 3.5 build

rkn avatar
rknpredicted NO at 99%

@rkn apparently microsoft says otherwise, guess we know why Bing's version was so shoddy (less tested/poor bing system prompts/???)

grahambo avatar
grahambopredicted NO at 52%

You really think BING got GPT-4 before ChatGPT?

grahambo avatar
grahambobought Ṁ10 of NO

@grahambo GPT-4 is rumored to have trillions of parameters, which means it's orders of magnitude more computationally expensive than ChatGPT or GPT-3 is. Which means there's no way they are scaling that for people to use it for FREE

R2D2 avatar
R2D2predicted NO at 52%

@grahambo The trillion parameters ba has been debunked by Sam Altaman himself. The GPT-4 models won’t be much bigger than the GPT-3.5 ones

R2D2 avatar
R2D2predicted NO at 52%

@R2D2 *trilion parameters bs.

grahambo avatar
grahambopredicted NO at 52%

@R2D2 Damn that's confusing. Still though, people seem to think Bing will be getting GPT-4 before OpenAI does lmao

R2D2 avatar
R2D2predicted NO at 51%

@grahambo I don’t know about that. Since Microsoft paid very good money, they might demand to be the first to release it. We don’t know the details of their commercial agreement. Anyway, that doesn’t matter for what it concerns this market. “Resolves based on what Bing was running on at the time of market creation” and there’s no way GPT-4 was out already when this market was created.

RobinGreen avatar
Robin Greenpredicted YES at 53%

@R2D2 Bing could have been running on a prerelease of GPT-4. Scott Alexander thinks this is the case and thinks they just fumbled it because they were less technically competent than OpenAI expected them to be. But he's not an AI expert.

RahulShah avatar
Rahul Shahpredicted YES at 50%

@grahambo OAI already has access to GPT-4 and plan to publicly release it soon. It is a smaller model as well

JacyAnthis avatar
Jacy Reese Anthisbought Ṁ100 of NO

I think it would be bad strategy from OpenAI's perspective to retroactively label the first iterations of Bing AI as GPT-4 because of all the issues people had with it—better to start discussions of GPT-4 with a clean slate by just labeling the better model they have now.

ML avatar
MLpredicted NO at 53%

@JacyAnthis Exactly -- whether some OpenAI product release gets touted as "GPT-4" or something else is a marketing question, not a technical one, and as you say it doesn't seem to be to OpenAI's advantage to claim early Bing as GPT-4 regardless of where Sydney's latest weights came from. Though, had things played out differently, without so much high-profile unhinged-itude, maybe it would have been.

RahulShah avatar
Rahul Shahbought Ṁ10 of YES

I heard from a friend who works at a different company (a very big unicorn that all of us have heard of) that they are using GPT-4 on a limited license that only 5 companies have access to. Their AI writing is kinda shit as well so it tracks with Bing being worse than 3.5, so I don't think it's that far of a jump to assume Msft/Bing are one of the other 4 considering Msft owns OAI who plan to release GPT4 next week publicly -- it should have been beta tested before public release and since there were no waitlists I know of I assume this was GPT-4.

MaximLott avatar
Maxim Lottbought Ṁ100 of NO

It had better not be, because it's worse than 3.5

JimHays avatar
Jim Hayspredicted NO at 52%

@MaximLott

https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/

“It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.”

Milli avatar
Millipredicted NO at 46%

If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system

I'd suggest resolving to a percentage instead of yes/no then. E.g. the so called GPT-3.5 could have been 50% for GPT-4 (if GPT-3 is the baseline).

JimHays avatar
Jim Hayspredicted NO at 40%

@Milli I don’t think people generally thought it was the base version of GPT-3. GPT-3.5 seems like a better baseline for 0.

LeoSpitz avatar
Leo Spitz

Would a smaller version of GPT-4 count as GPT-4?

LeoSpitz avatar
Leo Spitzbought Ṁ100 of NO

"Apparently Mikhail has spoken to Gwern personally and told him bingchat does not use gpt4. I'll take his word on it."

https://twitter.com/Teknium1/status/1633148325154418689

IsaacKing avatar
Isaac Kingbought Ṁ50 of YES

@LeoSpitz I believe that person on Twitter has misread Gwern's comment on LW. It doesn't say that Gwern talked to Mikhail, just that someone told Gwern it's not GPT-4, and that Mikhail is in charge of deployment.

That's the same comment this market updated on on February 20th, so no new information there.

LeoSpitz avatar
Leo Spitzsold Ṁ100 of NO

@IsaacKing You're right, just seeing that he has linked to that LW post as his source.

https://twitter.com/Teknium1/status/1633170248743661578

mkualquiera avatar
mkualquierapredicted YES at 57%
Lavander avatar
Lavanderpredicted NO at 57%

@mkualquiera this market is about model at the time of market creation, so this is (i guess) evidence that it was probably not gpt4 before.

LeoSpitz avatar
Leo Spitzbought Ṁ50 of YES

@Lavander I believe that @Techinfoeditor is drawing a comparison between the original ChatGPT and not a previous version of Bing.

NoaNabeshima avatar
Noa Nabeshimapredicted NO at 61%

Suppose Bing sometimes uses GPT-4 to generate tokens and sometimes uses a model that isn't GPT-4. How would this resolve?

NoaNabeshima avatar
Noa Nabeshimapredicted NO at 57%
IsaacKing avatar
Isaac King

@NoaNabeshima I think if it uses GPT-4 for more complicated or challenging queries, that should count.

RahulShah avatar
Rahul Shahpredicted NO at 55%

@IsaacKing That should not count. The question is "Is Bing's chatbot GPT-4?" and not "Does Bing's chatbot use GPT-4 in any way shape or form". You should make a new question if you have a different intent as these 2 questions have differing answers.

IsaacKing avatar
Isaac King

@RahulShah I don't see a meaningful difference between a chatbot "using" vs. "being" a specific language model.

IsaacKing avatar
Isaac King

Bing seems confused about whether it is or isn't GPT-4.

RobinGreen avatar
Robin Greenpredicted YES at 60%

@IsaacKing It is being a smart-alec. It clearly thinks it is GPT-4 plus some layer on top, and if that's true the market should resolve Yes.

But I don't understand how it does reasoning about itself, so this answer might be just hallucination.

IsaacKing avatar
Isaac King

@RobinGreen Oh I'm sure it is hallucination. There's no reason its training data or prompt would include such information about itself.

R2D2 avatar
R2D2predicted NO at 54%

@IsaacKing you confirm that this refers to Bing AI at market creation (February 16 2023), right? Bing AI keeps getting new releases (we’re at v96 right now), so at some point in time it will feature GPT-4…but I’m very confident it didn’t at the time of market creation

IsaacKing avatar
Isaac King

@R2D2 Yes, at the time of market creation. I'll edit the description.

JimHays avatar
Jim Hayspredicted NO at 51%

@IsaacKing Potential for a bounty if this ends up mattering

IsaacKing avatar
Isaac King

@JimHays Someone else already mentioned it below and I provided a clarification, so I don't think this one qualifies.

LeoSpitz avatar
Leo Spitzpredicted NO at 68%

If it turns out to be some intermediate model, this market resolves based on whether it's "closer" to GPT-3 and its derivatives such as Codex and ChatGPT, or "closer" to a new GPT-4 system.

This seems so vague to me. It's either going to be GPT-4 or not.

StrayClimb avatar
Reynoldspredicted YES at 60%

Why the recent change?

VictorLi avatar
Victor Lipredicted YES at 77%

@StrayClimb no clue, the person deleted their account as well so this smells lowkey fishy lmao

RobinGreen avatar
Robin Greenpredicted YES at 59%

@VictorLi "the person deleted their account" - which person are you referring to?

VictorLi avatar
Victor Lipredicted YES at 59%

@RobinGreen Mira, the current highest YES shareholder who was responsible for the temporary spike to 75%. when i click onto their profile it just says deleted

higherLEVELING avatar
higherLEVELINGpredicted NO at 59%

@VictorLi weird. i saw this in my feed just now. when i click though, it also says deleted.

firstuserhere avatar
firstuserhere

@higherLEVELING My guess is that they deleted their account (after that legendary market https://manifold.markets/levifinkelstein/will-this-market-be-in-the-interval).

This would mean that their user profile is deleted. However, it's possible that manifold did not delete the user's API key, only their profile/personal data, which is why the user can bet in a programatic fashion.

They don't delete the (for example) comment data (or trading history on markets that the user bet on) of that user either:

Probably to avoid a lot of [deleted]-[deleted] comments from a lot of different markets that the user had commented on?

higherLEVELING avatar
higherLEVELINGpredicted NO at 59%

@firstuserhere I don't believe the account is deleted though. They liked the comment i just wrote above.

Lavander avatar
Lavanderpredicted NO at 59%

@higherLEVELING witchcraft!

VictorLi avatar
Victor Lipredicted YES at 62%

@higherLEVELING yea they just liked my comment as well lmao wtf

IsaacKing avatar
Isaac King

@VictorLi Mira is the ghost of Manifold. Dead but not gone...

Thunderstar avatar
Thunderstarpredicted YES at 58%

@IsaacKing so much for taking a break from manifold 😂

Charlie avatar
Charliepredicted NO at 56%

This feels like a market that is careening toward a highly contested resolution even with @IsaacKing‘s best efforts to resolve it fairly.

Greedily bets more NO.

JacyAnthis avatar
Jacy Reese Anthispredicted NO at 51%

@Charlie I think the ambiguous "closer" criterion is the cause of most contestation. I would personally be more excited about a market that just resolves YES if Microsoft or OpenAI explicitly say that ChatGPT, as it was in February 2023, is based on GPT-4, and resolves NO otherwise. This has downsides, such as that F23-ChatGPT could be based on GPT-4 but Microsoft and OpenAI never get around to mentioning that. Personally, I think the unambiguous resolution would outweigh those downsides.

Charlie avatar
Charliepredicted NO at 41%

@JacyAnthis perhaps you can create that market. I’m sure @IsaacKing would welcome it, and I’d be interested in seeing any discrepancies.

IsaacKing avatar
Isaac King

@JacyAnthis I think that would end up being pretty similar to this one

hyperion avatar
hyperionbought Ṁ200 of YES

https://blogs.bing.com/search-quality-insights/february-2023/Building-the-New-Bing

More direct confirmation from Microsoft that Sydney is based on a "next generation GPT model" "more powerful than GPT-3.5"

Last Summer, OpenAI shared their next generation GPT model with us, and it was game-changing. The new model was much more powerful than GPT-3.5

JacyAnthis avatar
Jacy Reese Anthispredicted NO at 56%

@hyperion Note that "much more powerful than GPT-3.5" is probably a lower bar than question resolution. Bing's chatbot needs to be closer to GPT-4 than "GPT-3 and its derivatives such as Codex and ChatGPT," and, presumably, ChatGPT is already (much) more powerful than GPT-3.5.

R2D2 avatar
R2D2predicted NO at 47%

@JacyAnthis nope. 1) GPT-3.5 is not a single model but a series of models https://platform.openai.com/docs/model-index-for-researchers/models-referred-to-as-gpt-3-5 2) the last one of this models (text-davinci-003) was trained with PPO, the same way ChatGPT was trained https://openai.com/blog/chatgpt/ so there’s really no evidence at all that ChatGPT is more powerful than GPT 3.5

hyperion avatar
hyperionpredicted YES at 47%

@JacyAnthis Really? "much more powerful than 3.5" is exactly what we're expecting to get from GPT-4, expecting anything more is a bit silly. ChatGPT is GPT-3.5.

StrayClimb avatar
Reynoldspredicted NO at 60%
R2D2 avatar
R2D2predicted NO at 49%

@StrayClimb 1) Davinci is (one of the iterations of) GPT-3, not GPT-4 https://platform.openai.com/docs/models/overview 2) ChatGPT already has a context of 8k, so that’s nothing to be impressed about 3) 32k, now this sounds a bit more peculiar. However if GPT-4 is going to be a multi-trillion parameters model, as rumored, with many optimizations over GPT-3.5, then this isn’t it at all.

Mqrius avatar
Mqrius

Apparently Sydney sometimes says GPT4 unprompted. Hard to be sure though give that by now there's enough speculation online about it that she might have found

Mqrius avatar
Mqrius

Re this

R2D2 avatar
R2D2predicted NO at 49%

@Mqrius It could easily claim to be the Pope, but that’s not evidence it’s the Pope. These models lie like there’s no tomorrow. A non-technical intro: https://noahpinion.substack.com/p/why-does-chatgpt-constantly-lie

Mqrius avatar
Mqrius

@PippoPluto do you mean evidence in the yes/no sense, or evidence in the Bayesian update sense? Because it's definitely the latterz even if it's weak. In the world where Sydney is based on GPT-4, she's more likely to say so than in the world where she's not. But I agree that in the world where she's not based on GPT-4, it's still plausible for her to say she is. Just a bit less plausible.

R2D2 avatar
R2D2predicted NO at 50%

@Mqrius I meant in the yes/no sense, but even in the Bayesian update sense, P(claims to be GPT-4| it’s GPT-4) is not necessarily higher than P(claims to be GPT-4| it’s not GPT-4). Because of the LM objective (“predict the next token”), these models tell you what you want to hear. They’re trained to sound plausible, not to tell the truth. Have a look at the link I shared.

LeoSpitz avatar
Leo Spitzpredicted NO at 54%

Yes, yes, it's the next-generation model. But what will OpenAI call it? GPT-4 or something completely else? We know that Bing is running on something more powerful than GPT-3.5, yet OpenAI could release a completely different kind of model under the GPT-4 name.

I heard Sam Altman mention in an interview that they "need to figure out this naming thing."

LeoSpitz avatar
Leo Spitzbought Ṁ1 of YES

@LeoSpitz Or never release a GPT-4 at all.

R2D2 avatar
R2D2predicted NO at 50%

@LeoSpitz They will, since they’re now under high pressure to actually become profitable, given the huge investment made by Microsoft (who will be claiming their ROI). But no one knows when.

R2D2 avatar
R2D2

OpenAI models are not publicly released (meaning it's impossible to have an independent verification of the claim) and the names are there for marketing purposes only. For example, there are many models that, at some point in time, were called GPT-3 (Davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003...). Thus it seems difficult to determine what is the model behind Bing. Almost surely, it won't be a fixed model, but different models will be served through the same Bing AI service at different points in time. So "when" should Bing AI be GPT-4, for this to resolve as YES?

IsaacKing avatar
Isaac King

@PippoPluto As of market creation.

Stefan avatar
Stefanbought Ṁ50 of NO

Apparently Sydney has been running in several markets for a year, making it seem less likely it's based on GPT-4. https://twitter.com/MParakhin/status/1627330287276261381?t=eYUlpATUkoUzvX8WOLPjkA&s=19

ShortStick avatar
ShortStickpredicted YES at 84%

@Stefan i couldn't confirm if this is a official account and to be honest it screams a fake account

IsaacKing avatar
Isaac King

@ShortStick It's real, see the other comments below.

mkualquiera avatar
mkualquierapredicted YES at 84%

@Stefan

They have changed the underlying model a number of times.