@mods please resolve to N/A, as that was stipulated at the time of market creation and there’s no other fair resolution option

@nsokolsky done

GPT-4 is not visible on the main page anymore due to the sheer number of models, unfortunately. Hence planning to resolve to N/A unless someone figures out a way to get the data. @traders

We could do various extrapolations to claim that Gemini-exp-1206 would've had a 75%+ win rate, but my own rules clearly state that if the win % for at least one GPT-4 model is no longer available, the question must resolve to N/A.

@nsokolsky I tries to repro the chart with more models using the colab notebook, but the raw dataset they provide isn’t recent enough. If anyone has the pointer to a more recent one I could try to repro the leaderboard with that one?

@Sketchy yep sadly I don’t know how to get more recent data. I’ll resolve to N/A to ensure the resolution is in line with the stated criteria, but in spirit I believe the answer is YES.

@nsokolsky Agreed, I think it's YES and I'm saved only by pure luck.

Still at 71.3% for o1-preview

o1-preview is now back to a lead of just 71.6% over the strongest GPT-4 model after more votes came in.

New question with a more reasonable threshold.

Very small sample (<30 heads up matches) but o1-preview is now at a 75% win rate against both remaining GPT-4 models in the Arena. I'm now quite bullish on this market resolving to Yes.

I fear regression to the mean will wipe out this gap as we get more samples.

@SergeyDavidoff possible but 'o1' (non-preview) is supposed to come out by EOY, plus I assume o1-preview is actively being tweaked right now based off user feedback.

gpt-4o-latest is now at a 68.4% win rate against the latest GPT-4

Gemini 1.5 Pro (experimental) is at 62.7% as of today against the latest GPT-4

turbo isn't the latest gpt-4.

oh nm - gpt-4o doesn't count

Chat Arena is a flawed benchmark anyway, and based on the numbers it seems almost impossible this would happen. GPT-4o, the number 1 model on the leaderboard, only has a 71% win rate against its most lopsided matchup. That matchup is GLM-4, a model which is terrible in all my evaluations.

Even if the next gen of models are a huge step up over the current SOTA, I don't expect any of them to achieve that high of a win rate

@JaundicedBaboon

Very small sample (<30 heads up matches) but o1-preview has now done it :-) The market will resolve on January 1st and things might change with a bigger sample size but l'm becoming pretty bullish on this resolving to Yes.

GPT-4o has a 61% win rate against GPT-4-turbo. Pretty good but still needs a lot of work to get to 75%.

GPT-4o did not follow the gpt-4-x scheme, so it doesn't count as GPT-4 for the purposes of this question (it would've if it was called gpt-4*-*o). So if gpt-4o beats gpt-4-latest with a 75+% win rate in the leaderboard, this will resolve to Yes.

Limit up if anyone wants to take it

why is this market less than the gpt-5 comes out in 2024 market? The 'gpt-5 will be underwhelming' money?
https://manifold.markets/VictorLJZ/will-gpt5-be-released-before-2025

@JoeandSeth GPT-5 could come out and still fail to have a decisive advantage over GPT-4.