Step change anyone? https://open.substack.com/pub/maximumtruth/p/massive-breakthrough-in-ai-intelligence

o1 seems to count as both a new class and step change

also fits in line with JD's comment below "I hear whispers / speculation (e.g., strawberries) that something much better is coming and want a price on the likelihood on that happening by the end of the year."

I think the only potential squabble I could see is someone arguing that "a step change better" contains an implicit "and is not worse in any existing area" which seems dubious in a "best judgement" sense of this feeling more like the 3-to-4 "step" where new things became possible.

@JamesBaker3 I personally don't see it at the level of the 3.5 to 4 step change. GPT-4 cut error by ~50% across the board on all benchmarks - this is mostly about driving up math and (to a lesser degree) logical reasoning.

Win rate over gpt-4o hovers around 58%, with no gain for text/writing. That's about half the ELO difference between gpt-3.5 and gpt-4.

The Information article on strawberry even noted some testers felt the pause isn't worth the increased smartness (at least in some categories). I feel that using o1 for even reasoning tasks -- unlike with gpt-4, where I fully switched accepting the slowness.

@JamesBaker3 It obviously does not.

But the fact that someone can even claim such nonsense is good reason for people not to bet in this market. In contrast it is very much a reason to bet in the market about "GPT-5" coming out this year, which will not happen. OpenAI is not going to tarnish their brand in that way (by calling something GPT-5 when it is at most a slightly improved GPT-4).

@DavidBolin "does not" what? I'm guessing you mean the "step change" part, because "a new class of ai model" seems really really clear. James D gets to decide what "significantly better" (from the title) vs "narrowly better". I think that even if o1 mini & preview don't cross that line, their main o1 does/will (and will be even more so before 12/31).

Will be reviewing o1 vs the criteria above, but will be a little delayed - new baby came this week. Please bear with me.

@jdilla Sounds like the main improvement is training to do chain of thought without prompting, which doesn't really strike me as a step-change. https://x.com/GaryMarcus/status/1834293745782870488?t=qFvvCMC71-pJ3EWSjmODcg&s=19

@WilliamGunn You are quoting Gary Marcus saying "It is definitely impressive" and counting that as evidence against?🤣

@JamesBaker3 Did you see the part after "but"?

@WilliamGunn Bet against me then! 😂 I'm not planning to argue much here, I've made my bets and will trust James D to resolve in line using his best judgement in line with the spirit of the question.

I'm going with no still, because I think model improvement will be slow enough that there will be no obvious phase shift to "GPT-5 class model" in the way there was between GPT-3.5 and GPT-4

one of 3.5 Opus, Grok-3, or Gemini 2 will probably fit the bill for me. Idk if you will agree.

These aren't live yet, are they? As far as I can tell 3.5 Opus is still coming

(Aside: I hate Anthropic's naming system - I can never remember which one is which)

I don't like this market's design, so I'm not bidding.

My main issue is that a model with 10% of people randomly being allowed access to it is not publicly available. OpenAI has done this repeatedly with "beta" programs for GPT-4o and its previous models, or when it announced memory and that was rolled out to increasing numbers of users over weeks.

There's a reason why I use Anthropic models - they announce something and it is available to everyone who wants to pay for it on the same day. And, Google's AI studio allows immediate testing for everything they announce. These companies have actual models that can be used, while a model that isn't available to anyone who wants to pay is not publicly available.

Totally fair! I struggled with that criteria as well.

Given OpenAI's rollout process, I felt I would be unable to prove that any release of theirs was available to all users, but would be able to tell if it was invite only (as in, Ethan Mollick has it but no one else does).

With that said, if you have better ideas on how to improve this, I'll take them!

Don't change this market, as it's too late. However, in the future, I would only resolve a market as YES when the model can be used by any person in the United States who is willing to pay for the service on the same day that they sign up for it.

Yeah, definitely not going to make any changes that I believe substantially change the way the market has been defined to date. With that said, I do believe in clarifying in the spirit of what's written.

I'll consider that in the future!

This contains a several false statements.

Anthropic Claude 3 included the announcement of Haiku, but it wasn’t available until later. 3.5 Haiku and Opus are announced but not available.

Google Gemini Ultra was never made available to users. Pro 1.5 was provided to small cohorts, and 1M and 2M context lengths were also rolled out slowly after their initial announcement days.

It’s fair to be upset about not having access to something that you want, but it’s inaccurate to state that OpenAI is the only one doing progressive rollout launches.

I am specifically referring to instances where a model is demonstrated and made available to a small number of people. GPT-4o voice is an extreme example of vaporware like that.

None of Anthropic's products fit this criteria. They did state that other 3.5 models will be available in the future, but they never claimed they are ready yet and nobody has ever had access to them. They aren't making videos that depict 10 "beta" users running 3.5 Haiku or creating a "waitlist."

With Google, the Pro context lengths are indeed available to all users on the AI studio, which anyone can get access to if they know howto do it, and they actually were available to any user who wanted them.

So that leaves Gemini Ultra, which indeed is an example of Google promising something that wasn't delivered. I would say then that Anthropic is the most trustworthy and customer-focused, followed by Google, and then followed by OpenAI.

It's highly misleading that the title says "GPT-5" but the description talks of a model "by anyone (not just OpenAI)". I'd recommend changing the title

Title updated! With that said, if you can't be bothered to read the resolution criteria before betting your mana, I'm not sure I can help you.

Considering this is a lot higher than will GPT-4 come out this year this is free mana

Well, you are betting against anthropic. The next Opus model is definitely coming out this year. I just think it's too difficult to define what a step change would be

IIRC the question originally said GPT-5, but OP changed it. I want my mana back :(

this part of the resolution criteria has remained the same from the beginning:

> a new class of ai model is released by anyone (not just OpenAI). The model must be in wide availability. Some gates are acceptable (e.g., paid users only, 10% of users), but it cannot be available to selected AI influencers.

I think that's pretty clear, but let me know if you disagree.