Question for @pietrokc in particular after Tuesday's mathocalypse: How should we think about resolving this in the hypothetical world where frontier research is paused and GPT-7 or whatever internal model solved 372 more major open problems, supposedly with just a few hours of compute per problem, is never released?

My own feeling (note my conflict of interest) is that the mathocalypse is enough for an early YES. The path to NO is getting so narrow. GPT-7 (or whatever it gets called) would have to:

• fail to max out every benchmark (FAQ2)

• fail on specific, non-adversarial problems (FAQ3)

• fall on its face on enough easy problems to outweigh all the superhuman problem-solving (FAQ7).

Of course failing on FAQ2 and FAQ3 wouldn't be particularly surprising. And unfortunately FAQ7 is somewhat subjective. But if even the currently released models have no failures we can find that are enough to trigger FAQ7...

I mean, even if we do scrounge up those examples (and don't get me wrong, I very much want to see them regardless!) there's the word "routinely" in FAQ7.

(Let me also apologize again for the messy resolution criteria. I didn't really anticipate AI solving major open problems other than to add FAQ7 as a kind of afterthought. And I really should've. Extreme jaggedness of abilities has been the hallmark of AI for forever.)

@dreev Again I have to call attention to the asymmetry in information sources.

When AI solves a big open problem, the company that makes that AI will announce it far and wide. When AI fails to solve a "normal" problem, how are we going to hear about it?

The "mathocalypse", while very impressive, doesn't come close to showing what is required for this market to resolve YES. For many independent reasons.

The third batch of FirstProof just came out. These are normal problems that normal mathematicians come across in their day to day research and solve within a few days, as small steps towards larger goals. You can see their statements in the pdf. For the purposes of this market, the relevant table is the "correctness" table, where each of the 5 submissions is graded on whether their provided solution is correct. Note that no individual model got more than 7 out of 10 of these normal problems, and even a "proof council" that called "GPT-6 Astra Pro, GPT-6 Astra, Claude Fable 5.1, Claude Sonnet 5, and Gemini 3.1 Pro (preview), with web search enabled" only got 8.67 out of 10.

Michaël Trazzi (@MichaelTrazzi) on X

What do @traders think of pinning down the human/superhuman threshold as "what a team of 4 Fields Medalists can solve in 4 days"? Up to now I had only referred to what a "team of Fields Medalists" could solve in "a day or a week". Since 4 is the average of 1 day and 7 days, might as well go with 4?

I'm trying to be extra careful and transparent since I have a large position myself. Currently for YES but when this market was new I put thousands of mana on NO.

Mostly my thinking now is that the unit distance problem and the Jacobian conjecture and especially Navier-Stokes already put us far beyond what was required for YES. But as @pietrokc is arguing, we also need clarity on jaggedness. AI is far beyond the threshold on some problems but below it on others. FAQ#7 actually tried to anticipate this, posing what was meant as a wildly hypothetical edge case: AI that's "so blatantly superhuman that it cracks a famous open problem, yet it's routinely stumped or wrong on problems amateur human mathematicians can do." I went on to suggest that we'd resolve such ambiguity by asking if a human with the AI's abilities would be considered the greatest math problem-solving prodigy of all time.

Which is annoyingly fuzzy, now that we're actually in something like that wild hypothetical. But I'm not sure we need to adjudicate the prodigy question because in the hypothetical the ambiguity arose from the combination of cracking a famous open problem and routinely being shown up by amateur mathematicians. Can anyone (@pietrokc?) make the case that it's routine for amateur mathematicians to show up Astra or Fable (let alone the latest internal models)? In the example of the SAIR competition the human winners were professionals.

Further clarification on FAQ items 2 vs 7: FAQ#2 says we need to saturate every benchmark. FAQ#7 says the AI can potentially be routinely outclassed by amateurs and still count as superhuman if it also cracks a major open problem. The seeming contradiction is because most of the FAQ just presumed AI would not solve a major open problem. So it's basically saying "superhuman means solving everything humans can faster and cheaper OR solving major open problems without otherwise being too blatantly subhuman at math".

@dreev I would not consider current AI results to make someone the best maths problem solved of all time as per FAQ7. Although it's getting close and I expect it to be unambiguously the case by the end of the year.

@dreev Well... The market's deadline is "before 2030" which is approximately a lifetime and a half in this cursed singularity. If we already find ourselves debating precise definitions because of capability jaggedness late-2026 (more than three years to go still to smooth out the edges!), the industry will be able to clear any resolution criteria you throw at it within a year at most, and it looks to me like we might even get there by the end of this year. So I wouldn't spend too much time and effort endlessly refining criteria that will fall within the time boundary as an already foregone conclusion; my biggest concern here is actually "assuming no AGI yet"; i.e. we only need to clearly and unambiguously separate between these two major conditions, and the rest will be self-evident enough.

@dreev Some disorganized thoughts.

I think there is a contradiction between "considering AI the greatest raw math prodigy of all time" and the fact that AI can be scaled up indefinitely by just spending more money.

Here is the fact / intuition that I find it helpful to come back to. We have known that a computer could prove every provable theorem / solve every solvable problem since the 1930s. The technical result is "the set of theorems of ZFC is recursively enumerable" if you wish to google or ask models (in case you're not familiar). It means you can set off a computer with very simple initial configuration, and if you give it enough time and memory, it will eventually spit out every provable theorem, while spitting out no unprovable statements.

Of course there is nothing special about computers. A human could carry all this out by hand. Or you could parallelize it across a billion humans to speed it up. But this clashes with the colloquial meaning of "raw math prodigy". If Ramanujan had obtained his amazing results by just blindly listing all provable theorems for billions of years, then occasionally using a time machine to bring the theorems back to our time, would we consider him a prodigy? Or if he got the results by cloning himself a billion times and spreading this blind work?

So if we're going to put a 4-day cap on the "team of Fields medalists", we need to put a cap on AI as well, which is your FAQ#5, but I feel is being forgotten in your latest post.

This concern has already manifested in practice. OpenAI solved Navier-Stokes by spending more than $20M. Given that their result was just the final brick in an edifice that humans had been constructing for decades, and that $20M can pay the salary of ~10 top mathematicians for ~10 years, it is not at all clear that solving N-S with AI was cheaper.

the unit distance problem and the Jacobian conjecture and especially Navier-Stokes already put us far beyond what was required for YES

I don't think this follows at all. The market is explicitly about the AI dominating humans in ALL problem solving. If a few examples were enough, then this market would have resolved positively in the 1990s if not before: Robbins algebra - Wikipedia

@pietrokc Re: recursive enumerability: true fact! It's a little like how infinite monkeys on infinite typewriters eventually reproduce Shakespeare. And you're right about FAQ#5 (cost and speed). It's meant to plug that loophole.

I take your point that for Navier-Stokes in particular it's not obvious whether humans could've gotten there with a similar budget. Here are Claude's estimates on human-vs-AI costs for the biggest open problems AI has solved:

@pietrokc Re: the Robbins conjecture: the above open problems seem pretty different to me, but I'm not a mathematician. As for how to interpret this market, it was originally something like "presumably AI won't solve major open problems pre-AGI but will it mog humans?" and then FAQ#7 considered the case that it actually were to solve major open problems, implying that that would suffice if it weren't otherwise too dumb.

So how dumb is it otherwise, could be the key question.

If we don't have good examples of problems that amateurs can solve that AI can't, then I think that'll mean we're already at YES.

(I know this isn't likely to matter for the actual resolution since there are 3.3 years for these i's/t's to get dotted/crossed but I'm actually very interested in the question of when exactly the superhuman math problem-solving milestone should be considered to have been crossed. Probably sometime within 2026?)

@dreev What are the human costs in those estimates based on? Wild ranges aside, they kind of treat these problems as something that can be solved by throwing money at human mathematicians which, if it were actually the case, would've been done decades ago by big businesses who'd directly benefit from having those solutions on hand. But mathematical ability isn't a commodity; there's no guarantee that any number of people would be able to figure many of these problems out regardless of the available time and motivation. And if we're talking the best of them, money is still not the bottleneck.

@moozooh Yeah, those are just wild guesses at the market rates of Fields Medalists and how long those problems would've taken them. I agree we can't put much stock in those numbers and for the biggest open problems maybe the answer should be like a quadrillion dollars if the whole worldwide math community was stumped for decades. There's some ambiguity with Navier-Stokes but it seems like for everything else, the AI solutions cost much less than if you had to hire humans to solve them.

Mostly the point of talking cost is just to close the loophole where the AI solves math problems by quasi brute force with absurd amounts of compute. The point of automation is to do it faster/cheaper than humans.

@dreev Yeah, being cheaper is a foregone conclusion imo, we still have three years to drop intelligence costs and they're already more than competitive in this particular domain.

@dreev I don't have time for a longer response now but those Claude estimates are totally off. As is to be expected; what data would Claude be drawing on, to make these calculations?

I fear a lot of folks in these threads have a caricature of math in mind. It is absolutely not the case that "AI mogs humans" even in the case of Navier-Stokes, as I alluded to in my "last brick in an edifice" comment.

To give just one example: $200k/year is a very respectable tier-1 US research university salary. The salary is actually much lower in Europe! (These are the people who actually solve most problems, Fields medal publicity and idol worship notwithstanding.) But that salary doesn't buy you an Erdos unit distance solver; it buys a solver of whatever problems the prof feels like solving, following whatever strategy they feel is most likely to succeed, most fruitful even if unsucessful, etc. With AI, on the contrary, you can tell it exactly what to do, and you can spawn one copy for each strategy you can conceive of.

I claim, and I think this is general community consensus, that if you could have paid a combinatorial geometer to specifically work on disproving Erdos unit distances, there is a very good chance they would have been done in 6mo. You may read Gowers's remarks about this.

@pietrokc Yeah, I guess Claude imagined Fields Medalists would demand sky-high wages. But note that some rows in that table had AI as 1000x cheaper, so even paying grad student rates wouldn't change much.

As for Gowers and the unit distance problem, I think the cope around that was reasonable at the time, back in May. What does Gowers say now? Again, we don't have to fully adjudicate this. Any remaining ambiguity about cost/speed should resolve soon enough.

Interestingly, Toby Ord concluded the other day that the insane amount OpenAI spent on Navier-Stokes was purely to compress it into a week so they could get the scoop. He estimates that they could've spent 10% as much doing it in a month or so. A single agent, at 1% of the cost, could've done it in a year. https://www.lesswrong.com/posts/6cb7qd3RSkgnviCpf/swarm-scaling

Swarm Scaling — LessWrong

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? …

I think the best way to argue we're not at YES yet is to point to the biggest failures of frontier models we can find. Is there any math problem you can solve that a frontier model can't?

@dreev Yes, I believe I can produce such problems, and of course a lot of mathematicians can as well. I made a hobby of this up to about a year ago but to the great credit of AI companies it is no longer so easy to just pull one out. So I'm going to need some time. In the meantime you may be interested to look at the ongoing First Proof project which aims to measure some version of this. The last round ended in July and AI could not solve 3 out of 10 human-solved problems which were far, far below Navier-Stokes level. They expect to have the next round of results, examining the lastest models including Opus 5.5 and Astra, in October.

In general, I expect human-solvable problems on which AI fails to come from budding topics on which there are few or no papers, but which humans can still figure out due to our shared "inductive biases".

I'll say that the Ord essay is extremely unconvincing. He has 15 data points from one company for one task which is not math, but a security benchmark. But I mean, it doesn't even pass the smell test. Here's a simple example. I have a SAT instance with 20 variables. I ask each of 1024 agents to check 1024 possibilities for the T/F settings on the variables. So there are 1024 x 1024 = 2^20 possibility checks in all. If I ask a single agent to check them all, the agent will still need to check 2^20 possibilities. It will cost essentially the same as the 2^10 agents doing 2^10 possibilities each.

@pietrokc Edge of seat to see these math problems, for real!

I don't think Toby Ord is being unreasonable to do his best to extrapolate from what data we have on swarm scaling. It's an important question, how long a single agent would've taken to get that Navier-Stokes answer given that 10,000 agents did it in 88 hours. I probably agree that Ord's analysis is only weak evidence that the answer is around 1 year.

Again, not that that matters for this market, since this market could've resolved YES even without AI ever solving a single major open problem. I think the fairest interpretations of FAQ#3 and #7 is that a single failure on a non-adversarially posed problem is enough for NO, unless the AI is also solving major open problems, in which case we'll be forgiving about failures as long as they aren't too routine/embarrassing. When I put it that way I'm really struggling to see how to squeeze a possible NO out of this, but I'm anxious to see examples regardless!

@dreev I gave it a weak try yesterday with the first thing that occurred to me. There is a very niche variant of the Rubik's cube called the tangle cube, pictured below. Basically I described in text a simplified tangle cube with just 2 strings, and asked about some knot configuration after a simple twist of just one of the faces. To be clear, I didn't just paste a picture; I described it in very great detail in text, using coordinates and explicitly spelling out all the positions, like I did here. A human could understand the question just from the text, the image is not essential.

And here comes a complication: ChatGPT got it wrong the first time, but got it right the second time. (The question I asked, just from its form, is a priori very likely to have just one of two possible answers, so this is the performance you'd expect from a coinflip.) How would we resolve the market in this case?

Now, I'm not sure how much knot theory is formalized in Lean. It's possible that if OpenAI set 10,000 agents loose on this question, they would get it. But it's also a question that any human, once they finished reading it, could answer in under 10 seconds.

Maybe we'll get some better examples from the next round of First Proof.

@pietrokc Oof, that would immediately break my brain. What version of ChatGPT did you try? If you paste it to me I can try with GPT-6 Astra and Claude Opus 5.5. I'll be surprised if at least one of them doesn't get it right.

@dreev if the answer is a coin flip and one but not both of the latest Astra/Claude get it right…

@DavidHiggs Since we don't even require the AI to be publicly available, we can safely say that if any of the released models get the right answer, that counts. We should probably add clarification on how consistently it has to answer correctly to count. My inclination is to say the spirit of the question is to be forgiving on this dimension, but I'm biased at this point. The strictest possible interpretation is that we have to pick a single model that solves every problem humans can solve, on the first try. The least strict might be something like, pose the problem multiple times to each of multiple models and if any run comes back with a solution at reasonable time and cost, it counts as AI solving it.

Actually, as long as AI can pick a correct solution out of many candidates, which I don't think there's much question that it can, we can just make a harness that turns the latter scenario above into a system that one-shots the correct answer.

But all this is moot if we can't even find problems that Astra or Opus 5.5 can't solve, which the jury is still out on. As @pietrokc pointed out, we may learn more from the next round of First Proof.

@dreev Sorry, haven't had time to work on this, but small update: although ChatGPT got the right answer once (out of two times), upon closer reading the reasoning is completely wrong. As I'd mentioned, the question is such that it's very obvious the answer will be one of two options ("unlinked" or "Hopf link"), so the model hallucinated some nonsense and happened to pick the right option that one time.

While not necessarily agreeing with your proposed bar of "the AI just needs to solve the problem sometimes", I will note that this can increase the cost a lot. If an AI can only solve Navier-Stokes 1 out of 100 times, then the correct cost to assign to it is 100x the cost of the successful solution.

AI getting super human at math is my #1 favorite black swan moment of the year so far.

This feels like it's there in spirit, right? And within epsilon for the most persnickety possible reading of the market criteria. FrontierMath is close to saturated and AI aces the IMO and is close on the Putnam.

Reviewing each of the FAQ items:

1. No AGI yet: N/A

2. Saturating all benchmarks: ✅ - 𝜀

3. No Adversarial posing of questions: N/A

4. No multi-page problem statements: N/A [4b. Research taste not required: N/A]

5. Cost and speed: ✅

6. Public availability not required: N/A

7. Jaggedness: ✅

I don't see how any but FAQ#2 could militate for NO and we've still got over 3 years to go to close that epsilon gap. Even if we don't quite close that gap, FAQ#7 suggests that we could still get to a YES despite AI not completely dominating humans at all math problem-solving, as long as the peaks of its achievements are high enough. As of the Navier-Stokes solution, we've got that in spades.

PS: Navier-Stokes did cost millions in compute, but humans couldn't solve that one at all. Even if you think a team of Fields Medalists could've solved it, I'm pretty sure the cost, paying them market rates, would've been even higher. For the hardest contest and benchmark problems, the compute cost and speed seems to blow humans away. And, again, still 3 years for it all to get even better/cheaper/faster.

Btw, if I end up too biased, I'm happy to defer to a neutral arbiter for resolution. Just let me know what you think I'm missing above.

(And note that when this market was new I dumped thousands of mana into NO. Then reality played out.)

@dreev I would say it's not there yet based on public information. The open problems that have been solved are a small fraction of the most important. We can also see frontier maths open problems to have a more precise denominator. Further all the solutions were within human reach as far as I can tell and have been less clever than you would expect based on the problems prestige. In particular the most impressive results are all counter examples so far. Based on this I don't think it outperforms a team of fields medalists for a week on > 50% of problems. However if Open AIs internal results are as good as they suggest this might well be moot.

I'd give it some more time to resolve, especially to hear about performance in other subfields, but it's looking like we'll quite likely hit this before 2030.

@dreev I don't think your assessment is correct.

As I had warned below, the claim in this market is difficult to establish either way. However, recently there was a SAIR competition on the inverse Galois problem. Basically, for each one of the 25,000 transitive permutation groups contained in S_24, find a polynomial whose Galois group is that group. What's interesting about this competition is that everyone was highly encouraged to use AI, and (I believe) got infinite free credits to do so. So here is a scenario in which dozens (hundreds?) of people were trying to combine their efforts with all that AI could offer, to solve the same 25,000 problems. Everyone had two months to do it.

Nevertheless, the winning team did not use AI at all. And you can see that they were the only team to solve this problem for hundreds upon hundreds of specific groups.

More generally, mathematics is incredibly vast, and I worry that this market will be resolved based on vibes from announcements like "OpenAI solved 100 open problems". There is a big PR firehose spraying us with the (very impressive!) positive examples, but what is the mechanism for us to learn of the negative examples?

There's thousands of papers posted on arxiv every day, of which definitely hundreds solve some problem which hadn't been solved before, and of which quite likely dozens had AI tried on it and it didn't work.