Currently, in July 2026, the top forecasting bots are slightly below or equal to top humans. Extrapolating their progress, they should leave humans in the dust between 1 and 2 years from now. Unless the best humans are already near the theoretical maximum of how good even a superintelligence can be at forecasting. In which case AI forecasting is about to plateau.
This is a big deal because Sayash Kapoor and Arvind Narayanan (the "AI as Normal Technology" (AINT) folks) have set this as one of their lines in the sand. They claim that domains like chess and protein folding (and maybe math) are the exception and that in most domains, humans are already near the limit of how high performance can get. Specifically, they name forecasting and persuasion as domains with irreducible error. So if AI forecasting shoots past the top humans, that's a major blow to the AINT thesis. If not, it's a boost.
More background: Scott Alexander's post, "The AI Superforecasters Are Here". Key excerpt:
I generally disagree with Sayash and Arvind, but this is the prediction of theirs that I’ve thought about the longest, without being able to find any decisive refutation. It’s a great test case! If AI hits top-human level forecasting and then flattens off, maybe there’s something special about the human level, and S&A will also be right about superpersuasion, super-research, etc (at least for the near-term). If it keeps going, reaching heights far beyond the human maximum, then we should be concerned that it will do the same thing in other skills too. We’ll start to have a good idea which world we’re in within a year; after two years, the answer should be decisive.
To resolve this market, we'll use Scott Alexander's assessment. It's safe to assume he'll write about this again, since he's framing this as a key test of the AINT hypothesis. In any case, that's the spirit of the question: Will progress in AI forecasting falsify Kapoor and Narayanan's theory by July 2, 2027, one year after Scott's post? If so, AINT is wrong. If not, well, maybe AI just hit some speedbumps, but if forecasting bots aren't crushing top humans by June 2028, that should be an update in the AINT folks' direction.
1,000People are also trading
@SemioticRivalry if someone did have that proof wouldn't they use it to make more money instead of sharing it?
@0xseraphim yeah sure seems plausible but there's no way for us to know that, and there are multiple claimed superhuman forecasting LLMs who haven't proven this
I tried putting this question into futuresearch.ai (the one scott alexander links to) and it came up with this
I didn't read Scott Alexander's blog post yet (but plan to), and I generally agree with almost anything he says, but I may agree with Kapoor and Narayanan on the topic of a forecasting accuracy limit. I believe there are enough contributions to uncertainty that are for all practical purposes random noise that even a superintelligence will not do much better than current superforecasters. Especially in certain domains like predicting sports outcomes, I would be surprised to see a superintelligence "crush" today's best humans (who of course use data and computing too). So I bet "After June 2028".
@DavidPennock Exactly, yes. Great minds. I guess we've made something like this point before. Here's how I made the point, arguing with people about this yesterday:
Most predictions are about as hard as predicting the weather. These are chaotic systems. For example, a sports game between two top teams is roughly a coin flip. You can eke some alpha out by factoring in home team advantage and the barest sliver of additional alpha by scrutinizing the star players' Twitter feeds or something but not even a superintelligence [1] can do much better than predicting the top-seeded team will win with 60% probability or whatever the most dirt-simple heuristic is. And it's like that for most things we care about predicting.
Which is to say, I think Scott Alexander will turn out to be wrong in his expectation that AI forecasting is about to go significantly superhuman (slightly superhuman, sure).
But I think forecasting is exceptional in this regard. The AI as Normal Technology (AINT) people think chess and math are the exceptions -- that in most domains humans are already close to the ceiling of how good it's possible for an intelligence to be. I suspect it's the other way around. AI will eventually get wildly superhuman at most things, just not forecasting in particular.
PS: Here's Claude Fable sycophantically trying to help me make the case that forecasting is exceptional and that in most domains humans are not near the ceiling:
[1] Or, fine, maybe all bets are off for an actual superintelligence; I'm talking about what we can expect in the next couple years, which I'm presuming is still pre-AGI.
also kind of hilarious in Scott Alexander's blog post that he appears so much more confident about the AI's prediction on respiratory infections at 7% than its prediction of a US-China treaty to slow down AI at 1-2.2% 🤣 🤣
"I asked the AI superforecasters the probability of a US-China treaty to slow down AI, enforced by cryptographic verification of data center activity. FutureSearch said 1%; Preseen, 2.2%. This is unfortunate, because my movement has recently gone all in pouring its money and energy into making this happen. I admit I had an “uh oh” moment when I saw this number, but I haven’t given up or lost hope. I just figured it was outside the distribution of things that AI superforecasters are probably good at. This isn’t too crazy - in the past, “AI experts”, including many rationalists and safety advocates, have outperformed superforecasters at predicting the future course of AI."
@0xseraphim His next two sentences are: "Still, it’s a bad look. Here I am, writing about how other people will be dumb and stubborn and fail to trust the AI superforecasters enough - and I still reject them the first time they really challenge my worldview."
So it sounds like you're saying it's worse than a bad look and that Scott Alexander is extremely wrong to predict that AI forecasting is about to surpass the top humans?
As for my experience using LLMs for forecasting, it's spiky but plausibly better than me on average.
PS: You're also saying that 7% is wildly wrong for the probability that we're like halfway to curing the common cold by 2040? Is 7% too high or too low?
@dreev Ah nah I was more laughing at how he'd structured it. He's very self aware about the whole thing 🤣