Dan Hendrycks (Director of the Center for AI Safety) claims:
Let's very generously extend the timeframe to "anytime in 2024", instead of "within a month" as claimed.
First I established a baseline poll here about last year, 2023:
I will repeat that exact poll about the year 2024, in 2025-Jan.
If >=50% of poll respondents say "Yes" for the 2024 version, then we'll say Dan Hendrycks was right. Otherwise we'll say Dan Hendrycks was wrong.
Results that are botted, hacked or manipulated won't count (I'd openly consult with the mods).
Let's establish a better understanding of what is meant by "obsolete." The most stringent definition might involve people unilaterally deferring to AI forecasts instead of human ones before 2025. I think this is probably too high of a standard and likely not Dan's view, especially since even with very strong forecasting bots, it also seems unreasonable to require that humans will almost never be able to improve on or spot mistakes in the CoT of an LLM forecast.
Instead, I think a more reasonable interpretation of Dan's claim is that "human-only" forecasting will be obsolete by 2025. That is, purely human forecasts (even those of superforecasters), which do not reference any LLM-forecasting tools, will be considered consistently inferior to LLM-aided forecasting, such that LLM forecasts are treated as the "de facto" starting point in the process of making a forecast.
@AdamK I think a good interpretation is "for the majority of forecasting tasks, humans don't improve performance by more than a tiny amount over just using AIs". This doesn't require that in practice people do use AIs for these tasks (e.g. because people might be irrational). (Edit: changed from "vast majority" to just "majority")
I don't think '"human-only" forecasting will be obsolete' is a reasonable interpretation of the claim. Like "no-google" forecasting is obsolete, but we don't say that humans have been made obsolete by google. TBC, if LLMs are adding almost all the value, then I think this would count as humans being obsolete. I think 'LLM forecasts are treated as the "de facto" starting point in the process of making a forecast' seems a decent amount of the way there, but not sufficient.