Resolution Criteria:

I will resolve this question to yes if, across the majority of professional tasks that could be accomplished by a remote worker (e.g., legal, medical, math, engineering, computer science, etc.), AI systems can perform theproblem solving portionof the task better than a human expert. Humans may still be critical to the task. Maybe a human lawyer needs to talk to their client to figure out what the specific legal question is, and then the AI system can give the answer better than a lawyer. Maybe patients just prefer to interact with a human doctor, but the AI doctor can actually do the diagnostic better. All that's necessary here is that, for the actual problem-solving component, an AI system would generally perform better. I will require that it's fairly clear that the AI system is outperforming the human expert across a wide range of tasks.

Motivation and Context:

Current LLMs are beginning to score very highly on standard tests that we give them, including tests designed for people. Some people believe that this means models will generally outperform humans on most tasks. Others believe that progress on synthetic benchmarks is not indicative of progress on actual real tasks people care about. Which is it? Will future AI systems outperform humans on most tasks?

Question copied from: https://nicholas.carlini.com/writing/2024/forecasting-ai-future.html