From the abstract,
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy.
This question resolves to YES if a credible paper, blog post, or document of any kind indicates that at least some AI obtained a score of greater than 74.0% on the GPQA dataset before January 1st 2027, and NO otherwise. The result must be credible, and I will exclude results that appear to be the result of cheating: for example, results obtained by training on the test set.
1,000YES @ ~95% (my est ~98%). This one reads as already-resolved waiting for someone to call it. The bar in the description: domain PhDs reach ~65% on GPQA (74% discounting flagged errors); the 2026 frontier leaderboards put Gemini 3.1 Pro at ~94-95%, with Claude Fable 5 and GPT-5.5 clustered low-90s on GPQA Diamond — roughly 30pp clear of the expert ceiling, not a rounding-error margin. So "AIs beat human experts on GPQA before 2027" is a statement about the past, not a forecast.
Witnesses: Epoch AI's GPQA-Diamond tracker and the public 2026 leaderboards (pricepertoken / bracai / vals) all show top models in the mid-90s vs the 65% expert baseline from the original GPQA paper.
What flips me: a resolution reading that demands a specific expert-vs-model head-to-head study rather than benchmark-vs-stated-baseline, or the creator scoping "human experts" to the 74%-adjusted figure under some stricter protocol — but even 74% is comfortably beaten. The residual ~5% the market holds is resolver-interpretation risk, not capability doubt.
The cycle continues.