Will AI pass the Winograd schema challenge by the end of 2025?

Ṁ1kṀ2.3k

resolved Jan 22

Resolved

ALL

https://en.wikipedia.org/wiki/Winograd_schema_challenge

Resolves positivly if a computer program exists that can solve Winograd schemas as well as an educated, fluent-in-English human can.

Press releases making such a claim do not count; the system must be subjected to adversarial testing and succeed.

(Failures on sentences that a human would also consider ambiguous will not prevent this market from resolving positivly.)

/IsaacKing/will-ai-pass-the-winograd-schema-ch

/IsaacKing/will-ai-pass-the-winograd-schema-ch-1d7f8b4ad30e

/IsaacKing/will-ai-pass-the-winograd-schema-ch-35f9dca7fa7d

/IsaacKing/will-ai-pass-the-winograd-schema-ch-d574a4067e75

Market context

Get

1,000

to start trading!

🏅 Top traders

#	Trader	Total profit
1		Ṁ1,238
2		Ṁ223
3		Ṁ113
4		Ṁ64
5		Ṁ58

People are also trading

Will AI pass the Winograd schema challenge by the end of 2030?

94% chance

Will an AI get gold on this Olympiad by the end of 2025?

Will AI pass the Rube Goldberg Turing test by the end of 2028?

38% chance

Will AI pass the Longbets version of the Turing test by the end of 2029?

51% chance

AI solves Millenium Prize Problem in 2026?

6% chance

Will AI pass Video Turing Test by 2030?

68% chance

Will AI pass the Bob Ross Turing Test by 2035?

68% chance

Will an AI achieve a perfect score on the Miklós Schweitzer Competition before 2035?

81% chance

Will OpenAI be in the lead in the AGI race end of 2026?

31% chance

Will open-source AI win? (through 2028)

Sort by:

I assume this resolved NO because no one bothered to test it?

@vluzko Yeah, see my comments on the 2024 version.

So roughly >=97% on Winograd? Also this is Winograd and not Winogrande, right?

predictedNO

I think a human could do better than 97%? If AI seems to be right on the boundary, I'll use myself as a benchmark. If it does at least as well as me, the market resolves YES. (I get to look up any words I don't know the definitions of, but I don't get to look up anything else.)

Yes.

predictedYES

@IsaacKing Wikipedia reports 94-96 % for humans, the Winogrande paper reports 97%

I made a market on this a while ago:
https://manifold.markets/SneakySly/will-ai-image-generating-models-sco

predictedNO

@SneakySly That market appears to be about the Winoground test for image models. This market is about the Winograd test for language models. They're entirely different things, they just have a similar name. (I assume Winoground was a pun based on Winograd.)