Preface:

Please read the preface for this type of market and other similar third-party validated AI markets here.

Third-Party Validated, Predictive Markets: AI Theme

Market Description:

Break

As measured by, "Break," (non high-level) from Allen A.I. leaderboards here:

https://github.com/allenai/Break

https://allenai.github.io/Break/blogpost.html

"Significantly better," will be interpreted as meaning 30% better Normalized EM Score than the top post on this leaderboard at the time this market opened, compared to the end of the year, UTC.

https://leaderboard.allenai.org/break/submissions/public

Market Resolution Threshold:

At the time of authoring, the highest EM Score is:

0.4230
T5-Large

Tomer Wolfson, Tel Aviv University

So to qualify as, "Understanding the, "Meaning" of Questions Significantly Better By the End of 2023," for the purposes of this market, there would need to be a submission which scores >= 0.5499 by the end of the year, UTC.