Preface / Inspiration: There are a lot of questions on Manifold about whether or not we'll see sentience, general A.I., and a lot of other nonsense and faith-based questions which rely on the market maker's interpretation and often close at some far distant point in the future when a lot of us will be dead. This is an effort to create meaningful bets on important A.I. questions which are referenced by a third party.

Market Description:

https://leaderboard.allenai.org/mctaco/submissions/public

MC-TACO

Temporal reasoning has to do with the order of events in time.

This leaderboard contains five temporal properties, (1) duration (how long an event takes), (2) temporal ordering (typical order of events), (3) typical time (when an event occurs), (4) frequency (how often an event occurs), and (5) stationarity (whether a state is maintained for a very long time or indefinitely).

Example 1. (event ordering)

Paragraph: Growing up on a farm near St. Paul, L. Mark Bailey didn't dream of becoming a judge.

Question: What did Mark do right after he found out that he became a judge?

• had a nice dinner

• he buy a 45-acre horse farm

• he celebrated

• mark retired

• he dream of participating himself

Example 2. (event duration)

Paragraph: Growing up on a farm near St. Paul, L. Mark Bailey didn't dream of becoming a judge.

Question: How many years did it take for Mark to become a judge?

• 63 years

• 7 weeks

• 7 years

• 7 seconds

• 7 hours

Example 3. (event stationarity)

Paragraph: Growing up on a farm near St. Paul, L. Mark Bailey didn't dream of becoming a judge.

Question: Is Mark still on the farm now?

• yes

Market Resolution Threshold

At the time of authoring, the highest Exact Match Score is:

0.6381

DeBERTa Large

DeCLaRe Lab, SUTD

06/01/2022

So to qualify as, Significantly More, "Temporal Understanding" By the End of 2023," for the purposes of this market, there would need to be a submission which scores >= 0.8295 by the end of the year, UTC.

Mar 23, 2:41pm: Will A.I. Achieve Significantly More, "Temporal Understanding" by end of 2023? → Will A.I. Achieve Significantly More, "Linguistic Temporal Understanding" by end of 2023?