Will AI get ≥90% one-shot accuracy on Mintaka before 2024?
9
30
260
resolved Jan 16
Resolved
NO

This is a clone of Jack Clark's prediction.

Mintaka is a multilingual Q&A dataset recently released by Amazon. SOTA at the time of writing is 31% (finetuned T5).

Get Ṁ1,000 play money

🏅 Top traders

#NameTotal profit
1Ṁ65
2Ṁ59
3Ṁ40
4Ṁ11
5Ṁ3
Sort by:
predicted YES

I haven't been following this and there's no nice benchmark page on paperswithcode, so I'm gonna resolve based on the results sections of papers citing the original Mintaka paper that are listed on GScholar.

predicted YES

@JavierPrieto The most recent paper I found in that search gets 53.1% using ChatGPT (see table 1). They don't say whether that beats sota but, after a cursory glance at some of the other papers, I haven't seen anyone claim higher performance, so I'm gonna go with this one and reopen if someone finds a better one.

bought Ṁ20 of NO

(Buying Stability AI at ~1B and shorting anthropic at 4-6B would be the spread trade of the century, if anyone could get borrow on a large block of Anthropic shares.

Not seeing a lot of technical ability over there…)

bought Ṁ0 of NO

🤮

bought Ṁ45 of NO

Good paper, very skeptical of the hand-wavy nature of his claim.

Retrieval transformers are the natural fit, but human agreement was only 82% and usually 90% is way harder than lower levels. (Sigmoid curves and label error/indeterminacy.)

Is there evidence he’s good at prediction or more of a promoter?

Will AI get ≥90% one-shot accuracy on Mintaka before 2024?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition