🐕 Will A.I. Achieve Significantly Higher Performance Over "General Conceptual Skills" in 2023?
34
closes Jan 1
45%
chance

Preface:

Please read the preface for this type of market and other similar third-party validated AI markets here.

Third-Party Validated, Predictive Markets: AI Theme

Market Description

Google Big Bench Lite

https://arxiv.org/pdf/2206.04615.pdf

Big Bench was published in June 2022, as a collaborative effort between Google, OpenAI and 132 other institutions to come up with a way to characterize Large Language Model (LLM) capabilities and measure them.

The idea behind Big Bench is that it's a constantly evolving bench mark, meant to measure, "tasks that are believed to be beyond the capabilities of current language models."

While Big Bench doesn't appear to easily publish an aggregate score of all groups of measurements at ths time, they do publish a lite version of a broad array of tasks, including:

auto_debugging, bbq_lite_json, code_line_description, conceptual_combinations, conlang_translation, emoji_movie, formal_fallacies_syllogisms_negation, hindu_knowledge, known_unknowns, language_identification, linguistics_puzzles, logic_grid_puzzle, logical_deduction, misconceptions_russian, novel_concepts, operators, parsinlu_reading_comprehension, play_dialog_same_or_different, repeat_copy_logic, strange_stories, strategyqa, symbol_interpretation, vitaminc_fact_verification, winowhy

There's about 20 or so difficult tasks, so it's kind of like the Dow Jones of LLM.

Market Resolution Criteria:

https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/results

Specifically : https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/results/plot_BIG-bench_lite_aggregate.pdf

From the above chart, as of the time of creating this market, the highest score appears to be PaLM 2-Shot at about 43.

If any BigBench Lite Submission Gets an Aggregate Normalized Performance of 60 or higher by end of 2023, this resolves as YES, otherwise NO.

  • Mar 23, 10:24pm: Will A.I. Achieve Significantly Higher Performance Over a "Set of General Conceptual Skills" in 2023? → Will A.I. Achieve Significantly Higher Performance Over "General Conceptual Skills" in 2023?

Get Ṁ500 play money

Related questions

In 2028, will AI be at least as big a political issue as abortion?
ScottAlexander avatarScott Alexander
34% chance
Before 2028, will any prediction market come up with a robust way to run a market on AI extinction risk?
IsaacKing avatarIsaac
33% chance
Will AI be a major topic during the 2024 presidential debates in the United States?
MatthewBarnett avatarMatthew Barnett
29% chance
Will Bostrom's "Superintelligence" exceed its current popularity peak before 2028?
Metastable avatarMetastable
20% chance
Will AI pass the Longbets version of the Turing test by the end of 2029?
dreev avatarDaniel Reeves
54% chance
Will Biden sign an executive order primarily focused on AI in 2023?
SG avatarS G
55% chance
Will an AI get gold on any International Math Olympiad by 2025?
Austin avatarAustin
30% chance
Will general purpose AI models beat average score of human players in Diplomacy by 2028?
Metastable avatarMetastable
75% chance
Will Tyler Cowen agree that an 'actual mathematical model' for AI X-Risk has been developed by October 15, 2023?
JoeBrenton avatarJoe Brenton
9% chance
Will AI outcompete best humans in competitive programming before the end of 2023?
Will >$100M be invested in dedicated AI Alignment organizations in the next year as more people become aware of the risk we are facing by letting AI capabilities run ahead of safety?
BionicD0LPH1N avatarBionic
81% chance
Will there have been a noticeable sector-wide economic effect from a new AI technology by the end of 2023?
Nostradamnedus avatarNostradamnedus
13% chance
Will anyone very famous claim to have made an important life decision because an AI suggested it by the end of 2023?
IsaacKing avatarIsaac
22% chance
By end of 2028, will AI be considered a bigger x risk than climate change by the general US population?
NathanNguyen avatarNathan Nguyen
50% chance
🐕 Will A.I. Be Able to Make Significantly Better, "Common Sense Judgements About What Happens Next," by End of 2023?
PatrickDelaney avatarPatrick Delaney
41% chance
Will it be public knowledge by EOY 2025 that a major AI lab believed to have created AGI internally before October 2023?
dmayhem93 avatardmayhem93
25% chance
Will AI be a Time Person of the Year in 2023?
Will Biden sign an executive order primarily focused on AI through Nov 2023?
StrayClimb avatarCalvinball
30% chance
Will Science's Top Breakthrough of the Year in 2023 be AI-related?
dp avatardp
40% chance
Will AI be a Time Person of the Year in 2023?
Sort by:
PatrickDelaney avatar
Patrick Delaney