5
Will A.I. Be Able to Make Significantly Better, "Common Sense Judgements About What Happens Next," by the End of 2023?
87
closes 2024
53%
chance

Preface / Inspiration: There are a lot of questions on Manifold about whether or not we'll see sentience, general A.I., and a lot of other nonsense and faith-based questions which rely on the market maker's interpretation and often close at some far distant point in the future when a lot of us will be dead. This is an effort to create meaningful bets on important A.I. questions which are referenced by a third party.

In short, here's the leaderboard:

Market Description:

Resolved by submissions at:

https://leaderboard.allenai.org/visualcomet/submissions/public

Visual Comet

This leaderboard collects evaluations of current AI systems on Visual commonsense tasks that measure both the knowledge that these systems possess as well as their ability to reason with and use that knowledge in context of an event in an image.

Even from a single frame of a still image, people can reason about the dynamic story of the image before, after, and beyond the frame. For example, given an image of a man struggling to stay afloat in water, we can reason that the man fell into the water sometime in the past, the intent of that man at the moment is to stay alive, and he will need help in the near future or else he will get washed away.

An example input question contains the following fields in JSON format:

  {
     "img_fn": "lsmdc_3005_ABRAHAM_LINCOLN_VAMPIRE_HUNTER/3005_ABRAHAM_LINCOLN_VAMPIRE_HUNTER_00.27.43.141-00.27.45.534@0.jpg",
     "movie": "3005_ABRAHAM_LINCOLN_VAMPIRE_HUNTER",
     "metadata_fn": "lsmdc_3005_ABRAHAM_LINCOLN_VAMPIRE_HUNTER/3005_ABRAHAM_LINCOLN_VAMPIRE_HUNTER_00.27.43.141-00.27.45.534@0.json",
     "place": "at a fancy party",
     "event": "1 is trying to talk to the pretty woman in front of him"
  }

Example sets of images (not necessarily related to the above)

Market Resolution Threshold:

  • Note, at the time of authoring this, human performance is 0.5

Top leftmost score, BLEU 1 is 0.3500 at time of authoring. We would need to see 1.3*0.3500 or >=0.4550 by the end of the year for this market to resolve as YES, otherwise this resolves NO.

Score to Beat -> 0.4550

Sort by:
PatrickDelaney avatar
Patrick Delaney

Example markets:

Related markets

Will A.I. Get Significantly Better at Evaluating Scientific Claims by the end of 2024?48%
Will A.I. Get Significantly Better At, "Community Based Ethical Judgements," by the End of 2023?77%
Will A.I. Achieve Significantly More, "Linguistic Temporal Understanding" by end of 2023?26%
Will A.I. Be Significantly Better, "Able to Track Changes in State," By the End of 2023?8%
Will A.I. Achieve Significantly Higher Performance Over "General Conceptual Skills" in 2023?70%
Will A.I. Be Able to Meet Just Below Human Performance In Being Able to "Track Changes in State," By the End of 2023?27%
Will AI Be Able to Understand the, "Meaning" of Questions Significantly Better By the End of 2023?62%
Will there have been a noticeable sector-wide economic effect from a new AI technology by the end of 2023?58%
Will A.I. Be Able to, "Feel and React to Pain," Significantly Better By the End of 2024?34%
Will A.I. Be Significantly Better at, "Egocentric Navigation," by the End of 2023?41%
Will A.I. Become Significantly Better at Drug Discovery in 2023?52%
Will A.I., "Hallucinate Significantly Less," by the End of 2023?50%
Will A.I. Get, "Scarily Better at Helping To Resolve Shipping Supply Chain Issues," by the End of 2024? (Re:HuggingFace)64%
Will anyone very famous claim to have made an important life decision because an AI suggested it by the end of 2023?48%
Will AI xrisk seem to be handled seriously by the end of 2026?15%
Will an AI fully convince me that it is sentient before 2028?59%
Will AI surpass human intellect by 2030?96%
Will an AI play a pivotal role in solving an important political issue by 2033?67%
Which Institution Will Have Shown To Have Been the Most Prescient In Terms of the Direction of A.I. by the end of 2023?
Will AI outcompete best humans in competitive programming before the end of 2023?18%