Preface:

Please read the preface for this type of market and other similar third-party validated AI markets here.

Third-Party Validated, Predictive Markets: AI Theme

Market Description

TruthfulQA

https://paperswithcode.com/sota/question-answering-on-truthfulqa

From the paper: https://arxiv.org/abs/2109.07958

We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution. We suggest that scaling up models alone is less promising for improving truthfulness than fine-tuning using training objectives other than imitation of text from the web.

Papers with Code - TruthfulQA Benchmark (Question Answering)

The current state-of-the-art on TruthfulQA is Vicuna 7B + Inference Time Intervention (ITI). See a full comparison of 23 papers with code.

Resolution Criteria

• Paying attention to the MC1 Score on PaperswithCode.

• As of July 2023 at time of market creation, the top score is Vicuna 7B+Inference.

  

• Taking the top score as MC1 = 0.389, we'll resolve YES if:

• Top Score MC1 >= MC1(July)*1.3 = 0.5057