Will resolve to YES if Meta releases an open source model that acheives a higher average score than GPT-4 on the following benchmarks by the end of 2024:

HellaSwag (few-shot): 0.953

MMLU (few-shot): 0.864

AI2 Reasoning Challenge (ARC): 0.963

• Update 2025-03-01 (PST) (AI summary of creator comment): - Llama 3.1 405B achieves the following benchmark scores:

  • MMLU (zero-shot CoT): 0.886

  • ARC (zero-shot): 0.969

  • Hellaswag score is not reported due to potential contamination and is not considered in the resolution criteria.

  • The model is deemed open-source, and based on its performance and higher Elo on LMSYS, the market is resolved to YES.