This market is centered on whether an open source Large Language Model (LLM) will achieve a higher Elo rating than OpenAI’s GPT-4 on the Chatbot Arena platform by the end of 2024.

Chatbot Arena utilizes a crowdsourced, randomized battle platform where user votes contribute to computing Elo ratings. This specific market will be resolved based on the Elo ratings of the models as reported by Chatbot Arena.
The link to the website: https://chat.lmsys.org/

The resolution will consider the highest Elo rating recorded for an open source LLM compared to GPT-4’s Elo rating as of December 31, 2024.

It’s important to note that only the Elo ratings will be used for determining the outcome, not considering other benchmarks like MT-Bench or MMLU scores.

The latest update of the leaderboard by December 20, 2024, will be used for the final assessment.

• Update 2024-12-12 (PST) (AI summary of creator comment): - The market will consider any GPT-4 model variant present on the Chatbot Arena leaderboard, not just a specific version

  • An open source model needs to surpass any of the GPT-4 models listed on the leaderboard to resolve as YES

• Update 2024-12-12 (PST) (AI summary of creator comment): - For a YES resolution, an open source model must surpass any of the GPT-4 model variants listed on the leaderboard

  • Multiple GPT-4 model variants may be present on the leaderboard, and all will be considered for comparison