On January 1, 2025, how many organizations will have released a chatbot which is ranked above (or equal to) GPT-4-0314 in the LMSYS Chatbot Arena Leaderboard? As of March 10, 2024, there are three such organizations: OpenAI, Anthropic, and Google.

The intent of this question is to measure how many companies will have OpenAI-level capacity for developing frontier AI systems (and thus how large of a moat OpenAI has). Thus, see below clarifications on how fine-tunes of a shared base model are treated.

Details on fine-tunes of a single base model: the organization counted for a model M will be the one which pretrained the base model from which M was fine-tuned. For example, if Meta releases a LLaMA-3 model which surpasses GPT-4-0314 on the leaderboard, and then Stanford makes a fine-tuned Vicuña-3 which also surpasses GPT-4-0314, then this would only count for one organization (Meta) and not for two (Meta + Stanford). If LLaMA-3 ranks below GPT-4-0314 but Vicuña-3 ranks above, then it counts as one organization, and Meta is that organization.

Two organizations count as the same organization iff they have the same listed name in the Chatbot Arena Leaderboard.

If GPT-4-0314 no longer appears on the leaderboard on January 1, 2025, then I'll replace it with another comparable GPT-4-series model, according to my judgement. If there are no GPT-4 series models on the leaderboard or the leaderboard no longer exists, then this market will resolve N/A.