What will the highest elo of an open-source model on chatbot arena be at the end of 2024?
💎
Premium
28
Ṁ19k
Dec 31
1,284
expected

Resolves according the the LMSys leaderboard.

If the elo is above 1700, resolves at 1700; if below 1200, resolves at 1200.

As of the time of writing (May 24th), the highest elo is GPT-4o-2024-05-13 (at 1287), and the highest open-source elo is Llama-3-70b-Instruct (at 1203).

  • Update 2024-14-12 (PST) (AI summary of creator comment): A model is considered open source if a random member of the public can:

    • Legally download the weights

    • Publish outputs publicly

Get
Ṁ1,000
and
S3.00
Sort by:

@Tetraspace What do you consider "open source" for the purpose of this market? The Nexusflow Athene-V2 license is only for personal use. The Llama 3 licenses are much more permissive, but don't quite meet the OSI criteria because of the 700m user cap, even if we exclude OSI's "source code availability" requirement.

@yetforever The ChatGPT share above is o1-mini, but here's an o1-preview response I generated through the API:

No, the LLAMA 3.3 Community License Agreement you've provided is not an open source license as defined by the Open Source Initiative (OSI).

The OSI's Open Source Definition (OSD) outlines specific criteria that a license must meet to be considered open source. Upon analyzing the terms of the LLAMA 3.3 Community License Agreement against the OSD, several clauses in this license conflict with these criteria. Here are the key reasons why it does not qualify:

  1. Discrimination Against Persons or Groups (OSD Clause 5):

    • Section 2: Additional Commercial Terms imposes a restriction based on the size of the Licensee. If a Licensee or its affiliates have more than 700 million monthly active users (MAUs) as of the Llama 3.3 version release date, they must request a license from Meta, which may be granted at Meta's sole discretion. Until such a license is granted, they are not authorized to exercise any rights under the Agreement.

    • This means that large entities (those exceeding 700 million MAUs) are treated differently and are subject to additional hurdles and discretionary approval. The OSD Clause 5 explicitly states that licenses must not discriminate against any person or group of persons. By imposing special conditions on larger organizations, the license discriminates based on the size of the user base, which is a form of discrimination against a group.

  2. Discrimination Against Fields of Endeavor (OSD Clause 6):

    • Section 1.b.iv: The license requires compliance with an Acceptable Use Policy (AUP), which is incorporated by reference. Although the content of the AUP isn't provided within the license text, such policies often include usage restrictions that could limit how the software can be used.

    • The OSD Clause 6 states that the license must not restrict anyone from making use of the program in a specific field of endeavor. If the AUP includes prohibitions against certain types of applications or uses (e.g., military, commercial, or competitive uses), it would violate this clause.

  3. Restrictions on Redistribution (OSD Clauses 1 and 7):

    • Section 2 effectively prevents certain parties from redistributing the software without Meta's express permission. Since large entities are not authorized to exercise any rights, including redistribution, without obtaining a separate license, this violates:

      • OSD Clause 1: which requires free redistribution without restrictions.

      • OSD Clause 7: which states that the rights attached to the program must apply to all to whom the program is redistributed, without the need for additional licenses.

  4. Trademark and Branding Requirements:

    • Section 1.b.i and Section 5.a impose requirements on how derivatives are named and how the "Llama" mark is used. Specifically, any AI model derived from the Llama Materials must include "Llama" at the beginning of any such AI model name and comply with Meta's brand guidelines.

    • While the OSD allows for reasonable attribution requirements, mandating the inclusion of a specific trademark in derivative works' names and imposing brand guideline compliance can be seen as restrictive. It may conflict with OSD Clause 3, which allows for modifications and derived works under the same license without such naming constraints.

  5. Termination Upon Litigation (Patent Retaliation Clause):

    • Section 5.c states that if you institute litigation alleging that the Llama Materials infringe on your intellectual property, your license terminates immediately.

    • While patent retaliation clauses are present in some open source licenses (like GPLv3), the OSD accepts them only if they are reasonable and don't broadly discourage the assertion of legitimate intellectual property rights. However, automatic termination clauses can conflict with OSD Clause 7, which requires that the license must not be contingent on executing an additional license.

  6. Indemnification Requirements:

    • Section 5.c also requires Licensees to indemnify Meta against third-party claims arising from the use or distribution of the Llama Materials.

    • The OSD does not explicitly address indemnification clauses, but imposing such obligations can create barriers to use and distribution, conflicting with the spirit of open-source principles.

  7. Reference to External Documents Affecting Rights:

    • Incorporating external documents like the Acceptable Use Policy by reference, especially when they can change over time, introduces uncertainty about the rights and obligations under the license.

    • The OSD requires that the license must be clear about the rights granted, and not subject to hidden or changeable terms.

Conclusion:

Due to these restrictions and conditions, particularly the discrimination against large entities and potential restrictions imposed by the Acceptable Use Policy, the LLAMA 3.3 Community License Agreement does not comply with the OSI's Open Source Definition. Therefore, it is not an open source license as OSI defines it.

Recommendation:

If you are seeking to use or contribute to software under an open source license, you should look for licenses that are OSI-approved, such as the MIT License, Apache License 2.0, or GNU General Public License (GPL), which fully comply with the Open Source Definition.

@Tetraspace ping^ What counts?

@Bayesian it's open source if I in my role as a random public person can download the weights legally and publish outputs on my twitter

bought Ṁ75 YES

Highest rated open source model as of 19th November 2024 is Llama-3.1-Nemotron-70B-Instruct, at 1269

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules