Getting pretty close to resolution for the before 2025 one without the criteria being clear. Does something like allenai/OLMoE-1B-7B-0924 which has a fully open training set and intermediate checkpoints count? I'd guess ~no, it's not good enough.

@Lun we should na this question

@mods can someone help pls?

.

@weissz interesting, is this also ranked on Chatbot Arena?

@weissz and apologies for the late reply, I somehow missed your comment

@Soli yea, why is not Olmo fully open-source?

1. Context text embedder ≠ LLM.

2. Mistral is open source. The training data isn't, but it's neither practical nor necessary to publish the whole dataset, considering that it's huge and likely has some copyrighted works in it.

3. Really strange to open a market first and come up with resolution criteria later. Whatever you come up with will upset people who made their bets having to use their own assumptions.

@ProjectVictory i think you could have saved both of us time by not making this comment :)