I was browsing Twitter, and I saw a post by Karpathy positively talking about ChatBot Arena, which is a platform for ranking LLMs based on human ratings. As expected, OpenAI is holding positions 1, 2, and 3. I wonder which company will be #1 at the end of 2024.
Screenshot of the rankings table taken on the 13th of December:
Related questions
@traders Based on the comments below, I think it makes sense to resolve this question based on the ELO rating in case of a tie in "rank." When I created this question, a tie was not an option, so I doubt anyone even traded based on this assumption.
I created a similar question that only uses the rank. Feel free to trade on it.
Lmsys keeps throwing curveballs at prediction markets!
Can we have a clarification whether the rank is important (as is literally stated_) or ELO is important?
@WillSorenson hahaha, I never expected this to happen - I hope that at the end of the year, only one company will be in spot #1, but if this is not the case, then we can either resolve each winning option to a %, or we can rely on the ELO scores. What do you think makes more sense?
@Soli I am so surprised they're calling them all #1 (I guess because of confidence intervals, right?) I think quite clearly ELO should be what counts for this question. Feel like I can say this with a clean conscience as I think my positions just aren't that big.
@Soli Confidence intervals exist for a reason. They are there to tell us that there is a meaningful or significant difference between the two ELO scores. So to ignore that and only focus on ELO is wrong as you are leaving it up to chance.
The OP says "best LLM" (not highest ELO) and asks which will be rank #1. Both of these specifically reference the ranking. IMO it's clear in that respect.
@AJama I see where you are coming from. I have one more question: Does chess also have a confidence interval in the official rankings?
@Soli FIDE ranking doesn't. But we can only assume they do it like that for simplicity or historical reasons. Confidence intervals are more objective, and we ought to take the LLM rankings as is imo. If they use confidence intervals to determine the #1 rank, that's how they do it, right?
Hence "as decided by ChatBot Arena"
@Soli I think the way you've worded it initially (both #1, and as decided) means it should probably be resolved as a % if this happens.
Funnily enough, this much less precise wording ended up getting at the essence of the question better! https://manifold.markets/HankyUSA/who-will-own-the-model-at-the-top-o
@Soli I'm fine with many interpretations here given that we've still got a long time until this resolves. My slight inclination is to resolve to highest elo even if the CIs overlap with others.
@nsokolsky That was cheating. Bard is more than just a LLM.
if we allow agents on the board, GPT4+bing search can easily go to 1350 range
@Sss19971997 The resolution criteria only talk about the Chatbot Arena ranking. It should resolve to Google even if someone literally bribes Chatbot Arena to put Google on the first spot.
@nsokolsky google+search counts although i do find it a bit weird but bugs, obvious cheating, bribes won’t count.
@Soli hm... then you need to update the market description, as well as an explanation of what happens if ChatBot Arena shuts down.
@Sss19971997 Realistically may not be much of a problem. OpenAI will probably include a version that can search like ChatGPT at some point.
Also it more realistically reflects what using these LLMs are like in the wild.
@Sss19971997 Idk, I put my predictions based off the text of the question. You want to define it more narrowly go ahead - but I always bet on the exact words in the prediction market.
@Sss19971997 I also hate that Chatbot arena pits search-enabled vs pure LLMs against eachother but I agree with @nsokolsky that unless we're being very pedantic, the Bard model would count under the current wording.
@WillSorenson Created a new market here for the no-tool version:
https://manifold.markets/WillSorenson/who-will-have-the-best-nontool-enab