🏅 Top traders

Which LLM do you think is as good as GPT3.5? How did you evaluate it?

So which is the OS LLM in question? Or was it just closed to close it without serious thought?

@PeterWildeford Are you going to resolve this market? And if no, do you want to propose an alternate resolution system?

Delegate this to trustworthy users?

@bohaska two comments down he says he’s still going to resolve all his open markets

Peter's most recent comment says that he's quitting Manifold. Most likely mods will just N/A and return the funds, but it's something to keep in mind if you want to bet.

@tfae I still commit to resolve all my open questions

You gotta be careful with the benchmark though. I think the most commonly used version of GPT-3.5 (gpt-3.5-turbo) today is likely worse than text-davinci-003 in terms of peak performance, but usually preferred because it's a lot smaller/faster/cheaper.

@PeterWildeford Have you tried any of the open-source models recently? The software LM Studio is getting pretty popular and you can try pretty much any of the open source models

Will you take LLaMA2 as open source? It's technically not, but for most people it's as if it were (unless you're a business with 700 million users as of last month)

@firstuserhere If I can personally access the weights and it's not due to some unusual characteristic of me, then yes

@PeterWildeford yes you can personally access the weights but they do ask you to fill out a form prior to that. The form asks for just the name afair. It took me 15 seconds to fill the form and i had weights within a minute after that

@firstuserhere Sounds open source enough to me

https://chat.lmsys.org/?leaderboard seems potentially relevant

I made a GPT4 version: https://manifold.markets/PeterWildeford/will-i-peter-wildeford-think-that-t-c95ff3c1b385

Will I (Peter Wildeford) think that there is an open source LLM as good as GPT4 by EOY 2023?

53% chance. I previously asked about GPT3.5: https://manifold.markets/PeterWildeford/will-i-peter-wildeford-think-that-t [link preview]In this question, I will use my subjective judgement to decide if there has been an open source LLM as good as GPT4 by the end of this year. Grading the quality of…

Do you mean an RLHF or SFT model like ChatGPT or an inframodel?

@ampdot any of them

@PeterWildeford Guanaco-65b & Guanaco-33b have already beat ChatGPT on benchmarks, you might want to try those

@ShadowyZephyr Do you have a link?

@PeterWildeford Huggingface: https://huggingface.co/timdettmers/guanaco-65b

https://huggingface.co/timdettmers/guanaco-33b

Demo of 33B (65B is even better supposedly)
https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi

@PeterWildeford Just to be clear, if either an open source inframodel (base model) beats GPT-3.5 base or an open source tuned model beats ChatGPT, you will resolve this to YES?

@ampdot Correct

Though we should be careful to explain what "beats" means, and this will be subjective and come from a moderately skeptical approach.

https://tatsu-lab.github.io/alpaca_eval/

By “anyone can access it,” do you mean anyone can access the weights?

@Alana Presumably so, since that would make it open source.

@ShadowyZephyr @Alana yes