an LLM as capable as GPT-4 will run on a 3090 before March 2024
23
196
430
resolved Mar 12
Resolved
NO

e.g. Winograde >= 87.5%

Get Ṁ200 play money

🏅 Top traders

#NameTotal profit
1Ṁ118
2Ṁ74
3Ṁ45
4Ṁ37
5Ṁ31
Sort by:
predicted NO

the initial question was specifically about winogrande, but MMLU would also be acceptable. fits on a single 3090. quantization could be allowed if quantized performance is comparable, but that might be annoying to evaluate

predicted NO

Bump. Is the evaluation just Winograde? MMLU? And by run on a 3090, do you mean a single one with any other computer specs?

predicted NO

Winograde? Do you mean the Winogrande benchmark? Like specifically this one? (Any requirement for the LLM in question to be open?)
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

By 'run' do you mean 'fits on '?