Will an LLM not worse than GPT Davinci cost ten times less by 2024?
18
330Ṁ1733
resolved Apr 30
Resolved
YES

Will there be a publicly available completion API based on an LLM comparable with text-davinci-003, but costing ten times less than it is now?

Current cost for text-davinci-003 is: $0.0200 /  1K tokens: “You can think of tokens as pieces of words, where 1,000 tokens is about 750 words. This paragraph is 35 tokens.”[1]:

[1]: https://openai.com/api/pricing/

Being comparable means the following benchmark results, with example numbers for GPT-3 Davinci, and GPT-J which is one of the best downloadable, open-source LLMs[2]:

  • LAMBADA ppl <= 3.0 (Davinci: 3.00, GPT-J: 3.99)

  • LAMBADA acc >= 72% (Davinci: 75%, GPT-J: 69.7%)

  • Winogrande >= 70% (Davinci: 72%, GPT-J: 65.3%)

  • Hellaswag >= 75% (Davinci: 78%, GPT-J: 66.1%)

no general consensus that this new model, while performing well on the benchmarks, is significantly worse than text-davinci-003 for most tasks

[2]: https://huggingface.co/EleutherAI/gpt-j-6B

If text-davinci-003 itself costs less than $0.002 per 1K tokens on midnight, 1-1-2024, this is also a YES.

Get
Ṁ1,000
to start trading!

🏅 Top traders

#NameTotal profit
1Ṁ173
2Ṁ56
3Ṁ36
4Ṁ22
5Ṁ9
Sort by:
predictedYES

I think this can be resolved now

predictedYES

gpt-4's pricing is: Pricing is $0.06 per 1K prompt tokens and $0.12 per 1k completion tokens., doesn't fit

but I'm still waiting for the gpt3.5 benchmarks

predictedYES

@ValentinGolev "Pricing is $0.03 per 1k prompt tokens and $0.06 per 1k completion tokens."

predictedYES

@firstuserhere (for the earliest release)

predictedYES

@firstuserhere my quite is from here https://openai.com/research/gpt-4 can you show me your source?

but anyway doesn't fit

predictedYES

Actually I'm not sure if gpt-3.5-turbo fulfills my criteria, but I expect it does. I've closed the market for now, and I'll investigate a bit more before closing it

predictedYES

from today's OpenAI newsletter:

"The new Chat API calls gpt-3.5-turbo, ... It’s priced at $0.002 per 1K tokens, which is 10x cheaper than the existing GPT-3.5 models."

I'm really... it was finger in the sky how exact I managed to be in my prediction?..

© Manifold Markets, Inc.TermsPrivacy