Will the next major LLM by OpenAI use a new tokenizer?
44
1kṀ1273
Dec 31
77%
chance

  1. The GPT-2 model used r50k_base: vocab size = 50k

  2. The GPT-3 model used r50k_base: vocab size = 50k

  3. The GPT-3.5 model used cl100k_base: vocab size = 100k

  4. The GPT-4 model used cl100k_base: vocab size = 100k

AI
Get
Ṁ1,000
to start trading!

People are also trading

Will OpenAI release a tokenizer with more than 210000 tokens before 2026?
24% chance
Will OpenAI give their new LLM an anthropomorphic name?
10% chance
Will OpenAI release a tokenizer with vocab size > 150k by end of 2024?
42% chance
Will OpenAI's next major LLM (after GPT-4) feature natural and convenient speech-to-speech capabilities?
80% chance
Will OpenAI's next major LLM release support video input?
48% chance
Will the next LLM released by OpenAI be worse than GPT-4 at MMLU?
16% chance
Will a flagship (>60T training bytes) open-weights LLM from Meta which doesn't use a tokenizer be released in 2025?
29% chance
Will OpenAI's next major LLM (after GPT-4) solve more than 2 of the first 5 new Project Euler problems?
45% chance
Will OpenAI's next major LLM (after GPT-4) achieve over 50% resolution rate on the SWE-bench benchmark?
79% chance
OpenAI to release model weights by EOY?
88% chance
Sort by:

4o uses a different tokenizer (eg "gumdrop")

https://platform.openai.com/tokenizer

Is 4o "major"?

bought Ṁ50 YES

What if there are significantly more new tokens, e.g. representing images or audio, but the tokens representing text are pretty much unchanged?

@firstuserhere So YES if there's a GPT-4.5/5 that uses a tokeniser not on this list, and NO if there's a GPT-4.5/5 that uses a tokeniser that is on this list?

@chrisjbillington correct

Do you consider GPT-4-turbo to be a new iteration? What do you quantify as "next major LLM"

@GiftedGummyBee Bumping

@oh No, GPT-4 turbo is part of the same family, does not qualify as the next major LLM release

People are also trading

Will OpenAI release a tokenizer with more than 210000 tokens before 2026?
24% chance
Will OpenAI give their new LLM an anthropomorphic name?
10% chance
Will OpenAI release a tokenizer with vocab size > 150k by end of 2024?
42% chance
Will OpenAI's next major LLM (after GPT-4) feature natural and convenient speech-to-speech capabilities?
80% chance
Will OpenAI's next major LLM release support video input?
48% chance
Will the next LLM released by OpenAI be worse than GPT-4 at MMLU?
16% chance
Will a flagship (>60T training bytes) open-weights LLM from Meta which doesn't use a tokenizer be released in 2025?
29% chance
Will OpenAI's next major LLM (after GPT-4) solve more than 2 of the first 5 new Project Euler problems?
45% chance
Will OpenAI's next major LLM (after GPT-4) achieve over 50% resolution rate on the SWE-bench benchmark?
79% chance
OpenAI to release model weights by EOY?
88% chance
© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules