Will the next major LLM by OpenAI use a new tokenizer?

MANIFOLD

Ṁ1kṀ2.4k

Dec 31

89%

chance

ALL

The GPT-2 model used r50k_base: vocab size = 50k
The GPT-3 model used r50k_base: vocab size = 50k
The GPT-3.5 model used cl100k_base: vocab size = 100k
The GPT-4 model used cl100k_base: vocab size = 100k

Market context

Get

1,000

to start trading!

People are also trading

Will OpenAI release another open source LLM before end of 2026?

70% chance

Will OpenAI, Deepmind, or Anthropic be the next to release a frontier LLM?

Will there be a state-of-the-art LLM that is NOT based on next raw token prediction before 2029?

55% chance

Will OpenAI announce AGI before 2028 conditional on it centrally being an LLM?

48% chance

Before 2029, will OpenAI provide API access to a frontier LLM with 100,000,000+ context length?

49% chance

How much time will pass between an LLM being released that beats GPT4 and the next OpenAI LLM being released? (+ANSWERS)

Will any widely used LLM be pre-trained with abstract synthetic data before 2030?

Sort by:

bought Ṁ1,000 YES

Uses o200k which is not on the list.

bought Ṁ23 YES

Regardless of which release is considered major for the purpose of the question, OpenAI have moved to a new tokenizer with 4o, and all of their big models released since (o1, o3, 4.5, 5) have also used it.

4o uses a different tokenizer (eg "gumdrop")

https://platform.openai.com/tokenizer

Is 4o "major"?

bought Ṁ50 YES

What if there are significantly more new tokens, e.g. representing images or audio, but the tokens representing text are pretty much unchanged?

@firstuserhere So YES if there's a GPT-4.5/5 that uses a tokeniser not on this list, and NO if there's a GPT-4.5/5 that uses a tokeniser that is on this list?