When OpenAI announces GPT-5, this market resolves to the largest context size measured in tokens that they announce support for.
GPT-3: 2048 tokens
GPT-3.5: 4096
GPT-4: 8k, 32k
GPT-5: ???
Anthropic's Claude announced a 100k variant, there are rumors of upcoming 1 million context size models, and surely OpenAI would want the most impressive-sounding model on release.
In the unexpected case they don't mention a specific context size or their architecture is changed so fixed context sizes no longer make sense, I'll wait until I have access and test its recall using very large documents.
If the largest context size isn't on this table, then this market resolves to a weighting of the surrounding entries. k is a multiplier of size 1024. GPT-4 would resolve "32k". Claude would resolve "log2(100k) = 16.61, so 2^16 = 64k would get weight 39% and 2^17 = 128k would get weight 61%".
1,000
3.00Google Gemini is 2 million tokens already, with tests up to 10 million. If OpenAI competes with Google, I might have needed more levels on my scale...
It resolves 100% 4096k(4 million) no matter how larger the final context is, since I can't add options.
@Gen Is it possible to admin add levels to this? 8192k and 16384k would be nice to have. Anything larger would be basically infinite. The context sizes grew faster than I expected.
There's now a standard test suite for the kind of recall test I was thinking of doing: How Long Can Open-Source LLMs Truly Promise on Context Length? | LMSYS Org
Leaving it as a comment here so I can remember to find it again in 2 years, if it's needed.
@ShadowyZephyr This is an intentional choice because it allows higher leverage if you have a strong opinion on a narrow numerical range.
@Mira I heard from other people that the math of the multi-choice markets is not good for compensating people who make correct bets early on.
@ShadowyZephyr I would disagree with them, but I usually don't debate people. You can bet on @firstuserhere 's binary markets if you like, since I stole his market idea for this.
@dmayhem93 It would resolve to the largest entry if it can pass the test at any size without errors, in a single API call.
If it has an increasing error rate like RNNs often do, I'll resolve to the highest size I get at least 50% successful recall.
It will be a simple "locate a matching entry" task, so even if its performance degrades for more complex reasoning it's likely to be able to pass as having a high context size.