non-GDM companies have been surprisingly slow to adopt long context. I think it will happen soon-ish but it's not a top priority.

Wild that 4k is still at 3%.

Google Gemini is 2 million tokens already, with tests up to 10 million. If OpenAI competes with Google, I might have needed more levels on my scale...

It resolves 100% 4096k(4 million) no matter how larger the final context is, since I can't add options.

@Gen Is it possible to admin add levels to this? 8192k and 16384k would be nice to have. Anything larger would be basically infinite. The context sizes grew faster than I expected.

There's now a standard test suite for the kind of recall test I was thinking of doing: How Long Can Open-Source LLMs Truly Promise on Context Length? | LMSYS Org

Leaving it as a comment here so I can remember to find it again in 2 years, if it's needed.

This should be a higher/lower market with a log scale IMO

@ShadowyZephyr This is an intentional choice because it allows higher leverage if you have a strong opinion on a narrow numerical range.

@Mira I heard from other people that the math of the multi-choice markets is not good for compensating people who make correct bets early on.

@ShadowyZephyr I would disagree with them, but I usually don't debate people. You can bet on @firstuserhere 's binary markets if you like, since I stole his market idea for this.

By testing recall, how would that work for an RNN, having theoretically infinite context

@dmayhem93 It would resolve to the largest entry if it can pass the test at any size without errors, in a single API call.

If it has an increasing error rate like RNNs often do, I'll resolve to the highest size I get at least 50% successful recall.

It will be a simple "locate a matching entry" task, so even if its performance degrades for more complex reasoning it's likely to be able to pass as having a high context size.