Using characters instead of tokens because token size can be changed, and characters are what humans actually care about. If they advertise a context window in tokens, I'll convert it to characters at the average rate of that tokenizer on representative human text.
Let's keep in mind that LLM's are not purely generative in nature and don't have to be based upon a GPT or any pre-determined architecture, they are merely a probability distribution over sequences of words. So as written, this question has a wide interpretation. I would almost advocate for narrowing down the definition further to make it more interesting.
@PatrickDelaney It should probably be "equivalent to GPT-3 on some benchmarks", otherwise a random tree search or markov chain would qualify. (Well, a "large" markov chain)
Maybe the "Evals" repo that was introduced with GPT-4 would be a good one? openai/evals: Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks. (github.com)
I'm advancing the idea which of course helps my YES bet that this must include private LLM's not publicly disclosed LLM's, so if there is a leak or a news report about any kind of LLM with said context window, it qualifies.
Not with transformers since it scales quadratically but I'm sure somebody will train a test model using e.g. hyena operators just to test its limits.
No, because it would be totally superfluous... No reason to waste compute like that
@IsaacKing would be much easier to use some kind of external knowledge/state store rather than a massive context window
@jonsimon No reason? How do you know that a-priori, do you know every industry? Do you know every possible architecture that might come out in 6 years? 1 million characters is 600 to 800 pages. I could imagine existing some odd esoteric application for that. If it was 100,000 or 1 million pages, increasingly less likely. But what about...government/intelligence summarization?
@IsaacKing I don't want to argue too harshly because this is all metaphors we're talking about but...humans do not have an unlimited context window analogous to an autoregressive GPT's context window...right? You would have to be able to, "remember," e.g. "tokenize," every conversation you ever had in detail, e.g. including every word to fulfill the condition, "*unlimited* context window." I think you might mean...humans can have a context window that stretches back selectively for years if not decades...something like that? To me, "unlimited," means just a massive billions or trillions of words long corpus including everything one ever heard, read, wrote or spoke. I hardly can remember what I did 10 minutes ago.
Relevant: "GPT-4 is capable of handling over 25,000 words of text, allowing for use cases like long form content creation, extended conversations, and document search and analysis."
