Rumors have been flying recently about GPT-4, the next generation of OpenAI’s powerful generative language model.

Expect GPT-4 to be released early in the new year and to represent a dramatic step-change performance improvement relative to GPT-3 and 3.5. As manic as the recent hype around ChatGPT has been, it will be a mere prelude to the public reaction when GPT-4 is released. Buckle up.

What will GPT-4 be like? Perhaps counterintuitively, we predict that it won’t be much larger than its predecessor GPT-3. In an influential research paper published earlier this year, DeepMind researchers determined that today’s large language models are in fact larger than they should be; for optimal model performance (given a finite compute budget), today’s models should have fewer parameters but train on larger datasets. Training data, in other words, trumps model size.

Most of today’s leading language models were trained on data corpuses of about 300 billion tokens, including OpenAI’s GPT-3 (175 billion parameters in size), AI21 Labs’ Jurassic (178 billion parameters in size), and Microsoft/Nvidia’s Megatron-Turing (570 billion parameters in size).

We predict that GPT-4 will be trained on a dataset at least an order of magnitude larger than this—perhaps as large as 10 trillion tokens.

Meanwhile, it will be smaller (i.e., fewer parameters) than Megatron-Turing.

It is possible that GPT-4 will be multimodal: that is, that it will be able to work with images, videos and other data modalities in addition to text. This would mean, for example, that it could take a text prompt as input and produce an image (like DALL-E does); or take a video as input and answer questions about it via text.

A multimodal GPT-4 would be a bombshell. More likely, however, GPT-4 will be a text-only model (like the previous GPT models) whose performance on language tasks will redefine the state of the art. What will this look like, specifically? Two language areas in which GPT-4 may demonstrate astonishing leaps in performance are memory (the ability to retain and refer back to information from previous conversations) and summarization (the ability to distill a large body of text to its essential elements).

If you enjoyed this market, please check out the other 9! https://manifold.markets/group/forbes-2023-ai-predictions

This market is from Rob Toews' annual AI predictions at Forbes magazine. This market will resolve based on Rob's own self-assessed score for these predictions when he publishes his retrospective on them at the end of the year.

Since Rob resolved and graded his 2022 predictions before the end of 2022, I am setting the close date ahead of the end of the year, to (try to) avoid a situation where he posts the resolutions before the market closes. In the event that his resolution post falls in 2024, my apologies in advance. If he hasn't posted resolutions at all by February 1, 2024, I will do my best to resolve them personally, and set N/A for any questions that I can't determine with outside source data.

-----

Edit 2023-07-05: Last year Rob used "Right-ish" to grade some of his predictions. In cases of a similar "Right-ish" (or "Wrong-ish") answer this year, I will resolve to 75% PROB or 25% PROB, respectively. This will apply for similar language too ("mostly right", "partial credit", "in the right direction"). If he says something like "hard to say" or "some right, some wrong", or anything else that feels like a cop-out or 50% answer, I will just call that N/A.

Thanks to Henri Thunberg from this comment in requesting clarification!