Resolution Criteria:

A "closed-source" model is one that is not available for download (like today's GPT-4) An "open-source" model is one that is available for download (like today's Llama). Technically, yes, I'm talking about "open weight" and "closed weight" but these words are less well understood broadly. Specifically, let X be the date the current state-of-the-art open source model was released. Let Y be the date the earliest closed-source model was released that matched the capabilities of the open-source model. What is the difference between X and Y? (If there are multiple open-source models that only very marginally improve in capabilities, take the earliest release date of these.) The date of release for the closed-source model is the date that the model is first available to a broad audience (e.g., anyone in the US). The date of open-source release is the date that the model is available to be downloaded. "Matched" means that across a diverse range of benchmarks, the AI system achieves an average score at least as high as the closed-source model.

Motivation and Context:

At present, there seems to be a two year delay between the best closed-source model being released (GPT-4) and an open-weight model that reproduced its capabilities (Llama-3 405B). This 24-month delay has been fairly consistent since GPT-2 was first trained. Will open-source AI begin to fall further behind? Or will it catch up?

Question copied from: https://nicholas.carlini.com/writing/2024/forecasting-ai-future.html