Criteria for Resolution:
1. Definition of "New Lab":
- A "new lab" refers to any company, lab, or other entity that is not OpenAI, Anthropic, DeepMind, Google, Meta, Mixtral, xAI, Microsoft, Nvidia or any subsidiary or parent company of them.
2. Top-Performing Generally Capable AI Frontier Model:
- The AI frontier model must achieve no less than a robust second place by performance. This includes:
- Unambiguous first place.
- Unambiguous second place.
- Ambiguous first place.
- Sharing first place.
- Sharing second place does not qualify.
3. Performance Metrics:
- Performance will be judged based on the most well-accepted metrics and user opinions and approvals available by then.
- For example, metrics may include benchmarks such as MMLU, HumanEval, and other relevant AI performance benchmarks.
Related questions
The compute cost of training a cutting edge model is in the hundreds of millions currently. Epoch estimates that it's going to continue to go up by 0.2 OOM each year.
That's without accounting for the human capital costs. Training a cutting edge model is going to require a bunch of engineering schlep, which means hiring some world class people.
You need to have both deep pockets and a strong motivation to start an AI lab for this to make sense. So maybe a national govenrment?
It must be considered a general purpose model with general capabilities. A video generation model can in principle be in this class. If there is a capable video generation model that can be applied for various tasks and it demonstrates strong intelligence capabilities, it will qualify. If, for example, it is just the best model in the category of the most aesthetically beautiful short videos generators or the best advertisement producers, it will not qualify.