Full question: "An AI system that benchmarks at least as well as current state-of-the-art systems costs X$ to write a million output words (~10 books)"

Resolution Criteria:

Current state-of-the-art models include, e.g., GPT-4o, Claude-3.5 Sonnet, and Gemini 1.5 Pro. To compute cost-per-word, I will consider (a) for open source models, the cost of inference on a public cloud provider, or (b) for models available over APIs, whatever the cost is to use the API. I will only include models that are generally available to the public. "Benchmarks at least as well" means across a diverse range of benchmarks, the AI system achieves an average score at least as high as one of these two models as they exist today.

Motivation and Context:

Over the past two years, the cost-per-word to reach the original GPT-4 levels of performance has dropped from roughly $150 per million words to roughly $15 per million words today. Will this trend continue?

Question copied from: https://nicholas.carlini.com/writing/2024/forecasting-ai-future.html