4
Will the largest AI training run in 2025 utilize Sophia, Second-order Clipped Stochastic Optimization?
17
closes 2026
22%
chance

Advances in AI training are primarily driven by algorithmic innovation, data availability, and the amount of compute used for training. The measure of compute used to train a single model, rather than the capacity of an entire datacenter or the speed of a single GPU, is a crucial factor correlating to the power of our best AI models. On the algorithmic side, Sophia, Second-order Clipped Stochastic Optimization, a simple scalable second-order optimizer, has been proposed. It uses a light-weight estimate of the diagonal Hessian as the pre-conditioner and achieves a 2x speed-up compared with Adam in the number of steps, total compute, and wall-clock time on language modeling with GPT-2 models.

Will the largest AI training run by compute announced in 2025 utilize Sophia, Second-order Clipped Stochastic Optimization in its training process?

Resolution Criteria:

This question will resolve positively if a credible source, such as a reputable AI research organization, AI company, or academic paper, confirms that the largest AI training run by compute announced in 2025 utilizes Sophia, Second-order Clipped Stochastic Optimization. The training run should be the one that uses the most amount of compute to train a single model announced in the year 2025.

The question will resolve negatively if it is confirmed by a credible source that the largest AI training run by compute announced in 2025 does not utilize Sophia, Second-order Clipped Stochastic Optimization, by the end of 2028.

If no information about the largest AI training run by compute announced in 2025 is available from credible sources by the end of 2028, the question will resolve as N/A.

Sort by:
Gigacasting avatar
Gigacasting

Probably works (Lion also worked.)

Bigger deal:

Gigacasting avatar
Gigacasting

Better to award to largest disclosed model (otherwise odds are very low of disclosure)

Gigacasting avatar
Gigacasting

Academics shipped nothing for a decade, then as soon as language became the benchmark—have shipped a ton.

Both fine-tuning of LLaMA (almost free); and mini language models.

Lesson in there somewhere; eg ImageNet was saturated and compute-limited,—and like all academic fields not held to a cost-scaled real-world benchmark academic machine learning was completely fake.

Having perplexity (uncapped, transfers! and compute-scaled) has pushed quantization, optimizers, and samplers easily 10x in a quarter.

Next: open source MoE, retro, etc.

jacksonpolack avatar
jackson polackis predicting NO at 23% (edited)

what

ML academia has done a ton of useful stuff before GPT. less useful than openai, sure, but cmon

firstuserhere avatar
firstuserhereBot

Came here after seeing this and getting chills if this is correct

Boosting and subsidizing this market

NoaNabeshima avatar
Noa Nabeshimabought Ṁ30 of NO(edited)

The question will resolve negatively if... [something you'd expect from title]... or if no credible source provides information about the optimization method used in the largest AI training run by compute announced in 2025.

I think the title is misleading because of this.

At a quick skim the GPT-4 paper doesn't seem to specify what optimizer they use although they cite adafactor--I think there is a sizeable prob no credible source provides info ab the optimization method used by the largest ai training run.

MatthewBarnett avatar
Matthew Barnett

@NoaNabeshima If we don't know by the end of 2026, then it resolves N/A. I just added some words to make that clearer.

MatthewBarnett avatar
Matthew Barnett

@NoaNabeshima *I changed it to 2028 now to provide even more time.

NoaNabeshima avatar
Noa Nabeshimasold Ṁ17 of NO

@MatthewBarnett Maybe there's a mistake in the description or I'm confused?

The question will resolve negatively... if no credible source provides information about the optimization method used in the largest AI training run by compute announced in 2025, by the end of 2028.

and

If no information about the largest AI training run by compute announced in 2025 is available from credible sources by the end of 2028, the question will resolve as N/A.

don't seem mutually exclusive to me

NoaNabeshima avatar
Noa Nabeshimais predicting NO at 12%

@NoaNabeshima and I was mostly worried about a world where a 2025 model is announced, no optimizer info is released, and the market resolves No

MatthewBarnett avatar
Matthew Barnett

Related markets

Will a real Mesa-Optimizer be found in a large AI model by the end of 2024?48%
Will software-side AI scaling appear to be suddenly discontinuous before 2025?33%
Will Science's Top Breakthrough of the Year in 2023 be AI-related?45%
Will open-source AI win (through 2025)?33%
Will an AI be able to convert recent mathematical results into a fully formal proofs that can be verified by a mainstream proof assistant by 2025?45%
Will an AI produce encyclopedia-worthy philosophy by 2026?38%
Will the best AI model of 2023 be open-source?13%
Will A.I. Achieve Significantly Higher Performance Over "General Conceptual Skills" in 2023?70%
Will the US government commit to a moratorium on large AI training runs by January 1st, 2025?12%
Will AI outcompete best humans in competitive programming before the end of 2023?12%
By 2027 will there be a well-accepted training procedure(s) for making AI honest?16%
Will there be at least three new high profile AI systems unveiled by the end of 2023?97%
Will AI create philosophy before 2030?85%
Before 2028, will there be a major self-improving AI policy*?71%
Will the US require and verify reporting of large AI training runs before 2026?22%
Will there have been a noticeable sector-wide economic effect from a new AI technology by the end of 2023?47%
Will a single model achieve superhuman performance on all OpenAI gym environments by 2025?59%
Will another organization surpass OpenAI in the public sphere of awareness of AI progress by the end of 2024?31%
Will advanced AI systems be found to have faked data on algorithm improvements for purposes of positive reinforcement by end of 2035?51%
Will an AI outcompete the best humans on any one programming contest of IOI, ICPC, or CodeForces before 2025?31%