Relevant: AlgoPerf

The AlgoPerf: Training algorithms benchmark competition opens on November 28, 2023, and is scheduled to close on March 28, 2024. To enter the competition please see the instructions on the competition website. Additionally, the accompanying technical report motivates and explains the design choices of the benchmark.

Sponsorship & Prize Money

MLCommons is offering a total prize pool of $50,000, to be awarded by a committee, for the top-performing submissions in each tuning ruleset. 

We would also like to express our gratitude to Google for their generous support in providing computational resources to score the top submissions, and resources to help score promising submissions from submitters with more limited resources.

Jack Clark says:

With AlgoPerf, we might finally have a decent, principled  way to evaluate new optimizers and work out if they’re actually any good

I tried submitting to AlgoPerf.

A problem that I ran into is that only Google Cloud offers machines that match their requirements for benchmarking, and at a steep price, so I ended up not submitting

My Lilith will dethrone it :)

Adam was introduced in 2014: https://arxiv.org/abs/1412.6980

@NiplavYushtun Yup. And aside from minor variations it’s been the staple ever since. It seems to survive everything and works on new architectures that weren’t even invented in 2014, like transformers. Makes me think it’ll still be around for a couple more years at least.

How will we know whether SOTA models are using Adam if training details are not disclosed for the best models at the time?

@AdamK we'll infer the best we can. Atm, can also switch to the best open sourced models