METR = formerly ARC Evals (https://metr.org/)

if METR/Google reorgs and has a clear successor org, that org also applies for the purpose of this market

central YES cases:

• if Google releases a model card with something like Gemini Supermega 2024 edition with METR exfiltration eval results, like OpenAI did for GPT4 technical report

does not have to be the specific exfiltration eval.

does not have to be included in initial model release paper. does not have to be specifically in a paper.

does not have to be any specific eval granularity. "METR ran the eval and it was all OK" would be ... annoyingly vague from whoever would write it, but it would count.

has to be confirmed-ish by Google and/or METR. can't be just a Twitter rumor.

• Update 2025-02-20 (PST) (AI summary of creator comment): Minimum Valid Evidence Update:

  • The archived report claude-3-5-sonnet-report qualifies as sufficient evidence for a YES resolution.

  • This document serves as a minimum threshold that confirms METR ran the evaluation and the results were in line with the market’s criteria.

This update supplements the current resolution criteria without replacing the original requirements.