Does participation require scores being officially certified by IMO or can the companies run the model internally and then claim a certain score? @Bayesian

@DottedCalculator the intent is to count real cases and not count fake cases. The big companies are unlikely to make it up so internally and claiming a certain score seems fine, if it’s a very niche non frontier lab I would be more suspicious and my bar for believing them woukd be higher. Id prefer not relying on my judgement but i want to count real successes even if they arent done through a preferred channel. May need to have a uninvolved mod panel resolve instead if ambiguous.

@Bayesian Would Anthropic resolve yes now?

@DottedCalculator yes i’ll count it

do we know the source for google being so high?

@DottedCalculator

they did it last year, they’re competitors with OpenAI who are certainly doing it, they have no reason not to compete