See https://astralcodexten.substack.com/p/mostly-skeptical-thoughts-on-the (note that I'm asking about the inverse of his 4th prediction). Resolution criteria from the post:
"The AI will fail this task if there’s any kind of post I write that it can’t imitate - for example analyzing scientific data, or writing fiction, or reviewing books. It will fail this task it it writes fluently but says false things (at a rate higher than I do), eg if it makes up references. It doesn’t have to be able to coordinate complex multistep projects like the Book Review Contest".
Related markets:
@MordecaiWeynberg You should ask Scott. Unless he's being blantantly dishonest or unreasonable, I'll resolve however he decides
@PhilipGoetz Isn’t that not the usual meaning? I would assume top 25%. (Though I would be very surprised if an AI could do one but not the other)