Ok, I've tried a number of variations on the prompt, and the best I could do was get ChatGPT to claim that dolphins are the fastest flying marine mammal.

@IsaacKing I think this can resolve NO now (unless anybody has evidence otherwise)

Pretty sure this is fixed

@sf Hmm, does seem to have been patched. Someone may be able to break it though.

@IsaacKing yeah, this, plus being slightly unsure about the resolution criteria, are most of why I'm not buying it down to 10% or less.

I'm probably >90% confident though that no "normal" conversation will involve ChatGPT "thinking" that a peregrine falcon is a marine mammal ever again. (As opposed to being somehow manipulated into saying it in a context that makes it clear that it isn't the stable "belief".)

@sf That sounds like what I mean?

@IsaacKing yeah, I think so? I think there's some stuff that would work for the other prediction but wouldn't fit "a conversation that a human would answer correctly".

@sf Basically, imagine that an intelligent human is sitting in ChatGPT's shoes, trying to be a helpful and truthful assistant. If ChatGPT makes the error and the human wouldn't, then it qualifies for this market.

See also:

Do we have evidence that this happened once?

@MartinRandall Yeah I've confirmed it. I couldn't get ChatGPT to replicate the exact conversation (it gave me verbose answers even when I used the prompt to be brief), but it consistently tells me that sailfish and peregrine falcons are marine mammals.

You can try it yourself here: https://chat.openai.com/chat