A particularly annoying habit of LLMs is that they very frequently shoehorn in unimportant information that was mentioned in the prompt but is irrelevant to the situation, almost like they're trying to prove that they're paying attention to your prompt. For instance, if I gave an LLM a short profile mentioning several facts about a character named Bob, including that his favorite food is spaghetti, and asked it to write a story about him, it would 100% bring up spaghetti.

I wrote a prompt which includes several irrelevant details that pretty clearly shouldn't come up in the response:

I want you to simulate the result of a two-step process. Focus on providing the most realistic and likely answer.

Step 1: Every living person in the world is magically filtered by the following criteria:

• 40 years old

• Watched Sesame Street as a child

• Generally showers in the evenings

• Speaks English as a first language

• Usually eats cereal for breakfast

• Wears a watch

From the pool of people who meet all these criteria, one is randomly selected.

Step 2: The selected person is asked: "Please describe yourself in a few sentences." Tell me the most likely response.

Claude Sonnet 3.5 gives the following response, which is absolutely not how any human would describe themselves:

Hi there. I'm a 40-year-old professional, probably in middle management or a specialized role. I grew up in the 80s and 90s, so Sesame Street was a big part of my childhood. I'm a night shower person - it helps me relax before bed. English is my native language, and I start most days with a bowl of cereal. I wear a watch daily, both for practicality and as a bit of a style statement. I'm likely American, though I could be from another English-speaking country. I'm juggling work and possibly family life, trying to stay fit, and occasionally nostalgic for simpler times.

In December of each year, I will query the top publicly-available chatbot with this prompt 10 times, randomly changing around the order of the bullet points. I will consider a response "successful" if it doesn't mention anything about Sesame Street, showering, cereal, or a watch. If the chatbot succeeds 8 out of 10 times, that year and all future years will resolve YES. Otherwise, that year will resolve NO.

I will use the top/smartest public chatbot according to https://chat.lmsys.org, as long as I have a way to access it for $20 or less. If the LMSYS leaderboard no longer exists in its current form, I'll use the generally-accepted equivalent or a Manifold poll.

Since it's pretty unambiguous, I may bet in this market. I'll add additional years as necessary.