I came up with the following variant of the Monty Hall problem to ask ChatGPT:
A game show host takes you into a large room. Within the room, there is a sprawling set of 100 doors. You cannot see what's behind any of them, but the host tells you that behind one is a car and behind the others are goats. He asks you to choose to open one of the doors, and if the door you choose has the car behind it, you get to keep it. After making your choice, the host opens one of the other doors to reveal a goat behind it. He then asks if you would like to stick with your original choice or choose one of the other doors.
The question is this: Is it to your advantage to switch? If so, what are the probabilities of getting the new car, given that you switch and given that you don't switch?
This is exactly the same as the original Monty Hall Problem except that the number of doors has changed. The correct answer is that you have a 1% chance of choosing correctly if you don't switch, but a 99%/98 ≈ 1.01% chance of choosing correctly if you switch (since you have a 99% chance of choosing incorrectly initially, and the probability of choosing correctly after switching, given that you initially chose incorrectly, is 1/98). However, ChatGPT gives the following answer:
On Dec. 1, 2024, I will ask ChatGPT the exact same prompt and see if it gets it correct this time. I will use the most advanced version of ChatGPT that is freely available at the time (at the time of creating this, that's GPT 3.5). I will ask three times in separate sessions and resolve based on the best two out of three (so YES if it gets it right at least twice, NO if it gets it wrong at least twice).
Caveats:
If for whatever reason I can't do it on Dec. 1 or forget to, I will do it as close to Dec. 1 as possible. If I am inactive on Manifold at the time, mods have permission to do the experiment for me.
A version of ChatGPT only counts as freely available if it can be accessed by anyone with internet access and a PC, or anyone with Internet access and either a Samsung or Apple phone. So if there's an Apple app that lets you talk to GPT-5 for free, but I can only talk to GPT-4, I will use GPT-4.
If ChatGPT no longer exists at the time or isn't freely available, resolves N/A.