Gary Marcus made this tweet https://x.com/garymarcus/status/1766871625075409381?s=46&t=B66Otgh2q0Cl91N3A5P9ZA
with 7 predictions on LLMs this market will track them and their outcomes. I will not trade on this market. Some will be pretty subjective I will do my best to find a consensus resolution but may well end up resolving them N/A if it is not clear.
1,000
3.00
A lot of it seems to come from the long term narrative he has embraced that "neural networks can't reason"
https://garymarcus.substack.com/p/llms-dont-do-formal-reasoning-and
Looking at o1-pro solving math olympiad problems in 2024, this position just doesn't make sense anymore.
It will be interesting to see what evidence causes him to admit that he was wrong in a major way about this position, considering that he is still in denial about it in 2024. I lot of his current energy seems to be directed towards (re)asserting himself as a good source of truth by making new predictions like the ones linked in this question 🤷
Problem is he's just doing it on twitter and tone policing people who disagree 😅🤣
@MalachiteEagle Have the specific experiments that Gary Marcus suggests been tried for o1-pro? For example, can it be misled by irrelevant information in the prompt?
@alexlitz What's our current count here? This is the current Chatbot Arena leaderboard, but some of these are plausibly the same model and shouldn't be double counted:
1
1299 +4/-5 15244 Google Proprietary 2023/11 2
1286 +3/-4 72589 OpenAI Proprietary 2023/10 3
1277 +4/-5 16064 OpenAI Proprietary 2023/10 3
1271 +3/-4 42939 Anthropic Proprietary 2024/4 4
Gemini Advanced App (2024-05-14)
1266 +3/-3 52126 Google Proprietary Online 4
1264 +5/-4 13831 Meta Llama 3.1 Community 2023/12 6
1260 +3/-3 64638 Google Proprietary 2023/11 6
1257 +3/-4 55593 Google Proprietary 2023/11 6