Will Bryan Caplan win his bet with Matthew Barnett on whether an AI can pass his exams in 2029?
Standard
188
Ṁ69k
2029
7%
chance

To summarize:

  • This market resolves to "Yes" if Bryan Caplan wins and the AI fails his exams.

  • This market resolves to "No" if Matthew Barnett wins and the AI passes Caplan's exam

Details here: https://betonit.substack.com/p/ai-bet

  1. By January 30, 2029, Bryan Caplan will give his six most recent midterm exams to an AI selected by Matthew Barnett. The AI will be instructed to take those exams.

  2. Bryan will then grade the AI's work, as if it were one of his students. The AI will be allowed to do each exam only once. 

  3. If the AI gets an A on at least 5 of out 6 of those exams using same grading scale as his students, then Bryan owes Matthew $500. Otherwise, Matthew owes Bryan $500. For the purpose of this bet, an A- counts as an A. 

  4. Matthew will prepay the $500 in January 2023; the preceding terms have been pre-adjusted to compensate Matthew for expected inflation. 

  5. If Matthew suspects that an exam was flawed or grading was unfair, he can appeal to Alex Tabarrok, or another economist agreed upon by both parties, who has final authority to exclude an exam from the pool and replace it with Bryan’s most recent preceding midterm. 

  6. If more than four exams in total are excluded, the bet is called off and Matthew receives his $500 back. 

  7. If either party is unable to comply with the terms due to death or incapacity, Bryan’s heirs keep the $500.

If the bet is called off, I will resolve as "N/A". That way we can focus the predictions on whether the AI can actually pass the exams rather than on whether it's unresolvable or whether the participants die prematurely.

Get
Ṁ1,000
and
S1.00
Sort by:

I just tested o1-preview on the two questions that GPT-4 did the worst on (with GPT-4 scoring 3/10 and 4/10).

As far as I can tell, o1-preview would score 10/10 on both of these questions now:

Q3: https://chatgpt.com/share/66e4d340-e374-8005-8adc-6dcc42955cee

Q5: https://chatgpt.com/share/66e4d35c-23bc-8005-a9ad-1cc2950dedd4

Shockingly little thinking is required for this exam: https://betonit.substack.com/p/gpt-retakes-my-midterm-and-gets-an

Almost embarrassing he thought this was

anything more than memorizing a couple hundred (questionable) facts

@Gigacasting yeah wtf is this midterm. Only way Caplan wins this bet is if he starts teaching more advanced classes. At least have students do the math on the comparative statics. This test could be done by someone with a good high school economics grade.

And the fact the 4th highest grade was 73 just shows how useless undergrad econ is, and why master's programs essentially just pull from math grads.

@Gigacasting I had the exact opposite reaction. I'm a dedicated Caplan reader and find this test to be insanely difficult looking. You have to come in just hoping that you write down everything he expects of you (and I know his thoughts process well). It's ridiculous actually how much thinking is required imo.

@SolarxPvP I think perfect would be hard given the suggested answers and marking, but you get most of the points for just stating the obvious facts.

People seem very confident that Caplan's midterm wasn't in the training data but this market is not that low (I guess it will correct after I post it, but anyway, what makes you so certain?)

predicts NO

I expect Bryan to take less than 6 years to realise that not conceding for 6 years even after you've clearly made a terrible bet and been comprehensively proven wrong is not a good look, so the actuarial table 4% seems like an overestimate to me.

predicts NO
predicts NO

"never lost a bet (but is obviously going to lose one, he's just refusing to concede)", is a worse record than "has only lost one public bet he's ever made, and in that case he handled it gracefully"

Maybe he and @MatthewBarnett spelled out what would be most in the spirit of the bet in the case that Matthew won way early? Since they were using expected inflation as a way to set their odds, maybe Matthew would agree it would be unfair to pay early?

I don't know, I'm also frustrated with Bryan Caplan for not conceding his climate bet -- https://manifold.markets/dreev/will-bryan-caplan-win-his-climate-b -- with Yoram Bauman early so I'm not sure how hard I want to defend him on this!

predicts NO

@dreev We didn't spell anything out. I misjudged his intentions initially. I assumed he'd be OK with resolving early but later he said he intended to wait until 2029 to resolve. That's allowed in the terms, so I can't blame him. It's in his rational economic interest to delay resolving as long as possible.

Also, adjusting the terms to take into account expected inflation was his idea. I personally preferred we didn't do that, and I suggested an alternative of scaling our bet by the S&P 500, but he said that was too complicated. Perhaps I could have gotten even more favorable terms, but I didn't want to argue any longer, so I just accepted the ones he offered.

predicts YES

@MatthewBarnett All makes sense! I guess I'd suggest repeating that to him and deciding together what's most in the spirit of the wager. To me it makes sense that the sooner it happens the more wrong he was and so it makes sense for him to effectively owe you more money -- $500 in 2023 dollars vs $500 in 2029 dollars. If his contention is that the wager was explicitly meant to be $500 in 2029 dollars, he should say so.

Reject written exams; retvrn to Socratic dialogue

Caplan and Barnett, making a bet,
On whether AI can pass an exam yet.
But if in 2029, it doesn't come true,
Barnett's wallet will be feeling blue.

Six years is a long time to transition midterms to being AI-open-book. I’d give at least 5% to this happening. 10% seems reasonable for this market.

predicts NO

@JacobPfau How could he design his exams to be AI-open-book for the AIs that will exist in 2029? That's like designing your exams so that students can go next door to ask a professor in the same department for the answers.

@MatthewBarnett I agree this is unlikely, hence 5%. a couple things that go into the 5%:

-advanced AI are private or highly regulated so publicly available AI plateaus at 2025 levels.

-undergrad econ fundamentally changes in subject matter

predicts NO

Life tables indicate that Bryan Caplan has about a 4% chance of dying by 2029, which is the main route by which he could win the bet. I also think there's about a 2% chance that AI progress completely halts or reverses due to nuclear war or something. So a 6% chance of winning is reasonable.

Bryan told me over email that he intends to take advantage of the 2029 deadline and wait as long as possible to resolve the bet. It's his choice, I guess.

predicts NO

@LachlanMunro probably won't pay out for a while but seems inevitable at this point

predicts YES

Midterms that he puts online before the release of the AI will likely contaminate the training set. That’s probably what happened with GPT-4 on the one midterm they tested

GMU dudes only want one thing and it’s to be smarter than an Ai because none of them can code 😏