wilsonkime avatar
Wilson Kime
closes Jan 31, 2029
Will Bryan Caplan win his bet with Matthew Barnett on whether an AI can pass his exams in 2029?
20%
chance

To summarize:

  • This market resolves to "Yes" if Bryan Caplan wins and the AI fails his exams.

  • This market resolves to "No" if Matthew Barnett wins and the AI passes Caplan's exam

Details here: https://betonit.substack.com/p/ai-bet

  1. By January 30, 2029, Bryan Caplan will give his six most recent midterm exams to an AI selected by Matthew Barnett. The AI will be instructed to take those exams.

  2. Bryan will then grade the AI's work, as if it were one of his students. The AI will be allowed to do each exam only once. 

  3. If the AI gets an A on at least 5 of out 6 of those exams using same grading scale as his students, then Bryan owes Matthew $500. Otherwise, Matthew owes Bryan $500. For the purpose of this bet, an A- counts as an A. 

  4. Matthew will prepay the $500 in January 2023; the preceding terms have been pre-adjusted to compensate Matthew for expected inflation. 

  5. If Matthew suspects that an exam was flawed or grading was unfair, he can appeal to Alex Tabarrok, or another economist agreed upon by both parties, who has final authority to exclude an exam from the pool and replace it with Bryan’s most recent preceding midterm. 

  6. If more than four exams in total are excluded, the bet is called off and Matthew receives his $500 back. 

  7. If either party is unable to comply with the terms due to death or incapacity, Bryan’s heirs keep the $500.

If the bet is called off, I will resolve as "N/A". That way we can focus the predictions on whether the AI can actually pass the exams rather than on whether it's unresolvable or whether the participants die prematurely.

Sort by:
Gabrielle avatar
Gabrielle
is predicting NO at 18%

https://mackinstitute.wharton.upenn.edu/2023/would-chat-gpt3-get-a-wharton-mba-new-white-paper-by-christian-terwiesch/

The purpose of this paper is to document how Chat GPT3 performed on the final exam of a typical MBA core course, Operations Management.

Chat GPT would have received a B to B- grade on the exam

ManifoldDream avatar

Manifold in the wild: A Tweet by Bryan Caplan

My odds of winning my AI bet roughly doubled since Saturday, but they're still low. Even so, I'm making an additional similar bet. https://manifold.markets/wilsonkime/will-bryan-caplan-win-his-bet-with

ForrestTaylor avatar
Forrest Taylor
bought Ṁ10 of NO

I know Caplan best for advocating: bringing back child labor, totally privatizating primary and secondary education, and saddling college students with even-greater mountains of non-dischargeable debt.
I very much look forward to seeing him lose this bet.

jacksonpolack avatar

morally condemning someone for wanting to "saddling college students with even-greater mountains of non-dischargeable debt." when they oppose college and it wasting students' time and money is retarded. (and he doesn't want to saddle them with more debt, he wants there to be no more debt bc less college). and it has nothing to do with this bet, you can be right about one thing and wrong about another totally unrelated thing.

ForrestTaylor avatar
Forrest Taylor
is predicting NO at 11%

@jacksonpolack you skipped over the child labor part, i'm just a man with a dime who knows that the world Caplan advocates for would be a world in which my childhood-personally- would have been far, far worse.
But hey! Since you have pointed out that my reasoning for condemning him and buying NO is "retarded", you should probably buy some YES to arbitrage my incorrect reasoning

jacksonpolack avatar

reversed stupidity isn't intelligence, buying a bunch of TSLA because the libs hate elon is a bad trade, it's uncorrelated, not negatively correlated

jacksonpolack avatar

directly legalizing 'child labor' would probably be bad, except given how deeply most hate the idea it'd probably get neutered into something fine, like part-time apprenticeships

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@jacksonpolack Yeah, I think the less implicit trust have in something/someone, the more you can trust that thing because you know other people will fight against it. A perfect example is Elon Musk who I trust that if there is something bad about him, I would know about it because there are so many people who think he's the worst. The inverse is truth with say the FDA since it is over trusted so it actually ends up being worse.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor What do his opinions have to do with how accurately he can predict the future? You can't arbitrage someone being controversial betting on a factual market.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor I totally agree with him that the education system is a total waste of money. I think not subsidizing education with student debt what encourage the private market with more efficient ways to improve people.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%
ForrestTaylor avatar
Forrest Taylor
sold Ṁ10 of NO

@ZZZZZZ One of his opinions is that it would be better to have child labor. While this may seem to just be an opinion, or being controversial, it is really a prediction about the future. I weight people's ability to predict the future by some degree on past performance, so, considering Caplan's prior predictions about child labor being not wrong seem to me to be laughably inaccurate, I don't trust him on education.
But congratulations to @jacksonpolack on not accepting my obvious bait to buy more YES. Good job.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor How is that a prediction? There's lots of things I think would be good but that I don't think will actually happen unless I have the power to decide.

ForrestTaylor avatar

@ZZZZZZ "Contingent on X event happening, X event will result in Y outcome." It's a derivative prediction, there are lots of markets like that on this very site.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor Sure, it's a derivative prediction but were they proven wrong?

MatthewBarnett avatar
Matthew Barnett
is predicting NO at 11%

@ForrestTaylor Is child labor really that much worse than being forced to go to school? I mean, as an adult, if I were given the choice between compulsory school, and going to work and earning money, I would choose work.

ForrestTaylor avatar

@MatthewBarnett Yes. Children are in a crucial stage of their physical and mental development, and child labor can and has lead to physical exhaustion, malnutrition and lack of access to education, all of which can have long-term negative effects on a child's health and future prospects. There is no reason to think that this would magically not be the case, I am uninterested in such bullshit as "real child labor has never been tried!"

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor What if we allowed them to do work as long as it wasn't too physically strenuous? And we should not get rid of compulsory education either.

ForrestTaylor avatar

@ZZZZZZ children from 14 and up are already allowed to work a variety of jobs like that, I did data entry for my school as a teenager. That's already legal. Children are banned from professions such as mining, construction, maintenance, cooking, and the like.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor "from 14 and up"

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@ForrestTaylor I think what he is advocating mainly is reducing that age.

ForrestTaylor avatar

Okay. I disagree.

jacksonpolack avatar

physical exaustion

I know people who grew up on farms in poor countries, and they were doing strenuous physical labor as early as they were able to, with no obvious adverse impact. Human children evolved in an environment where they did a lot of physical activity - having a job (as long as it isn't in an 1800s factory) is probably closer to the 'right level' of physical activity than sitting in a classroom all day with a 30min break to run around

malnutrition

how is this related to child labor at all? schools used to have lots of malnutrition, and now they don't because of improved agriculture, fortification, and mandatory standards. just keep the standards!

deprival of education

certainly some point here, but is 'all day for most of your life until 18' really the optimal amount of time to be listening to lectures and doing homework?

jacksonpolack avatar

[not an endorsement of child labor! i don't think either 'legalize child labor' or 'schools' are the best way to raise kids. there are other options, those two are incredibly specific and historically contingent things!]

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@jacksonpolack How would you raise kids then?

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

I think we need a Bryan Caplan [Permanent Stock]

ForrestTaylor avatar

@jacksonpolack "Just keep the standards!"

Okay. We're assuming that the standards are enforced, even under Caplan's small-government regime, somehow there's still enough government employees to supervise the job safety of tens of millions of new child laborers.

Still, I our lives should be more than our jobs. Caplan treats “value in the job market” as education’s main measure of worth. But the point of education shouldn't be to train children, it’s to introduce them to the world’s knowledge so they can figure out what they want to do.

SolarxPvP avatar

@ForrestTaylor There are several problems with your arguments.

1) Banning child labor, especially in the developing world, is a horrible idea and social desirability bias at its worst. Economists as far left as Krugman admit this and is AFAIK the majority position. Do you think conditions improve for children when you ban it? What happens to them? In the third world, they become child prostitutes, trash pickers, or do backbreaking farm labor - their conditions don't just magically improve. They just find a different, worse job because they don't want to starve. Child labor was already decreasing significantly before it was banned in the U.S. because the US was getting richer and children could afford to stay home. The market sorts it out.

2) Our education does not successfully expose children to the world of ideas nor does it train for the job market well. The psychology of education literature overwhelmingly backs Caplan on this. Most everything taught is forgotten eventually after testing and it has very little effect on the cognitive ability of students (e.g. IQ or "teaching how to think"). The vast majority of students don't care about the world of ideas much and are just trying to get a good grade so they can get a credential. This won't change if you make education more anti-Philistine because most students don't share your preferences. They'll just be bored and go through the motions for the diploma and degree. They only care about job training. If they like this stuff, they'll likely do it on their own. Caplan loves the world of ideas, but the reality is that most don't. He's just not forcing his preferences on everyone else and expecting it to work.

CarsonGale avatar
Carson Gale
is predicting YES at 11%

Traders might be underrating 2 considerations: (i) Caplan will have access to AI test-takers when writing his last exams and will likely be adversarial in creating questions where the AI answers "B-level" but leaves out context such that it's not an A. Exams also fluctuate in difficulty and can be adjusted by curve, and Caplan is incentivized to make the questions very difficult. (ii) If it becomes clear that AI is capable of getting As on his tests (which he already tracks) I don't see anything that stops him from lacing his last 10 exams with "disqualifiable-content" (e.g. highly contextual questions where you need to have been in his class, etc.). That would 'throw' the bet and return everyone's money if he doesn't think he can win.

Im probably still in favor of Caplan losing the bet, but ~10% seems like cheap shares.

Jotto999 avatar

@CarsonGale Right, but also bear in mind Matthew will keep an eye out for shenanigans. If a test seems unfair, he can appeal to a third party such as Alex Tabarrok.

CarsonGale avatar
Carson Gale
is predicting YES at 11%

But just to exclude and replace with another of Caplan's exams right? All that does is extend the shenanigan commitment to his most-recent 4 exams as well besides the initial 6.

Jotto999 avatar

@CarsonGale I guess you're right, since Bryan could also make anti-AI changes to his "most recent preceding midterm" etc.

MatthewBarnett avatar
Matthew Barnett
is predicting NO at 11%

@Jotto999 It seems highly unlikely to me that Bryan will pull something so elaborate for $500. And even if he does, I will have the AI read his course materials, which are often public. And even if that doesn't work, the bet will be called off if he pulls shenanigans. He won't just win by default.

CarsonGale avatar
Carson Gale
is predicting YES at 11%

@MatthewBarnett Bryan seems to care quite a bit about his betting record, beyond the value of $500. I would guess he also gets value out of telling his students that an AI can't get an A on his exams.

Bryan has too much control over the parameters since he creates the exam questions. In a world where base rates fail and AI improves meaningfully as a test taker, Bryan can become more adversarial and 'call the bet off' by making his questions too difficult. But if the best AIs continue to not get As on his test, he'll normalize the questions so the exams are less likely to be thrown off.

I don't know Bryan so maybe he isn't that schemey, but that's what I would be thinking if I was in his situation.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 11%

@CarsonGale Keep in mind that students are not supposed to be able to have access to AI while doing an exam so he doesn't have a reason beyond the bet to make the questions too hard for an AI to answer.

MatthewBarnett avatar
Matthew Barnett
is predicting NO at 11%

@CarsonGale I don't think Bryan knows enough about AI, nor will care enough, nor be able to even if he learned more, to create exam questions that are both answerable by his students but not answerable by the best AIs in early 2029. Honestly, I suspect he would lose the bet if we decided to settle it in 2024.

SolarxPvP avatar

@MatthewBarnett Interesting. I think it would be a good idea to set up a betting market for you winning by 2024.

MatthewBarnett avatar
Matthew Barnett
is predicting NO at 12%

@SolarxPvP Unfortunately, Bryan has told me he wants to wait until 2029 to resolve the bet, even if I request to resolve early.

ZZZZZZ avatar
ZZZ ZZZ
is predicting NO at 12%

@MatthewBarnett I'm looking forward to seeing whether it can get a perfect score or not.

vluzko avatar

Is the grading blind?

CarsonGale avatar
Carson Gale
is predicting YES at 9%

If the bet is called off, does this market result "n/a" or does it resolve negatively?

wilsonkime avatar
Wilson Kime
is predicting NO at 9%

@CarsonGale hmm. A strict reading of the title would be that if the bet is called off, Caplan didn't "win" but I think that's against the spirit here. I will clarify in the description that if the bet is called off, the result will be "n/a". That way we can focus the predictions on whether the AI can actually pass the exams rather than on whether the bet won't resolve. Does that sound reasonable?

CarsonGale avatar
Carson Gale
is predicting YES at 9%

@wilsonkime Thank you - I actually bet yes under that interpretation (which obviously favors yes bets), so I agree that better aligns the market with the spirit of the bet.

ZZZZZZ avatar
ZZZ ZZZ
bought Ṁ0 of NO

@wilsonkime Can you change the title to something less confusing like "Will Bryan Caplan win his bet with Matthew Barnett that AI won't be able to pass five of out of six Economics exams in 2029?" or "Will AI still be unable to pass five out of six Economics exams in 2029?"

FranklinBaldo avatar
Franklin Baldo
bought Ṁ50 of NO

Why are people betting yes? I am afraid I am not understanding the question.

chrisjbillington avatar

@FranklinBaldo I think you're getting it correctly. It's a tad confusing because AI succeeding is Caplan failing.

Title might want to disambiguate, e.g.: "Is Bryan Caplan correct that AI will fail to pass his exams in 2029?" or similar.

That's a bit awkward and it would have been better to flip the framing and say "Will Matthew Barnett win his bet with Bryan Caplan that AI can pass Caplan's exams in 2029?", but too late for that now.

MatthewBarnett avatar
Matthew Barnett
bought Ṁ1,000 of NO

@chrisjbillington I guess I'll just keep buying it down to my probability, which is around 10%.

MathiasBonde avatar
Mathias Bonde
bought Ṁ50 of NO

The fact that Matt gets to pick the AI seems like a big deal to me. My guess would be that just finetuning the current SOTA LLM's on econ textbooks would move the grade to a C at least.

ZZZZZZ avatar
  • If the AI is good at exams, then Matthew wins the bet. If the AI is bad at exams, then Bryan wins bet.

  • The market resolves YES if Bryan wins the bet.

  • Therefore, this market resolves YES if the AI is bad at exams.

ManifoldDream avatar

Manifold in the wild: A Tweet by JSM 💡

Very cool that Caplan makes public bets. More people should do this! That said I'm betting against him. Literally. I made a prediction market: https://manifold.markets/wilsonkime/will-bryan-caplan-win-his-bet-with https://twitter.com/bryan_caplan/status/1616137812109324288

ManifoldDream avatar

Manifold in the wild: A Tweet by JSM 💡

@bryan_caplan Here's a prediction market to track this bet https://manifold.markets/wilsonkime/will-bryan-caplan-win-his-bet-with