How well will ChatGPT do on the LSAT?
4
330Ṁ200
resolved Dec 3
Resolved
148

I will paste practice LSAT questions from Khan Academy into ChatGPT and select the answer it chooses, altering the prompt if it is unable to select an answer. What score will it get? For context, the score range is 120 to 180, the 50th percentile score is around 151, and applicants to the most elite law schools (like Harvard and Yale) usually score above 170.

Close date updated to 2022-12-02 9:00 pm

Get
Ṁ1,000
to start trading!

🏅 Top traders

#NameTotal profit
1Ṁ6
Sort by:
predictedHIGHER

$5 of compute versus 2.5 hours for an average college graduate

It definitely struggled most on Analytical Reasoning; I think its overall score might improve with better prompt engineering (instead of just copy/pasting which leaves some web artifacts in). This is still good enough to get into some law schools, though, which impresses me for a general language model.

@a (And obviously if you train it on LSAT solutions it will do much better! But that would probably be less predictive of other logic-and-analysis applications.)

How does it compare to untuned gpt3? 🤔

Are you just doing the multiple choice questions?

@vluzko Yes, the written section isn't scored in an objective way as far as I can tell, so there's no easy way for me to evaluate GPT-3's performance.

Also very good at reading.

They did a lot of work on “summarization” to reason through to the right answer.

Smarter than most people at top law schools already.

predictedHIGHER

Very uneven on reading. ~150 overall.

Brilliant at analogical arguments and summarization, no ability to do symbolic reasoning, quite bad at contrasting different views, certainly very obvious path to perfect.

Latency is about 10x lower than Manifold betting—props to the brilliant engineers who did the deployment.

I gave it a few questions.

Logic games: nonsense.

Logical reasoning: flawless.

Might go perfect on half, zero on a quarter, and be quite good at reading. 150-160, with very easy 170s once they learn math.

How well will ChatGPT do on the LSAT?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules