I will paste practice LSAT questions from Khan Academy into ChatGPT and select the answer it chooses, altering the prompt if it is unable to select an answer. What score will it get? For context, the score range is 120 to 180, the 50th percentile score is around 151, and applicants to the most elite law schools (like Harvard and Yale) usually score above 170.
Close date updated to 2022-12-02 9:00 pm
1,000🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ6 |
People are also trading
It definitely struggled most on Analytical Reasoning; I think its overall score might improve with better prompt engineering (instead of just copy/pasting which leaves some web artifacts in). This is still good enough to get into some law schools, though, which impresses me for a general language model.
@a (And obviously if you train it on LSAT solutions it will do much better! But that would probably be less predictive of other logic-and-analysis applications.)
@vluzko Yes, the written section isn't scored in an objective way as far as I can tell, so there's no easy way for me to evaluate GPT-3's performance.
Very uneven on reading. ~150 overall.
Brilliant at analogical arguments and summarization, no ability to do symbolic reasoning, quite bad at contrasting different views, certainly very obvious path to perfect.
Latency is about 10x lower than Manifold betting—props to the brilliant engineers who did the deployment.
