Will GPT-4 be able to consistently solve college-level discrete math problems?

970Ṁ4738

resolved May 16

Resolved

ALL

I will use the class CS 70 at UC Berkeley to test this (https://www.eecs70.org/), once I get access to GPT-4. I will take all text-only homework problems from the most recent iteration of the class, and copy and paste them in, maybe with a minimal amount of prompt engineering (e.g. "pretend you are a brilliant mathematician" or something). Even if GPT-4 has image recognition abilities, I won't use problems with images.

After this, I will grade GPT-4's responses. I am currently a TA for the class, and I was a grader for the class for two semesters, so the grading will be as close to real life as it can be. If GPT-4 scores above 73%, I will resolve this market positively. (If a student in CS 70 scores above 73% on a homework, they get 100% on it.) Otherwise, I will resolve this negatively.

If GPT-4 releases under a different name, I'll test that model.

Note: ChatGPT and Bing Chat both cannot do this, they produce good-looking answers but consistently make incorrect statements like "9 is prime" or something.

Math

GPT-4 speculation

Get

1,000

to start trading!

🏅 Top traders

#	Name	Total profit
1		Ṁ176
2		Ṁ70
3		Ṁ68
4		Ṁ50
5		Ṁ46

People are also trading

Will GPT-5 be able to solve A::B system puzzles consistently

78% chance

Will Gemini outperform GPT-4 at mathematical theorem-proving?

62% chance

Will GPT-5 not be terrible at the "Numbers Game"?

88% chance

Will GPT-5 be capable of some form of online learning?

29% chance

Will OpenAI's next major LLM (after GPT-4) solve more than 2 of the first 5 new Project Euler problems?

45% chance

GPT-4 #5: Will GPT-4 be a dense model?

1% chance

Will GPT-5 be able to get gold on the International Mathematical Olympiad?

23% chance

Will GPT-4 escape?

6% chance

Will GPT-5 get the Monty *Fall* problem correct?

84% chance

Will GPT-5 be capable of achieving superhuman performance in at least one exam that is typically taken by humans?

Sort by:

Resolves to no (around 65%, so not too far off)

predictedNO

@dominic had you had a chance to check? If not, could you provide an estimate?

predictedNO

@dominic any update?

@ValeryCherepanov I'm not Dominic but I got GPT4 and tried it out on some CS 70 questions; it did not meet the 73% threshold for my sample, so I doubt it will do better whenever GPT4 improves/he tests this in the future.

The only way this resolves Yes is if, by chance, GPT4 is asked questions it was trained on (questions that were already on the internet).

@qumeric working on it, have graded the first 3 homeworks so far

I can't get it to correctly solve slightly convoluted versions of "where do the trains cross" problems, so I highly doubt it.

gpt 4 limitations are making this take a little longer to resolve, plan to try to resolve soon though

predictedNO

Given how Bing's Chatbot is now confirmed as GPT-4 (https://blogs.bing.com/search/march_2023/Confirmed-the-new-Bing-runs-on-OpenAI%E2%80%99s-GPT-4) and that Bing's ChatBot can't do CS70 problems consistently, this should be resolved as no. Though you could buy ChatGPT pro to test if that will be fine-tuned for math more.