Since I do not have access to GPT-4 Vision yet and someone else might, I will not reveal the year of the paper I will test GPT-4 vision on. Here is a MD5 hash specifying the year, so as to confirm later:
1064009a1779660808da7382ff1b094b
JEE Advanced is one of the hardest tests at a pre-college level in the world.
The marking scheme is as follows:
+4 ONLY if (all) the correct option(s) is(are) chosen
+3 If all the four options are correct but ONLY three options are chosen
+2 If three or more options are correct but ONLY two options are chosen, both of which are correct
+1 If two or more options are correct but ONLY one option is chosen and it is a correct option
0 If none of the options is chosen (i.e. the question is unanswered)
-2 In all other cases
All the options are [A,B) format where A is included and B is excluded. For example, 10-15 will be chosen if the score is 10 or 11 or 12 or 13 or 14 but not if the score is 15.
1,000
1.00@firstuserhere sorry, I've not gotten around to doing this still, so i will n/a the questions for now . I was curious but sadly didn't find time for this project
How will you prompt and evaluate? I fed him the electrostatics problem with the hexagon. He got the first one right on the first try but refused to solve the other three because “we need to do the calculations”. When told to go ahead and do the calculations it gives some answer… but not in SI. It says stuff like “(B) cannot be confirmed without the value of ε0 or the context that relates the given expression to the correct formula.” Pressing it further may or may not result in the correct answer, but I expect it to make a huge difference.
@mariopasquato yeah won't be going rigorous on trying my best to make it solve a problem. I'm trying to see the models ability, not my prompting skills. Just a general prompt telling that it's solving a problem from this subject and this level of difficulty and to think logically etc. Nothing more.
