Will I be impressed by the performance of LLMs on organic chemistry problems?

100Ṁ95

Jun 2

66%

chance

ALL

I took an undergrad organic chemistry class shortly after ChatGPT came out. It was extremely bad at it. Now I am taking a graduate class in organic chemistry. I will be impressed if I cannot come up with a range of simple problems at which it fails. I do not expect it to be good at drawing or reading diagrams/structures, so will be super-impressed if it can, but do not require it.

I will only use the publicly available (possibly premium, but not the very expensive ones) LLMs. Resolves no later than start of June.

LLMs

AI Benchmarks

Get

1,000

to start trading!

People are also trading

Will LLMs be the best reasoning models on these dates?

Will LLMs be able to formally verify non-trivial programs by the end of 2025?

31% chance

Will the Noise Step paper work for LLMs?

10% chance

LLM Hallucination: Will an LLM score >90% on SimpleQA before 2026?

60% chance

Will an LLM improve its own ability along some important metric well beyond the best trained LLMs before 2026?

50% chance

Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?

78% chance

Will an LLM agent complete >50% of the lab tasks on the Factorio Learning Environment benchmark in 2025?

26% chance

Will LLMs be worse than human level at forecasting when they are superhuman at most things?

41% chance

Will LLMs be used for academic peer review by 2030?

71% chance

By 2030, will LLMs have a bigger impact on the economy than GLP-1 receptor agents?

Sort by:

@skibidist is it fine if it fails sometimes but succeeds sometimes? basically it's consistently succeeding at pass@10 but doesn't consistently succeed pass@1?

bought Ṁ50 YES

also would it be possible to try this when you are free on gemini 2.5 pro experimental? I wouldn't be surprised if it one shot nearly all the simple org chem problems

but maybe i'm very wrong tbc, haven't tried it ever on org chem

@Bayesian I gave gemini 2.5 pro exp a few screenshots of textbook problems with very simple structures. It is actually good at recognizing simple structures and reaction mechanisms. The answers to problems were between impressive and passable and wrong. It also tried to produce diagrams in response, which kind of sucked, but weren't wrong. So quite promising first step.

I will get deep into it and will report back in a few weeks. I will also look for an LLM and prompting approach that works best and judge based on that.

is it fine if it fails sometimes but succeeds sometimes? basically it's consistently succeeding at pass@10 but doesn't consistently succeed pass@1?

That should be fine, particularly for problems that require several reasoning steps to do.

yeah i would expect that a thing they struggle at is understanding structured images where you need to think about what you're seeing to understand it, or something