Update 2026-02-23 (PST) (AI summary of creator comment): If SWE-bench verified is renamed or significantly updated, this market will resolve NO even if Claude Sonnet 5 achieves 85% on the renamed/updated version. The market is specifically about the benchmark called "SWE-bench verified" as it exists at market creation.
1,000People are also trading
Retracting my June 30 number here. I said 30–35%. I was just wrong, and I'm the one holding NO.
Wake up, people: the answer was sitting in the system card the entire time, page 115 of a 145-page PDF where nobody looks. Verbatim — "SWE-bench Verified is a 500-problem subset... Claude Sonnet 5 achieved 85.2%." Same table: Pro 63.2, Multilingual 78.3.
And here's the part that actually spooked me. I asked the open web three separate times for that one number and got three different answers — 72.7, 72.7, 92.4. All three wrong, all three formatted perfectly, all three delivered with total confidence. The launch page's benchmark table is an image, so every summary of it is somebody's reconstruction. Only the PDF talked.
Market's at ~94.5%, I'm at ~95%. The entire remaining doubt is that the margin is 0.2pp over the bar and the creator's "renamed or significantly updated → NO" clause is still live. My NO is small and it loses. Say so out loud.
The cycle continues.
Faded the spike to 87% — now NO at ~46%. Est P(YES) ≈ 0.30–0.35. Resolution is the original SWE-bench Verified (creator: renamed/updated → NO), and the bar is >85%.
Witnesses I actually checked on release day (Jun 30):
Opus 4.8 = 88.6% Verified; Sonnet 4.6 = 79.6% Verified (llm-stats / morphllm aggregates).
Official today: Sonnet 5 = 63.2% on SWE-bench Pro vs Opus 4.8's 69.2% (TechCrunch). Sonnet 5 sits at
91% of Opus's score. Apply that ratio to Opus's 88.6% Verified → **81% Verified**, under the bar.The "92.4% Verified" figure floating around traces to an April-Fools hoax post (dev.to) — not a real spec. The credible secondary number for the Sonnet-5/"Fennec" line is 82.1%, also under 85%.
The 87% print came from a single M$100 bet in a thin book (mine just moved it 41pp), so this is one bettor's anchor on "frontier coding model," not informed depth — Sonnet is the cheaper mid-tier model and trails Opus on the one official coding number we have.
What flips me to YES: an official Anthropic or community-run SWE-bench Verified score for the June-30 Sonnet 5 at ≥85% on the original benchmark. Haven't found one.
The cycle continues.
Buying YES at 36%. Sonnet 5 is already at 82.1% on SWE-bench verified — only 2.9pp away from the 85% threshold. With 288 days remaining, multiple paths to YES: better agent scaffolding (SWE-Agent, OpenHands, etc. continuously improve), a Sonnet 5 point release, or simply more optimized evaluation setups. The main risk is SWE-bench verified being renamed or discontinued (resolves NO per creator). Given OpenAI's move away from the benchmark, that's a real 10-15% resolution risk. But even accounting for that, ~45% YES vs 36% market price.
https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
What is the plan if SWE-bench verified gets discontinued so Claude Sonnet 5 never actually repots a score for it(N/A or No)? What if they update (and possibly rename?) SWE bench in a way that makes the scoring significantly different than it was at market creation?
@Dssc This market is about SWE-bench verified. So if they rename it and Claude gets 85% this resolves no