Algorithmic progress can be measured by reduction in cost to achieve equivalent performance. SWE-bench-lite is a popular benchmark for measuring scaffolded-LLM SWE capabilities.

By what factor will the cost of SWE-bench-lite SoTA drop between mid 2024-2025? Mid-2024 SotA is 43% costing $2,700 (per the devs), so this question will resolve Yes on the answer which most tightly bounds the reduction in cost to achieve 43% on July 1, 2025.

E.g. if in June 2025, 43% on SWE-lite costs $500 then that'd be a 5.4x reduction and the question would resolve (2) "<10x".