SWE-bench Leaderboards (original) (raw)

SWE-bench Verified is a human-filtered subset of 500 instances; use the Agent dropdown to compare LMs with mini-SWE-agent or view all agents [Post].
SWE-bench Multilingual features 300 tasks across 9 programming languages [Post].
SWE-bench Lite is a subset curated for less costly evaluation [Post].
SWE-bench Multimodal features issues with visual elements [Post].

Each entry reports the % Resolved metric, the percentage of instances solved (out of 2294 Full, 500 Verified, 300 Lite & Multilingual, 517 Multimodal).

News

• [11/2025] Introducing CodeClash, our new eval of LMs as goal (not task) oriented developers! [Link]
• [07/2025] mini-SWE-agent scores 65% on SWE-bench Verified in 100 lines of python code. [Link]
• [05/2025] SWE-smith is out! Train your own models for software engineering agents. [Link]
• [03/2025] SWE-agent 1.0 is the open source SOTA on SWE-bench Lite! [Link]
• [10/2024] Introducing SWE-bench Multimodal! [Link]
• [08/2024] SWE-bench x OpenAI = SWE-bench Verified [Report]
• [06/2024] Docker-ized SWE-bench for easier evaluation [Report]
• [03/2024] Check out SWE-agent (12.47% on SWE-bench) [Link]
• [03/2024] Released SWE-bench Lite [Report]

Acknowledgements

We thank the following institutions for their generous support: Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI, and Anthropic.