SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

The problem it targets

Most evaluations of code-generating language models ask for something small: complete a function, repair a snippet, produce a program from a short description. Real software maintenance looks different. A developer receives an issue report, has to locate the relevant code inside a large repository, and change it without breaking the parts that already work. A model that can write a tidy function in isolation may still be far from doing that.

SWE-bench was built to measure that kind of work. The paper was submitted to arXiv on 10 October 2023 by Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan^[1].

How the benchmark is built

The benchmark consists of 2,294 software engineering problems drawn from real GitHub issues and the pull requests that resolved them, collected from 12 popular Python repositories^[1]. Each problem hands a model a codebase together with a description of an issue to be resolved, and asks it to edit the codebase so that the issue is addressed^[1]. Real fixes are rarely confined to a single spot, so solving a task frequently requires understanding and coordinating changes across multiple functions, classes and even files at the same time^[1].

That design choice is the point of the benchmark. A model cannot succeed by pattern-matching a single function signature; it has to connect a written complaint to specific places in a codebase and produce a coherent set of edits. The tasks also carry the messiness of real projects, in which the reporter and the person who fixed the problem were different people writing at different times.

What the first results showed

The initial evaluation was sobering. Both state-of-the-art proprietary models and the authors' own fine-tuned model, SWE-Llama, could resolve only the simplest issues, and the best-performing model, Claude 2, solved 1.96% of them^[1]. In other words, on roughly 98 of every 100 tasks the model failed to produce a fix that counted. That is a very different picture from the impression left by models that write fluent code when asked for a small, self-contained program.

The Lite subset and retrieval settings

The authors also report results on SWE-bench Lite, a smaller subset covering 11 of the original 12 repositories^[2]. Because a whole repository does not fit into a model's context, the experiments include a step that decides which parts of the code the model is allowed to see. Models were evaluated under sparse retrieval and under an oracle retrieval setting, in which the relevant files are supplied directly; the authors treat that second setting as less realistic^[2].

Why it matters

The authors present real-world software engineering as a rich, sustainable and challenging testbed for evaluating the next generation of language models^[1]. The low solve rate makes the gap concrete: models that can write plausible-looking code still struggle to carry out a maintenance change across a real repository. Because the problems come from issues and pull requests rather than from hand-written puzzles, a score on the benchmark says something about how far a model is from being useful as a working contributor to a codebase, and about which parts of the job — finding the code, understanding the report, making consistent edits — remain hard.