OpenAI releases SWE-bench Verified

Background

Benchmarks for AI software engineering are meant to compare systems on the same tasks, but the comparison is only as good as the tasks and tests. Two problems can distort a score. A task can be underspecified: the written request leaves out requirements, so a solver cannot know what a correct answer should include. A test can be too brittle: it can reject a valid solution because the solution differs from the expected implementation. Both problems turn measurement into a mix of model ability and benchmark noise. SWE-bench Verified was announced by OpenAI on August 13, 2024, as a human-validated subset of SWE-bench, developed with the benchmark's original authors from Princeton.^[1]

What was released

The subset was selected from a random sample of 1,699 SWE-bench test samples.^[1] From that pool, 500 validated samples formed SWE-bench Verified.^[1] The review was done by 93 software developers with Python experience.^[1] Each sample was annotated by three independent annotators, and each annotator first had to pass an onboarding test of 50 samples.^[1] The release also introduced a Docker-based evaluation harness developed together with the SWE-bench authors.^[1]

How the filtering worked

The review did not make small edits around the edges. Of the reviewed samples, 38.3% were flagged as underspecified, and 61.1% were flagged as having unit tests likely to falsely reject valid solutions.^[1] Overall, 68.3% of samples were filtered out.^[1] Those figures explain why the final subset is much smaller than the starting sample. The aim was not to make tasks easier in a simple sense, but to remove cases where the benchmark could mislabel a good solution or where the request did not contain enough information for a fair attempt. A benchmark with such defects can reward guessing at hidden requirements or punish a correct but different design.

Results

The new subset changed measured performance. GPT-4o solved 33.2% of SWE-bench Verified, more than double its 16% on the original SWE-bench.^[1] The open-source Agentless scaffold doubled from 16% to 32%.^[1] The same direction appears for GPT-4o and the open-source Agentless scaffold: both score higher on the validated subset.^[1] That makes the subset useful for comparing systems under cleaner conditions, while also showing that scores from the original benchmark and the verified subset are not interchangeable.

Why it matters

SWE-bench Verified treats benchmark quality as part of the evaluation. It identifies two concrete failure modes — vague task statements and tests that reject valid alternatives — and removes samples that show them.^[1] For users of benchmark results, the lesson is that a score is tied to a particular sample set, filtering rule, and test harness. The Docker-based harness, built with the SWE-bench authors, adds a shared way to run and evaluate solutions.^[1] The release therefore matters less as a new ranking than as a correction to an existing one: it gives researchers a smaller, reviewed set and makes the effect of sample quality visible in the score changes.