Separating signal from noise in coding evaluations

Accurate measurement of model capabilities is crucial for safety and deployment decisions, but widely used coding benchmarks can mask serious flaws. OpenAI’s investigation of SWE-bench Verified revealed fundamental design and contamination issues, making it unreliable. They then turned to SWE-bench Pro, which was designed for longer-horizon tasks and saw model pass rates jump from 23.3% to 80.3% in eight months. However, a subsequent audit uncovered that roughly 30% of SWE-bench Pro tasks are broken. This undermines the signal these evaluations are supposed to provide, directly affecting safety cases and research priorities.

The audit used a multi-stage pipeline. An automated filter flagged 286 potentially broken tasks based on instructions, model attempts, and test failures. This subset then underwent two parallel reviews: a human-supervised agent review using Codex-based investigator agents that could run tests and inspect code, and a human annotation campaign where five experienced software engineers independently reviewed each flagged task. The engineers identified 249 broken tasks (34.1%), with issues falling into four categories: overly strict tests that enforce specific implementation details, underspecified prompts that omit requirements enforced by hidden tests, low-coverage tests that let incomplete fixes pass, and misleading prompts that point toward wrong behavior. Human reviewers found more broken tasks than the automated pipeline, especially in low-coverage tests.

For builders, the takeaway is clear: treat benchmark numbers with healthy skepticism. OpenAI retracted its earlier recommendation to adopt SWE-bench Pro, acknowledging that curation is extremely difficult when tasks are sourced from real pull requests that were never designed as clean evaluation units. The work also shows that agentic systems themselves are becoming useful for scalable benchmark validation—the same models being tested can help audit the tests. Serious evaluation requires benchmarks that are hard to game, easy to trust, and genuinely reflective of capability. This episode underscores the need for the community to develop benchmarks built explicitly for evaluation, with human oversight throughout.