Why we no longer evaluate SWE-bench Verified

Company news OpenAI Source language: English

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.

Excerpt

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.

Short excerpt from the source — open the original for the full text.

Sources

← Back to the daily brief