METR Blog company blog
Many SWE-bench-Passing PRs would not be merged
We find that roughly half of test-passing SWE-bench Verified PRs written by recent AI agents would not be merged into main by repo maintainers. A naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more elicitation or human feedback.
- Points
- 278
- HN comments
- 153
- Submissions
- 1
- Cited by
- 0 commenters
Software EngineeringAI & Machine Learning2026
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Mar 11, 2026 | Many SWE-bench-Passing PRs would not be merged by mustaphah | 278 | 153 |
