METR is a nonprofit that studies the most advanced AI agents to understand whether their developers are on track to lose control of them.
Visit metr.org → RSS feedOn Hacker News Jul 2025 – Sep 2026
AI & Machine Learning 5Society, Policy & Culture 1Security & Privacy 1Software Engineering 1Open Source 1Career & Management 1
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.
We find that roughly half of test-passing SWE-bench Verified PRs written by recent AI agents would not be merged into main by repo maintainers. A naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more elicitation or human feedback.
We propose measuring AI performance in terms of the length of tasks AI agents can complete. We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. Extrapolating this trend predicts that, in under a decade, we will see AI agents that can…
Two METR staff members and a Redwood Research contractor investigated an incident in which OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned message board.
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.
Our measurements of the time horizons for public frontier language models.