Half the AI-written code that passes your eval still gets rejected by a human reviewer.
METR had real maintainers review 296 AI-written pull requests that had already passed SWE-bench Verified. Merge rates came in about 24 points below the automated score. A green checkmark was never the same thing as ship-ready.
On March 10, 2026, METR published a short research note that should bother anyone who has shipped a feature because the tests turned green. They asked four active maintainers from scikit-learn, Sphinx and pytest to review 296 pull requests written by AI coding agents, from Claude 3.5 Sonnet up through Claude Sonnet 4.5 and GPT-5. Every one had already passed the SWE-bench Verified benchmark. The maintainers would have rejected roughly half. On average, merge rates landed about 24 percentage points below what the automated grader reported.
Maintainers gave specific reasons. Code quality was the most common one for the newer models, things like style and following the repo's own standards, and METR found GPT-5 notably weaker there than the Anthropic models. After that came patches that broke other code, and patches where the core fix was wrong in a way the tests never caught. None of those show up as a failed test. They show up when someone with context reads the diff.
METR is careful about what this does and doesn't mean, and the caveats are worth keeping. The agents got one attempt with no feedback, while human developers iterate. Better prompting could close some of the gap. And the original human-written patches only got merged about 68% of the time, so maintainers are tough graders in general. METR also calls the note an early update with less review than its full papers. That said, a 24-point gap on code that already passed is a lot to explain away.
This is a familiar trap. Once "pass the eval" becomes the target, the pass rate stops being a reliable stand-in for quality. Running QA on GenAI data pipelines, I saw the data version of this. A batch reporting full coverage tells you the job finished. It doesn't tell you the labels underneath were right. A coding agent clearing SWE-bench tells you the tests ran. It doesn't tell you a maintainer wants the patch.
So keep the automated eval, and treat it as a filter instead of a finish line. Every release, pull a sample of eval-passing outputs and have the person who actually owns the merge or ship decision score them. Track that human acceptance rate as its own number, next to the automated pass rate, and watch the gap between them.
When the gap widens, the eval is what's broken, not the reviewers slowing things down. Fix it like any test suite with a blind spot. Pull the rejected cases back in as new eval cases, the same way you'd add a regression test after a production bug. Do that every release and the automated number starts meaning what everyone already assumed it meant.
This is the same idea I wrote about when acceptance criteria became evals. An eval is a written-down guess at what good looks like. It needs checking against the people who live with the result, or it drifts into measuring whatever is easiest to pass.
An automated eval tells you the code ran. The person who has to maintain it tells you whether it should have.