Back to work

A team quality gate built on reviews and AI

I reviewed over 740 teammate PRs and stopped defects before launch, then turned the knowledge built up in reviews into the team's AI agent rules. I investigate together with AI, but confirm conclusions with code and evidence before applying them.

Situation

As AI coding agents made it possible to write code quickly, the bottleneck moved to review and verification. AI can be wrong in convincing ways, so we needed ways to check that keep up with the faster pace.

Decisions

Reviews: As a standing reviewer on the team, I have reviewed over 740 teammate PRs, and nearly half of the comment threads I opened led to actual fixes. I do not stop at the server diff. Together with AI, I also check client code, design docs and real data. A large guild feature had been merged across five PRs, so I combined it into one diff and reviewed it as a whole with AI, checking interactions between features that are hard to see at the PR level, which led to follow-up fixes.

Agent rules: I analyzed over 2,000 of the team's past review comments and its code patterns with AI agents, and distilled the recurring feedback into the team's AI agent rules and recipes for each task type. I fact-checked the draft with a separate agent and fixed 14 errors, and left out items where reviewers disagreed.

Release and bot reviews: After launch, for each release, I open a review-only PR and run human and AI reviews on it together, and I maintain the AI review bot a teammate introduced. I limited the shell commands the bot can run to read-only ones.

Finding causes with AI: Once, the CI load test failed on every branch when only the date had changed. I narrowed down the cause together with AI and confirmed that the development logger was using up all the CPU on logs that had grown larger because of content that opened that day. I fixed it by switching to the same logger as production. An earlier change of mine had worked around the failure by lowering the load. I stated in the PR that it had misidentified the cause, and reverted it.

Result

I state where AI results come from and check the evidence before applying them. Once I applied only 4 of the review bot's 6 findings and rejected 1 with reasons. In a release review, I confirmed the design intent behind items the AI rated as high severity, and dismissed them.