Accelerating the Software Factory: Code Review for an AI-Native Team

Bright curved horizon of a planet glowing against the dark backdrop of space.

Being AI-native isn't about which tools you use. It's a habit of asking, every week, what's standing between our customers and more value, and then removing it. When agents took over writing our code, the answer turned out to be code review.

The day we closed the IDE

Earlier this year, something changed in how our engineers work. With Opus 4.6, agents stopped being a smarter autocomplete and started doing the work end to end. An engineer describes the change, an agent writes it and opens the pull request. Most of us stopped opening an IDE at all.

The effect on output was immediate. We now open more than 3.5 times as many pull requests per week as we did in January, and the curve is still getting steeper. From September to October alone, weekly volume grew about 1.6 times.

Pull requests opened per week, compared to January

But every one of those pull requests still waited for a human to approve it. We wrote new detections, fixes, and features in minutes and then they sat in a queue. What slowed value on its way to customers was no longer writing code. It was reading it.

What code review is actually for

Before trying to speed review up, we asked what it's for. We landed on two goals.

Shared knowledge. Review is how a team keeps a common picture of the system: what it does, how it's changing, and why.
Protecting customers. A second pair of eyes is the last check before a change reaches production.

Neither goal needs a human to read every change. The changes worth learning from together are the ones that touch core flows, data access, or security. Nobody learns much from a dependency bump. Protecting customers comes down to how far a mistake could reach. A change that could affect a customer deserves an expert's full attention. A copy fix doesn't need the same depth.

So we reframed the problem. The goal was never faster review. It was review in proportion to risk: experts spend their attention on the changes that matter, and everything else moves to customers without waiting in line.

Two lanes

Every pull request has to answer two questions before it merges. The first is is it correct? CI runs, and an AI code reviewer reads the change looking for bugs. The agent that wrote the change fixes whatever it finds and pushes again, usually before a person ever looks. The second is how far could it reach if we're wrong? That answer puts the change in a lane: Low is approved automatically once its safeguards pass, and High waits for an expert.

The path every pull request takes

The safeguards are deliberately simple. A Low change still needs a clean AI code review, a size a person could comfortably review, and an author with a track record in the repository. Anything touching authentication and permissions, database migrations, or CI and deploy configuration always goes to a human. Every approval is tied to the exact commit being merged, and a new push withdraws it.

The hard part is telling the lanes apart. We started by hand-labeling a sample of real pull requests. Simple rules based on file paths and size would have cleared only a small slice of them. Human judgment found about nine times as many changes that were safe to ship. That gap is judgment, and it's why we built an AI judge instead of a longer list of rules.

Changes found safe to ship, relative to path and size rules

Building a judge we can trust

The AI scores, the code decides

When a pull request opens, an agent investigates it. It reads the change, follows it through the rest of our codebase, and checks the state of related feature flags. Then it scores the change's customer impact on several dimensions, including customer experience, security, data access, and reliability.

The agent never makes the final call. Fixed rules in code turn its scores into Low or High, so the same scores always produce the same verdict. When someone disagrees with a verdict, the conversation is about a specific score and the evidence behind it.

We also treat the pull request itself as untrusted input. Its description is a claim, not evidence, and the rules the judge applies are read from the main branch. Nothing inside a pull request can change how it gets judged.

An eval with zero false negatives

To know whether the judge was good enough, we built our own eval set from real pull requests in our repositories, each labeled "fine to ship" or "a human must see this."

We decided early which mistakes mattered. A risky change slipping through is the only error that can hurt a customer. A safe change sent to a human just costs a few minutes. So the bar was simple: every change our engineers had tagged for human review must land High, on every run.

Then we iterated. Every change to the agent ran the full eval in CI. We read the reasoning behind each miss, fixed the guidance, and ran it again. Because a model can answer the same input differently, we only trusted a result once it held across repeated runs.

We stopped when the judge matched every case our engineers had tagged for review, run after run, while still sending roughly two thirds of changes to the Low lane.

Shaping the work to fit

When agents write most of the code, you can change how the work is split, not only how it's reviewed. We built a planning skill that breaks each feature into a stack of small changes that can each clear the gate on their own, and gathers every risky line into one focused change for an expert.

You get a focused PR with all of the risky parts.

Customers get the low-risk parts as soon as they're ready, and the expert review is better because it's concentrated where it counts.

What our customers get

In the repositories behind the gate, about 1.5 times as many changes reach production each week. The median time from opening a pull request to merging it is about 3.4 times faster, and twice as fast for small changes. Nearly half of all changes now flow through the Low lane, and the revert rate hasn't gone up.

The less visible change matters just as much. Our reviewers now spend their attention on the High changes, which is exactly where both goals of code review are served.

The next bottleneck

Agents changed how fast we can build. Matching review to risk is how that speed turns into value in our customers' hands, with expert attention still on every change that could affect them.

Review was this year's bottleneck. Being AI-native means we're already looking for the next one.