One agent reviewing its own diff agrees with itself. A worked pipeline instead: independent agents find candidate issues per dimension in parallel, a second pass of independent agents adversarially tries to refute each finding, and only what survives gets posted as a PR comment. Why agreement beats one long reviewer prompt.
why self-review is weak
The obvious way to get an agent to check its own diff is to ask it to. Right after it writes the change, prompt it again: "now review this for bugs." This is cheap and it is also close to useless, for a specific reason rather than a vague one.
The agent that wrote the diff produced it from a chain of reasoning it still holds in context: this variable is safe to reuse here because of an assumption made three messages earlier, this edge case does not matter because of a constraint stated in the original request. When you ask it to review, it reviews with that same reasoning still loaded. It is not lying when it says the diff looks correct. It is checking the diff against the beliefs that produced the diff, and of course they agree.
A reviewer that never held that reasoning does not have this problem. It reads the diff cold, with no memory of why a particular shortcut seemed fine at the time, and it either rebuilds a justification for the shortcut or it does not. When it does not, that is a real signal, not noise.
the shape of the pipeline
The pipeline described here has two passes, each made of independent agent runs, and a gate between them. A finding is not posted to the PR unless it survives both.
The two passes run different agents on purpose, not the same agent asked twice. Independence is the whole mechanism: a finding that one process invented and a different process could not talk itself out of is a stronger signal than a finding one process both invented and approved.
1. finding candidates, in parallel
Rather than one reviewer prompt covering correctness, security, and simplification at once, run three narrower agents in parallel, each with a diff and one dimension:
- Correctness. Does the code do what the diff claims it does. Off-by-one errors, null handling, wrong operator, a condition that cannot be reached.
- Security. Injection, missing auth checks, secrets in logs, unvalidated input crossing a trust boundary.
- Simplification. Duplicate logic, dead code, a function doing what a library call already does.
Splitting by dimension is not about running more agents for its own sake. It changes what each context window has to hold. A prompt asking for all three at once spends the same attention budget across three different postures, and a security-shaped read of a diff looks nothing like a simplification-shaped read of the same diff. Narrow prompts produce more specific findings than broad ones do, because there is less competing for the model's attention inside a single pass.
Each finding from this pass is required to state a concrete failure scenario, not a vibe:
{
"dimension": "correctness",
"file": "src/billing/charge.ts",
"line": 42,
"summary": "amount is not re-validated after currency conversion",
"failure_scenario": "a request with amount=100 and currency=JPY is
converted to USD cents, but the post-conversion value is never
checked against the minimum charge, so a sub-cent charge attempt
can be submitted to the payment provider"
}A finding without a file, a line, and a scenario is discarded before it reaches the second pass. This is a small filter and it removes a real category of noise: vague style opinions phrased as bugs.
2. trying to kill each finding
Every finding that survives the filter is handed, alone, to a fresh agent with a different job: not to review the diff, but to argue the finding is wrong. It is given the finding and the surrounding code, told explicitly to look for the reason the reported failure cannot actually happen, and told to say so if it finds one.
You are given a claimed issue and the code it refers to.
Your job is to try to prove the claim wrong.
Look for: a guard earlier in the call path that already handles
this case, a type constraint that makes the described input
impossible, a test that already covers this exact scenario, or
a misreading of what the code does.
If you find a real reason the claim does not hold, explain it
and mark the finding REFUTED.
If you cannot find one after genuinely trying, mark it SURVIVED.The instruction to argue against the finding matters more than it looks. An agent simply asked "is this finding correct?" tends to answer with a mild yes, because agreeing costs nothing and the finding was already phrased as if it were true. An agent told its job is specifically to find the counter-evidence looks harder, because it now has something concrete to search for.
This agent has no memory of writing the original code and no memory of producing the finding. Both of those absences are the point. It is checking a claim on its merits, not checking whether it agrees with an opinion it holds.
what survives, and what does not
Three outcomes are possible for a given finding, and only one of them becomes a comment.
That third row is deliberate and easy to get wrong. "I am not sure" from the refuter is not the same as a successful refutation, and it should not be scored as one. If the refuter did not genuinely try to find the counter-example, the finding has not actually been tested and should not be posted on the strength of an absent objection. In practice this means the refutation prompt above asks for the reason, not just a verdict, and a verdict with no reasoning behind it is treated the same as no verdict.
Worth being honest about the failure mode this does not remove: a finding that is wrong for a subtle reason neither the finder nor the refuter happens to check will still get through. Adversarial refutation raises the bar a single opinion has to clear; it does not make the bar infinite. It is a filter, not a proof.
the cost, honestly
This pipeline is more expensive per PR than a single review pass, in a way that is worth stating in plain numbers rather than glossing over. As an illustration, not a measurement: three finder agents plus one refuter per surviving finding is, for a diff that produces say six candidate findings, ten separate agent calls against a diff that a single-pass reviewer would have read once. If each call costs roughly the same as reading the diff once, that is on the order of ten times the token spend of one reviewer pass.
Whether that is worth it depends entirely on what a missed bug costs versus what ten review calls cost, and that tradeoff is not the same for a hobby project and a payments system. The pipeline is not the right default for every diff. It is a reasonable default for the diffs where being wrong is expensive: anything touching money, auth, or data deletion is a natural place to spend the extra calls, and a straightforward refactor is a natural place not to.
building this yourself
None of the pieces here require special infrastructure. The finder pass is three ordinary agent calls run in parallel, each with the diff and a one-paragraph description of its dimension. The gate is a small filter checking that a finding has a file, a line, and a scenario, which can be enforced with a JSON schema on the finder's output. The refuter pass is one agent call per finding, run independently, with the refutation prompt above. Posting is a loop over whatever findings come back marked SURVIVED.
- Keep the passes on separate context. The refuter must not see the finder's reasoning, only its claim. If it inherits the finder's context it inherits the finder's confidence too, and the independence is gone.
- Require a reason for every verdict, not just a label. A REFUTED or SURVIVED with no explanation is not auditable, and you will want to read the refutation reasoning the first few times to check the filter is behaving sensibly.
- Log what gets dropped. Findings refuted with a weak reason are the signal that tells you whether the finder prompts need narrowing, not just the ones that survive.
- Scope which diffs get the full pipeline. Running ten calls on every one-line typo fix is waste; reserve it for the diffs where a miss is expensive.
The result reads, from the PR author's side, like a reviewer who only speaks up when they have actually checked. That restraint is not a personality trait of the model. It is the adversarial pass doing the checking that a single prompt would have skipped.