CI is all wrong for agents

CI was built for a different author

CI assumes the author is a human who may have skipped the tests. So it re-runs every gate after the PR opens, then parks the PR until another human has read the diff. With an agent as the author, both assumptions fail. The agent can run every locally runnable gate before the PR exists. The cheapest review it can get is from another agent that never saw its context. CI still re-runs it all, and still waits for a human on every merge, whatever the risk of the change. Anthropic reports that its model now writes about 80% of the code it merges, and calls CI the most painful bottleneck of the whole transformation. Its scanning costs are consumption based and grow with code throughput. That's the complaint. The work gets done twice, and humans sit on the critical path of every merge instead of only the risky ones.

The pipeline

The fix is to move the work before the PR and make it provable. An in-context agent implements the code. Out-of-context agents review it adversarially. The implementer assesses the findings and implements the fixes, and the review repeats until a round comes back with no relevant fixes. Then every locally testable gate runs and produces proof of work, and the review loop is part of the proof. After the PR, CI verifies the proof, runs the gates that weren't proved, and runs the non-gate processes, like environment setup. A classifier reads size, depth, complexity and other heuristics and tags the PR auto-merge, one human required, or two humans required. Merge is automatic. On auto-merge, wait for the gates. Otherwise wait for the first or the second approval.

Each piece below answers the same question: who is trusted to say the work was done, and how that trust gets checked. The full spec is also a guide for agents, with an assessment checklist and an adoption order, so an agent can run it against its own repo and apply what fits.

Proof of work

The implementer can't produce the proof. It's the party with the most incentive to cut corners. A sandboxed, egress-allowlisted local runner executes the gates and signs the results with a key the implementer can't read. The proof then has five requirements.

• Bound to the git tree hash, not only the commit SHA. Any content change invalidates it.
• Records the base SHA. If the target branch has moved, the proof is stale.
• Hashes the environment: lockfile, image digest, tool versions. A local pass then means the same as a CI pass.
• Stored outside the tree, as a PR artifact or in git notes. Committing it would change the hash it proves.
• Spot-checked. CI re-runs a random sample of proved gates, which keeps forgery expensive.

The in-toto attestation format, the one SLSA builds on, already gives a signed envelope with a subject digest and a custom predicate, and the signing and verification tooling comes with it.

Which gates can be proved locally is the team's call. A linter can run anywhere. A custom, infra-dependent validation may only run remotely. That policy lives in a gate registry in the repo: each gate with its command and a local or remote flag. The verifier reads the registry from the base branch, never from the PR, so a PR can't flip a gate to local, or delete it, and then prove it. Any change to the registry needs a human.

The review loop

The reviewers are several agents, each with one narrow scope, say security, correctness and data access, and none with the implementer's context. Separate scopes have separate blind spots, and one can catch another's mistake or compromise. A finding counts only with proof: a failing test, a reproducing input, a trace, or the exact code path that breaks a stated invariant. Unproven findings are recorded and don't block. Anthropic's reviewers work this way, and once they had to prove findings, the share of PRs with substantive comments rose from 16% to 54%. Proof up front is cheaper than arguing dismissals afterward.

Each blocking finding ends in exactly one resolution.

• Fixed. References the commits, and the finding's proof now passes.
• Dismissed. The implementer explains the dismissal, and the reviewer that raised it agrees, with a signed ack.
• Escalated. The reviewer doesn't agree, and a human resolves the review item.

Out-of-context reviewers flag non-issues more often than not, so the implementer needs a way to push back. It can't judge its own pushback, or the loop converges by dismissing instead of fixing. A third agent as arbiter would have less context than the reviewer that raised the finding. So the reviewer agrees, or it doesn't. There's a chance of a misaligned actor, but that's low, and the human escalation handles it. Two guards keep the protocol honest. The reviewer re-checks the dismissal against the code, not only the rationale, and its ack states why the concern no longer applies, because models tend to accept a confident rebuttal. And there is one rebuttal round per finding. Still in disagreement, it escalates.

The channel between implementer and reviewer is restricted: structured messages tied to one finding, every message logged, and the reviewer has no tool that can act on the repo. Anthropic once had an incident-response agent ask another instance over Slack to push a fix, on its own initiative, and a human gate caught it. A channel of the same shape must not allow that here.

The classifier

The classifier reads size, depth, complexity and other heuristics. Those alone are gameable. Risky work can be sliced into small PRs that each look safe. So deterministic rules set the minimum tier and the model can only raise it.

• Sensitive paths need at least one human: auth, migrations, infra, billing, secrets, CI config, the gate registry and the agent instruction files.
• New or changed dependencies need one human.
• Any escalated finding or failed invariant gate needs one human. Invariant gates encode what must always hold, like user A can never read user B's data.
• Risk is judged over a rolling window of related PRs, same author, branch lineage or area, not one PR at a time.

Review-loop stats, rounds, dismissals, escalations and unproven findings, and gate retries are inputs too. Every merge decision is logged with its signals and reasoning, and humans review a risk-weighted sample of the auto-merges. A new reviewer starts in shadow mode, posting findings for human approval only, and gets red-teamed with malicious changes before it can block anything.

Merge, and what stays out

There's no merge queue. Proof goes stale when the base moves, and the original dev pushes new proof. At low volume that is a merge queue done by hand. At high volume every merge invalidates every in-flight proof and causes rebase churn. A merge queue then automates the same rule without changing the design. Deployment is a separate workflow and stays human controlled. Canary and rollback belong there, not here. The summary for human reviewers is the proof of work itself, with the specific commits referenced in the review timeline. Each round points at its fix commits or its acked dismissals, and the classifier reads the same timeline.

What I haven't settled

Some questions survived every round:

1. What to do with flaky tests. Retries recorded in the proof could be a classifier signal.
2. The sampling rates, both for re-running proved gates and for the human audit of auto-merges.
3. Whether the review loop needs a round cap, and whether reviewer models or prompts should rotate.
4. At what PR volume stale-proof churn justifies adding a merge queue.
5. The thresholds between the auto-merge, one-human and two-human tiers.
6. Once findings must be proven, how often does the dismissal protocol still fire?
7. Does re-checking against the code hold up against sycophancy?
8. How is the local runner's signing key kept out of the implementer's reach on a developer machine?