How the Claude Code Team Thinks About Verification

On the Claude Code team, about 90% of code is written and reviewed by Claude. The team argues that verification loops are what make this scale possible: without them, you can run only one or two agents because every output needs human checking; with them, you can run dozens or hundreds in parallel. Internal testing around structured file generation found that using a separate grader to check Claude’s work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems. The economics of verification also shift: tests, benchmarks, and specs are now cheap for an agent to generate, and because they run as software they can be repeated without spending many additional tokens. The results can feed back into the system, for instance by turning a repeated failure into a hook or an explicit rule in CLAUDE.md.

Not all tests are worth equal investment. Flaky tests teach the agent to rerun rather than fix; slow tests reduce coverage; heavier tests should run before committing. The team prioritizes three kinds. End-to-end tests are hardest to fool because they exercise real user flows and have definitive done criteria; a unit test may report a timeout while an E2E test can show a screenshot of what broke. Agents can be given hands and eyes on the running system through a skill that boots, drives, and inspects the app using CDP, simulators, or API harnesses, with a feature map acting as guide and rubric. Property tests attack edge cases the original author did not anticipate: you state an invariant and let a library like fast-check, Hypothesis, or proptest generate thousands of inputs trying to break it. Useful properties include round-trip encoding, invariants such as a non-negative account balance, idempotency, and equivalence between optimized and naive code; they are especially valuable for rewrites where the spec has drifted from production behavior. Differential tests are for migrations: run old and new systems on the same inputs, normalize irrelevant differences, and maintain a list of approved changes.

On the Claude Code team, verification starts with running the change end to end, not just tests. Claude can test the whole app the way a human QA engineer would: Chrome extension for web, iOS/Android MCP for mobile, computer use for desktop, or running the service locally for backend. The team codifies this in a verify skill stored in .claude/skills/ near the relevant code; Claude is expected to use it before committing, and the team invests in making it fast and reliable as a smoke-test harness. Failing checks are a routing mechanism for human attention: Claude tries to fix failures and stops after three failed rounds to bring in a person. Code owners write their own checks and refine them until they flag what the owner would flag; when a check pulls in a reviewer, it should say why.

Reliability checks are tied to diff triggers. A 39-item checklist includes entries such as the one for –resume loading old transcripts; each item has a trigger, a check, and the incidents or PRs behind it. Claude reads the diff and only runs the 3 to 10 matching items, so no one has to memorize past outages. Reviews use a single primary reviewer guided by REVIEW.md instead of stacking more reviewers, and later rounds build on earlier context and past failures. A change cannot modify the check that judges it: Code Review reads REVIEW.md from the base or default branch, never from the PR’s changes. Reviews are reserved for commits, because a passing review on an out-of-date commit is meaningless. Cheap checks come first: Claude runs tests, build, and linter before opening a pull request, enforced with a hook or a /goal of a passing test suite.

New gates start in observe mode. The team stamps each commit with how it was verified, including the value none, and has a commit gate with off, observe, and enforce modes that can block commits lacking /code-review or /verify. Observe mode logs what would have been blocked to catch false positives; enforce mode blocks, but after two blocks in a session the next commit is allowed through so Claude does not get stuck. The team also evaluates the reviewers themselves, tracking reverts, rejected changes, and clean size-matched controls. The Code Review team’s formal evaluation recreates repos at review time, uses a one-sentence answer key, and scores only findings that correctly describe the bug; merged stable changes act as a control against over-flagging. They measure recall for raw findings, post-verification, and final review separately, and tag issues as Important, Nit, or Pre-existing. Recall is tuned only up to the point where further improvement would mean overfitting the eval.

Misses are converted into rules. One method finds patterns in human review comments and PR reverts that appear in at least two PRs and are identifiable from a diff; three agents try to disprove each candidate before a human approves it in a PR. The other replays three bug finders against a PR linked to a known incident; if at least two find the bug it is detectable, and a miss on a small diff becomes a gap worth adding to REVIEW.md. Flaky tests get the same treatment: Claude cannot skip them, add sleeps, or retry until they pass; if it cannot find the root cause, it deletes the test and records why. Rules and verification material are pruned routinely, since suites and .md files only grow if left alone. The payoff is shifting the human job from reading every diff to defining what done means and reviewing evidence, which is what makes many Claudes in parallel feasible.

How the Claude Code team thinks about verification

View Original