Anthropic has added eval design and hillclimbing workflows to the claude-api skill in Claude Code. The skill exposes two commands: /claude-api build-eval creates an evaluation inside your codebase, and /claude-api hillclimb improves your application against that evaluation one change at a time while using a held-out set to catch overfitting. The article first lays out eval design principles, then describes how the skill operationalizes them, and closes with two examples.

The core eval design advice is to make eval tasks mirror production. Sampling tasks only because they are easy to generate or grade can produce a distribution that does not represent what matters. A good eval also has headroom: the most capable model at the highest effort should be well below 100%, since otherwise changes in the system cannot be reliably measured. The gap should not come from impossible or ambiguous tasks—a task that fails on every run regardless of replicates is a tell. Run-to-run variance should be low, and variance often comes from ambiguous tasks, graders that change verdicts on identical output, inconsistent effort settings, or leftover environment state such as files or git history that hand the agent the answer. The article also warns against adversarial sampling by picking cases the current model fails; that measures a model’s failure fingerprint rather than intrinsic difficulty. Instead, pick cases because a human judged them hard, and ground examples in production traffic, bug reports, and tickets, while remembering that user traffic can skew easy because users try what they expect to work.

The build-eval command turns these principles into an interactive workflow. Claude interviews you, builds the eval, and pauses for approval at specific points. Inputs are sampled in a fixed order: production transcripts (after asking about retention and sensitive data), bug reports and support tickets, five to ten hand-written cases, and cases synthesized from your codebase anchored in real examples. The skill shows every input for confirmation. For grading, it proposes the cheapest grader that fits the output: programmatic checks for constrained outputs, and an LLM-as-judge for open-ended outputs where a second reader scores against a rubric written as checkable claims rather than a 1-to-5 scale. If a baseline exists, the judge reads both answers in random order without knowing which is the baseline. The user picks the judge model, which should not be the model under test, and Claude grades a handful of cases and asks if you would score any differently. The skill then reports evaluation-set size and estimated time, runs the baseline, and prints a score with a confidence interval. Outputs include cases, grader, runner, one JSON line and full transcript per case, and a plain page listing per-case scores. During baseline runs, Claude checks grader consistency, infrastructure noise such as timeouts or API errors, and headroom—if the baseline is already around 95% or higher, it warns that hillclimbing should target cost or latency instead of quality.

Hillclimbing is positioned as a way to tune cheaply modifiable and attributable surfaces. Text such as prompts and skills is easy to change and revert, while open-ended harness changes are expensive and hard to attribute. A well-scoped objective matters: open-ended improvement requests on a saturated eval tend to stall, so cost reduction at parity is often a strong target. To avoid overfitting, the article recommends splitting cases into train and test, never pasting failure content into the prompt, and keeping answers structurally out of the model’s reach.

The hillclimb command operationalizes those principles. Before starting, Claude asks what to optimize—for example performance, or cost while holding performance—and splits the eval into random test and train sets. It checks that eval noise is smaller than the smallest improvement you would act on, and suggests more repetitions or cases if it is not. Each round, Claude reads the previous round’s train transcripts, proposes one change as a patch, runs the eval, and accepts or reverts based on train and test results. If train improves but test stays flat, it suspects overfitting and reverts. When the score stalls for two or three rounds, Claude reads remaining train failures and sorts them by root cause rather than making another edit; it also does this early if no single fix could exceed noise. This reflection step can surface ambiguous cases, harness bugs, or run-to-run variance, and only legitimate failures are used in further rounds. At the end, Claude leaves the code at the version that scored best on the test set, reports the test result against baseline with confidence intervals, and says so if the gain is within noise and recommends against merging.

Two examples illustrate the workflows. On an internal customer support benchmark with 44 tickets (30 used for search, 14 held out), the hillclimber started with Opus 4.8 at high effort at 74.4% accuracy and 4.6 cents per ticket. After auditing the prompt and moving to Opus 5.5 on low effort, accuracy rose to 87.8% at 1.9 cents per ticket. Stepping down to Sonnet 5 at low effort kept accuracy around 88.9% at about 1 cent per ticket, and prompt improvements with routing rules and a refund-cap cross-reference brought it to 98.9% at roughly the same cost. On the 14 held-out tickets, the final configuration scored 90.5% versus the original 78.6%, at about one fifth of the cost. In the second example, the claude-api skill itself was evaluated against an eval set derived from the documentation. Before hillclimbing, the skill scored 66%. Adding sections for eight missing features improved it to 74%, fixing C# and Java type tables to 77%, and a reflection step that found the skill content was present but Claude relied on older API shapes led to a migration table near the top of the skill, pushing the score to 80%. Fixing flawed tasks and grader instructions brought performance to about 88%.

To get started, run claude update first because the claude-api skill ships inside Claude Code and updating gets the latest commands. Then run /claude-api build-eval to generate an eval set with your approval on examples and grader, or /claude-api hillclimb to improve an existing eval with overfitting checks and eval-bug detection.