The real difficulty in building an agentic system is measurement, not construction. The author, who has run evaluation engineering initiatives at Google, describes the Google Cloud Developer plugin as a live testbed for stress-testing eval principles across real developer workflows. The post distills four lessons for setting up systematic, repeatable tests that reveal whether an agent actually behaves correctly.

The first lesson is to grade execution steps instead of the final summary. An LLM-as-judge that only inspects the agent’s final text reply can penalize correct behavior. In the plugin’s initial 11-prompt evaluation suite, the automated grader marked three prompts as failing, but reading the raw tool-call logs showed the agent had actually passed all three. On a Cloud Run cleanup prompt, the rubric required checking command syntax before running anything; the agent ran gcloud help run services list as its first step, but its final message focused on warning about data loss instead of mentioning that check. The grader guessed wrong because it never saw the tool-call trajectory. The takeaway: score the raw execution trajectory, not the closing pitch.

Second, evals must be designed for asynchronous operations. How telemetry is extracted matters as much as what is read. Flattening agent telemetry into a sequential list obscures concurrent tool dispatch, a common cause of behavioral failure. Modern harnesses often batch multiple tool calls in a single assistant turn. In one evaluation, the agent was asked to inspect an environment and look up documentation; the harness dispatched a skill-loading call and two documentation searches simultaneously. Because activating a skill is multi-stage, the skill instructions arrived after the searches had already executed, so the guidance never influenced the actions. If telemetry pipelines strip out turn structure, engineers will waste days trying to understand why an agent ignored explicit instructions embedded in a skill. Telemetry extractors should preserve turn structure.

Third, test live runs and plan-only runs side by side. Live runs use real sandboxed credentials and catch syntax issues, serialization problems, and whether the agent exercises restraint before destructive actions. But live runs often hit permission errors early and abort before reaching the behavior under test. Plan-only runs strip credentials and ask the agent to explain what it would do, exposing the model’s pure reasoning and routing intent. Roughly half the most important findings came from the delta between the two conditions. In one onboarding prompt, the live run was blocked immediately on a project list call, while the plan-only run produced an 8,000-character plan that ignored organization policies, admin approvals, and governance. The live run masked a major cognitive flaw that the plan-only run exposed in seconds.

Fourth, use golden sets as durable baselines. Frontier model architectures, agent harnesses, and plugin manifest standards all change, so the author treated evaluations as continuous integration testing for a non-deterministic environment. That means stress-testing the composite stack: the harness, the live model, and the tools together on realistic tasks. Instead of guessing from final text output, capture the whole trajectory, including what the agent thought, what commands it ran, and whether it solved the problem without breaking anything. Score those runs against golden sets curated by human experts who know the ideal path. Models change, but baseline runs serve as a safety net.

The post closes with a warning against one-shot demos, where a single prompt produces an impressive result and is treated as an eval. That is a vanity metric. Enterprise-grade agent work requires evaluation suites that test multi-step interactions, permission handling, and avoidance of destructive outcomes. Shipping a trustworthy agent plugin or MCP server requires significantly more evaluation work, and the only thing that outlasts model churn is a stress-tested eval suite.