How Google Ships Agent Skills Like Products: Versioning, Evals, and AI Review

Two months after the first behind-the-scenes post, Google Agent Skills has grown from 15,000 to 21,000 GitHub stars and now hosts more than 170 skills from teams across Google, distributed across multiple AI platforms. The follow-up describes the infrastructure the team built to keep that repository reliable: automated quality gates, an AI review agent, and a set of lessons from six months of commit history.

The central principle is that a skill is a living product, not a one-off document. Documentation explains concepts to a human reader; a skill steers an agent’s reasoning loop, tool selection, and execution path, so a misplaced phrase can cause invalid command flags, infinite loops, or wasted tokens. Every instruction edit is therefore treated as a behavioral change. Before a skill is exported to GitHub, every pull request must pass six automated gates plus human approval: structural conformance, skill linters, semantic versioning, evaluation suite validation, link validation, and an AI review agent.

Structural conformance and linters enforce that folder names match metadata names, SKILL.md and description length limits are respected, required frontmatter fields exist, only curated file types are allowed, Markdown formatting is consistent, and the README and machine-readable index.json are up to date. Semantic versioning adds stricter rules: versions must be quoted strings in YAML, new skills start at 1.0.0, any behavioral change requires a bump, and standard reset rules apply for MAJOR and MINOR. The version is for developers and tooling, not for the model at runtime; skills do not auto-update, so explicit versions let developers compare local skills with the upstream catalog, run npx skills update, or lock critical skills to a known-good version.

Evaluation suite validation ensures that evals are trustworthy before they are used. The validator resolves file-based prompt references, enforces atomic expectations (exactly one criterion per expectation), flags duplicate prompts and conflicting expectations, requires an evaluation report for every skill change, and checks that the linked report ran against the latest commit. Link validation runs before merge for relative links and then weekly via a GitHub Actions workflow using lychee across external links, reporting broken links to skill owners.

The AI review agent uses a dedicated review skill to catch design anti-patterns. Its first rule is skill suitability: authors often fall into the deterministic trap, writing rigid CLI command sequences into Markdown, which invites hallucinations because LLMs are probabilistic. Rigid sequences, parsers, and deployment steps belong in executable scripts under scripts/; Markdown is reserved for judgment tasks. The second rule is the L1/L2/L3 modular architecture: concise metadata in frontmatter, core instructions in SKILL.md kept under 500 lines, and reference docs in a flat references/ directory with single-hop indirection only, because chained references add latency and agents often stop before following a second hop. The third rule is non-interactive script design: scripts must accept flags or environment variables, provide –help, include –dry-run for destructive operations, write structured data to stdout, and write errors to stderr. After all checks pass, a human subject-matter expert approves the change; new skills also need approval from the Central Skills team.

The repository history yields four takeaways. First, scope CI checks carefully: overly broad patterns once blocked a one-line typo fix for a missing bug-tracker ticket while newly added skills in deeper directories went uninspected. Second, use atomic expectations: compound expectations produced inconsistent LLM-as-a-judge ratings until split. Third, documentation drift happens fast, so index files should be generated and enforced by CI rather than edited by hand; index.json even carries a generated-file warning. Fourth, gate expensive LLM checks behind fast deterministic ones: structural checks, linters, and versioning block the pipeline before the AI reviewer and evaluation runs, and further cost guardrails such as manual triggers for heavy checks are worth considering.

The conclusion restates the operating model: ship skills like products, enforce semantic versioning and atomic expectations with CI gates, use an AI reviewer to catch deterministic-trap anti-patterns, and check links both before merge and on a schedule. The post also links to the google/skills repository, the previous post, the launch announcement, and a skills tutorial.

Behind the scenes, two months later: How we ship Google Agent Skills like products

View Original