This blog post introduces a benchmarking harness for evaluating how well coding agents can use software libraries, using Hugging Face‘s transformers library as a case study. The authors argue that traditional benchmarks, which only check final answers, miss crucial information about the cost and effort an agent expends. Two agents might reach the same correct result, but one could take a 40-line Python script with debugging while the other uses a single CLI command. The harness measures not just success rate, but also median time, token usage (cached, generated, new), error rates, and a novel concept called ‘markers‘ — named patterns that capture specific agent behaviors such as invoking a CLI or using a high-level API.

The harness runs each task under three tiers: bare (pip install transformers only), clone (full source checked out), and skill (packaged documentation and task examples loaded in context). Every run is a Hugging Face Job on identical hardware, with results stored in a Bucket. The authors benchmark across two axes: for large open models (e.g., Kimi, GLM, MiniMax), they fix the model and vary the transformers revision to measure effort; for small models, they fix the revision and sweep model sizes to measure accuracy and cost.

Results show that adding a dedicated CLI and a Skill (documentation package) to transformers reduces median time for large models but increases token consumption on the clone tier because agents read the new CLI source code. For small models, the change can be harmful: Qwen3-4B sees median new tokens jump from ~2.4k to ~23k with no accuracy gain, and Qwen3-14B’s match rate drops from 67% (bare) to 43% (skill) because it mistakes the Skill for an executable tool and gives up. The harness thus reveals a tradeoff that maintainers would otherwise miss — a change that speeds strong models can confuse weaker ones.

The authors emphasize that the cost of reading the CLI is amortized over multiple tasks in real usage, so the measured token bump is a worst case. The harness is released as a CLI tool called agent-eval, intended for local use with security precautions. It is profile-based and can be adapted to any library by defining tasks and expected answers. The post concludes that this kind of fine-grained evaluation is essential for optimizing libraries for agentic use, and that it caught a surprising regression that would have been shipped on faith.