Benchmarking open models on agentic tooling with transformers
Most agent benchmarks only check the final answer, ignoring the cost of getting there. This article argues that as coding agents increasingly drive libraries like transformers, library design must be evaluated not just on correctness but on agent effort: turns, tokens, time, and error paths. A clunky API or stale documentation now sends agents down longer, more expensive paths, and traditional evaluation misses this entirely. The authors wanted to expose how much work an agent actually does, and how that shifts across models, library revisions, and tasks.
To measure this, they built a harness called agent-eval that runs open models (like Kimi-K2.6, MiniMax-M2.7, Qwen3) against different tiers of tooling support: a bare pip install, a full source clone, and a packaged Skill (CLI docs + task examples). They fanned out runs across models × revisions × tasks on Hugging Face Jobs with identical hardware. The key metrics are match rate, median time, tokens (new vs. cached), and marker adoption—labeled behaviors like using the CLI vs. writing raw Python. A live report and per-run traces allow deep inspection.
The harness caught a critical tradeoff: adding a CLI + Skill commit to transformers reduced time for large models but hurt smaller ones. For example, Qwen3-14B’s sentiment classification match rate dropped from 100% (clone) to 0% (Skill) because it misinterpreted the Skill as a direct tool call. The takeaway is clear: agent-facing APIs must be tested across model sizes, because an affordance that helps strong models can confuse smaller ones. The harness is open-source and adaptable to any CLI-operable library.
