Agent Seer, by Harish Karumuri, Mahesh Vemula, and David Lopes Pegna, is a pipeline for synthesizing evaluation scenarios for tool-using AI agents directly from a single Model Context Protocol (MCP) specification. It requires no example dialogues, no live tool access, and no domain-specific tuning. The authors argue that function names, natural-language descriptions, and typed parameter schemas in tool specifications contain enough semantic information to generate realistic, graded multi-turn test scenarios. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded dialogues that demonstrate strong tool-calling correctness and conversational coherence.

The pipeline was evaluated on seven MCP specifications covering diverse domains and tool-suite sizes. It achieved strong quality across all domains and complete tool coverage on small and medium specifications. Two findings stand out: parameter schema complexity is the strongest correlate of quality variation, while tool-suite size plays a smaller, orthogonal role; and argument value accuracy is the dominant failure mode among imperfect scenarios—a sub-dimension that coarse-grained name-match metrics miss. The work was accepted at the Fifth Workshop on Natural Language Generation, Evaluation, and Metrics at ACL 2026.