Daily curated AI insights you can't miss.
As agents evolved to include reasoning, tool calls, and multi-step loops, each capability jump broke the existing evaluation methods.
Deterministic checks catch definable behaviors, while LLM as a judge adds analysis that fixed rules cannot provide.
Agent as a judge discovers unknown failure modes and can automatically open pull requests to fix issues it finds.
Aparna Dhinakaran argues that agent evals must evolve as fast as the agents they grade. Early agents were little more than prompts, and deterministic checks sufficed. As agents gained reasoning, tool calls, and multi-step loops, each capability jump broke the previous eval. The response has been a progression: deterministic checks for definable behaviors, LLM as a judge for analysis fixed rules cannot provide, and now agent as a judge.
The newest step—agent as a judge—does not rely on human-defined rules or prompts. Instead, it hunts for failure modes a human would never think to check for and can open a pull request to fix what it finds. This creates a self-improving loop where one agent grades and repairs another. Dhinakaran sees this arc from static checks to an agent grading another agent as the natural next direction for evals, driven by the accelerating complexity of modern agents.
Aparna Dhinakaran argues that agent evals must evolve as fast as the agents they grade. Early agents were little more than prompts, and deterministic checks sufficed. As agents gained reasoning, tool calls, and multi-step loops, each capability jump broke the previous eval. The response has been a progression: deterministic checks for definable behaviors, LLM as a judge for analysis fixed rules cannot provide, and now agent as a judge.
The newest step—agent as a judge—does not rely on human-defined rules or prompts. Instead, it hunts for failure modes a human would never think to check for and can open a pull request to fix what it finds. This creates a self-improving loop where one agent grades and repairs another. Dhinakaran sees this arc from static checks to an agent grading another agent as the natural next direction for evals, driven by the accelerating complexity of modern agents.
