Using Surprisal to Map Agent Retrieval Performance

Standard pass/fail benchmarks tell you whether an agent cleared a bar, but they hide the terrain underneath. For data agents doing retrieval over enterprise warehouses, a single missing token can send a perfect score to zero without the benchmark registering it. The real question isn’t whether the agent passes, but how vague the question can get before it breaks — and that demands a map of capability, not a scalar verdict. The article exposes this tension clearly: a static exam stamps a query ‘solved’ when the agent is actually standing on a cliff edge.

The concrete path uses surprisal, an information-theoretic measure of how much uncertainty remains about the correct dataset given a query. The authors built Discovery Bench, a meta-benchmark that modulates difficulty by adding or removing high-information terms. Their iterative surprisal-based query refinement (iSQR) loop generates high, medium, and low ambiguity variants of each evaluation case, grounded in bits rather than subjective labels. On KramaBench, they found that dropping a single distinguishing token like “TLE” crashed F1 from 1.00 to 0.00 for the same query, agent, and ground truth. They also discovered a sweet spot where medium ambiguity outperformed both neutral and low ambiguity — more specificity is not monotonically better. And when they dug into the benchmark itself, they found broken ground truth: tables that didn’t answer the query, oversized shards, and date mismatches. The measuring stick was defective.

The takeaway for serious builders is to stop trusting evals as black boxes. Treat difficulty as a measured quantity, sweep it, and plot the curve. Your agent’s pass score may mask a cliff that a slightly vaguer phrasing exposes. The same lens should be turned on the benchmark itself: evaluate your evals. The authors are candid that difficulty-as-entropy is only as reliable as the entropy estimator, but retreating to pass/fail is worse. Build evaluations that produce signals, not just verdicts — and keep the evaluator under the same scrutiny as the agent.