ITBench-AA Benchmark: Frontier Models Score Below 50% on Agentic SRE Tasks
The first benchmark for agentic enterprise IT tasks reveals a striking gap: frontier AI models score below 50% on ITBench-AA, a new evaluation for Site Reliability Engineering (SRE). The benchmark tests models on diagnosing live Kubernetes incidents by reading logs, tracing dependencies, and identifying root-cause entities across complex infrastructure. Despite advances in general reasoning, models still struggle with the precision and concision required for real-world incident response.
ITBench-AA, built by Artificial Analysis and IBM Research, includes 59 tasks with Kubernetes incident snapshots. Models run in a standardized open-source Stirrup harness with shell access and submit a JSON diagnosis. Scoring uses average precision at full recall: if a model misses any ground-truth root cause, it scores zero; extraneous entities penalize precision. Claude Opus 4.7 leads at 47%, followed by GPT-5.5 at 46% and Qwen3.7 Max at 42%. Turn counts vary nearly 3x across models, but longer trajectories do not correlate with higher accuracy—Gemini 3.1 Pro Preview averages 83 turns at 30%, while Gemma 4 31B (Reasoning) averages 58 turns at 37%.
For builders, the takeaway is clear: agentic SRE tasks reward concise root-cause identification over exhaustive investigation. Open weights models like GLM-5.1 (Reasoning) at 40% ($1.23 per task) and Gemma 4 31B (Reasoning) at 37% ($0.14 per task) offer strong cost-performance ratios, beating much more expensive models like Gemini 3.1 Pro Preview ($2.23, 30%). The recall-gated precision metric is unforgiving—models must not only find the correct entities but avoid false positives. This benchmark sets a new standard for evaluating models on real, constrained agentic workflows.
