The article tackles a reliability problem in LLM agents that standard benchmarks hide: a workflow that succeeds once may fail when the same request is repeated. The authors define the consistency gap as Mean@k minus Pass^k, where Mean@k is the average pass rate over k runs and Pass^k is the fraction of tasks that succeed on all k runs. On AppWorld test_normal, a ReAct agent using GPT-4.1 achieves Mean@5 of 77.4% but Pass^5 of only 53.0%, a 24.4-point gap. The gap widens on hard tasks, reaching 30 points.

To diagnose this, they build the Consistency Analyzer, which takes a single recorded trajectory and resamples each decision step by requesting k completions (default k=5) from the model, without re-running the task or needing ground truth. This produces a per-step consistency score, identifying flat probability distributions where near-tied tokens can flip under small perturbations. These flagged steps become consistency guidelines in the ALTK-Evolve pipeline, injected back into the agent’s context at inference time. The guidelines target instability rather than failure, so they catch steps that happened to be correct this run but could easily flip next time.

Results on AppWorld show the consistency gap is cut roughly in half: aggregate Pass^5 rises from 53.0% to 69.0%, while Mean@5 rises from 77.4% to 81.0%, narrowing the gap to 12.0pp. Nearly a third of previously inconsistent tasks become fully consistent. Medium and hard tasks gain the most in relative terms (+44% and +45% respectively), and Mean@5 never drops at any difficulty level, preserving average accuracy as a hard requirement. The guidelines generalize to similar tasks within the same scenario, lifting Pass^5 by +13.0pp, only 3 points below the same-task gain. Tests with a weaker model, gpt-oss-120b, also show improvement (+6.0pp same-task, +8.7pp similar-task), suggesting the method captures reusable failure patterns rather than memorizing a single trajectory.

The article concludes with practical advice for shipping agents: report Pass^k alongside Mean@k, expect the gap to widen with difficulty, and don’t reach for a bigger model first because consistency is orthogonal to capability. The diagnosis requires only one extra LLM call per decision step, making it usable on production traffic where end-to-end replay is often impossible. The open-source ALTK-Evolve toolkit now includes the Consistency Analyzer and consistency-guideline generation, and the full methodology is available in a technical report on arXiv.