ALTK-Evolve is a prompt-level memory method: it mines behavioral guidelines from an agent’s own training trajectories using GPT-4.1 and appends those guidelines to the system prompt. Applied to τ²-bench, Sierra’s customer-service benchmark, the approach lets a stock ReAct agent using the older, non-reasoning gpt-4.1 reach 67.5% Pass^4 on retail — above every retail entry in the October 12, 2026 leaderboard snapshot, including RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data. No weights are updated and no human annotation is needed; the same model performs guideline extraction and acts as the agent.

τ²-bench pairs the agent with an LLM-simulated customer who has a goal, gives it domain tools and a written policy document, and counts a task as solved only when the final database state matches the expected outcome — a strict binary reward. The authors use τ²-bench‘s own ReAct scaffold with a two-phase THINK→ACT loop, gpt-4.1 as the agent model, gpt-5.5 as the customer simulator, and four trials per task. The only modification is a plain-markdown list of learned guidelines inserted after the policy. Sample rules distilled from failures and successes include authenticating the user before any order-related action, requiring explicit user approval before database-updating actions, taking one action per step, and cleaning a name before looking a user up. Each guideline carries a support count — the number of distinct training tasks it was mined from — and the paper compares baseline, a high-support core (support ≥ 3), and a lossless set that keeps every guideline after consolidation.

On held-out retail tasks, the lossless configuration raises mean accuracy from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5%. On held-out airline tasks, lossless raises mean accuracy from 57.5% to 65.0%, but the strict Pass^4 is highest with the compact sup3 core (45.0%) rather than lossless (40.0%). The optimum guidance amount is domain-dependent: retail wants the full playbook, while airline reliability prefers a smaller, higher-support set. The cost of guidelines shows up in input tokens: total LLM cost per conversation rises about 65% in retail ($0.084 to $0.139) and 32% in airline ($0.081 to $0.107). The agent does not talk longer — on matched runs solved under both baseline and guidelines, lossless slightly reduces assistant turns (retail 10.59 → 10.16; airline 9.15 → 8.82), so end-to-end latency improves a little and the input overhead can be largely reclaimed with prompt caching, since the guideline block is identical across calls.

The authors also check whether learned guidelines only patch a weaker model. Rerunning airline with gpt-5.5 as both agent and guideline extractor, with guidelines re-mined from the agent’s own runs, none of the extracted guidelines reached support ≥ 3 (support tops out at 2), so the high-support core is sup2. The guidelines still help: accuracy rises from 72.5% to 82.5% and Pass^4 from 55.0% to 70.0% (+15 pts) — a larger reliability gain than with gpt-4.1 on the same tasks, from a much higher floor. The lossless configuration reproduced exactly across reruns, while the baseline fluctuated.

On the full base split (train ∪ test) required by the leaderboard, the gpt-4.1 retail agent with the lossless playbook finishes #1 at 67.5% Pass^4, ahead of every frontier model and the fine-tuned RAFT-30B-A3B. On airline, the gpt-4.1 agent with the sup3 core reaches 44% Pass^4 (8th), beating GPT-5.2 without reasoning and nearly matching Claude Sonnet 4.5; the gpt-5.5 agent reaches 82.0% and tops the field, whose frontier entries sit at 66–72%. The overall result is that agents can be improved without weight updates — prompt-level, automatically extracted guidelines bring an older non-reasoning model into frontier company on retail and push a frontier model to the top on airline, with no test-set leakage.