A blog post from the team behind GPT‑5.6 Sol explains how two underappreciated API settings—retaining reasoning and enabling compaction—tripled the model’s score on the ARC‑AGI‑3 benchmark and cut output tokens by 6×. The model had initially scored only 7.8% on the benchmark, and GPT‑5.5 scored 0.4%. By examining the official ARC‑AGI‑3 harness, the team found that the harness discarded all private reasoning after each action and used a rolling truncation window that dropped older actions as the conversation grew. These two design choices meant the model could not remember its past thinking or see its earlier actions, forcing it to re‑interpret the game from scratch at every turn.

With the official harness, GPT‑5.6 Sol scored 13.3% on the public set, measured by Relative Human Action Efficiency (RHAE). The average human tester scored 48%. The team then switched to an implementation using their Responses API, which natively supports two features absent from the official harness. First, retaining reasoning: by passing the previous response ID, the model kept its private thinking messages across tool calls and turns. This allowed GPT‑5.6 Sol to spend less time re‑analyzing the game each turn and to develop coherent strategies over time. Second, compaction: instead of rolling truncation, which discards the oldest messages when the context exceeds 175,000 characters, compaction summarizes the conversation and continues. This preserved the model’s learned knowledge across longer runs.

Together, the two settings raised GPT‑5.6 Sol’s score from 13.3% to 38.3% on the public set, while reducing output tokens by 6×. The model also solved all six levels of the leaderboard game, whereas no frontier model had solved any level beyond the first with the official harness. The team notes that benchmarks rarely measure models in isolation; they also measure the harness design, API settings, and prompting choices. They recommend that API developers use the same settings deployed in ChatGPT and Codex: the Responses API, retained reasoning, and compaction. The post credits ARC for their work on AGI evaluation and for inspiring the closer look.