Daily curated AI insights you can't miss.
Agent evaluation should be a continuous rollout process, like software deployment.
Harbor closes the loop of evaluation, sandbox, and optimization, enabling agents to learn automatically.
Terminal-Bench specifically tests edge cases in the rollout of terminal interaction agents.
In this talk, Alex Shaw and Ryan Marten of the Laude Institute argue that the core primitive for building capable AI agents is the rollout — a structured interaction trace in a controlled environment. They present this as a unifying concept that spans agent evaluation, data generation, and optimization.
They introduce Harbor, a framework for running agents in sandboxed environments (Docker containers) and capturing rich rollouts. A rollout logs every step: tool calls, outputs, environment state changes, and reward signals. Harbor provides a standardized interface for replaying, inspecting, and scoring these rollouts. The environment gives the agent a private filesystem, network, and shell, making it suitable for tasks like code editing, bash scripting, and web browsing.
Terminal-Bench is a benchmark that uses Harbor to create reproducible evaluation rollouts. It focuses on terminal-based software engineering tasks — e.g., editing a config file, running a test suite, or diagnosing a build failure. Each task has a ground-truth solution and a reward function that checks the final environment state (e.g., ‘does the server start and respond correctly?’). The speakers emphasize that Terminal-Bench is designed to measure real task completion, not just token-level accuracy.
Beyond evaluation, rollouts are the training data for optimization. Shaw and Marten describe how captured rollouts can be fed into RL algorithms: successful rollouts become positive examples, and failures become negative ones. They show a loop: run agents in Harbor -> collect rollouts -> score them -> train a better policy -> repeat. The framework supports both online (training the agent in the loop) and offline (curating a dataset from past runs) workflows.
A key theme is that rollouts make agent behavior inspectable and debuggable. Because every action and its effect is logged, a developer can replay a failure, examine the exact tool output, and tweak the agent’s prompt or reward function. This is presented as a necessary condition for pushing agent reliability beyond one-shot demos.
The speakers also discuss OpenThoughts-Agent, a companion project that releases pre-trained agent trajectories and reward models, aiming to lower the barrier for reproducing and extending their results. They note that all projects are open-source and that the framework is designed to be environment-agnostic, accepting custom sandbox definitions.
Throughout the talk, Shaw and Marten stress that the rollout paradigm is not just a research idea — it is already the operational core of their engineering workflow. They encourage the audience to adopt rollout-driven iteration as a standard practice for building agents, rather than treating evaluation and training as separate disciplines.
In this talk, Alex Shaw and Ryan Marten of the Laude Institute argue that the core primitive for building capable AI agents is the rollout — a structured interaction trace in a controlled environment. They present this as a unifying concept that spans agent evaluation, data generation, and optimization.
They introduce Harbor, a framework for running agents in sandboxed environments (Docker containers) and capturing rich rollouts. A rollout logs every step: tool calls, outputs, environment state changes, and reward signals. Harbor provides a standardized interface for replaying, inspecting, and scoring these rollouts. The environment gives the agent a private filesystem, network, and shell, making it suitable for tasks like code editing, bash scripting, and web browsing.
Terminal-Bench is a benchmark that uses Harbor to create reproducible evaluation rollouts. It focuses on terminal-based software engineering tasks — e.g., editing a config file, running a test suite, or diagnosing a build failure. Each task has a ground-truth solution and a reward function that checks the final environment state (e.g., ‘does the server start and respond correctly?’). The speakers emphasize that Terminal-Bench is designed to measure real task completion, not just token-level accuracy.
Beyond evaluation, rollouts are the training data for optimization. Shaw and Marten describe how captured rollouts can be fed into RL algorithms: successful rollouts become positive examples, and failures become negative ones. They show a loop: run agents in Harbor -> collect rollouts -> score them -> train a better policy -> repeat. The framework supports both online (training the agent in the loop) and offline (curating a dataset from past runs) workflows.
A key theme is that rollouts make agent behavior inspectable and debuggable. Because every action and its effect is logged, a developer can replay a failure, examine the exact tool output, and tweak the agent’s prompt or reward function. This is presented as a necessary condition for pushing agent reliability beyond one-shot demos.
The speakers also discuss OpenThoughts-Agent, a companion project that releases pre-trained agent trajectories and reward models, aiming to lower the barrier for reproducing and extending their results. They note that all projects are open-source and that the framework is designed to be environment-agnostic, accepting custom sandbox definitions.
Throughout the talk, Shaw and Marten stress that the rollout paradigm is not just a research idea — it is already the operational core of their engineering workflow. They encourage the audience to adopt rollout-driven iteration as a standard practice for building agents, rather than treating evaluation and training as separate disciplines.
