REVERSAL-BENCH addresses a core limitation in autonomous reinforcement learning: continuous policy training without external resets. The authors observe that existing reset-free RL paradigms implicitly rely on environmental reversibility—the ability to undo state changes—which is absent in real-world manipulation tasks such as pushing objects off a table or spilling granular substances. To study this gap, they introduce a benchmark that controls reversibility through a continuous parameter ρ ∈ [0, 1] and provides a reset oracle, a ground-truth verification mechanism for state recoverability. The benchmark spans eight manipulation settings across five physics engines.

Using this setup, the authors evaluate a broad spectrum of policy architectures, including standard actor-critic algorithms, safe RL methods, and specialized reset-free frameworks. They report a sharp reversibility cliff: as ρ increases, reset-free agents are consistently absorbed into irrecoverable states, while episodic agents maintain steady learning. This failure mode appears across autonomous reset-free baselines and constrained RL approaches. Because reset-free agents lack external resets, any transition into an irrecoverable state results in permanent absorption, halting further learning. The authors confirm that this absorption phenomenon persists in full physics simulations under learned manipulation policies.

To isolate the cause, they compare against geometrically identical reversible counterparts, demonstrating that the breakdown is causally driven by irreversibility rather than obstacle complexity. The paper releases the benchmark suite, a large multi-simulator dataset labeled with recoverability, and the reset oracle. They also evaluate a safety shield that intervenes before irreversible failures occur. Results show that while recoverability can be predicted accurately, active recovery primarily succeeds only when the agent can physically steer clear of the trap. This suggests that prediction alone is insufficient; the agent must retain the ability to avoid irrecoverable states in the first place.