Reinforcement learning (RL) post-training for large language models (LLMs) suffers from severe infrastructure inefficiency. In both synchronous and asynchronous setups, GPUs sit idle 40–60% of the time because sampling and training phases alternate sequentially, and even overlapping attempts leave trainer accelerators starved for fresh rollout data. The llm-d project introduces co-operative time-slicing to solve this structural waste. By treating discrete RL steps—sampling rollouts and gradient training—as dynamic, schedulable phases, the infrastructure can interleave independent RL jobs onto shared physical hardware. When Job A enters an idle phase (e.g., waiting for sampling to finish), the system checkpoints Job A’s device state to host DRAM and restores Job B’s saved state, swapping in its active phase. This checkpoint/restore mechanism avoids framework-level interference and out-of-memory faults because only one job’s state occupies the accelerator at a time. Initial benchmarks show aggregate accelerator duty cycles rising from a ~40% baseline to 70% without impacting model convergence or accuracy.

The architecture is organized into three layers. The workload-scoped layer is the application runtime, where the user’s code (training loops, inference servers, RL frameworks) runs unmodified. A time-slice client library exposes two gRPC APIs: acquire() to request exclusive accelerator access and yield() to release it. The user wraps any accelerator-touching phase with these calls. The cluster-scoped layer is the control and orchestration plane. It maintains a lock queue for jobs sharing the same physical accelerators; only the job at the head of the queue holds the lock and runs on the hardware. When the running job calls yield(), the orchestrator passes the lock to the next job and triggers a coordinated context switch across every node in the group. The node-scoped layer performs the actual checkpoint/restore. A snapshot agent, implemented as a privileged DaemonSet, receives directives from the orchestrator, pauses accelerator processes, serializes device state to host DRAM, and restores the swapped-in job’s state. The first backend uses cuda-checkpoint, and future backends will support faster snapshot mechanisms and selective memory offloading (e.g., LoRA adapters).

The flow works as follows: when a workload finishes its current accelerator phase, its client library calls yield(). The orchestrator instructs the snapshot agent on each node to freeze the yielding workload’s processes and move its device state from accelerator memory to host DRAM. Then the orchestrator grants the lock to the next waiting job, directs the agents to restore that job’s saved state from host DRAM back into accelerator memory, and unblocks the pending acquire() call. The workload resumes exactly where it left off—no container restart, no framework reinitialization, no model reload from storage. The yielding workload remains warm in host DRAM, ready to be swapped back in when it regains the lock.

The developer experience is designed to be minimal. Researchers using Ray or similar platforms to orchestrate RL jobs can adopt time-slicing with little to no client-side changes, especially if training and sampling jobs are already queued separately at the platform level. The release includes the full stack: the Snapshot Agent, the Accelerator Orchestrator, and Python client libraries, each with a user guide. The roadmap highlights latency and state optimization (faster checkpoint/restore backends, application-aware selective snapshotting), automated scheduling (profiling processes to identify time-sliceable structures and handle job placement dynamically), and cross-hardware compatibility (extending beyond GPUs to TPUs and custom accelerators).

The article also describes the broader llm-d stack for RL infrastructure efficiency, which includes throughput-driven inference (llm-d-router), a high-velocity agent sandbox (recipe) for secure sub-second code execution, and core pipeline primitives like the Weight Propagation Interface (WPI) for reliable weight transfer. The focus remains on eliminating accelerator idle time, which the co-operative time-slicing approach addresses directly. The authors note that this is not just a synchronous RL problem—asynchronous variants still suffer from idle windows because generation remains the bottleneck, and bounded staleness limits how far generation and training can drift apart. Time-slicing reclaims those fragmented idle gaps by dynamically interleaving jobs.