On the AI Infrastructure team at Ai2, the goal for large distributed training workloads is framed as a pyramid of four metrics: availability, occupancy, impact, and utilization. This post focuses on impact—choosing the most valuable workloads—for a fleet of thousands of NVIDIA H100, B200, and B300 GPUs in clusters of 88 to 1024 GPUs serving about 150 internal researchers. Demand consistently exceeds supply by 2-3x, so roughly 2-3 research workloads compete for every available GPU hour.

The previous priority-based scheduler let workloads opt out of preemption and gave each team a limit on concurrent protected GPUs. This produced predictable pathologies: GPU squatting via parked no-op workloads, priority inflation until 100% of scheduled workloads ran at HIGH priority, and on-call engineers spending most of their ticket time negotiating shutdowns of non-preemptable workloads on hosts needing maintenance. The team recognized a tragedy of the commons: individuals maximizing their own outcomes produced a poor global result and abused the shared resource. They trace this dynamic to the resource-allocation literature, citing the 2011 Dominant Resource Fairness paper by Ghodsi et al., which recounts users inserting infinite loops to artificially inflate utilization.

The fix was to allocate GPU time instead of GPUs. Earlier attempts to assign teams monopolies over GPU sets created idle capacity due to the seasonality of research, effectively turning scheduling into a manually solved knapsack problem. The new system allocates portions of GPU time: managers proportionally allocate time to the projects and researchers they are responsible for, translating program strategy directly into guaranteed shares of cluster capacity. Every request must be funded by a budget or it is not protected from preemption. This shifts the debate over how much GPU time each project deserves from case-by-case operations to a transparent administrative budgeting process, and it makes gaming expensive—a squatting workload spends team budget on nothing. The key stated requirements are frequent opportunities for researchers to advocate for time, with decisions made by managers holding the most context on tradeoffs.

The scheduler layer is hierarchical fair-share over a sliding lookback window, defaulting to 7 days, with workloads from under-utilized allocations sorted above those from over-utilized ones. The source notes the algorithm is not new, tracing this lineage to the Hadoop Fair Scheduler in 2009 and to SLURM’s Fair Tree and YARN’s Fair Scheduler today; what is new is that the tree mirrors the research program structure and the weights are manager-set budgets rather than static quotas. The scheduler distinguishes allocated occupancy, which is charged to a budget and protected during a minimum runtime window, from unallocated occupancy, which is free, unprotected, and preemptible by any allocated request. Unallocated time keeps GPUs fully occupied when funded demand does not match supply.

To handle long-running jobs, the team introduced a scheduling contract: in exchange for cluster access, a workload must declare its minimum runtime, the shortest occupancy needed to make meaningful progress, with 8 hours chosen as the maximum allowed value. During that window the workload is protected; afterward it may be preempted and automatically requeued if resumable. Setting minimum runtime to zero marks the time as unallocated. This time-slicing lets fair share converge, disincentivizes squatting, and lets unhealthy hosts drain workloads automatically as they reach minimum runtime: repairs requiring a human in the loop fell by 74%.

Before rollout, the team built a small simulation environment that takes workloads and their submission schedule as input, then jumps ahead to schedulable moments to report queue wait times, preemption events, and GPU-time distribution across projects. They used both historical submission data and hand-built scenarios, since the historical record lacked enough debug-like workloads. Simulations predicted debug workload p90 wait times would fall from about 6 hours to 5 minutes. A cluster-by-cluster rollout began at the end of July. Over a 30-day test period, teams were delivered 98% of the GPU hours they were owed, counted as allocation capped hour by hour at actual demand; 13 of 15 team allocations received 95% or more, with the worst case at 90%. Occupancy held steady at 98% before and after, with 18% of delivered GPU time unallocated. Real outcomes outperformed predictions: debug workload p90 queue wait fell from 2 hours to 30 seconds, against the simulated 6 hours to 5 minutes, with the source noting the smaller baseline sample size means higher variance. On the largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds and p90 from 2.8 hours to 1.8 hours. A quoted researcher, Chris Clark, said the new scheduler feels like having an extra 30% compute, because idle slot capacity can be reclaimed later by bursting beyond allocation limits.

Challenges surfaced too. The learning curve was steeper than assumed, and old interface terminology such as workload priority kept its name while its meaning changed; live explanatory sessions with real examples worked better than documentation. Post-launch visualizations now show how closely assigned GPU time tracks allocations and expose the queue-sorting metric. The notable regression was interactive development sessions: previously a researcher could hold one for up to a week, but under time-slicing they are subject to the 8-hour protected-runtime cap and become preemptible beyond their allocation, forcing manual state rebuilding. The roadmap includes a CPU-only cluster next to on-prem storage for dev and data-prep sessions, plus restorable sessions. A potential issue under investigation is capacity fragmentation, where minimum-runtime protection may reduce opportunities to interrupt many jobs at once to place large pending workloads. Next work targets the pyramid’s capstone, utilization: improving the efficiency of bootstrapping, checkpointing, and the training applications themselves.