Pando: OpenAI’s Git Backend for Agent-Scale Source Control

OpenAI’s monorepo grew rapidly as Codex enabled engineers and agents to work in parallel, producing far more branches, pushes, merges, fetches, and clones from dev environments, CI, and research jobs. By late last year, the systems behind the monorepo were struggling: slow fetches and pushes, failed merges, and reliability sometimes below 90%. Instead of abandoning Git, OpenAI chose to rebuild the backend while keeping the familiar Git workflow.

The result is Pando, a new backend for Git repositories that now powers all development in OpenAI’s monorepo. Pando has two goals: horizontally scalable writes optimized for concurrent pushes to many branches, and maximum read throughput so clients can saturate their network link. It remains compatible with the Git protocol and client, so engineers and agents adopt it without changing how they work.

Scaling a Git service has two dimensions: many writers pushing different branches and many readers fetching the same history. OpenAI evaluated two common patterns—three-phase commit (3PC) across complete replicas, and a shared write-ahead log (WAL) in blob storage—and found both create a central bottleneck. 3PC puts replica disk performance and quorum round-trips on the push path; WAL makes every push compete for the same manifest update, and replicas can fall behind indexing incoming packs.

Pando instead relaxes repository-wide consistency. It provides causal consistency: pushes to unrelated refs proceed independently, with serializable updates per branch. A push stores the incoming packfile as an immutable blob in a blob store and records a ref version in the metadata store, Turtle, via a multi-key transaction. The cost is that a client fetching multiple branches can observe them at different points in time; OpenAI says users and agents are agnostic to global ordering and rely on per-branch order.

To keep packs reusable, Pando advertises only a subset of refs during a push—by default, a designated trunk branch. A full ref advertisement can exceed 100 MiB with millions of branches, and advertising all refs makes client-generated packs depend on too much history. A push produces a pack that depends only on the advertised base; Pando can then serve that same packfile to any reader that already has that base. This makes median pushed packfiles under 20 KiB in OpenAI’s monorepo (99th percentile under 3 MiB). Writes require O(1) metadata reads and writes.

Fetching uses the Git protocol’s packfile-uri extension. Pando maps the client’s haves and wants to a sequence of ref versions, selects the stored packfiles covering the needed objects, and returns URIs; clients download them from Express, a pack delivery service with regional caches. Fetch planning needs O(log Δ) metadata reads. This design lets Pando serve many clients from cache with little CPU.

Background compaction, similar to LSM-tree compaction, groups small L0 packfiles into larger covers up to L4. Covers act as skip lists, making pack planning logarithmic in how far the client is behind, and keep compression good without discarding old packs—all levels remain available to serve fetches.

In production, Pando cut median push latency by nearly 5x (an ~80% reduction), improved merge reliability, and delivers nearly a petabyte of Git data per day using only a handful of CPU cores. It has served more than 57 TB of packs in a single hour, with over 99% of bytes from local disk cache, and validated clone/fetch saturation of 10 Gbps network links on standard cloud VMs. The core was designed by one engineer, implemented by agent teams, and deployed by a few engineers—from kickoff to production in 10 weeks. Agents continued hardening it: using GPT-6 Astra, they wrote Lean and TLA+ formalizations of a “theory of Pando” to find and fix bugs, and ran continuous red-team sessions against Git and Pando.

OpenAI also contributed improvements back to Git: packfile-uri tracing and fixes, parallel pack download and index, explicit haves for –force-with-lease, and validation when accepting an existing pack for a fetch. Some are already upstream. Beyond the monorepo, OpenAI is exploring Pando for other agent-heavy shared-repository workloads and ways to share the technology.

Scaling Git at OpenAI

View Original