
OpenAI’s Performance Factory Automates ChatGPT Regression Fixes

OpenAI’s Performance Factory is an autonomous, cloud-based system that uses Codex agents to find and fix performance regressions in ChatGPT. It was built because the pace of agentic development made traditional performance engineering unsustainable. The first workflow, the Regression Workflow, automates the entire process from alert to recovery: an agent investigates a slowdown, records findings, proposes a fix for engineer review, and after deployment another agent verifies that performance has returned to baseline. In just over three months, it tracked more than 1,700 investigations and root caused over 600 cases — about 17 automated investigations per day.
The system evolved through stages: before Codex, engineers manually examined metrics, logs, and traces; with Codex as a partner they could ask why a service got slower and receive a guided investigation; a shared skill called perf-forensics captured the team’s investigation playbook; a human proxy phase had engineers copy alerts into Codex and paste findings into Slack; then investigations moved to the cloud; finally the Performance Factory coordinated end-to-end workflows.
The factory has four components. A task service holds configurable workflows defined in Markdown, listing states, transitions, and engineer-required steps. A dispatch loop periodically assigns ready tasks to agents by starting a Codex turn. An internal API runs Codex in the cloud inside the network, with the same repositories and observability tools engineers use, under a service identity with scoped permissions. Subscriptions let tasks wait on external events like GitHub review comments or CI failures and become ready again when the event arrives.
In the Regression Workflow, the root-cause agent identifies which metrics worsened, when the slowdown began, and which requests were affected, then checks deployments, feature flags, and code. Because agents sometimes stopped at the first plausible explanation, an adversarial Codex reviewer tries to disprove the diagnosis, checking counterfactuals such as whether unexposed requests stayed healthy or a rolled-back change recovered the metric. In one case the reviewer reopened an investigation because an image-download diagnosis did not explain continuing failures after the file-locating service had recovered. In a manual review of 100 root-cause suggestions, 70% were graded accurate and 30% needed refinement or did not fully establish causality.
Fixing the regression has been harder than finding the cause. Agent-proposed changes can be more complicated than necessary or introduce new issues, so the workflow also runs an adversarial review on proposed fixes. Some fixes require architectural judgment, such as adding a cache that improves performance but risks stale data, so engineers remain in the loop for review and deployment. The workflow links related incidents, groups them by metrics, and ranks estimated impact to protect engineers’ attention. Task reports include generated summaries with metrics time series, CPU flame graphs, and traces so engineers can inspect evidence quickly.
OpenAI evaluates changes to agents with backtests against past production incidents. Investigation tests are scored on timing, impact, affected requests, evidence use, and ranking of possible causes, with care to avoid expired telemetry or leaked answers. Agent-to-engineer handoffs are evaluated using snapshots of saved tasks, checking whether an unfamiliar engineer can understand what happened, whether useful charts are kept, and whether the proposed next step is supported. A daily monitoring agent also scans logs for tool failures and access errors, and the factory can create self-repair tasks when the same error blocks multiple investigations.
The Regression Workflow is one instance of a general agent-factory pattern, and other workflows can reuse the same machinery with different states and prompts. OpenAI is exploring proactive monitoring, using fine-grained performance data to spot regressions alerts miss, encoding known signals like service rightsizing into recurring workflows, and eventually letting agents ship low-risk fixes without a human in the loop.

