
Scaling Airflow pipelines with AI agents on Managed Airflow

Managed Service for Apache Airflow (formerly Cloud Composer) now ships agentic help across the pipeline lifecycle. In the Google Cloud console, Managed Airflow Agent investigates failed DAG runs, recommends environment improvements, and checks upgrade readiness; in an IDE or coding agent, Data Agent Kit writes, migrates, and deploys DAGs. The post walks through a fictional company, Cymbal Shops, to show how those agents move a production pipeline from Airflow 2.11.1 to Airflow 3.
Cymbal Shops runs cymbal-prod, a Managed Airflow (Gen 3) environment with 2-vCPU/4 GB workers autoscaling between 1 and 6. Clickstream events go through Pub/Sub to a Cloud Storage subscription that writes newline-delimited JSON files into hourly folders, and analysts want the data in BigQuery, cleaned and hashed, as soon as possible. Airflow 2 has no native external-event trigger, so the team settles for an hourly batch. Amira builds the DAG with Data Agent Kit in Antigravity using Gemini: the plugin inspects cymbal-prod, generates an Airflow 2.11-compatible DAG, deploys it, and confirms Airflow loads it. The DAG is readable, idempotent, and handles a normal hour of about 200 files/350 MB under 2 GB.
In late October, a campaign and a mobile release make events roughly three times larger; an hour grows from about 350 MB to 1.4 GB, and holding the full hour in pandas exceeds a worker’s 4 GB. Autoscaling doesn’t help because each task runs on one worker. On October 23, the 14:00 UTC run fails all three attempts with no error in the task log. Amira clicks Investigate on the failed run, and Managed Airflow Agent correlates worker memory metrics with task retries. Its report recommends three fixes: load files directly into BigQuery, process them in batches, or add worker memory, with evidence for each conclusion.
She takes the quick fix first and asks the same panel how much memory cymbal-prod needs. The agent recommends 8 GB workers and flags three production-readiness gaps: database retention is disabled, the environment isn’t highly resilient, and there are no scheduled snapshots. The agent is read-only, so Amira makes the change herself and schedules the rest. That buys time, but the root problem remains: the unit of work is a clock hour, so memory grows with traffic. Black Friday is forecast at 8-10x normal peak, which would require well over 8 GB.
Airflow 3 removes that constraint with native event-driven scheduling. An asset watcher can listen to a Pub/Sub subscription; when messages arrive, it updates an asset and any DAG scheduled on that asset starts a run. The watcher polls inside Airflow’s triggerer, so no sensor-only DAG or external REST API service is needed. Each run receives the new messages in its payload and can process one file per task, making memory depend on file size rather than hourly volume.
Before migrating, Amira asks Managed Airflow Agent to check readiness for Airflow 3.3.1 (composer-3-airflow-3.3.1-build.3). The agent scans DAGs and reports two blockers: a maintenance DAG that manually cleans the Airflow database, which Airflow 3 no longer allows, and marketing_attribution still using execution_date, which Airflow 3 removed. Four DAGs also need schedule_interval renamed to schedule. The report estimates the work and points to Data Agent Kit‘s migrations skill.
Instead of upgrading in place, Amira creates a new Airflow 3 environment, cymbal-prod-af3. The migrations skill refactors the DAG: PubsubPullTrigger becomes an event trigger used directly by an AssetWatcher, the triggerer checks the subscription every 10 seconds, and dynamic task mapping creates one task per file. Since files are capped at 128 MB, a busier hour means more tasks rather than bigger ones. In a load test replaying eight times normal peak traffic, clickstream_ingest simply ran more often and memory per task stayed flat, well under 1 GB; time from click to BigQuery dropped from over an hour to a few minutes.
The story’s division of labor is explicit: the console agent inspects production telemetry with user permissions and changes nothing, the coding agent works where code is reviewed and deployed, and agents propose while people decide. Two other capabilities are noted for readers: the Managed Airflow MCP server (generally available), which lets any MCP-compatible agent list and trigger DAGs, check run status, and inspect failed task instances under IAM control with read-only tools flagged, and the Apache Airflow Common AI provider, which adds model-driven tasks to Airflow 3 DAGs for LLM branching, agentic retries, and schema drift detection. The post notes that Cymbal Shops is fictional and its agent outputs, traffic, and memory numbers are illustrative.

