Pine59, a location intelligence company, runs data pipelines on Google Cloud that produce analytical metrics on schedules ranging from hourly to quarterly. Its most intensive workload, Daily Foot Traffic, computes data for as many as 14 million distinct locations in a single job, with heavy processing in BigQuery and orchestration handled by Managed Service for Apache Airflow (formerly Cloud Composer). As data volume and machine learning workloads grew, Pine59 modernized its monorepo, which contains hundreds of directed acyclic graphs (DAGs), and shared the results of that transition.

Before migrating, Pine59 stress-tested production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. Initial tests showed immediate and significant improvements in processing speed, task scheduling, and overall stability, leading the company to move fully to the new environment.

A core part of the migration was optimizing orchestration for ML inference. Pine59 had previously used standard Kubernetes operators for these tasks. With Managed Airflow (Gen 3), the team set up a dedicated Google Kubernetes Engine (GKE) cluster optimized for model inference and integrated it into the pipelines. This separation of orchestration from heavy ML execution compute let data processing and model inference run efficiently, positioning Managed Airflow as a scalable backbone for enterprise MLOps.

Pine59 also benefited from Airflow 3’s improved developer workflow and user interface. The team built custom plugins integrated directly into the new Airflow UI: BigQuery Auto-linkify, which detects BigQuery table references in Airflow Logs and XCom tabs and generates direct links to BigQuery Studio for faster debugging (available as a public GitHub gist); and DAG Run Configuration Search, a custom search form on the DAG overview page that lets engineers query key-value pairs in DAG run payloads and instantly surface matching runs. The team also deployed a compatibility shim layer within its monorepo that dynamically abstracts logic between Airflow versions, streamlining operator migration.

The migration produced clear results. Previously, tasks often stayed queued during peak surges; with Gen 3, queue latency dropped dramatically, allowing tasks to start almost immediately. A comparison of more than 300 runs of the same DAG between Managed Airflow Gen2 with Airflow 2.11 and Gen3 with Airflow 3.1 showed a significant reduction in queued time. Combined with DAG optimizations made during the transition, the Daily Foot Traffic pipeline dropped from nearly 38 minutes to under 26 minutes, roughly a 32% reduction in processing time.

Pine59 now runs all production workloads on Managed Airflow (Gen 3), citing improved MLOps capabilities, better developer tooling, and a faster, more resilient foundation. The post is authored with contributor Alexandre Crespo-Perez and was posted in Supply Chain & Logistics, Data Analytics, and Infrastructure Modernization categories.