Enterprises often run heavy Spark jobs on Databricks and concurrent analytics on BigQuery. Historically, bridging the two meant copying data between environments, which raised storage costs, added latency, and fragmented security controls. Google Cloud’s borderless Lakehouse now lets BigQuery query Databricks-managed Apache Iceberg tables directly. This is a zero-copy approach: BigQuery reads the same Parquet files Databricks writes, and new changes appear after Lakehouse’s next scheduled metadata refresh, as often as every 5 minutes.

Managed Iceberg tables generate Iceberg-compatible metadata alongside native Delta logs. Data teams store Parquet files once in Google Cloud Storage (GCS); Lakehouse reads the Iceberg metadata and BigQuery‘s vectorized engine operates directly on the open-format files. Lakehouse also extends federation to external clouds like Amazon S3 and Azure Blob Storage by pairing the open Apache Iceberg REST Catalog with cross-cloud connections. Each federated catalog lives in a Google Cloud location chosen by the administrator, ideally near the data. Remote-cloud egress charges still apply, but Cross-Cloud Interconnect (AWS) and Lakehouse caching can reduce them. Because the platform is built on an open Apache Iceberg foundation, BigQuery can execute multi-statement SQL commands directly on external tables.

Security is handled through delegated access. Administrators configure a BigQuery Cloud Resource Connection tied to a service account managing IAM rights for GCS buckets. Granting users access to the Lakehouse table entity lets BigQuery enforce row-level and column-level security and dynamic data masking on external Iceberg tables while Databricks handles ingestion. Catalog federation uses the Apache Iceberg REST Catalog API, with Databricks Unity Catalog acting as the central metadata layer. At query time, Lakehouse contacts Unity Catalog for current table state, and short-lived tokens are issued through credential vending to grant BigQuery temporary read access to GCS.

Partition pruning leverages Iceberg metadata: BigQuery plans the query and reads only the relevant files. Iceberg’s hidden partitioning lets filters use logical columns such as timestamps without knowing the directory layout. However, Databricks managed Iceberg tables don’t support partition transforms; Databricks recommends liquid clustering instead. Standard WHERE clauses on partition columns let BigQuery use manifests to skip non-matching partitions, and column-level statistical bounds allow it to bypass irrelevant manifest records and unneeded Parquet files, reducing scanned bytes, slot consumption, and storage scan costs.

Iceberg manifests already identify which files to read, so BigQuery does not need a metadata cache for Iceberg tables. For cross-cloud tables such as those in Amazon S3, Lakehouse caching (Preview) keeps retrieved data blocks in Google Cloud so repeat queries read them locally. BigQuery reads the Iceberg manifest files directly during query planning; there is no metadata caching to enable.

Active maintenance is necessary to avoid performance degradation. Databricks APIs can trigger regular compaction pipelines to merge tiny Parquet files into larger, optimized layouts, and Predictive Optimization can automate upkeep. Databricks VACUUM clears unreferenced files outside active snapshots. Frequent writes, continuous ingestion, and streaming workloads produce thousands of small Parquet files, creating a storage bottleneck that limits BigQuery‘s read efficiency. Running compaction through Liquid Clustering or manual OPTIMIZE aggregates fragments into larger blocks, typically 256MB to 512MB, improving scan throughput. The binpack approach prioritizes total file size balancing, while alternative sorting strategies organize data along specific column keys.

Every transactional update, modification, and compaction creates fresh snapshots; old detached Parquet objects become orphaned over time. These legacy files support time-travel analytics but can drive up GCS storage costs. Databricks VACUUM scans and purges abandoned files, with a standard seven-day threshold (deletedFileRetentionDuration) before deletion to prevent active jobs from encountering missing-file errors. Scheduling routine VACUUM jobs, or letting predictive optimization run them, removes unreferenced data files and old Iceberg metadata.

Pairing Databricks Iceberg reads (UniForm) with Lakehouse federation, Lakehouse caching, and regular OPTIMIZE and VACUUM maintenance builds an efficient, zero-copy lakehouse. The source says this approach can lower storage spend, avoid fragile pipeline maintenance, and prepare data environments for modern analytics and emerging AI workloads. Registering Databricks Iceberg tables with the Lakehouse runtime catalog enables bi-directional federation without moving data files across platforms.