Databricks’ Agent Cloud: Why Open Source and LTAP Matter for AI

Highlights

00:09:52

Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness.

00:31:43

Reynold walks through Databricks' database dream: why CDC is brittle enough to joke that it means 'continuous data corruption,' and why LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine.

00:18:05

Databricks' infrastructure runs 50–60 million virtual machines a day and processes exabytes before breakfast, underscoring the scale needed for the agent cloud.

The article exposes a tension in the current AI agent landscape: coding agents and enterprise agents face the same unsolved infrastructure problems—portability, session history, collaboration, security, and spend control—yet the ecosystem lacks a common layer above individual agent harnesses. Databricks co-founders Matei Zaharia and Reynold Xin argue that as agents move from demos to real work, the gap between agent capabilities and the underlying data and compute infrastructure becomes a bottleneck. Without standardized APIs for agent sessions, files, streams, tool calls, and cancellation, each new agent or tool doubles the integration effort, and security policies remain ad hoc rather than contextual and stateful.

Databricks‘ concrete response is Omnigent, an open-source meta-harness designed to combine, control, and share agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Omnigent provides a common API for persistent sessions, cloud sandboxes, sharing, search, and collaboration, sitting above agent harnesses rather than replacing them. Separately, LTAP (Live Transactional and Analytic Processing) rethinks the database stack by writing transactional data into column-oriented formats like Parquet, unifying storage rather than collapsing query engines—an approach they argue is “HTAP done right.” This matters for agents: live operational context from databases, not just telemetry, is needed for agents to act deterministically. Databricks also emphasizes agent security through contextual and stateful policies, and spend controls to prevent an agent from burning $500 reading logs.

The serious takeaway for builders is that the next wave of AI software may be defined by a simple thesis: “get the data there, slap some agent on top.” Traditional software gets rewritten once data is in the right place and agents sit on top. Databricks is betting that the agent era will be built on open formats, a common agent API, and a unified storage layer—not on new database engines or proprietary agent protocols. For startups, the article flags opportunities around coding-agent analytics, quality, skills, and spend, as well as the insight that vector databases should never have been a separate category because they are just a query engine capability. The core infrastructure battle is shifting from data warehousing to the agent cloud, and Databricks is open-sourcing the layer above agents to set the standard.

The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin

View Original