
A data simulator to test agents

Building and evaluating data agents against a real enterprise warehouse is hard: engineers often sit outside the data-owning team, production access is restricted, catalog descriptions are incomplete or conflict with docs and chat threads, there is no answer key across thousands of tables, and it is impossible to systematically vary complexity. Teams therefore fall back on toy datasets and hope the agent holds up in production.
To fix this, Google built an internal data simulator that works like a wind tunnel. Instead of copying proprietary schemas, an engineer provides a short plain-English description such as “Simulate a two-plant brake manufacturer with an ERP, a shop-floor execution system, and a three-shift schedule where second-shift supervisors track downtime differently from operations and finance.” The simulator invents a structurally analogous company (“Calder Brakes“) and generates schemas, rows, event history, documents, and chat threads, plus a benchmark of hard questions and a verified answer key. The synthetic estate preserves what the agent reasons over—table relationships, business events that create records, where institutional knowledge lives, and messy definitions—while sharing no sensitive content with production.
Clean test databases miss the judgment enterprise data demands. In the article’s worked example, a naive agent asked about downtime at Calder Brakes Plant 2 finds a confident, articulate policy memo from Bob, a second-shift supervisor, stating that downtime means unplanned stoppages only. The agent adopts that definition and filters out planned changeovers from its SQL. Both the qualitative and quantitative answers are wrong: the plant’s official metric counts changeovers as planned downtime, Bob’s belief flatters his shift’s numbers, and the metric owner quietly corrects the record in a later thread. Correct answers require looking past a document’s tone, checking who owns the metric before writing SQL, and tracing later corrections.
The benchmark’s trustworthiness comes from separating blueprint design from data generation. In the Compile blueprint stage, a language model turns the prompt into a frozen blueprint containing systems, people, metrics, and individual beliefs—some deliberately wrong, so traps and solutions are planted together. That is the only stage where a language model decides what is true. Simulate & build tables then runs deterministically: employees act on schedules, generating tickets, shift handovers, and database rows, while row-level flaws such as retry-storm duplicates, unmerged keys, and mid-quarter unit changes are injected and recorded in a messiness log so the ground-truth SQL can adjust. Render documents is the only later model call, turning fixed facts into memos and chat threads in each person’s style. Evaluate agent then runs against read-only tools.
Three build-time checks keep the benchmark honest. Quantitative questions ship with the SQL that computes the gold answer against messy tables; for example, a defect-count question pairs automatically with COUNT(DISTINCT) when the messiness log records retry duplicates, and decoy SQL queries representing common mistakes are run to confirm traps change the answer. Qualitative questions ship with a rubric grounded in the metric owner, official definition, and correction trail, and items are dropped if the estate lacks data to verify them. Rendered documents are checked against the blueprint to ensure no unrecorded numbers, dates, or names appear. Hallucination sentinels plant questions about invented metrics like “Basalt Velocity Ratio”; a reliable agent abstains and reports the metric does not exist.
Evaluation fidelity is tested at three levels. Statistical realism reports measure join fan-out, key skew, null and duplicate rates, event durations, day-of-week activity curves, and leading-digit frequencies (Benford’s law). Domain experts in early enterprise engagements queried synthetic estates modeled on their domains to judge similarity. An independent calibration suite of more than a hundred checks covers statistical realism and answer-key SQL verification; internally, capability upgrades moved accuracy in the same direction and ranked agent configurations in the same order on synthetic and human-authored benchmarks. Broader studies across paired real and synthetic customer estates are underway.
The approach builds on prior data-agent benchmarks—BIRD and Spider 2.0 moved from clean schemas to messy databases, while WebArena, τ-bench, and Generative Agents simulate websites or role-play users—but here employees follow deterministic rules tied to roles and schedules, so one seed replays the same organizational history. It also borrows from domain randomization in robotics: vary surface details while keeping underlying physics faithful. In this case, every surface name, number, and document is synthesized, while causal structure, cross-document links, and institutional messiness stay realistic.
Using synthetic estates removes the bottleneck of sharing production tables and hand-labeled question sets across organizational or compliance boundaries. Google uses the simulator internally to harden Google Cloud’s data agents without touching customer data, and the blueprint-first architecture can be reused by other teams building evaluation suites. The stated risks are overfitting to simulated patterns of messiness and drift as a live business evolves; the recommended discipline is periodic calibration against real-world runs, keeping benchmark difficulty aligned with production failure modes, and treating the simulation as a living model.


