Building an AI agent for a high-stakes operational domain like maritime surveillance is, above all, a reliability problem. The wrong answer could send a patrol vessel miles off course, costing scarce resources and endangering personnel. The article describes how the Skylight team built Shippy, an agent for real-time maritime domain awareness, and the real work was not the model but the system design to ensure correctness, stay within limits, and hold up across diverse tasks. The team had to verify everything against live data, not a static snapshot, making reliability the central tension.

Shippy is architected as three layers: a soul (system prompt defining persona and boundaries), skills (versioned markdown files with workflows for querying APIs, looking up boundaries, interpreting vessel tracks), and config (harness, LLM, runtime settings). A key technical move was building a deterministic CLI that wraps the Skylight API, collapsing complex API calls into predictable commands with typed filters, built-in pagination, and file-based output. This avoids the subtle bugs that arose when the agent constructed raw API calls. Each user session runs in an ephemeral, isolated Kubernetes pod on Mothership, an agent hosting platform that injects the user’s JWT and restricts network access. The evaluation system scores the whole agent—model, skills, sandbox—against live data using subject-matter expert scenarios, an LLM judge, and a pass/fail threshold, catching regressions before they reach users.

The practical takeaway is that for operational domains, reliability comes from infrastructure and guardrails, not just model quality. The layered abstraction—typed API, deterministic CLI, agent skills that reference CLI commands—narrows what each layer can get wrong. Sandboxing and isolated sessions prevent data leakage, and the eval pipeline ensures that changes to skills, model, or data are tested against real workflows. The patterns of failure (overstepping into tactical recommendations, geometry-sensitive misses, invented CLI commands) directly inform improvements. These architectural lessons are already being applied to other Ai2 platforms like EarthRanger and OlmoEarth, showing that the approach generalizes beyond maritime.