
Building real-time agents for EV charging using Gemma and TPUs

lemonflow builds AI agents purpose-built for EV charging: support agents for drivers and monitoring agents for charging networks. The support agents answer drivers around the clock in more than 90 languages and today help an EV driver every 90 seconds. The company was founded after observing how often charging goes wrong: between 15% and 25% of EV charging attempts in Europe end in failure, and with battery-electric car sales growing 30% in 2025, stranded drivers often face slow call-center queues. Live voice support adds technical constraints: serving latencies above 800 ms risk conversational barge-ins; managed LLM APIs spiked from 1.0-1.2 seconds to over 1.7 seconds during peak hours and returned HTTP 429 rate-limit errors. The ecosystem is also fragmented, requiring a standardized conversational engine to bridge around 30 distinct Charge Point Management Systems (CPMS), each with its own APIs, data models, and quirks, and the platform must serve hundreds of tailored agents with unique prompts at full speed.
The solution was to migrate from managed APIs to self-hosting the open Gemma 4 model on Google Cloud TPU v6e hardware in the EU. Gemma‘s compatibility with Gemini let lemonflow transition its existing prompt infrastructure with minimal modification. Self-hosting provided architectural control to establish a deterministic core layer that handles EV-specific guardrails, guards against prompt injections, and standardizes fragmented backend protocols before the LLM. The team built an intelligent routing gateway that prioritizes workloads by operational urgency: live voice calls first, web chats second, and automated re-simulations deferred to low-traffic periods. Custom optimizations to the open-source vLLM serving engine and memory tiering between TPU and host memory allow a single TPU slice to keep many more operator agents ready for instant answers without recalculating their context. Automated fallback routing to self-hosted GPUs and Gemini Enterprise endpoints exists, but thanks to TPU reliability Gemma answers more than 99.9% of all live conversation turns; every request is served from secure EU compute.
Architecturally, the platform runs two kinds of agents: support agents served by Gemma on Cloud TPUs for minimal latency, and monitoring agents that analyze conversations and charger logs on Gemini so operators can fix faults before the next driver is affected. A translation layer converts varied CPMS interfaces into a standardized format. During a call, the Conversational Engine pulls the charger’s live status and Open Charge Point Protocol (OCPP) logs from the operator’s CPMS in real time, feeds structured log summaries into Gemma on TPU, and streams token output selectively: partial text tokens stream instantly to the Text-to-Speech engine, while structured tool-call tokens stream asynchronously to backend systems for actions such as remotely restarting chargers, unlocking stuck cables, stopping sessions, or notifying technicians. Deterministic security controls screen inputs before they reach the TPU-hosted LLM, and a typical response starts in around 350 milliseconds.
The reported outcomes are substantial: agents typically answer every call, handle more than 80% of conversations end to end without human handover, and earn 95% driver satisfaction. Prompt update and test cycle times dropped from 2-3 hours to about 10 minutes, a more than tenfold faster improvement loop. P50 latency fell to around 350 milliseconds and P90 to around 600 milliseconds, roughly three times faster than the managed-API baseline and enough to sharply reduce barge-ins. Dedicated TPU infrastructure eliminated HTTP 429 errors and kept latency flat through peak afternoon traffic, while idle TPU cycles became testing capacity. The agents now support more than 40 charging networks in 17 countries, covering more than 250,000 charge points, around the clock. With additional compute capacity, lemonflow introduced Agent Coach, a background engine that provides agents with relevant context during live calls and turns real conversations into permanent test cases, plus nightly automated regression testing across thousands of fixed conversational scenarios.
The team says it is preparing to move to Ironwood, Google’s seventh-generation TPU, as soon as it is available in its EU region, citing six times the high-bandwidth memory per chip, which would multiply the number of operator agents a single TPU can keep ready in memory. The stated end goal is that no driver is ever left stranded at a charger: instant expert answers in the driver’s language, remote charger reboots when necessary, and monitoring agents that catch the next fault before another driver notices.

