How Trustpilot built real-time data enrichment with fine-tuned Gemma

Trustpilot faced a concrete tension: processing millions of user reviews in near real-time for review intelligence, while keeping costs predictable and latency strict. Large closed models like Gemini offered strong performance, but shifting from variable per-token pricing to fixed operational costs is financially necessary at scale. Relying on a third-party API also risks sudden changes or updates to the model that are outside Trustpilot‘s control, which is unacceptable for a pipeline that is core to the business. The real challenge was getting frontier-model-quality extraction (NER, categorization, sentiment) without the variable expense and vendor lock-in of closed APIs.

Their solution was to build a streaming pipeline using fine-tuned lightweight models based on google/gemma-2-9b, running on Gemini Enterprise Agent Platform with vLLM on A100 GPUs. The team used a consensus annotation process with Gemini 2.0/2.5 Pro/Flash teacher models to generate high-quality training data for specialized tasks. The architecture decoupled business logic (FastAPI-based classifier endpoint) from raw LLM inference (a separate Agent Platform endpoint), keeping the Dataflow job clean and allowing independent scaling. Performance tuning focused on vLLM backend configuration — adjusting data types, enabling prefix caching, and building a reusable load testing framework to find optimal serving capacity and tune auto-scaling.

The takeaway for builders is that fine-tuned open-weight models can deliver accuracy within a few percent of frontier teachers at a fraction of the cost, but getting there requires careful architectural decoupling and rigorous infrastructure tuning. Trustpilot’s setup processes millions of reviews daily in near real-time, achieving Gemini-like performance on fixed infrastructure. Notable friction points remain: private networking between distinct endpoints lacked native support, endpoint deployment observability was opaque, and securing A100 GPUs in EU regions required reservations rather than on-demand provisioning. These are practical integration hurdles that any team pursuing a similar stack should plan for early.