Best practices for customizing Gemini with managed RL fine-tuning

Reinforcement learning has been a keystone of LLM post-training, but it normally needs large training clusters and access to model internals that external customers cannot have with proprietary models such as Gemini. Google Cloud has packaged it into a managed RL fine-tuning service (RLFT): customers bring prompts and a reward function, and the service handles infrastructure and proprietary model internals. The service adapts Gemini from a reward signal you define rather than from labeled answers, which targets tasks that are hard to demonstrate but easy to score. At each training step it generates multiple candidate responses, scores them with your reward, and improves the model so higher-scoring responses become more likely while staying close to the original Gemini. The reward is the main thing the customer owns and the main determinant of results.

The article identifies three properties that define what RLFT can and cannot do. It learns from the model’s own outputs, refining what the model already produces rather than copying an external target, so it tends to disturb unrelated capabilities less than SFT. It rewards outcomes, not paths, so any response that reaches a good result earns reward, fitting open-ended tasks with many valid solutions. It amplifies existing competence: it makes occasional success reliable, but it cannot teach a skill the model never demonstrates. Prompting and SFT should be exhausted first. RLFT is worth using when you can grade a response but cannot cheaply author it, when SFT has plateaued on the metric that matters, or when many equally valid answers would make a single reference target wrongly penalize valid responses. Direct RLFT works when the base model already succeeds part of the time; if the base success rate is too low, use a light SFT warm start and continue into RL with Continuous Tuning, which initializes RL from the SFT checkpoint.

Use cases show where the pattern delivers value. For AI-powered NPCs in games, a Gemini LLM-as-a-judge autorater scores persona, flow, and game-state syntax, with penalties for format and language errors; the reported results are that loops and language drift disappeared and state syntax held. For structured entity extraction from documents, a rule-based precision/recall reward forces every field to be grounded in the source text; field-level accuracy rose on noisy real-world documents where tuning had stalled. For content moderation, a Cloud Run reward pairs format validation with a deterministic grader to enforce multi-step policy adherence; the model handled exemption carve-outs, sharply cut false positives, and stopped reward hacking, reducing human escalation.

For code measured by execution, such as SQL or API calls, a code-execution reward runs the code in a secure sandbox and pays out only if it compiles, executes, and returns the correct result. The model produced first-attempt executable queries at closed-frontier quality and lower inference cost, letting non-technical users query proprietary data in natural language. For presentation slide generation via HTML, a code-execution reward renders the slides and scores visual design, layout integrity, structural completeness, and rubric adherence; the model emitted modular, well-styled decks with no layout overflow.

Where to start? A diverse dataset of prompts with a held-out validation split is enough for a first run, with train and eval kept strictly separated to avoid contaminated evals. The reward function is the task specification as code or configs and the dominant driver of quality; a good reward correlates with human preference, handles malformed output with a clearly negative score rather than crashing, and resists reward hacking through ensembles, length penalties, floors on degenerate outputs, and verifiable checks over model opinion. Validate the reward offline before launch, then start from the service defaults, watch the reward and eval curves in the console, and take the checkpoint where validation reward saturates rather than the last step.

Best practices guide for customizing Gemini models

View Original