This guide presents a lightweight, reproducible recipe for improving a small language model’s structured-output compliance using Group Relative Policy Optimization (GRPO). The authors fine-tune LFM2.

5-350M with TRL on about 500 training samples for 100 steps, targeting three reward functions: JSON format correctness, field count accuracy, and schema validation.

The LoRA adapter trains roughly 6M parameters (1.66% of the model). After fine-tuning, the model is converted to GGUF and served via llama.cpp for evaluation on the IFStruct benchmark.

The base model scores 22.6% on IFStruct; the GRPO-tuned model reaches 29.7%, a gain of 7.1 percentage points. The improvement is concentrated on JSON outputs (18.0% → 31.

9%) and bare-list top-level structures (16.6% → 29.7%), matching the training data augmentations. YAML scores remain nearly unchanged (27.2% → 27.5%).

The training pipeline fits on a free-tier Colab or Kaggle GPU, and the evaluation runs on a MacBook.

The results show that task-specific reinforcement learning with a small, targeted reward signal can substantially close the performance gap between a 350M model and larger models (e.g., Qwen3.

5-2B scores 33.15%). The full code and instructions are publicly available on GitHub.