This guide walks through parameter-efficient fine-tuning of NVIDIA Cosmos Predict 2.5 using LoRA or DoRA, aimed at adapting the large world model to specific domains like robot manipulation without the cost of full fine-tuning. The motivation is that collecting real-robot trajectory data is slow and expensive, so generating synthetic trajectories with a fine-tuned video world model offers a scalable alternative. LoRA and DoRA inject small trainable adapter modules into the frozen base model, reducing memory requirements and keeping adapter files portable, making it practical to fine-tune on a single GPU and flexibly swap adapters for different domains at inference.

Training uses a dataset of 92 robot manipulation videos with text prompts describing pick-and-place tasks. The DiT’s attention projections (to_q, to_k, to_v, to_out.0) and feedforward layers (ff.net.0.proj, ff.net.2) are targeted for LoRA adapters, while all VAE, text encoder, and DiT weights remain frozen. The loss follows the rectified flow formulation: the model predicts the velocity that linearly transports a noise sample toward the clean data, trained with MSE loss only on non-conditioned frames. Training for 100 epochs on a dataset of robot videos takes approximately 2.5 hours on 8 H100 GPUs.

Evaluation uses Sampson Error (both temporal and cross-view) and an LLM-as-a-Judge scoring (Cosmos Reason2) for physical plausibility and instruction following. Qualitative results show the base model struggles with out-of-distribution robot hands, hallucinating human hands, incorrect hand usage, and jitter; fine-tuning with LoRA or DoRA addresses all three issues. Quantitative results show that both LoRA and DoRA converge to similar performance, with larger rank (32 vs 8) boosting instruction following but not geometric consistency or physical plausibility, likely because geometric and physical priors are largely captured by the frozen world model weights.