Surgical robotics is moving from teleoperation toward vision-language-action policies, but evaluating these systems remains difficult due to the cost of physical hardware and the complexity of surgical scenes. NVIDIA‘s Cosmos-H-Surgical-Simulator previously demonstrated that world foundation models can generate future surgical video from an initial scene and a sequence of robot actions, enabling faster-than-physical evaluation. Now, Cosmos-H-Dreams moves this capability into the real-time regime, producing an interactive, action-conditioned generative simulator that runs on a single NVIDIA RTX PRO 6000 GPU.

The core challenge is preserving useful surgical dynamics while reducing generation cost. The approach uses a teacher-to-student distillation pipeline. The teacher is a bidirectional model fine-tuned from Cosmos-H-Surgical-Simulator on the JHU dVRK tabletop mixture, which includes both successful demonstrations and failure episodes (e.g., needle drops, missed throws). The teacher’s temporal horizon is progressively increased during training for stability. A causal student is initialized from the teacher and first warmed up by imitating cached teacher denoising trajectories, learning to operate with causal attention and a streaming key/value cache.

To address the compounding-error problem that arises when autoregressive models condition on their own imperfect outputs during deployment, the pipeline uses self-forcing distillation. During training, the student rolls forward using its own generated context, and a frozen teacher provides distribution-matching supervision to guide those self-generated rollouts toward realistic surgical video. The resulting student model supports few-step diffusion, with as few as two denoising steps per latent frame.

The distilled student is served through FlashDreams, an accelerated inference library for autoregressive world and video models. FlashDreams applies optimizations including streaming KV cache, CUDA Graph capturing, and model compilation, bringing inference from roughly ten frames per second to approximately 160 frames per second on a single RTX PRO 6000 GPU. Cosmos-H-Dreams provides multiple human-machine interfaces: a browser client over WebRTC, a Meta Quest client via WebXR, and direct integration with learned policies for closed-loop operation.

While the released checkpoint is specialized for dVRK tabletop suturing, the system is designed to be extensible. A complete recipe for teacher fine-tuning and self-forcing distillation is provided for adapting to new embodiments. The article notes that next steps include evaluating more than visual quality—measuring tool-tip accuracy, gripper fidelity, idle stability, counterfactual diversity, and agreement between simulated and real policy outcomes. Potential downstream applications include latency-aware telesurgery, interactive surgical rehearsal, procedure planning, and intraoperative decision support, but Cosmos-H-Dreams is explicitly positioned as a research and development platform, not a diagnostic system or replacement for intraoperative imaging.