Cosmos3-H-Dreams is an interactive generative surgical simulator from NVIDIA built on Cosmos 3 and FlashDreams. It targets clinical cholecystectomy scenes recorded on the CMR Surgical Versius platform, extending the earlier Cosmos-H-Dreams tabletop suturing simulator built on Cosmos-Predict2.5. Instead of explicitly computing scene geometry, tissue deformation, or physical interactions, the world foundation model learns visual dynamics directly from synchronized video and robot kinematics; given an initial image and supplied tool actions, it generates each new block of video in real time. The authors say this is, to their knowledge, the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.

Inside the body, small instrument movements affect the entire scene: tissue stretches, reflections shift, structures become occluded, and smoke changes visibility. Temporal consistency is essential because small errors can accumulate, gradually changing instrument geometry or anatomical structure. Visual realism is a starting point for validation, not proof of biomechanical accuracy; a rollout must also follow translation, rotation, and gripper commands, so a plausible-looking scene that ignores those commands cannot serve as an interactive environment. Low latency generation is therefore required for a live control loop.

The model builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, with a bidirectional teacher specialized on curated cholecystectomy video and kinematics. Sample selection targets two visible, engaged instruments, a left Cadiere grasper and a right monopolar hook, and limits camera commands to emphasize instrument-driven interaction. It uses 28 action dimensions: pose and gripper information for both arms, plus energy controls, thumbstick axes, and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz.

A teacher-to-student pipeline supports long autoregressive rollouts. The teacher is first adapted on 24 future frames, then the horizon is extended to 72 frames, corresponding to 7.2 seconds at 10 Hz. Distillation converts the bidirectional teacher into a causal student that generates video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation using recorded context, and Self-Forcing trains it on its own generated context, guided by the teacher and a learned critic, to counter error accumulation during interactive rollouts. The first released student uses four denoising steps per generated block; the second checkpoint is separately trained with a two-step schedule. Fewer denoising steps give faster inference at lower visual quality, letting developers balance quality and cost.

FlashDreams provides the accelerated streaming runtime. Long rollouts also require managing the model’s attention history, so the integration uses an adapted FadeMem module that consolidates the growing key/value cache during long rollouts. This complements Self-Forcing: distillation addresses how the model learns from generated history, while cache consolidation addresses how that history is stored during streaming generation. With this stack, Cosmos3-H-Dreams runs interactively on a single NVIDIA RTX PRO 6000 GPU.

Demonstrations explore generalization beyond training data. Two rollouts begin with the same initial frame but follow different action sequences recorded with a VR headset, producing distinct continuations of the same scene with what the source describes as physically accurate tissue deformations. These rollouts make it possible to inspect whether instruments follow the controls, whether tissue responds coherently to interaction, and whether anatomical and instrument appearance remain consistent over time.

Cosmos3-H-Dreams is released with four-step and two-step cholecystectomy checkpoints and can be adapted to other datasets and embodiments with synchronized video and compatible action representations. A step-by-step guide covers data preparation, public teacher-training tools, and Self-Forcing distillation stages. Controller integrations must use the same action conventions and normalization as training. The source is explicit that the future directions, including modeling the full procedure, combining generative and numerical simulation, evaluating with surgeons, supporting more instruments, and developing autonomous surgical policies, are research directions, not established clinical capabilities. Cosmos3-H-Dreams is a research simulator, not a validated system for patient care.