MolmoMotion: Language-guided 3D motion forecasting from video
Predicting future motion from video remains harder than perceiving past motion, yet many systems — from robots reaching for objects to video generators producing physically plausible frames — require forward-looking anticipation rather than retrospective tracking. Existing 3D motion datasets and forecasters are limited by small scale, domain-specific templates, or reliance on pixel-space video prediction, which is expensive and camera-dependent. MolmoMotion targets this gap: a model that, given a video frame, 3D query points on an object, and a natural-language action description, forecasts where those points will move in 3D space over the following seconds.
The model uses Molmo 2 as a backbone to ground language instructions to objects and points in the image. Motion is represented as object-attached 3D points in world space — a representation that is class-agnostic, view-stable, and directly consumable by downstream systems like robot policies or video generators. Two training variants are released: MolmoMotion-AR, an autoregressive model that predicts quantized 3D coordinates step-by-step for smooth, well-defined futures, and MolmoMotion-FM, a flow-matching model that transforms noise into continuous trajectories, better suited for multimodal or uncertain futures. To train it, the team built an automatic pipeline that extracts object-grounded 3D trajectories from 1.16M unconstrained videos, filtering noisy tracks and clipping to meaningful motion windows, yielding the MolmoMotion-1M dataset. They also release PointMotionBench, a human-validated benchmark with 2.7K clips spanning 111 object categories and 61 motion types.
On PointMotionBench, MolmoMotion outperforms all existing forecasting methods including pixel-space video generators and parametric 3D approaches. In a simulated robotics pick-and-place task, a control policy built on MolmoMotion achieves 76.3% success versus 56.0% for the same policy on plain Molmo 2, and learns much faster — reaching 51% after 10K training steps where the baseline tops out at 19%. For video generation, feeding MolmoMotion‘s predicted trajectories into an image-to-video model improves motion quality on all five measured metrics, beating a much larger model on four of them. The key takeaway: object-centric 3D trajectory forecasting, learned from ordinary video at scale, provides a general, reusable motion representation that can accelerate downstream tasks in robotics and controllable video generation, and the open release of model weights, data, and benchmarks invites the community to build on this foundation.
