Multi-view transformers that process tokens from posed images face a hard positional encoding problem: patches must be uniquely encoded, the encoding must be SE(3)-invariant to allow consistent attention across views, similarity should be multi-frequency, and the scheme must adapt to scene geometry. The authors argue that prior absolute or relative encoding schemes fail all these requirements simultaneously. This tension is the core motivation for the paper: existing position encodings either break invariance or ignore 3D structure, leaving multi-view attention with degraded generalization across different views or scenes.

RayRoPE attacks this by representing each patch via a predicted point along its associated ray rather than just the ray direction. This yields a geometry-aware encoding that naturally respects depth. To achieve SE(3) invariance, the encoding computes projective coordinates in the query frame and uses them for multi-frequency similarity. A clever analytical mechanism handles uncertainty in the predicted 3D point, computing the expected positional encoding under noisy depth estimates without requiring Monte Carlo sampling. This design keeps the encoding differentiable and efficient while remaining robust to imperfect geometry.

The method is validated on novel-view synthesis and stereo depth estimation. On CO3D, RayRoPE achieves a 15% relative improvement in LPIPS over alternative positional encodings. It also seamlessly incorporates RGB-D input, where the inability of baseline encodings to leverage depth leads to even larger gaps. For builders working on multi-view models, the takeaway is that positional encodings benefit from being grounded in explicit 3D geometry, and that uncertainty-aware ray-based encoding is a practical drop-in improvement for many attention-based vision pipelines.