Language models can reason about code, math, and logic, but can that ability transfer to controlling robots in the physical world? This extensive evaluation reveals a nuanced picture: a model’s robotics capability depends far more on how it is connected to the robot than on the model alone. When models must directly command motor torques, they mostly fail. But when they supervise pretrained controllers—gait policies for legged robots or vision-language-action (VLA) models for robotic arms—their performance jumps dramatically. The same model can look weak or strong depending purely on the control interface, making interface design a central safety and capability concern.

The authors built the Embody benchmark suite spanning multiple robot bodies: a simulated quadruped and humanoid, a fixed-base Franka Panda arm, and a physical Unitree Go2 quadruped. They tested four control interfaces: direct torque control, writing Python controllers, supervising pretrained policies, and training RL policies from scratch. On high-level locomotion, newer Claude models can navigate simple environments using a pretrained gait policy and a compass heading. On manipulation, pairing models with a VLA raises task success far above direct control. Newer models are also better supervisors—they recognize when the VLA is wrong more often, recovering most of the policy’s value instead of destroying it. However, direct low-level control improves much more slowly, and models still fail at tasks requiring sustained spatial memory or long open-loop plans.

The practical takeaway for builders and safety researchers is that a model’s real-world influence can change by orders of magnitude depending on its access to pretrained controllers and perceptual tools. A general-purpose chat model with no robotics training can already, on a good run, walk a quadruped through a maze or pick a plate off a counter—but only when given a competent controller to supervise. Safety evaluations must therefore treat the control interface as a core part of the system, not an afterthought. Newer Claude models get more from raw images, adapt better after failure, and need less practice to learn a navigation course, but the bottleneck remains stable spatial bookkeeping and long-horizon planning. The rapid, if uneven, improvement across model generations suggests these gaps will continue to close.