Daily curated AI insights you can't miss.
An LLM judge can catch video drift early, but requires calibration against human preference to avoid its own drift.
Don't score, compare: pairwise preference holds up where absolute scoring collapses to one dimension.
Pairing real footage with deliberately broken videos trains a quality detector that evaluates story coherence, not just visual gloss.
Evaluating AI-generated video requires temporal coherence across frames, but standard metrics like CLIP score reward visual quality per frame while ignoring whether the character stood frozen for four seconds. Maor Bril‘s team at Character.ai found that absolute scoring collapses to a single dimension that misses storytelling failure. A team watching clips on Friday does not scale, and any AI judge you wire up drifts from human preference unless you measure the drift. Video breaks the text playbook because it has to hold temporal consistency, shot continuity, and a coherent story across frames, not just look good in a single still.
The fix that stuck was to stop scoring and start comparing. Absolute scores collapsed to one dimension, but pairwise preference (is B a better story than A?) held up. The team trained a Qwen3-VL judge with Bradley-Terry loss on pairs of real and deliberately broken footage to catch slop before it ships. Drift is cheapest to catch early, especially on longer-form video, so the judge runs as a regression gate in CI: every AgentX release at Character.ai clears an eval wall, calibrated against human scores, before users ever see it.
Pairwise comparison captures story and sound failure modes that absolute scores miss. The talk covers why a small model can work as a judge if trained on the right pairwise data, and the trade-off between small and large models. A key insight: when the judge scores vibe over substance, it rewards a static clip with good gloss. The trained judge avoids that by learning from pairs of real footage and deliberately broken versions, including temporal inconsistencies and shot discontinuity.
The team also explored self-verification in the generation loop, where the model checks its own output before finalizing. The judge is packaged as JudgeJudy, an open-source repository (github.com/character-ai/judgejudy) for reproducible evaluation. The talk concludes with Q&A on practical deployment, model scaling, and the importance of calibrating against human raters to prevent drift in the judge itself.
Evaluating AI-generated video requires temporal coherence across frames, but standard metrics like CLIP score reward visual quality per frame while ignoring whether the character stood frozen for four seconds. Maor Bril‘s team at Character.ai found that absolute scoring collapses to a single dimension that misses storytelling failure. A team watching clips on Friday does not scale, and any AI judge you wire up drifts from human preference unless you measure the drift. Video breaks the text playbook because it has to hold temporal consistency, shot continuity, and a coherent story across frames, not just look good in a single still.
The fix that stuck was to stop scoring and start comparing. Absolute scores collapsed to one dimension, but pairwise preference (is B a better story than A?) held up. The team trained a Qwen3-VL judge with Bradley-Terry loss on pairs of real and deliberately broken footage to catch slop before it ships. Drift is cheapest to catch early, especially on longer-form video, so the judge runs as a regression gate in CI: every AgentX release at Character.ai clears an eval wall, calibrated against human scores, before users ever see it.
Pairwise comparison captures story and sound failure modes that absolute scores miss. The talk covers why a small model can work as a judge if trained on the right pairwise data, and the trade-off between small and large models. A key insight: when the judge scores vibe over substance, it rewards a static clip with good gloss. The trained judge avoids that by learning from pairs of real footage and deliberately broken versions, including temporal inconsistencies and shot discontinuity.
The team also explored self-verification in the generation loop, where the model checks its own output before finalizing. The judge is packaged as JudgeJudy, an open-source repository (github.com/character-ai/judgejudy) for reproducible evaluation. The talk concludes with Q&A on practical deployment, model scaling, and the importance of calibrating against human raters to prevent drift in the judge itself.
