Evaluating AI Video Quality with Pairwise Comparison

The Breakthrough

Character.ai moved from absolute scoring metrics (like CLIP score) to a pairwise preference model trained on a small VLM, which allows the system to judge video quality based on relative storytelling and temporal consistency rather than individual frame aesthetics.

What Actually Worked

• Pairwise Preference Training: Instead of assigning a 1-10 score, the team trained a Qwen3-VL model using Bradley-Terry loss to determine which of two video clips tells a better story, which proved more reliable for human alignment.
• Small Model Distillation: The team distilled a committee of expert evaluators into a single small VLM to achieve a 3-second inference time for 15-second videos, making it fast enough to run as a regression gate in CI.
• Synthetic Badness: To train the judge, the team manufactured negative examples by deliberately corrupting high-quality footage or generating random artifacts, ensuring the model learned to identify specific failures like hovering physics or extra limbs.
• Agentic Validation Loop: The evaluation logic is integrated directly into the generation pipeline, allowing agents to verify their own work and regenerate specific shots if they drift before the final video is assembled.

Before / After

• Before: Absolute scoring models often rewarded "vibe" and gloss, giving high scores (9.2/10) to static images where the camera did not move for four seconds.
• After: The pairwise judge identifies specific failures like physics violations, pacing issues, and temporal drift, providing actionable feedback on why a clip is low quality.

Context

Video generation models often produce content that looks high-quality in a single frame but fails across time due to temporal incoherence, physics errors, or broken storytelling. Traditional metrics like CLIP score or LPIPS are insufficient for long-form video because they do not account for narrative pacing or shot continuity. Character.ai developed this approach to catch these errors early in the generation process, reducing the cost of regeneration and ensuring consistent output for user-facing applications.

Notable Quotes

• "Don't score, compare."
• "The model was wrong, but it was wrong in a very confident way."

Content References

• {"type": "tool", "title": "judgejudy", "url": "https://github.com/character-ai/judgejudy", "context": "recommended"}