RL for Video: The State of the Space

This note comes out of my recent exploration in video post-training and RL for video: a short map of where the space stands in mid-2026. Everything below is from public papers, tech reports, and press coverage, all linked at the bottom; the opinions are mine.

1.Where Video Models Are

The pretraining recipe has converged. Every serious model of the past two years is a latent diffusion transformer trained with flow matching on top of a 3D causal VAE, with joint image and video training, multi-stage progressive schedules, and dense captions written by a VLM. Movie Gen spelled the recipe out in public at 30B parameters on 6,144 H100s^[1], Wan reported scaling laws for it^[2], and Seedance 1.0 documented the pipeline shape everyone now runs: pretraining, continued training, SFT, then RLHF^[3]. Seaweed-7B made the counterpoint that a well-designed 7B model trained for 665,000 H100 hours competes with much larger peers, which says data curation and design still buy more than raw scale at current budgets^[4].

The products moved fast this year. Native audio became standard, durations jumped from eight seconds to fifteen and beyond (Seedance 2.0 generates 4 to 15 seconds natively^[5], and its 2.5 successor advertises single-pass 30-second clips), multi-shot direction and reference-based character consistency shipped (Kling 3.0's Omni line takes per-shot instructions^[6]), and Google folded video into an any-to-any Gemini Omni family instead of shipping a Veo 4^[7]. The market also delivered a verdict nobody predicted in 2024. OpenAI introduced Sora 2 in September 2025 as possibly "the GPT-3.5 moment for video"^[8], then discontinued the app and API within six months; reporting put inference cost near a million dollars a day against a shrinking user base^[9]. Kling meanwhile passed a reported 500 million dollars in annualized revenue, and the top of the public text-to-video arena in mid-August 2026 is a Google, Black Forest Labs, ByteDance, and Meta race, with MiniMax's open-weights H3 sitting sixth^[68]. Frontier video is a real business, and it consolidated hard.

Consistency progress came from the causal turn. Bidirectional diffusion over a fixed clip produces beautiful eight-second videos and nothing longer. The methods that matter distill the bidirectional teacher into an autoregressive student (CausVid^[10]), train that student on its own rollouts so exposure bias stops compounding (Self Forcing^[11]), then stretch the horizon with rolling windows, anchor frames, and KV-cache management (Self-Forcing++^[12], Rolling Forcing^[13]). This works well enough for real-time streaming on a single GPU, and not well enough to trust: the measured effective context of state-of-the-art long-video models is 3.0 to 9.2 seconds, beyond which errors accumulate and the video drifts^[14]. Interactive world models (Genie 3^[15], Cosmos^[16]) are the same problem restated: they hold a scene for minutes only by making generation causal and memory explicit.

2.What Scale Is Not Fixing

Put the benchmark numbers next to the demo reels. Physics-IQ tested whether models continue real videos with physically correct outcomes and found visual realism uncorrelated with physical understanding^[17]. On VideoPhy-2's hard split, the best evaluated model of early 2025 reached roughly 22 percent joint semantic-plus-physics score^[18]. MMGR scores Wan 2.2 near 84 percent on visual realism but 51 percent on physical commonsense, falling to 24 percent once every requirement must hold at once; Sora 2 leads the overall success rate at 70 percent and still spawns objects and passes solids through glass^[19]. The direction is not hopeless; Sora 2 reaches 42 percent of the Physics-IQ ceiling where the original Sora managed 10^[17]. The gap that refuses to close is between that curve and the polish curve. And the newest closed models, Gemini Omni Flash, Seedance 2.5, and MiniMax H3, ship with arena Elo attached and no public physics scores I could find; polish gets measured weekly, physics does not.

The sharper result is about state. A June 2026 study of 23 models across 9,600 videos found all of them treat the world as a tracking shot: an object that leaves the frame resumes in its abandoned state instead of evolving while unseen, and the failure holds across model families and increments of scale^[20]. VBench-2.0 reads the same way on its faithfulness dimensions^[21]. These are not rendering defects that a bigger VAE fixes. They are world-model defects.

More data does not obviously save you either. Passive internet video teaches how scenes look, and only thinly how they respond. V-JEPA 2 pretrained on over a million hours of video and still needed a stage on 62 hours of robot interaction data before it could plan actions^[22], and Silver and Sutton's "Era of Experience" essay makes the general argument that the next capability increments come from feedback and interaction rather than from more recorded data^[23]. Video pretraining will keep improving texture, layout, and style. The failures above need a training signal that says that bounce was wrong, and that signal is what post-training exists to deliver.

3.The RL Stack for Video, Today

The stack so far mirrors language with a lag (Fig 3). Reward fine-tuning came first: InstructVideo borrowed image reward models and backpropagated them through short denoising chains in 2023^[24], and VADER did it properly for video with differentiable rewards, including object detectors and action classifiers^[25]. DPO ports followed^[26], and DenseDPO found the first video-specific landmine: whole-clip preference pairs teach the model that static videos are safe, so it optimizes motion away unless preferences are collected per segment^[27]. Purpose-built video reward models arrived in 2025. VideoAlign trained on 182,000 human-labeled triplets and split the reward into visual quality, motion quality, and text alignment because one scalar could not carry all three^[28], and VisionReward decomposed the judgment further into interpretable checklists^[29].

On-policy RL became practical in 2025. Flow-GRPO converted the deterministic ODE sampler into an SDE so there is something to explore, and cut denoising steps during rollouts to keep cost sane^[30]; DanceGRPO unified the recipe across text-to-video and image-to-video^[31]. This is production technology now, and quietly so. Seedance 1.0 ships video-tailored RLHF with separate reward models for motion, aesthetics, and alignment^[3]; Meituan's LongCat-Video published its full multi-reward GRPO configuration^[32]; NVIDIA states Cosmos-Predict2.5 is refined with RL post-training^[16]; Kling's Omni report describes an SFT-then-RL phase^[6]. OpenAI's Sora 2 system card and Google's Veo materials say nothing about RL post-training in either direction.

The newest thread is the one I find most promising: verifiable rewards. RLVR-World scores world models with task metrics computed directly on their outputs^[33]. NewtonRewards extracts velocity and mass proxies from optical flow and penalizes violations of Newtonian motion^[34]. PhyMotion lifts generated humans to 3D meshes, runs them through a physics simulator, and scores kinematic plausibility, reporting a 68-point Elo gain in motion realism after RL^[35]. Wan-R1 and VideoRLVR use procedurally generated tasks, mazes, Sokoban, and trap avoidance, where correctness is checkable by rule, and both report verifiable rewards beating learned reward models on their tasks^[36][37]. This is the video version of the move that took language models from RLHF to reasoning RL.

There is also a test-time branch: spend more inference compute, search over candidates with a verifier, keep the best (Video-T1^[38]). It reliably buys quality, on the same curve language models rode, and its cost problem is the same one RL training has. CachedSearch sits here: cached exploration makes best-of-N affordable by re-generating only the winning candidate at full compute^[39].

4.Why RL for Video Is Hard

Start with the arithmetic. A language model rollout is a few thousand forward passes through a KV cache and costs a fraction of a cent. A video rollout is 30 to 50 denoising passes over a latent volume of a hundred thousand or more tokens, then a VAE decode, then a reward model forward, per candidate, and GRPO wants a group of those per prompt. That is how a weekend of H100 time disappears. Flash-GRPO puts aligning a 14B video model at hundreds of GPU-days per experiment^[40], and DanceGRPO ran on 64 H800s^[31]. At API list prices, frontier video costs 10 to 40 cents per generated second, so one 8-second rollout burns what thousands of LLM samples cost. An efficiency literature is moving fast in response: branched rollouts that share denoising prefixes across the group^[41], optimization at a single timestep per rollout^[40], and rewards computed in latent space so nothing gets decoded at all^[49]. The infrastructure half of the problem, keeping a fleet busy while rollout, scoring, and update stages have wildly different shapes, is its own topic, covered separately in these GPU infrastructure notes.

The reward is the deeper problem. The best public video reward model reports 73.6 percent pairwise accuracy with ties excluded on its own benchmark, and 61.3 percent when ties count^[28], and VLM judges show position bias and scoring instability on physics judgments^[44]. Newer reasoning judges push the no-ties number into the mid-80s on the same benchmark, and ties still drag them back toward 72^[67]. Anyone who has watched a reward curve climb while the samples quietly got worse knows where this goes. Optimize hard against a reward that weak and you get Goodhart's law, and video RL now has a documented catalogue of it: models that satisfy a physics reward by collapsing motion until nothing moves^[34], DPO pairs that quietly prefer static clips^[27], composite rewards gamed through their easiest component^[45], ratio drift that over-optimizes flow GRPO from inside the clipping range^[46], and policies that climb VideoAlign for a while and then visibly degrade^[47]. SAGE-GRPO adds that the ODE-to-SDE conversion itself injects off-manifold noise that makes video GRPO less reliable than its LLM counterpart^[48]. Fig 6 shows the quieter version of the same problem, straight from the papers: the optimized axis jumps, and everything else barely moves.

Then there is credit assignment. A language model gets its reward attached to a specific token sequence with per-token log-probabilities. A video policy gets one scalar for an artifact that emerged from 50 denoising steps times a hundred-plus frames, and has to work out which step and which region earned it. The current answers are partial: reweighting by timestep^[42], tree-structured advantages over branching rollouts^[41], or giving up on trajectory-level credit and optimizing a single step^[40].

None of these gaps is conceptual. Each one is an engineering-and-data problem that several groups are actively chipping at, which is roughly how language post-training looked in 2023.

5.Why RL Is Still the Right Bet

The reason to push anyway is that this movie has run once before. A 1.3B InstructGPT model beat a raw 175B GPT-3 on human preference^[50]. DPO made preference tuning a commodity^[51]. Verifiable rewards^[52] and large-scale reasoning RL^[53] then produced the last two years of language capability, and post-training became the main source of new capability in every frontier release. Video is tracing the same arc a year or so behind and closing (Fig 3), and it has now reached the part where the gains got large.

Video has one structural advantage language never had. Language rewards top out at human judgment unless the domain happens to have an executable check. Video is filmed physics, so optical flow, geometry, reconstruction error, simulators, and inverse dynamics^[55] all yield reward signals grounded in the world rather than in a preference model. NewtonRewards and PhyMotion are early and narrow, and they are the right shape: they are the unit tests of video^[34][35]. On the capability side, chain-of-frames results show base video models already do zero-shot maze solving and visual reasoning, jumping from 14 to 78 percent on 5×5 mazes at pass@10 between one Veo generation and the next^[54], which is the raw material reasoning RL is built to amplify.

There is a strategic argument on top. If video models become world models, RL shows up twice: RL post-training is how an interactive world model learns to respond correctly to actions (WorldCompass^[56], inverse-dynamics rewards^[55], Cosmos^[16]), and policy learning inside those world models is the payoff that justifies building them (DeepMind already runs its SIMA agent inside Genie 3's generated worlds^[15]). Most of the US field pivoted to world models this year, and that pivot signs everyone up for video RL whether they framed it that way or not.

6.What Comes Next

Here is how I map the current stacks, and where the empty quadrant is:

Ranked roughly by how much I think each one matters:

• Cheap rollouts. One-step and few-step policy optimization, branched rollouts, and latent-space rewards^[40][41][49]. If rollout cost falls two orders of magnitude, every experiment in this note becomes a weekend run, and iteration speed is the binding constraint on everything else here.
• Reward models worth optimizing against. The frontier moved from 50 to a claimed 87 percent in two years on one benchmark (Fig 10), but tie-aware agreement still sits near 72^[28][67], and closing the rest is worth more than any single algorithmic idea in the GRPO literature. Reasoning-style judges are the visible path.
• Physics as a verifier, at scale. Newton rewards, simulator scoring, and geometric consistency^[34][35] each cover a narrow slice. The open problem is composing them into broad-coverage verifiable reward suites, the video equivalent of the math-and-code RL corpus.
• A reward for minute three. Long-horizon methods can hold a scene for minutes^[12][13], but nothing scores whether minute three is still coherent with minute one, so nothing optimizes for it^[14][20]. Persistence-aware rewards are wide open.
• Reasoning RL on video. Procedurally generated visual tasks with rule-checkable answers^[36][37] plus chain-of-frames capability^[54] look like GSM8K circa 2023: small, synthetic, and about to matter.
• The world-model loop. RL to make world models action-faithful, then policy learning inside them^{[15][16][55][56]}. This is where video generation stops being a content tool and becomes infrastructure for robotics.

The pattern from language suggests the distance between "post-training exists" and "post-training drives the release notes" is about two years. For video, that clock started around DanceGRPO in mid-2025.

References

1. Polyak et al., Movie Gen: A Cast of Media Foundation Models, 2024.
2. Wan Team, Wan: Open and Advanced Large-Scale Video Generative Models, 2025.
3. ByteDance Seed, Seedance 1.0: Exploring the Boundaries of Video Generation Models, 2025.
4. Seaweed Team, Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model, 2025.
5. ByteDance Seed, Seedance 2.0: Advancing Video Generation for World Complexity, 2026.
6. Kling Team, Kling-Omni Technical Report, 2025.
7. Google, Introducing Gemini Omni, 2026.
8. OpenAI, Sora 2 Is Here, 2025.
9. TechCrunch, Why OpenAI Really Shut Down Sora, 2026.
10. Yin et al., From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CausVid), 2024.
11. Huang et al., Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, 2025.
12. Cui et al., Self-Forcing++: Towards Minute-Scale High-Quality Video Generation, 2025.
13. Rolling Forcing: Autoregressive Long Video Diffusion in Real Time, 2025.
14. Context Forcing: Consistent Autoregressive Video Generation with Long Context, 2026.
15. Google DeepMind, Genie 3: A New Frontier for World Models, 2025.
16. NVIDIA, Cosmos-Predict2.5: World Simulation with Video Foundation Models for Physical AI, 2025.
17. Motamed et al., Do Generative Video Models Understand Physical Principles? (Physics-IQ), 2025.
18. Bansal et al., VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation, 2025.
19. MMGR: Multi-Modal Generative Reasoning, 2025.
20. Current World Models Lack a Persistent State Core, 2026.
21. Zheng et al., VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness, 2025.
22. Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, 2025.
23. Silver and Sutton, Welcome to the Era of Experience, 2025.
24. Yuan et al., InstructVideo: Instructing Video Diffusion Models with Human Feedback, 2023.
25. Prabhudesai et al., Video Diffusion Alignment via Reward Gradients (VADER), 2024.
26. Liu et al., VideoDPO: Omni-Preference Alignment for Video Diffusion Generation, 2024.
27. Wu et al., DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models, 2025.
28. Liu et al., Improving Video Generation with Human Feedback (VideoAlign / VideoReward), 2025.
29. Xu et al., VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation, 2024.
30. Liu et al., Flow-GRPO: Training Flow Matching Models via Online RL, 2025.
31. Xue et al., DanceGRPO: Unleashing GRPO on Visual Generation, 2025.
32. Meituan LongCat Team, LongCat-Video Technical Report, 2025.
33. Wu et al., RLVR-World: Training World Models with Reinforcement Learning, 2025.
34. Le, What About Gravity in Video Generation? Post-Training Newton's Laws with Verifiable Rewards, 2025.
35. Huang et al., PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation, 2026.
36. Liu et al., Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning, 2026.
37. Zhu et al., Video Models Can Reason with Verifiable Rewards, 2026.
38. Liu et al., Video-T1: Test-Time Scaling for Video Generation, 2025.
39. Saini et al., CachedSearch: Training-Free Cached Exploration for Test-Time Search in Video Diffusion, 2026.
40. He et al., Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization, 2026.
41. Li et al., BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models, 2025.
42. TempFlow-GRPO: When Timing Matters for GRPO in Flow Models, 2025.
43. VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding, 2025.
44. Cao et al., Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation, 2026.
45. Li et al., Rethinking Reward Signals in Video GRPO: When Scores Become Targets, 2025.
46. GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping, 2025.
47. Wang et al., Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning, 2026.
48. Zheng et al., Manifold-Aware Exploration for Reinforcement Learning in Video Generation (SAGE-GRPO), 2026.
49. Mi et al., Video Generation Models Are Good Latent Reward Models, 2025.
50. Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (InstructGPT), 2022.
51. Rafailov et al., Direct Preference Optimization: Your Language Model Is Secretly a Reward Model, 2023.
52. Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024.
53. DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025.
54. Wiedemer et al., Video Models Are Zero-Shot Learners and Reasoners, 2025.
55. Ye et al., Reinforcement Learning with Inverse Rewards for World Model Post-training, 2025.
56. Wang et al., WorldCompass: Reinforcement Learning for Long-Horizon World Models, 2026.
57. Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO), 2024.
58. Clark et al., Directly Fine-Tuning Diffusion Models on Differentiable Rewards (DRaFT), 2023.
59. Lambert et al., RewardBench: Evaluating Reward Models for Language Modeling, 2024.
60. OpenAI, Learning to Reason with LLMs (o1), 2024.
61. Awesome-RL-for-Video-Generation, community-curated paper list, snapshot August 2026.
62. Zhang et al., OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models, 2026.
63. He et al., VideoScore: Building Automatic Metrics to Simulate Fine-Grained Human Feedback for Video Generation, 2024.
64. Wang et al., LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment, 2024.
65. Wang et al., Unified Reward Model for Multimodal Understanding and Generation, 2025.
66. VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning, 2025.
67. Reward Lightning: Fast Video Generation via Homologous Preference Distillation, 2026.
68. Arena, Text-to-Video Leaderboard, snapshot of August 14, 2026.

Citation

@misc{saini2026rlforvideo,
  author       = {Saini, Shreshth},
  title        = {RL for Video: The State of the Space},
  year         = {2026},
  month        = {August},
  howpublished = {\url{https://shreshthsaini.github.io/blogs/rl-for-video.html}},
  note         = {Blog post}
}