RL for Video: The State of the Space
A field note on post-training for video generation, revised October 2026. The field left the SDE likelihood for forward-process regression at about 12x lower cost, and the reward model became the measured bottleneck.
Thesis. Every lab that documents its pipeline runs RL post-training, but the best-described lab recipes are offline DPO and reward-gradient ascent, not GRPO. Academic work in 2026 left the manufactured SDE likelihood for forward-process regression, at about 12x lower cost for equal or better VBench.
State. The binding constraint moved from rollouts to rewards. The best public video reward model scores 60.8 percent on a frame-order test where people score 100, and a VLM judge rated 19.4 percent of VideoAlign's rewards reasonable once a policy was optimizing against them.
Implication. Treat the reward as the experiment: real video as the winner, a verifier where one exists, a motion term by default, and best-of-N with a world-model scorer at inference, worth 3 to 7 Physics-IQ points on any base.
Written in August 2026 and rewritten in October, after the field changed its mind about the sampler. Everything is from public sources linked at the bottom; the opinions are mine.
Where video models are
The pretraining recipe has converged: latent diffusion transformers trained with flow matching over a 3D causal VAE, joint image and video data, progressive schedules, dense VLM captions. Movie Gen spelled it out at 30B parameters on 6,144 H100s[1], Wan reported scaling laws for it[2], Seaweed ran it at 7B[3]. Figure 1 is the pipeline every 2026 report describes, with the four places a reward can enter.
The products consolidated. Native audio is standard and Seedance 2.0 generates 4 to 15 seconds natively[11][5]. OpenAI introduced Sora 2 as possibly "the GPT-3.5 moment for video"[12], announced its shutdown within six months and removed the API on September 24, 2026, at a reported million dollars a day against fewer than 500,000 users[13][14]; Kling's annualized revenue approached 500 million dollars[15], and Google shipped Gemini Omni, not a Veo 4[16]. The arena of September 21, 2026 has Gemini Omni 1.1 Flash first at 1516, Sora 2 Pro eleventh, and MiniMax H3 as the only open-weights model in the top ten∗[17]; Artificial Analysis puts Wan 3.0 first and Omni tenth[18].
∗"Open" with caveats: the H3 license excludes US, EU, UK and Korean users, only CFG-distilled base checkpoints are released, and the promised tech report has not appeared[17][19].Consistency came from the causal turn: distill the bidirectional teacher into an autoregressive student[20], train it on its own rollouts[21], stretch the horizon with rolling windows and KV-cache management[22][23]. It streams in real time on one GPU[23] and drifts past a measured 3.0 to 9.2 second context window[24]. The distilled, causal model is what ships, and it is the model RL has to run on.
What scale is not fixing
Physics-IQ found realism uncorrelated with physical understanding[25]; its 2026 third-party audit, Physics-IQ Verified, corrects 57.6 percent of sample scores and agrees with the original ranking only at Kendall tau 0.46[26]. On that board the frontier is measured, and low: Seedance 2.5 at 42.4, MiniMax H3 at 39.8, Gemini Omni 1.1 Flash at 35.3, Veo 3.1 Lite at 31.8, Sora 2 at 26.5 (Fig 3)[27]. The original board scores Sora 2 at 42.3, and the Verified paper notes Sora 2 performed worse in April 2026 than in October 2025, which an arena cannot see‡.
‡On the six models scored under both protocols, Sora 2 moves 25.3 to 26.5 and Wan 2.2 33.5 to 32.2; the headline 42.3 comes from an older run[26].The other benchmarks agree and have stopped moving: VideoPhy-2's hard split still tops out at 21.9 with no 2026 submissions[28], VBench-2.0 lists no 2026 frontier model[29], and MMGR's first version scored Wan 2.2 at 68.1 on physical understanding but 24.2 once every requirement must hold on 3D navigation[30]. The 2026 causality benchmarks find salience rather than physics, with no model above 52 percent on What-If World[31]; on PhysWeep the seed, not the prompt, picks the physical quantity[32]. The sharpest result is about state: across 23 models and 9,600 videos, an object that leaves the frame resumes in its abandoned state instead of evolving while unseen[33]. More passive video does not fix this: V-JEPA 2 needed under 62 hours of robot interaction on top of a million hours of video to plan[34][35]. These failures need a signal that says that bounce was wrong, and delivering it is what post-training is for.
One target, many estimators
Every method here chases the same distribution. Maximize expected reward under a KL penalty to the reference and the optimum is in closed form[36]:
\[ \pi^*(x \mid c) \;=\; \frac{1}{Z(c)}\, \pi_{\text{ref}}(x \mid c)\, \exp\!\big(r(x,c)/\beta\big) \tag{1} \]
GRPO climbs toward Eq. (1) with likelihood ratios, DPO and DGPO cancel \(Z(c)\) with pairs or balanced groups, DiffusionNFT, RAM and RVM regress toward it with reward-weighted targets and an anchor, and Reward Forcing and Stream-R1 distill toward it with \(\exp(r)\) weights.
Ratios on a manufactured likelihood
A flow model's sampler is a deterministic ODE with no per-step likelihood and nothing to explore, so Flow-GRPO converts it to an SDE with the same marginals[37]:
\[ dx_t = \Big[ v_\theta(x_t,t) + \frac{\sigma_t^2}{2t}\big(x_t + (1-t)\,v_\theta(x_t,t)\big) \Big] dt + \sigma_t\, dw_t, \qquad \sigma_t = a\sqrt{\tfrac{t}{1-t}} \tag{2} \]
\(v_\theta\) the learned velocity, \(a\) the injected noise level, \(w_t\) a Wiener process
Each step of Eq. (2) becomes a Gaussian with covariance \(\sigma_t^2 \Delta t\, I\), which supplies a ratio \(r_t^i\) and a closed-form KL, and the group-normalized advantage replaces the critic:
\[ \begin{aligned} \hat A^i &= \frac{R(x_0^i,c) - \operatorname{mean}_G}{\operatorname{std}_G}, \\[4pt] \mathcal J &= \mathbb E\Big[ \tfrac{1}{G}\sum_i \tfrac{1}{T}\sum_t \Big( \min\big(r_t^i \hat A^i,\ \operatorname{clip}(r_t^i, 1\pm\varepsilon)\,\hat A^i\big) - \beta\, D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) \Big) \Big] \end{aligned} \tag{3} \]
\(G\) rollouts per prompt, \(T\) denoising steps, \(\varepsilon\) the clip width
This line, Eq. (3), made 2025 video RL possible, from DanceGRPO on 64 H800s[38] to MixGRPO's sliding SDE window[39] and the LongCat-Video report[40]. Its failures are documented: near-tied scores become "illusory advantages"[41], pairwise preferences cycle[42], the ratio drifts below one and dodges the clip[43][44], the injected noise drifts off the data manifold[45] and leaves artifacts in the rollouts[46], and on a distilled few-step model it returns almost no gradient (below).
Pairs without a policy
Diffusion-DPO replaces the likelihood with a denoising-error gap against the reference, Eq. (4)[47]:
\[ \mathcal L = -\,\mathbb E \log \sigma\Big( -\beta T\, \omega(\lambda_t) \big[ \Delta_\theta(x^w) - \Delta_\theta(x^l) \big] \Big), \qquad \Delta_\theta(x) = \|\epsilon - \epsilon_\theta(x_t)\|^2 - \|\epsilon - \epsilon_{\text{ref}}(x_t)\|^2 \tag{4} \]
\(x^w, x^l\) the preferred and rejected sample, \(\omega(\lambda_t)\) the timestep weight
No rollouts and one forward pass per sample, which is why Kling-Omni "favor[s] DPO over alternative algorithms like GRPO"[5] and HunyuanVideo 1.5 runs DPO first because "existing reward models struggle to effectively differentiate fine-grained motion quality"[6]. Its blind spots: whole-clip pairs bias annotators toward low-motion clips[48], and updates with a component normal to the data manifold leave it[49]. The cheapest 2026 source of pairs is real video as the winner, against the model's own reconstruction[50], a regenerated masked region[51], or a rollout group[52]: no reward model, no annotation, and the winner is on the manifold by construction.
DGPO runs DPO over an entire GRPO group, split by the sign of the normalized advantage and weighted by \(|A|\) so the sides balance and \(Z(c)\) cancels, with \(\Delta_\theta\) now the \(x_0\)-regression gap[53]:
\[ \begin{aligned} \mathcal L_{\text{DGPO}} &= -\,\mathbb E \log \sigma\Big( -\lambda_t \beta T \Big[ \sum_{x \in \mathcal G^+} |A_x|\, \Delta_\theta(x) - \sum_{x \in \mathcal G^-} |A_x|\, \Delta_\theta(x) \Big] \Big), \\[4pt] \mathcal G^+ &= \{A > 0\},\qquad \mathcal G^- = \{A \le 0\} \end{aligned} \tag{5} \]
The group-relative signal survives in Eq. (5) while the stochastic policy, the ratio and the SDE disappear: GenEval 0.97 against Flow-GRPO's 0.95, about 20 times faster. Every DGPO number is text-to-image, so I read it as evidence about what GRPO buys (the sign of the advantage) rather than as a video method.
Leave the reverse process alone
The defining 2026 move puts the reward into a forward-process regression and never touches the sampler. DiffusionNFT did it first with an implicit positive and negative policy, \(v^\pm_\theta = (1 \mp \beta)\, v^{\text{old}} \pm \beta\, v_\theta\)[54]; reward-based velocity matching states the general form[8]:
\[ \begin{aligned} \mathcal L_{\text{rvm}} &= \tfrac12\, \mathbb E_{x_0^{1:G} \sim \pi_{\text{old}}} \Big[ r^i\, \big\| v_\theta(x_t^i,t) - (\epsilon^i - x_0^i) \big\|^2 + \beta(r^i)\, \big\| v_\theta(x_t^i,t) - v_{\text{anc}}(x_t^i,t) \big\|^2 \Big], \\[4pt] x_t^i &= (1-t)\,x_0^i + t\,\epsilon^i \end{aligned} \tag{6} \]
\(r^i\) the mapped reward of rollout \(i\), \(v_{\text{anc}}\) the anchor velocity, \(\beta(r^i)\) its weight
Sample, score, re-noise, regress toward your own good samples; NFT is a special case of Eq. (6) and Reinforce Adjoint Matching (RAM) and Advantage Weighted Matching are update-equivalent, each a choice of reward map, anchor and \(\beta\)[8][55], and a path-space analysis shows Flow-GRPO and forward matching are the same estimator up to variance reduction[56]. RVM reports VBench 84.13 on Wan2.1-1.3B against 83.10 base, 82.95 NFT, 79.46 DanceGRPO and 75.91 Flow-GRPO, at about 525 GPU-hours against an estimated 6,171 for DanceGRPO[8]; DiffusionART matches NFT's reward on HunyuanVideo-13B with 12.02x fewer GPU-hours and Flow-GRPO-Fast's with 11.72x[57], and RPO's two-rollout routing costs 24.1 H100-hours on Wan2.2-5B (Fig 4)[58]. The levers rank themselves: leave the reverse process alone (12x on video, 20x on images), shrink the group (3x to 4x), share prefixes, window the SDE or skip recomputation (2x to 5x)[59][39][60]. The family's shared warning is a 5 to 15 percent velocity-norm inflation co-adapted into the weights[61].
What labs run, what academia publishes
Read side by side, the 2026 reports with a training section disagree with the arXiv feed. Seedance 1.0 "directly maximizes the composite rewards" of three reward models by reward gradient and reports that "comparative experiments against DPO/PPO/GRPO demonstrate that our reward maximization approach is the most efficient"[4]; Seedance 2.0's report has no training section[11]. Kandinsky 6.0 Video is the most detailed open recipe of the year[7][62].
| Report | Preference or RL stage, as stated | Algorithm |
|---|---|---|
| Seedance 1.0[4] | three RMs, composite reward maximized directly | reward gradient |
| Kling-Omni[5] | human pairs on motion and visual integrity; GRPO rejected on cost; 10 NFE from 150 | DPO |
| HunyuanVideo 1.5[6] | DPO on ~10K prompts, then online RL with a 4-axis VLM reward; final-model good/same/bad (GSB) +17.12% vs Wan 2.2 | DPO, then online RL (objective unstated) |
| LongCat-Video[40] | 3 RMs, G=4, SDE at one random step in 0 to 6 of 16, 0.5k iterations | GRPO |
| Cosmos-Predict2.5[63] | VideoAlign reward, 8 rollouts x 20 steps, diffusion loss as regularizer | GRPO-style |
| Kandinsky 6.0 Video[7] | K=10, eight rewards over video, audio, sync; KL to SFT at β=1e-4; speech WER 0.235 to 0.124 | OmniNFT (forward process) |
| DreamX-Creator[64] | four rewards in three advantage groups; DeSync 0.190 to 0.135 | OmniNFT-style advantages, loss unstated |
| Veo 3.1, Sora 2, Gemini Omni[65][16] | watermarking and production filtering; nothing on RL | undisclosed |
The labs that explain themselves chose offline or reward-gradient methods for cost and for reward-model weakness; the two that went online kept a diffusion loss or a DPO-like form in the objective. Nobody has published an RL compute budget, so the academic numbers (525 GPU-hours for a 1.3B run, about 1,000 for a 1.3B autoregressive run, "hundreds of GPU-days per experiment" at 14B[8][10][66]) are the only public anchors. The tooling gap closed: Flow-Factory and verl-omni ship Flow-GRPO, DiffusionNFT, DGPO and MixGRPO on Wan[67], and SAGE-GRPO is Tencent's official repo on HunyuanVideo-1.5[45]; an eight-GPU reproduction of a lab-shaped recipe is possible for the first time.
Why it fails
The reward is the experiment
VideoReward reports 73.6 percent pairwise accuracy with ties excluded and 61.3 with ties counted[68]; Reward Lightning's reasoning judge reaches 86.6 and 72.2[69]. The gap is the tie, and it is a modelling choice: VideoAlign treats a tie as a third outcome, and SoliReward's win-tie loss pulls every "pass" sample toward the same score[72]:
\[ \mathcal L_{\text{BT-WT}} = \mathbb E\big[ -\mu \log \sigma(\Delta r) - (1-\mu)\log \sigma(-\Delta r) \big], \qquad \mu = 1\ (\text{win}),\quad \mu = \tfrac12\ (\text{tie}) \tag{7} \]
Collapsing the passes with Eq. (7) shrinks the advantage spike GRPO chases, and post-training HunyuanVideo with it lifts its VideoAlign motion-quality score to 0.33 against 0.17 for plain Bradley-Terry. People are not a clean ceiling either: annotators agree at Fleiss kappa 0.68[69], and pairwise video preference reaches only Krippendorff alpha 0.35[72].
Figure 6 shows how rewards break under optimization. IsoVRBench puts the best reward model at 60.8 percent on temporal shuffle where people score 100 and 71.6 on frame hold where people score 93, and VideoGen-RewardBench accuracy predicts none of it[71]. Reward as an Agent ran DanceGRPO on embodied world models and catalogued occlusion hiding defects, motion reduced to trivial movement and invalid physics; a Doubao judge rated 19.4 and 23.3 percent of VideoAlign's and UnifiedReward-Flex's rewards reasonable[73]. Latent rewards stay high while quality falls[74], AesRM's judges flip with position on 16 to 30 percent of pairs[75], composite rewards get gamed through one component[76][77], and Wan-R1 states the general case: "multimodal reward models could lead to degenerate solutions"[78].
Motion collapse is the universal hack with a universal patch: RVM's ablation on Wan2.1-1.3B drives VBench Dynamic Degree to 2.78 under VideoAlign alone, 5.56 with HPSv3, and 75.00 with a RAFT optical-flow reward[8]. Every fix is hand-built[79]; no reward model yet learns a motion preference that survives optimization. The constructive responses are real video as the winner, zero-label synthetic pairs (IsoVRM-Q2 edges VideoReward as a training reward, 49.8 to 49.0 VBench-2.0, 51.5 when seeded from it[71]), verified checklists[80] and yes-token rewards scaled to 26B[81].
Credit assignment
A language model gets its reward attached to tokens; a video policy gets one scalar for 50 denoising steps and a hundred-plus frames. The 2025 answers, on images, reweighted by timestep[82] or branched[59][83]; Flash-GRPO optimized one step[66]. The 2026 answers go per token and per objective: TVRL turns a frozen VLM's answer likelihood into the reward and its input gradients into a credit map inside the clipped ratio, for +3.60 VBench-2.0 over base[84], with per-step rewards from predicted clean images[85], learned step credit per objective[86], physics scores routed to the aligned steps of a streaming student[87], and prompt credit separated from generator credit[88]. All of them need a judge that exposes a gradient or a per-step score, which pulls the problem back to the reward; Figure 7 marks where each constraint binds.
The streaming model is the policy now
Nearly everything deployed is a distilled few-step model, and on those the SDE construction fails outright: on Self-Forcing "the SDE-based variant fails to improve the reward throughout training", because replacing the solver's intermediate noise changes almost nothing[89]. The four 2026 recipes avoid it: AR-CoPO forks 12 perturbed noises at a pivot chunk and updates that chunk only, with Neighbor GRPO's softmax-over-distances surrogate policy[90]; Astrolabe runs the NFT loss over a rolling KV cache on clip windows[9]; KVPO routes KV-cache entries under a Gibbs surrogate policy[10]; RAVEN treats a consistency step as \(\mathcal N(z_s;\ \alpha_s \hat x_\theta,\ \sigma_s^2 I)\) and runs GRPO on that transition[91]. The gains are real and small, 0.05 to 1.7 VBench Total points with larger moves on HPSv3 (Fig 8), at roughly 1,000 GPU-hours for a 1.3B model[10]; the hazard is measured too, a text-alignment-only run driving motion quality from 1.68 to 0.25[89].
A parallel thread aligns while it distills: Reward Forcing reweights DMD toward high-motion samples at 23.1 FPS on one H100[92], Stream-R1 weights it by \(\exp(r)\)[93], DM-Align runs a single stage[94], and MeanFlowNFT's four-step Wan2.1 reaches VBench 84.33 against 82.57 for LongCat's 50-step RL model[95]. DreamX-World states that RL "recovers camera control and visual quality after distillation"[96]. Treat RL on the deployed sampler as the default and RL on the 50-step teacher as the research setting.
Physics, verifiers and world models
Verifiable rewards are the thread I find most promising, and they moved to inference time first. NewtonRewards penalizes Newtonian violations read from optical flow[97], PhyMotion simulates generated humans as meshes for a 68-point Elo gain[98], Wan-R1 trains on procedurally generated mazes and traps, +29.1 points exact match and +51.4 on trap avoidance over SFT[78][99][87]; HiPhy trains the prompt rewriter, generator frozen[100]. None of this tops a board. Search with a world-model verifier does: WMReward scores candidates with a V-JEPA 2 latent world model[101], and best-of-8 with it (with minimum-Bayes-risk consensus) puts Odyssey-3 Pro first on Verified at 66.10 and adds 3 to 7 points to every base[27]; Off-Manifold Refinement adds a frozen V-JEPA 2.1 surprise gradient to Wan2.2-A14B's velocity and lifts VideoPhy-2 from 47.0 to 52.0 at 1.71x runtime[102]. CachedSearch is our attempt at the cost of that search[103][104].
For world models the objective itself split. WorldCompass post-trains WorldPlay with negative-aware fine-tuning on interaction-following rewards[105], and inverse-dynamics and task-metric rewards check that the video did what the action said[106][107]. DRILL trains the world model for the policy's return after an inner GRPO update and beats WMPO by 13.1 success points despite 17 percent higher next-frame MSE[108][109]. DreamTrue takes AgiBot interaction defects from 48.12 to 6.25 percent[110], and DeepMind already runs its SIMA agent inside Genie 3's worlds†, the payoff that justifies the program[111].
†Cosmos-Predict2.5's report has a full RL section[63]; the Genie 3 post says nothing about post-training[111].What is missing, and what to try first
The lower left of the map is crowded and production-grade, the right edge is climbing one clip at a time, the top row is where the money is going with learned rewards, and the verifier-plus-interactive-model corner holds only DRILL, a manipulation simulator. Ranked by how much I think each matters:
- Rewards that survive optimization. Measure the reward under the policy with intervention pairs and a second judge[71][73]; refit on current samples anchored to real video[74]; count ties[72].
- A motion term by default. Dynamic Degree 2.78 without one and 75.00 with one is the cheapest ablation in the field[8].
- Likelihood-free RL on the deployed model. Forward-process objectives cut cost by an order of magnitude and are the only ones that work on distilled AR samplers[8][57][9]. Nobody has shown one above 14B with a public cost number; that is the experiment I would fund.
- Physics at training time that beats search. Best-of-8 with a V-JEPA scorer adds 3 to 7 Physics-IQ points to any base and no training-time physics reward has matched it[27].
- A reward for minute three. Context windows run 3.0 to 9.2 seconds[24], every model forgets off-screen state[33], and the AR recipes train on clip windows[9]. Nothing in the reward literature scores a minute.
Honest edges. Every cost number above is as reported in its own paper; none are compute-matched, and the Verified board mixes third-party and self-reported entries. DGPO has no video result, and the Sora 2 system card could not be fetched, so that row rests on the Veo card and Omni posts[65][16]. The paper list the August version counted stopped in April 2026[113], and nothing has replaced it. For a wider map, the TMLR survey is where I would go next[114].
References
- Polyak et al., Movie Gen: A Cast of Media Foundation Models, 2024.
- Wan Team, Wan: Open and Advanced Large-Scale Video Generative Models, 2025.
- Seaweed Team, Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model, 2025.
- ByteDance Seed, Seedance 1.0: Exploring the Boundaries of Video Generation Models, 2025.
- Kling Team, Kling-Omni Technical Report, 2025.
- Tencent Hunyuan, HunyuanVideo 1.5 Technical Report, 2025.
- Team Kandinsky, Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation, 2026.
- Choi et al., Scaling Reinforcement Learning for Diffusion Models via Velocity Matching (RVM), 2026.
- Zhang et al., Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models, 2026.
- Zhang et al., KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration, 2026.
- ByteDance Seed, Seedance 2.0: Advancing Video Generation for World Complexity, 2026.
- OpenAI, Sora 2 Is Here, 2025.
- TechCrunch, Why OpenAI Really Shut Down Sora, 2026.
- OpenAI, API Deprecations (Sora API removed September 24, 2026), 2026.
- Kuaishou Technology, First Quarter 2026 Results (Kling AI annualized revenue approached USD 500 million), 2026.
- Google, Introducing Gemini Omni, 2026.
- Arena, Text-to-Video and Image-to-Video Leaderboards, snapshot of September 21, 2026.
- Artificial Analysis, Video Generation Arena (AA-Video-T2V v2.1), fetched October 31, 2026.
- MiniMax, MiniMax-H3 model card on Hugging Face (community license, CFG-distilled checkpoints), 2026.
- Yin et al., From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CausVid), 2024.
- Huang et al., Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, 2025.
- Cui et al., Self-Forcing++: Towards Minute-Scale High-Quality Video Generation, 2025.
- Liu et al., Rolling Forcing: Autoregressive Long Video Diffusion in Real Time, 2025.
- Chen et al., Context Forcing: Consistent Autoregressive Video Generation with Long Context, 2026.
- Motamed et al., Do Generative Video Models Understand Physical Principles? (Physics-IQ), 2025.
- Rädsch et al., Physics-IQ Verified, 2026.
- Google DeepMind, Physics-IQ Benchmark Leaderboard, snapshot of October 2026.
- Bansal et al., VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation, 2025.
- Zheng et al., VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness, 2025.
- Cai et al., MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation (v1, December 2025; v3, September 2026, restructures the tables), 2025.
- Cai et al., What-If World: A Causal Benchmark for General World Models in Embodied Scenarios, 2026.
- Khanbayov and Kurban, PhysWeep: Does a Video Generator Realize the Physics You Ask For?, 2026.
- Lu et al., Current World Models Lack a Persistent State Core, 2026.
- Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, 2025.
- Silver and Sutton, Welcome to the Era of Experience, 2025.
- Rafailov et al., Direct Preference Optimization: Your Language Model Is Secretly a Reward Model, 2023.
- Liu et al., Flow-GRPO: Training Flow Matching Models via Online RL, 2025.
- Xue et al., DanceGRPO: Unleashing GRPO on Visual Generation, 2025.
- Li et al., MixGRPO: Unlocking Flow-Based GRPO Efficiency with Mixed ODE-SDE, 2025.
- Meituan LongCat Team, LongCat-Video Technical Report, 2025.
- Wang et al., Pref-GRPO: Pairwise Preference Reward-Based GRPO for Stable Text-to-Image Reinforcement Learning, 2025.
- Lin et al., When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO, 2026.
- Wang et al., GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping, 2025.
- Wang et al., λ-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource, 2026.
- Zheng et al., Manifold-Aware Exploration for Reinforcement Learning in Video Generation (SAGE-GRPO), 2026.
- Wang and Yu, Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching, 2025.
- Wallace et al., Diffusion Model Alignment Using Direct Preference Optimization, 2023.
- Wu et al., DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models, 2025.
- Han et al., Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking (ThermoDPO), 2026.
- Liu et al., Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion (cIPO), 2026.
- Huang et al., Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models (LocalDPO), 2026.
- Cai et al., PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation, 2025.
- Luo, Hu and Tang, Reinforcing Diffusion Models by Direct Group Preference Optimization (DGPO), 2025.
- Zheng et al., DiffusionNFT: Online Diffusion Reinforcement with Forward Process, 2025.
- Bergmeister et al., Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models, 2026.
- Xu et al., Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View, 2026.
- Wan et al., DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models, 2026.
- Shen et al., Routing Policy Optimization for Online Diffusion Reinforcement Learning (RPO), 2026.
- Li et al., BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models, 2025.
- Wang et al., LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL, 2026.
- Pan et al., NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning, 2026.
- Zhang et al., OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation, 2026.
- NVIDIA, World Simulation with Video Foundation Models for Physical AI (Cosmos-Predict2.5), 2025.
- Zhu et al., DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution, 2026.
- Google DeepMind, Veo model page and Veo 3 model card (updated January 13, 2026), 2026.
- He et al., Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization, 2026.
- Ping et al., Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models, 2026.
- Liu et al., Improving Video Generation with Human Feedback (VideoAlign / VideoReward), 2025.
- Cheng et al., Reward Lightning: Fast Video Generation via Homologous Preference Distillation, 2026.
- Wang et al., VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning, 2025.
- Li et al., IsoVRBench: Video Reward Modeling Without Human Feedback, 2026.
- Lian et al., SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models, 2025.
- Li et al., Reward as an Agent for Embodied World Models, 2026.
- Su et al., CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models, 2026.
- Han et al., AesRM: Improving Video Aesthetics with Expert-Level Feedback, 2026.
- Li et al., Rethinking Reward Signals in Video GRPO: When Scores Become Targets, 2025.
- Wang et al., Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning, 2026.
- Liu et al., Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning, 2026.
- Ban et al., Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models, 2026.
- Zhang et al., FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling, 2026.
- Wu et al., RewardDance: Reward Scaling in Visual Generation, 2025.
- He et al., TempFlow-GRPO: When Timing Matters for GRPO in Flow Models, 2025.
- Ding and Ye, TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models, 2025.
- Wang et al., Token-Level Video Reinforcement Learning (TVRL), 2026.
- Deng et al., DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment, 2026.
- Li et al., Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation (OTCA), 2026.
- Zhao et al., Distilling Physical Priors into Streaming World Models (PhyS), 2026.
- Zhu et al., FlowCred: Hierarchical Credit Assignment and Functional Coordination for Joint Text-to-Video Post-Training, 2026.
- He et al., AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization, 2026.
- He et al., Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models, 2025.
- Lu et al., RAVEN: Real-Time Autoregressive Video Extrapolation with Consistency-Model GRPO, 2026.
- Lu et al., Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation, 2025.
- Wu et al., Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation, 2026.
- Lin et al., Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching (DM-Align), 2026.
- Huang et al., MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generators, 2026.
- DreamX Team, DreamX-World 1.0: A General-Purpose Interactive World Model, 2026.
- Le et al., What About Gravity in Video Generation? Post-Training Newton's Laws with Verifiable Rewards, 2025.
- Huang et al., PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation, 2026.
- Zhu et al., Video Models Can Reason with Verifiable Rewards, 2026.
- Kazimi et al., HiPhy: Hierarchical Alignment for Physically-Plausible Multi-Principle Video Generation, 2026.
- Yuan et al., Inference-time Physics Alignment of Video Generative Models with Latent World Models (WMReward), 2026.
- Nguyen-Truong et al., Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model, 2026.
- Saini et al., CachedSearch: Training-Free Cached Exploration for Test-Time Search in Video Diffusion, 2026.
- Liu et al., Video-T1: Test-Time Scaling for Video Generation, 2025.
- Wang et al., WorldCompass: Reinforcement Learning for Long-Horizon World Models, 2026.
- Ye et al., Reinforcement Learning with Inverse Rewards for World Model Post-Training, 2025.
- Wu et al., RLVR-World: Training World Models with Reinforcement Learning, 2025.
- Kang et al., DRILL: Training World Models to Improve Policies, Not Predict Pixels, 2026.
- Che et al., Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training, 2026.
- Li et al., DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training, 2026.
- Google DeepMind, Genie 3: A New Frontier for World Models, 2025.
- Bardhan et al., Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning, 2026.
- Awesome-RL-for-Video-Generation, community-curated paper list, last updated April 2026.
- Li et al., Video Generation Models: A Survey of Post-Training and Alignment, 2026.
Citation
@misc{saini2026rlforvideo,
author = {Saini, Shreshth},
title = {RL for Video: The State of the Space},
year = {2026},
month = {August},
howpublished = {\url{https://shreshthsaini.github.io/blogs/rl-for-video.html}},
note = {Blog post}
}