RL Is Worth N Samples
RL fine-tuning of an image model is worth a fixed number of best-of-N samples, and that number depends on who judges. One Flow-GRPO sample matches more than 32 base samples on PickScore, about 4 on ImageReward, and fewer than one on CLIPScore.
Thesis. One Flow-GRPO sample from SD3.5-Medium is worth more than 32 base best-of-N samples under PickScore, about 4 under ImageReward, and fewer than one under CLIPScore, where even a single base sample scores higher.
Main technical point. Best-of-N follows \(\mu + \sigma e_N\) with R² between 0.98 and 1.00, so the exchange rate is set by RL's lead in base standard deviations of each judge. Under that fit a lead of one standard deviation is worth 4 samples and three are worth 444. Flow-GRPO raises the mean and cuts the spread by 24 to 47 percent on the rewards it improves, which shrinks what search adds, but the base does not catch up within 32 samples.
Practical implication. Report RL checkpoints with best-of-N curves and a search-equivalent budget on held-out judges, and check prompt faithfulness at large N: on 80 percent of prompts Flow-GRPO's best of 32 has lower CLIPScore than the base model's best of 32.
Why count RL in samples
RL fine-tuning raises a reward model's score for one image. Best-of-N uses a scorer at inference instead: draw N images from different seeds and keep the one it ranks highest. Its gain depends on the spread of scores across candidates, and visual diversity alone does not guarantee a useful high-scoring tail[1]. We wanted to know whether an RL checkpoint leaves enough variation for search to keep helping.
The question came up while we were planning to train diffusion models for best-of-N instead of for the average, an idea from pass@k training for language models[2][3][4]. That plan only makes sense if today's RL models are bad at best-of-N, and language models give a reason to suspect it: after RL with verifiable rewards, the base model overtakes the RL model on pass@k once k is large[5]. So before training anything, we measured the image case.
RL maximizes the average reward of one sample from the model \(p_\theta\):
\[ J_{\text{RL}} = \mathbb{E}_{x \sim p_\theta}\,[\,r(x)\,] \]
At deployment we draw N samples and keep the best, so the quantity we care about is
\[ J_N = \mathbb{E}_{x_1,\dots,x_N \sim p_\theta}\big[\max_i r(x_i)\big] \approx \mu + \sigma\, e_N . \]
Here \(\mu\) and \(\sigma\) are the mean and spread of the reward over a prompt's samples, and \(e_N\) is the expected best of N standard normal draws∗: 0.56 for N = 2, 1.42 for N = 8, 2.07 for N = 32. RL raises \(\mu\). If it also shrinks \(\sigma\), every extra sample buys less, and RL stays ahead at budget N only while
∗\(e_N\) has closed forms only for very small N. We integrate it numerically; it grows roughly like \(\sqrt{2\ln N}\).\[ \Delta\mu \;>\; e_N\,(\sigma_{\text{base}} - \sigma_{\text{RL}}) . \]
Setup
We compared public RL checkpoints of Stable Diffusion 3.5 Medium[7] with the base model, each under the sampling setting its authors evaluate with†. Flow-GRPO turns the sampling ODE into an SDE so that each step has a likelihood for a GRPO update[6]. DiffusionNFT needs no likelihood and puts the reward into a flow-matching regression on the forward process[8]. Both train on reward models built from human preference data[9][10].
†Flow-GRPO reports results at classifier-free guidance 4.5[11]; DiffusionNFT samples without guidance. Each RL model is compared with its base under the same setting.- Flow-GRPO, trained on PickScore, with and without its KL penalty, sampled at CFG 4.5.
- DiffusionNFT, trained on a mix of rewards that includes PickScore, sampled without guidance.
Each model drew 32 images for each of 100 DrawBench prompts[12], starting from the same noise, and we scored every image with six reward models: PickScore, HPSv2, HPSv3[13], ImageReward[14], CLIPScore[15] and an aesthetic predictor[16]. We then repeated the study on Z-Image-Turbo[17]. For best-of-N we do not resample. From n scored images, the expected best of a random subset of size N has a closed form in the sorted scores \(s_{(1)} \le \dots \le s_{(n)}\), the max@k version of the unbiased pass@k estimator[18]:
\[ \mathbb{E}[\max \text{ of } N] = \sum_{m=N}^{n} \frac{\binom{m-1}{N-1}}{\binom{n}{N}}\, s_{(m)} . \]
This is exact over subsets of the collected pool; the pool itself is finite, so the bootstrap below measures the remaining uncertainty. We ran the evaluation end to end on our own GPUs and did not retrain the RL models: we used the checkpoints their authors released, so the numbers describe the models people download.
| Item | Setting |
|---|---|
| Models | SD3.5-Medium base. Flow-GRPO PickScore, with KL (jieliu/SD3.5M-FlowGRPO-PickScore) and without. DiffusionNFT multi-reward (worstcoder/SD3.5M-DiffusionNFT-MultiReward). Z-Image-Turbo base and a PickScore RL LoRA from DiffusionOPSD. |
| Sampling | SD3.5-M: 40-step Euler flow ODE at 512 px; Flow-GRPO and its base at CFG 4.5 with an empty negative prompt, DiffusionNFT and its base without guidance. Z-Image-Turbo: 9-step flow-matching Euler, no guidance, bf16 transformer and fp32 VAE, 1024 px for curves and 2048 px for the gallery. |
| Noise | Each image starts from its own generator seeded by (seed, prompt index), so every model begins from identical noise for the same prompt and seed. |
| Prompts | SD3.5-M: first 100 unique DrawBench prompts, 32 seeds each (3,200 images per model). Z-Image-Turbo: first 50 DrawBench prompts with 16 seeds, 8 written prompts with 16 seeds for the panels, 12 with 4 seeds at 2048 px for the gallery. |
| Scorers | PickScore v1 (CLIP ViT-H/14), HPSv2.1, HPSv3, ImageReward, CLIPScore (CLIP ViT-L/14), aesthetic predictor. DreamSim distance and Vendi score for diversity. |
| Statistics | Exact order-statistic best-of-N, averaged over prompts. 95% intervals from 2,000 paired bootstrap resamples of prompts. Search-equivalent budget interpolated on the base curve in log N. |
| Scope | Independent best-of-N only. NVIDIA GH200 GPUs on TACC Vista. |
Finding 1: the exchange rate depends on the judge
With guidance, one Flow-GRPO sample scores 23.44 on PickScore. The base model reaches 23.24 only at best of 32. The RL curves stay above the base curves at every N on every reward that RL improved to begin with.
That gives one number per checkpoint and judge \(r\), the search-equivalent budget: the smallest N at which the base model's best-of-N reaches the RL model's mean.
\[ N_{\text{eq}}(r) = \min\Big\{ N \ge 1 \;:\; \mathbb{E}_{x_{1..N} \sim p_{\text{base}}}\big[\max_i r(x_i)\big] \;\ge\; \mathbb{E}_{x \sim p_{\text{RL}}}\,[\,r(x)\,] \Big\} \]
We read it off the measured base curve, interpolating in log N, so it need not be an integer‡. Under the fit the condition becomes \(e_{N_{\text{eq}}} = z\), with \(z = \Delta\mu / \sigma_{\text{base}}\), RL's lead measured in base standard deviations of that judge. Because \(e_N\) grows so slowly, \(N_{\text{eq}}\) grows faster than exponentially in \(z\): a lead of one standard deviation is worth 4 samples, two are worth 28, three are worth 444. A judge on which RL lost has \(z < 0\) and no budget at all.
‡Under the fitted ImageReward parameters the base reaches the RL mean between N = 5 and 6, against 3.9 on the measured curve. We use the measured curve for budgets and the fit for shape and extrapolation.The same Flow-GRPO checkpoint exceeds the base at N = 32 on PickScore and matches roughly N = 4 on ImageReward. On CLIPScore, even the base at N = 1 scores higher. A single number for “RL improvement” would hide that disagreement between judges.
Finding 2: RL shrinks what search can add
Fitting \(\mu + \sigma e_N\) to each curve explains the pattern. Flow-GRPO raises the mean and cuts the spread by 24 percent on PickScore, 44 percent on ImageReward and 47 percent on HPSv3. In raw numbers, best of 32 adds 0.94 PickScore to the guided base and 0.72 to Flow-GRPO, and 0.61 versus 0.34 on ImageReward. DiffusionNFT loses more: search adds 1.58 ImageReward to its base and 0.21 to the RL model.
Extrapolating the fitted curves puts the crossover near 180 samples on ImageReward (Fig. 1), tens of thousands on HPSv3, and further out on PickScore. These are projections of a two-parameter fit well beyond our 32-sample measurements, and they depend on reward tails we never observed, so read them as orders of magnitude. Within the measured range the image result differs from the language-model one: on every reward that RL improved, the base never overtakes it by N = 32.
The usual answer to RL collapse is to report a diversity score, such as DreamSim distance[19] or the Vendi score[20], and to keep a KL penalty toward the base. Neither tracks what search can recover. Without the KL penalty Flow-GRPO is less diverse (Vendi 2.9 against 3.3, with 5.3 for the base) and its best-of-N gains are about the same on every reward. From base to Flow-GRPO, Vendi falls 37 percent while the search gain on PickScore falls 24 percent.
Guidance also changes diversity. CFG 4.5 cuts the base model's Vendi score from 12.7 to 5.3 over 32 seeds. DiffusionNFT, which runs without guidance, has a Vendi score near 2, an effective diversity rather than a count of two unique images. The lost spread may also matter during training. GRPO divides each reward by its group's standard deviation, and Pref-GRPO shows that when scores sit close together this inflates trivial gaps into “illusory advantages” the model over-optimizes[21]. We did not test whether collapse feeds that loop.
Finding 3: search does not buy back prompt faithfulness
CLIPScore is lower for the RL model at every tested budget. It is a proxy for image-text similarity, not a complete test of prompt faithfulness. On 80 percent of prompts, Flow-GRPO's best of 32 has lower CLIPScore than the base model's best of 32, and the examples below show prompt details that search fails to recover.
The judge used for reranking matters too. Ranking the same 32 images by PickScore or by HPSv3 picks different winners at almost every N, and sometimes the winner breaks the prompt. The training-side version is documented: in text-to-image RL, preference rewards and prompt-consistency rewards each cause reward hacking, and ensembling them only partly mitigates it[22].
The same pattern on a distilled model
We repeated the study on Z-Image-Turbo, a 6B model sampled in 9 steps, against a version fine-tuned with RL on PickScore§: 50 DrawBench prompts, 16 seeds each, at 1024 px. The base already has low measured diversity, with a Vendi score of 2.3, falling to 1.6 after RL. This comparison alone does not isolate how much of that low diversity comes from distillation.
§The RL LoRA is the one released with DiffusionOPSD[23], the only public RL checkpoint for Z-Image-Turbo we found.With so little spread, search helps neither model much. Best of 16 adds 0.51 PickScore to the base and 0.46 to the RL model, while the RL lead is 2.25. On held-out scorers one RL sample is worth about 3 base samples to HPSv3 and about 10 to ImageReward. On CLIPScore the RL model starts lower and the gap more than doubles by N = 16.
The gallery below uses six prompts we wrote, generated at 2048 px with four seeds per model, and we picked one image per model and prompt by eye. PickScore rates the RL image higher in every pair. RL rewards heavier texture, dramatic backlight and extra props: the "minimalist" headphone shot gains mist and plants, and the reading nook gains clutter.
Base and after RL. “A narrow street in Kyoto at night in the rain, paper lanterns glowing, reflections on wet stone”
Base and after RL. “A lighthouse on a rocky cliff during a storm, huge waves crashing, dramatic sky”
Base and after RL. “A hummingbird hovering next to a red hibiscus flower, frozen wings, macro photography”
Base and after RL. “A cozy reading nook with a window seat, afternoon sun, stacks of books and a sleeping cat”
Caveats
We measured one released checkpoint per method. Every budget beyond 32 is an extrapolation of a two-parameter fit. CLIPScore stands in for prompt faithfulness and misses details a human would check.
We tested independent sampling only. Noise-space search and particle methods steer generation instead of filtering it[24][1], and they may interact with RL very differently.
Both RL checkpoints also predate the 2026 shift to ODE-native training. DGPO keeps GRPO's group-relative signal but drops the policy gradient, so it trains on the deterministic ODE sampler and runs about 20 times faster than earlier methods[25]. Reinforce Adjoint Matching reaches Flow-GRPO's peak reward on SD3.5-M in up to 50 times fewer steps[26]. A path-space analysis argues that Flow-GRPO-type and forward-matching updates differ by variance reduction rather than RL principle[27], which suggests these methods face the same mean-up, spread-down trade. We have not measured them.
In practice
- Report RL in samples, under more than one judge. Next to mean reward, publish the best-of-N curve and \(N_{\text{eq}}\) on at least one held-out judge. The estimator below turns 32 samples per prompt into the whole curve.
- Track search gain, not a diversity score. Vendi fell 37 percent from base to Flow-GRPO while the PickScore search gain fell 24, and dropping the KL penalty left search gains about the same. The best-of-N curve is the direct measurement.
- Rerank with a different judge. Training on PickScore and reranking with a prompt-faithfulness scorer could recover part of what RL removed. We have not tested how far this goes.
- Do the cost arithmetic. For \(N_{\mathrm{eq}}>1\), equal per-sample costs give a rough break-even of \(C_{\mathrm{train}}/((N_{\mathrm{eq}}-1)C_{\mathrm{sample}})\) queries. This excludes scoring, serving overhead and differences between samplers, and we did not measure the full comparison.
- Train for the search you deploy. If the product reranks N images, the objective should be \(J_N\), not \(J_{\text{RL}}\). Pass@k and max@k objectives are one route[2][3], and a first max@k method for text-to-image diversity already exists[28]. The target is to raise \(\mu\) while keeping \(\sigma\).
- Post-train distilled models without spending their spread. Turbo models start with little variety, so RL for them should raise the mean without using up what is left.
from math import comb
import numpy as np
def best_of_n(scores, N):
"""Exact E[max of N] from n >= N scored samples of one prompt."""
s = np.sort(scores); n = len(s)
w = [comb(m - 1, N - 1) / comb(n, N) for m in range(1, n + 1)]
return float(np.dot(w, s))
References
- N. Ma, S. Tong, H. Jia et al., Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps, 2025.
- Z. Chen, X. Qin, Y. Wu et al., Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models, 2025.
- C. Walder, D. Karkhanis, Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems, 2025.
- A. Balashankar, Z. Sun, J. Berant et al., InfAlign: Inference-aware language model alignment, 2024.
- Y. Yue, Z. Chen, R. Lu et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025.
- J. Liu, G. Liu, J. Liang et al., Flow-GRPO: Training Flow Matching Models via Online RL, 2025.
- P. Esser, S. Kulal, A. Blattmann et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, 2024.
- K. Zheng, H. Chen, H. Ye et al., DiffusionNFT: Online Diffusion Reinforcement with Forward Process, 2025.
- Y. Kirstain, A. Polyak, U. Singer et al., Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, 2023.
- X. Wu, Y. Hao, K. Sun et al., Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis, 2023.
- J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, 2022.
- C. Saharia, W. Chan, S. Saxena et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, 2022.
- Y. Ma, Y. Shui, X. Wu et al., HPSv3: Towards Wide-Spectrum Human Preference Score, 2025.
- J. Xu, X. Liu, Y. Wu et al., ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, 2023.
- J. Hessel, A. Holtzman, M. Forbes et al., CLIPScore: A Reference-free Evaluation Metric for Image Captioning, 2021.
- C. Schuhmann, CLIP+MLP Aesthetic Score Predictor (improved-aesthetic-predictor), 2022.
- Z-Image Team, Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer, 2025.
- M. Chen, J. Tworek, H. Jun et al., Evaluating Large Language Models Trained on Code, 2021.
- S. Fu, N. Tamir, S. Sundaram et al., DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, 2023.
- D. Friedman, A. B. Dieng, The Vendi Score: A Diversity Evaluation Metric for Machine Learning, 2022.
- Y. Wang, Z. Li, Y. Zang et al., Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning, 2025.
- Y. Hong, K.-C. Kao, H. Zhou et al., Understanding Reward Hacking in Text-to-Image Reinforcement Learning, 2026.
- W. Zhou, X. Zhu, L. Kong et al., On-Policy Self-Distillation in Diffusion Models, 2026.
- R. Singhal, Z. Horvitz, R. Teehan et al., A General Framework for Inference-time Scaling and Steering of Diffusion Models, 2025.
- Y. Luo, T. Hu, J. Tang, Reinforcing Diffusion Models by Direct Group Preference Optimization, 2025.
- A. Bergmeister, S. Jegelka, N. Nüsken et al., Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models, 2026.
- Y. Xu, Y. Zhang, S. Luo et al., Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View, 2026.
- K. Onoda, P. Parmas, H. Furuta et al., Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation, 2026.
Citation
@misc{saini2026rlworthn,
author = {Saini, Shreshth},
title = {RL Is Worth N Samples},
year = {2026},
month = {September},
howpublished = {\url{https://shreshthsaini.github.io/blogs/rl-worth-n-samples.html}},
note = {Blog post}
}











