Rectified Flow: A Technical Note on Objectives, Geometry, and Training

01Intuition: Straight Lines Are Cheap

Every generative model of images is secretly solving a transport problem. You start from something trivial to sample (a Gaussian noise tensor) and you must end at something hard to sample (a natural image). The models differ in the route they take between those two points, and the route is not an aesthetic detail. At generation time, every step along the route costs one full forward pass of the network (one NFE, number of function evaluations), so the geometry of the route is literally your latency budget.

A DDPM learns to undo a gradual noising process, and at sampling time it retraces that reversal as a long sequence of small denoising moves. The resulting trajectory is heavily curved, and a curved route can only be followed accurately with many small steps. Rectified Flow asks the blunt question: if the only requirement is to move mass from noise to data, why not learn a route that is as close to a straight line as possible? Straight routes are the best case for any ODE solver. One Euler step follows a perfectly straight path exactly, so the straighter the learned flow, the fewer steps you need before quality stops improving.

The rest of this note builds that picture up with the actual objects: the coupling and interpolation that define the training targets, the regression view of the loss, where RF sits relative to DDPM and flow matching, why curvature is the quantity that controls few-step quality, and how reflow buys straightness. The theme throughout is that RF is not exotic. It is least-squares regression on a velocity field, plus a geometric agenda.

02Setup: Coupling, Interpolation, and Targets

Let \(x_0 \sim p_0\) be a base sample (usually Gaussian), and \(x_1 \sim p_1\) be a data sample. A coupling \(\pi(x_0, x_1)\) defines how these endpoints are paired^1[1]. Given \(t \in [0,1]\), define the linear interpolation

\[ x_t = (1-t)x_0 + t x_1. \]

The instantaneous displacement target is

\[ u(x_0,x_1) = x_1 - x_0. \]

Rectified Flow trains a neural vector field \(v_\theta(x,t)\) to predict this displacement from \((x_t,t)\). If \(v_\theta\) matches the conditional mean velocity, integrating \(\dot{x}_t = v_\theta(x_t,t)\) pushes \(p_0\) toward \(p_1\) with straightened trajectories^[1][2].

One subtlety deserves emphasis early, because it explains most of what follows. Each pair \((x_0,x_1)\) is connected by a perfectly straight segment, but the learned field is not automatically straight. Many different pairs pass through the same point \((x,t)\), and the network can only output one velocity there, the average over all pairs passing through:

\[ v^\star(x,t)=\mathbb{E}\bigl[x_1-x_0 \,\big|\, x_t=x\bigr]. \]

Wherever segments cross, that average points somewhere none of the individual segments point, and the marginal flow bends. Under an independent coupling of noise and data, crossings are everywhere, so a first-pass RF model is straighter than a DDPM trajectory but still noticeably curved. Removing the crossings, by re-pairing endpoints so segments no longer intersect, is exactly what reflow does in Section 07. Keep this picture in mind: straight pieces, curved average, and rectification as untangling.

03Loss Function and the Regression Optimum

The basic objective is

\[ \mathcal{L}_{\mathrm{RF}}(\theta) = \mathbb{E}_{(x_0,x_1)\sim\pi,\; t\sim\rho} \bigl[\|v_\theta(x_t,t) - (x_1-x_0)\|_2^2\bigr]. \]

Here \(\rho(t)\) is the time-sampling distribution (uniform is the default; non-uniform choices are often better in practice). The minimizer at each \((x,t)\) is the conditional expectation

\[ v^\star(x,t) = \mathbb{E}[x_1-x_0 \mid x_t=x,\; t]. \]

This follows from the projection theorem in \(L_2\):

\[ \mathbb{E}\|v-u\|^2 = \mathbb{E}\|v-v^\star\|^2 + \mathbb{E}\|u-v^\star\|^2, \]

so optimization can only reduce the first term; the second is irreducible variance from ambiguous pairings. This decomposition is useful when debugging: if training loss plateaus early with large data variance, the bottleneck may be coupling noise, not model capacity.

In matrix form for a batch of size \(B\), define \(X_t \in \mathbb{R}^{B\times d}\), \(U \in \mathbb{R}^{B\times d}\), and \(V_\theta(X_t,t)\in\mathbb{R}^{B\times d}\). Then

\[ \mathcal{L}_B(\theta)=\frac{1}{B}\|V_\theta(X_t,t)-U\|_F^2. \]

This Frobenius form makes clear that RF training is a standard least-squares regression over features \((x_t,t)\), which is why most optimization tricks transfer directly from diffusion training pipelines^[5].

04Where RF Sits: DDPM, Score SDE, Flow Matching

RF is easiest to place once you see that the whole family shares one template. Pick a path that interpolates between a noise sample and a data sample,

\[ x_t = \alpha_t\, x_1 + \sigma_t\, x_0, \qquad \dot{x}_t = \dot{\alpha}_t\, x_1 + \dot{\sigma}_t\, x_0, \]

then train a network to predict something that identifies the path at \((x_t, t)\): the noise \(\epsilon\), the clean sample \(x_1\), the score \(\nabla_x \log p_t(x)\), or the velocity \(\dot{x}_t\). These targets are affine functions of one another at fixed \((x_t,t)\), so the choice is a parameterization decision, not a modeling decision. What genuinely differs across the family is the schedule \((\alpha_t, \sigma_t)\) and the sampler you run afterwards.

Two consequences of this table are worth internalizing. First, RF is flow matching with the linear path, plus an explicit program (reflow) for straightening the marginal flow; the training losses are the same MSE template. Second, a DDPM is not doomed to a thousand steps because of its loss. Its probability-flow ODE can be integrated deterministically like any flow. It needs many steps because the VP schedule bends trajectories hard near the ends, which is precisely the geometric defect the linear interpolant avoids^[3].

05Matrix Sanity Check: Deterministic Linear Map

Consider a deterministic linear transport \(x_1 = A x_0\)², with \(x_0 \sim \mathcal{N}(0, I)\). Then

\[ x_t = \bigl((1-t)I + tA\bigr)x_0 = M_t x_0, \quad M_t := (1-t)I+tA. \]

The target displacement is \(u=(A-I)x_0\), so expressed in terms of \(x_t\):

\[ u = (A-I)M_t^{-1}x_t = K_t x_t, \quad K_t := (A-I)M_t^{-1}. \]

Therefore the optimal velocity is exactly linear in \(x_t\) at each \(t\). If you fit a linear model class

\[ v_W(x,t)=W_t x, \]

then \(W_t=K_t\) is the minimizer, and normal equations recover it in closed form in finite data when \(X_t^\top X_t\) is full rank:

\[ W_t^{\star} = (X_t^\top X_t)^{-1}X_t^\top U. \]

This toy case explains why RF often converges quickly on low-dimensional synthetic distributions: the regression target is well-conditioned and nearly linear. In high-dimensional image manifolds, the same formula holds locally, but \(K_t\) varies sharply across regions, which is where network capacity and time embeddings matter.

A concrete instance makes the geometry vivid. Take \(A\) to be the 90° rotation \(A=\begin{pmatrix}0 & -1\\ 1 & 0\end{pmatrix}\). Then

\[ M_t=\begin{pmatrix}1-t & -t\\ t & 1-t\end{pmatrix}, \qquad \det M_t = (1-t)^2 + t^2 \;\ge\; \tfrac{1}{2}, \]

so the interpolation never collapses, and \(K_t=(A-I)M_t^{-1}\) is a rotation-scaling whose angle sweeps continuously as \(t\) goes from 0 to 1. Even this fully Gaussian, fully linear problem has a marginal flow that bends: the independent coupling forces mass to swap places, and the swap shows up as curvature. The resolution is the same as everywhere else in this note. For Gaussian endpoints, iterating reflow provably converges to the linear optimal-transport (Monge) map, so the bending is a coupling artifact, not a modeling limitation^[12].

06Sampling Dynamics and Why Curvature Matters

Generation solves the ODE

\[ \frac{dx_t}{dt} = v_\theta(x_t,t), \qquad x_{t=0}\sim p_0. \]

Euler integration with \(N\) steps uses

\[ x_{k+1} = x_k + \Delta t\, v_\theta(x_k,t_k), \qquad \Delta t = 1/N. \]

The second time derivative along the trajectory is

\[ \ddot{x}_t = \partial_t v_\theta(x_t,t) + J_{v_\theta}(x_t,t)\,v_\theta(x_t,t), \]

and this directly controls local truncation error:

\[ e_{\text{local}} = \frac{\Delta t^2}{2}\ddot{x}_t + \mathcal{O}(\Delta t^3). \]

The practical message is simple: for a fixed step budget, you want small effective curvature. Reflow/rectification methods can be interpreted as pushing the learned coupling toward self-consistency so that the solver follows straighter paths^[1].

Straightness can be made quantitative. For a flow with trajectories \(x_t\), define

\[ S \;=\; \int_0^1 \mathbb{E}\,\bigl\|\, (x_1 - x_0) - \dot{x}_t \,\bigr\|^2 \, dt . \]

\(S=0\) exactly when every trajectory is a straight line traversed at constant speed, in which case one Euler step is exact. Liu et al. prove that each round of reflow does not increase \(S\), which is the formal version of “rectification straightens the flow”^[1]. Tracking an empirical estimate of \(S\) during validation is a cheap way to predict few-step sampling quality without running a full eval.

Three practical layers sit on top of the error formula. First, solver order: Euler's global error is \(\mathcal{O}(\Delta t)\), while Heun's second-order corrector reaches \(\mathcal{O}(\Delta t^2)\) at two network calls per step; the EDM study found Heun to be the sweet spot for diffusion-family ODEs at moderate budgets^[5], and dedicated exponential integrators such as DPM-Solver exploit the semi-linear structure of diffusion ODEs to do even better at 10 to 20 steps^[11]. Second, where the curvature lives: for VP-style schedules it concentrates near the data end of the trajectory, which is why EDM's \(\rho\)-schedule crowds its steps there, and why RF training recipes that emphasize boundary times (Beta sampling) or mid-path variance (logit-normal) both help in different regimes^[7]. Third, stiffness at the data end: any conversion between the velocity and endpoint views multiplies by \(1/(1-t)\), so late-time steps amplify prediction error; production samplers clamp the final time at \(t_{\max}=1-\varepsilon\) and treat the last step specially.

And the deeper “why” behind all of it is the crossing argument from Section 02: curvature in the marginal flow is the geometric price of averaging over a coupling whose segments intersect^[1][3]. Solvers manage that price; only rectification actually removes it.

07Reflow, Distillation, and Few-Step Generation

A common rectification loop is:

1. Train base \(v_{\theta_0}\) with the standard RF objective.
2. Sample \(x_0\sim p_0\), then integrate \(\dot{x}=v_{\theta_0}(x,t)\) to obtain \(\hat{x}_1=\Phi_{\theta_0}^{0\to1}(x_0)\).
3. Use pairs \((x_0,\hat{x}_1)\) to retrain a new field \(v_{\theta_1}\).

This process is closely related to reflow/trajectory-straightening ideas in RF literature and to practical guidance adaptations in conditional flow sampling^[1][4].

The retrained objective is

\[ \mathcal{L}_{\mathrm{reflow}}(\theta) = \mathbb{E}_{x_0\sim p_0,\; t\sim\rho} \left[\left\|v_\theta\bigl((1-t)x_0+t\hat{x}_1,t\bigr) - (\hat{x}_1-x_0)\right\|^2\right]. \]

In addition, weighted losses are usually more stable than plain MSE in large latent spaces:

\[ \mathcal{L}_{\lambda}(\theta) = \mathbb{E}\left[\lambda(t)\,\|v_\theta(x_t,t)-u\|_2^2\right], \qquad \lambda(t)=\frac{1}{t(1-t)+\tau}. \]

The \(\lambda(t)\) factor up-weights boundary regions where errors are most visible. In practice, choose \(\tau\in[10^{-3},10^{-2}]\) to avoid exploding weights.

From a broader viewpoint, this sits inside the stochastic-interpolant family: change the interpolation law and you change both the conditional targets and the geometry of the learned field^[3].

It is worth separating reflow from distillation, because the two are often blurred together. Reflow retrains the velocity field on self-generated couplings. The result is still a genuine ODE model: you can run it at 1 step or 50, and quality degrades gracefully with fewer steps. Distillation instead trains a student to reproduce the teacher’s solver output at a fixed budget, typically one step. It squeezes out the last factor of latency but freezes the step count and inherits any bias in the teacher’s samples. The strongest few-step systems do both in sequence: reflow once or twice to straighten, then distill the nearly-straight flow into a one-step generator, which is exactly the InstaFlow recipe^[8]. In practice one reflow round captures most of the straightening, and additional rounds trade diversity for marginal gains, so measure \(S\) and FID per round rather than iterating blindly.

The reflow operator also comes with guarantees that are stronger than the pictures suggest. Writing \(Z^{k+1}=\mathrm{Reflow}(Z^k)\) for the coupling produced by integrating the \(k\)-rectified field, three properties hold^[1][12]:

• Marginal preservation. Every \(Z^k\) has exactly the marginals \(p_0\) and \(p_1\); rectification never trades sample fidelity for speed at the population level.
• Transport-cost monotonicity. \(\mathbb{E}\,[c(x_1-x_0)]\) is non-increasing in \(k\) for every convex cost \(c\), simultaneously; reflow moves the coupling toward optimal transport without picking a cost in advance.
• Straightness monotonicity. \(S(Z^{k+1})\le S(Z^k)\), with equality only at fixed points; for Gaussian endpoints the fixed point is the linear OT map^[12].

It also helps to place reflow against its neighbors in the few-step landscape. Progressive distillation halves the step count each round by regressing a student on two teacher steps, keeping the diffusion parameterization intact^[9]. Consistency models train a map \(f(x_t,t)\mapsto x_1\) that is self-consistent along trajectories, deleting the ODE solve entirely at inference^[10]. Reflow is different in kind: it keeps a bona fide velocity field and improves the geometry that any solver will face. The approaches therefore compose rather than compete, and the strongest recipes chain them, reflow for geometry, then a distillation or consistency stage for the final collapse to one step^[8]. The shared caveat is that every post-hoc stage trains on synthetic pairs, so the teacher's sample quality is a hard ceiling; production pipelines keep some real-data fine-tuning or preference signal in the loop to avoid compounding the teacher's biases.

08Implementation Corner

Parameterization identities (useful in implementation)

If \(v\) is predicted, then endpoint estimates follow directly:

\[ \hat{x}_1 = x_t + (1-t)\,v_\theta(x_t,t), \qquad \hat{x}_0 = x_t - t\,v_\theta(x_t,t). \]

These formulas are often used for auxiliary losses: e.g., reconstruction losses on \(\hat{x}_1\), or consistency regularizers between endpoint predictions at neighboring times.

Minimal training snippet with weighting and stable target scaling

def rf_weighted_loss(model, x1, tau=1e-2):
    x0 = torch.randn_like(x1)
    # Beta sampling can emphasize boundaries without singular weights
    t = torch.distributions.Beta(0.9, 0.9).sample((x1.size(0),)).to(x1.device)
    t = t[:, None, None, None]

    xt = (1.0 - t) * x0 + t * x1
    target_v = x1 - x0
    pred_v = model(xt, t)

    w = 1.0 / (t * (1.0 - t) + tau)
    per_example = ((pred_v - target_v) ** 2).flatten(1).mean(1, keepdim=True)
    return (w.flatten(1) * per_example).mean()

Time sampling: where the gradient signal goes

The distribution \(\rho(t)\) decides which part of the trajectory the network practices most. Uniform sampling is the neutral baseline. Beta distributions with parameters below one push mass toward the endpoints, which sharpens the final approach into the data manifold. Logit-normal sampling does the opposite: it concentrates training in the middle of the path, where the velocity target has the highest conditional variance, and it is the choice that shipped in SD3 after a broad ablation^[7].

Resolution changes the clock: timestep shift

A subtlety that bites when you scale resolution: the same \(t\) corrupts a 1024px latent far less, perceptually, than a 256px one, because noise averages out over more spatially-correlated pixels. SD3 compensates with a resolution-dependent time reparameterization^[7],

\[ t' \;=\; \frac{s\,t}{1 + (s-1)\,t}, \]

with shift \(s\approx 3\) at 1024px, which drags both training emphasis and the inference-time step grid toward the noisy end. Two practical corollaries. If you fine-tune a pretrained flow at a new resolution and keep the old grid, your steps land in the wrong part of the trajectory, and quality drops for no visible reason in the loss. And the shift must be applied consistently at train and sample time; mixing shifted training with an unshifted sampler is one of the most common silent bugs in RF fine-tuning pipelines.

Field notes (bugs I have actually chased)

• Time conditioning. Use continuous Fourier or sinusoidal embeddings of \(t\in[0,1]\); reusing a discrete 1000-step DDPM timestep embedding and rescaling it interpolates poorly and shows up as banding at low NFE.
• Numerical guards. Every identity in this section divides by \(t\) or \(1-t\). Sample \(t\in[\varepsilon, 1-\varepsilon]\) with \(\varepsilon\approx 10^{-5}\) during training and clamp the final sampler step; NaNs from the boundary are rare enough to hide for days and correlated enough to poison EMA when they hit.
• Latent statistics. In latent-space RF the base distribution is \(\mathcal{N}(0,I)\), so the VAE latents must actually be scaled to unit-ish variance (the SD-VAE scale factor exists for exactly this reason). A mismatched latent std tilts every trajectory and the model wastes capacity relearning an affine map.
• Loss weighting is schedule design. A uniform-\(t\) velocity MSE is implicitly a particular SNR weighting of the equivalent \(\epsilon\)-loss; if you port a weighting recipe from a diffusion codebase, re-derive what it does to the velocity view before trusting it^[5][7].
• Variance reduction via pairing. The irreducible term in the loss decomposition of Section 03 is coupling variance. Pairing noise and data within caption or class buckets lowers it measurably for conditional models; random global pairing is the right default only for unconditional runs.
• EMA discipline. Decay 0.999 to 0.9999 scaled with batch size, evaluate with EMA weights only, and never let a non-EMA checkpoint into a reflow round: the pairs it generates bake its noise into the next model.

09Practical Notes (Detailed Checklist)

• Time sampling: Uniform \(t\sim U[0,1]\) is a clean baseline, but beta distributions such as Beta(0.9, 0.9) often improve endpoint behavior. If you observe noisy textures near the final steps, increase mass near \(t\approx1\).
• Loss scale management: In latent diffusion backbones, \(\|x_1-x_0\|\) can vary by channel and resolution. Normalize targets per channel or use adaptive gradient clipping; otherwise optimization focuses on high-variance channels.
• EMA is non-optional: Maintain EMA weights for sampling. For RF, EMA typically improves perceived smoothness and text alignment because it reduces high-frequency oscillation in \(v_\theta\).
• Solver choices: Euler is useful for debugging, but midpoint/Heun usually gives better quality at fixed steps. If the model is trained for 8-16 step inference, verify quality with the exact solver used in deployment, not only with high-step validation.
• Guidance scaling: In conditional models, high classifier-free guidance can bend trajectories and amplify curvature terms. A practical compromise is guidance annealing: lower scale early, higher scale near \(t\to1\).
• Reflow scheduling: One reflow pass often gives a large gain; additional passes may have diminishing returns. Measure both FID-like scores and user-facing prompt adherence before paying extra retraining cost.
• Batch construction: Randomly permuting \(x_1\) each step is cheap and surprisingly effective in unconditional settings. For conditional tasks, pair inside each condition bucket (same text class or style bin) to reduce target variance.
• Matrix diagnostics: Track \(\|J_v\|_F\) proxies (finite differences) and trajectory curvature statistics during validation. If curvature grows while training loss decreases, expect low-step sampling regressions.
• Guidance in a window: Applying CFG only in a middle interval of the trajectory, off near both ends, improves flow models the same way it improves diffusion models; full-trajectory guidance is where most oversaturation comes from^[14][4].
• A little stochasticity can help: At moderate budgets, EDM-style churn (inject a small amount of noise, then take a slightly larger step) can break up solver-induced artifacts even on flows; treat it as a knob, not a default^[5].
• Report an NFE sweep: Validate at 1, 2, 4, and 8 steps every time, not only at the high-step setting. Straightness regressions are invisible at 50 steps and catastrophic at 4, and the sweep costs minutes.
• Match the step grid to the shift: If training used a resolution-dependent timestep shift, the sampler grid must use the same shift; a uniform grid on a shifted model concentrates steps where nothing is happening^[7].
• Track \(S\) across reflow rounds: Straightness (from Section 06) should drop monotonically; if it plateaus after round one, further rounds are spending compute to trade diversity for nothing^[1].

10What Ships Today

The linear-interpolant recipe stopped being a research curiosity and became the default. SD3 trains exactly the weighted RF objective from this note at scale, with logit-normal time sampling and an MM-DiT backbone^[7]; Flux and most current video generators follow the same template. One-step and few-step T2I systems stack reflow and distillation on top^[8].

One caution when you move from this note to a production flow model: guidance does not transfer untouched. Classifier-free guidance was tuned on curved diffusion trajectories, and naively extrapolating along a straight flow amplifies curvature exactly where you worked to remove it. That failure mode, and a predictor-corrector fix for it, is the subject of our Rectified-CFG++ work^[4].

References

1. Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, 2022.
2. Lipman et al., Flow Matching for Generative Modeling, 2022.
3. Albergo and Vanden-Eijnden, Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, 2023.
4. Saini et al., Rectified CFG++ for Flow Based Models, NeurIPS 2025.
5. Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, 2022.
6. Ho et al., Denoising Diffusion Probabilistic Models, 2020.
7. Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (SD3), 2024.
8. Liu et al., InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, 2023.
9. Salimans and Ho, Progressive Distillation for Fast Sampling of Diffusion Models, 2022.
10. Song et al., Consistency Models, 2023.
11. Lu et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling, 2022.
12. Liu, Rectified Flow: A Marginal Preserving Approach to Optimal Transport, 2022.
13. Fjelde, Mathieu, and Dutordoir, An Introduction to Flow Matching, 2024.
14. Kynkäänniemi et al., Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, 2024.

Citation

@misc{saini2025rectifiedflow,
  author       = {Saini, Shreshth},
  title        = {Rectified Flow: A Technical Note on Objectives, Geometry, and Training},
  year         = {2025},
  month        = {September},
  howpublished = {\url{https://shreshthsaini.github.io/blogs/rectified-flow.html}},
  note         = {Blog post}
}