Distilling Diffusion Models

Distillation leaves the network its size and shrinks the path from noise to image. DMD2's four-step SDXL student scores FID 19.32 against 19.36 for its teacher at 100 steps.

Thesis. A diffusion sampler is a numerical integrator, and distillation replaces it with a learned map. The three families differ in what the student must match (the teacher's trajectory, the teacher's distribution, or a discriminator's verdict), and that choice fixes what the student loses.

State. Distribution matching (DMD2, SiD) and adversarial training (LADD, APT) hold the one-to-four-step records for images: DMD2 posts FID 1.51 on ImageNet 64 in one step against 2.32 for its teacher's ODE sampler. For video the production recipe is a causal few-step student trained by distribution matching on its own rollouts; Self Forcing streams at 17 FPS with a 0.69 s first frame on one H100 where Wan2.1-1.3B needs 103 s per clip.

Implication. Pick the family by what you can afford to lose. Trajectory matching keeps diversity and the teacher's coupling and inherits every teacher error; distribution matching can beat the teacher's ODE sampler on FID and still drops modes (SiD's one-step ImageNet 64 student has recall 0.63 against its teacher's 0.67); a discriminator gives the sharpest frames and the least stable training. Diversity is the number almost nobody reports, so measure it yourself.

The problem

One SDXL image at 50 sampler steps with classifier-free guidance costs 100 UNet evaluations, two per step for the two guidance branches; DMD2 lists its SDXL teacher at exactly that[1].∗ The same paper clocks the 50-step SD 1.5 teacher at 2.59 s per image and its one-step student at 0.09 s[1]. Video multiplies this by frames. Our CachedSearch post measured a single 81-frame, 50-step Wan2.1-1.3B rollout at about 68 seconds on our GPU; the Wan defaults on one H100 are timed in the video section.

∗Implementations batch the two branches into one call at batch size two, so the FLOPs are the same and the kernel launches halve. The SDXL refiner, if used, adds its own passes.

One big step instead of fifty fails because a denoiser trained with the diffusion MSE loss at noise level \(t\) converges to the posterior mean \(\mathbb E[x_0 \mid x_t]\), the minimum-MSE estimator[2]. Tweedie's formula, which Robbins reported in 1956[3][4], ties that mean to the score of the noised marginal:

\[ \mathbb E[x_0 \mid x_t] \;=\; \frac{x_t + \sigma_t^2\, \nabla_{x_t} \log p_t(x_t)}{\alpha_t} \tag{1} \]

\(x_t = \alpha_t x_0 + \sigma_t \epsilon\), \(p_t\) the marginal density of \(x_t\)

At low noise the posterior is sharp and the mean sits on a mode. At high noise it spreads across every plausible image and the mean sits between them, the MMSE estimate that "frequently 'falls off' the natural image manifold"[2], Dieleman's "average of many plausible images"[5]. Under linear flow matching the exact velocity at \(t = 1\) points at \(\mathbb E[x_0]\), since pure noise carries no information about \(x_0\), so one Euler step from \(t = 1\) lands every sample on the data mean, which the hero figure shows at one step and Fig. 1 shows in one dimension.

teacherstudentloss, gradientdata DATA p(x0) mode mode t = 0.5 xt pt E[x0 | xt] t = 1.5 xt pt drifts t = 3 xt pt pulled to the middle
A bimodal \(p(x_0)\) and its noised marginals \(p_t\) at three noise levels, with one observation \(x_t\) held near the right mode. The violet dot is \(\mathbb E[x_0 \mid x_t]\) from Eq. (1), the point an MSE-trained denoiser returns; as the noise grows it slides from the mode toward the middle, where no data lives[2][5].

Distillation here means training a second network to take the few-step path the teacher cannot. Pruning and quantization change the network; distillation keeps it, "the same architecture and number of parameters as the teacher model"[6]. The teacher represents a denoiser at every noise level; the student needs the map at one, two or four levels and is "only ever evaluated on a small discrete set of times"[6]. Same parameter count, far less to represent, which is why distillation can look free; the next sections track what it costs.

Distillation before diffusion

Hinton, Vinyals and Dean distilled classifiers in 2015 by training the student on the teacher's softened probabilities[7]. The loss is two cross-entropies, against the teacher's soft targets at temperature \(T\) and against the hard labels at \(T = 1\), with the soft-target gradients multiplied by \(T^2\) because their magnitude scales as \(1/T^2\):

\[ \begin{aligned} \mathcal L \;=\;& T^2\, H\!\big(\mathrm{softmax}(v/T),\ \mathrm{softmax}(z/T)\big) \\ &+\; \lambda\, H\!\big(y,\ \mathrm{softmax}(z)\big) \end{aligned} \tag{2} \]

\(v\) teacher logits, \(z\) student logits, \(y\) the labels, \(H\) cross-entropy, \(\lambda\) a weight on the hard term that Hinton et al. found works best well below the soft term's weight

On MNIST an 800-unit net made 146 test errors alone and 74 with soft targets at \(T = 20\), against 67 for the 1200-unit teacher[7].† The student trains on real inputs against a dense per-class target, which in diffusion terms is trajectory matching, where the teacher turns a noisy regression target into a denoised one and the student fits it.

†In the high-temperature limit with zero-meaned logits the soft term's gradient becomes \((z_i - v_i)/(NT^2)\), so distillation is logit regression, \(\tfrac12 (z_i - v_i)^2\)[7].

Kim and Rush moved the target from tokens to sequences[8], training the student by plain likelihood on the teacher's beam-search outputs. A 2x500 student scored 18.9 BLEU with greedy decoding against 14.7 for its baseline and 17.7 for the 4x1000 teacher. The student fits the teacher's samples by likelihood, as reflow and InstaFlow do for images.

MiniLLM switched the divergence[9]. Forward KL "forces the student distribution to overestimate the low-probability regions of the teacher distribution", so the student minimizes reverse KL over its own samples with the teacher as judge; a LLaMA-7B student of LLaMA-13B scored 29.0 Rouge-L on DollyEval against 26.3 for SFT, with the 13B teacher at 29.7. GKD named the reason[10], a "distribution mismatch between output sequences seen during training" and those the student generates at inference, and recorded the cost, since moving from forward to reverse KL "leads to decreased diversity". Thinking Machines ran the recipe at scale in 2025, grading every token of a student trajectory with the teacher's log-probabilities[11], and about 150 on-policy steps took Qwen3-8B-Base from a 400k-prompt SFT checkpoint to 70 percent on AIME'24. A scalar reward gives a fixed number of bits per episode; distillation gives \(O(N)\) for an \(N\)-token trajectory.

Diffusion took the same road, for the same reason; the table maps the steps.

LM methodDiffusion counterpartWhat the student trains on
Soft targets[7]Progressive and consistency distillationReal inputs, a denoised teacher target
Sequence-level KD[8]Reflow, InstaFlow, plain one-step KDTeacher samples; coupling kept
Reverse KL on student samples[9]Diff-Instruct, DMD, SiDIts own samples, scored by the teacher
On-policy distillation[10][11]Self Forcing, DMD2's backward simulationIts own rollouts, in the states it will visit
Scalar reward (RL)Reward terms in T2V-TurboOne number per sample

Why diffusion makes it a specific problem

A trained diffusion model is a vector field, and sampling is the numerical integration of it. At every noise level the network returns the score of the noised marginal, or an equivalent velocity, and a sampler such as DDIM, DPM-Solver or Euler turns that into a path from noise to data whose error falls with the step count. Wan2.1's 50 steps and SDXL's 50 are solver settings. Distillation replaces the solver. The student is trained so that one of its evaluations lands where the teacher's integrator would have landed after many, and the methods in this post differ in how they define that landing point and whether they store it.

An LM teacher hands over its target for free, a distribution over the next token at every position of a real sequence[7]. A diffusion teacher does not. Its weights define a score at every noise level, and the few-step map the student needs exists only implicitly, as the endpoint of an ODE that costs 50 to 100 evaluations to run. The target has to be made, by one of three routes. Run the teacher and store the result, as DMD does with its regression pairs[1] and CausVid with its causal ODE warm start[12] (both costed in Matching the distribution and the Wan2.1 walkthrough); Self Forcing ships the result of that stage as a checkpoint[13]. Define it locally, as consistency models do, by asking that adjacent points on one teacher trajectory map to the same output[14]. Or never write it down, as distribution matching does, and let a second network report where the student's samples differ from the teacher's[15].

The naive target is the worst one. Regress a one-step student onto real images and it converges to the posterior mean of Eq. (1), the blur of Fig. 1; regress it onto the teacher's own output and the target is "fully determined given the teacher model and starting point"[6], which is what progressive distillation, reflow and the ODE warm starts all exploit.

The object being distilled is rarely the trained network. At inference a text-to-image teacher runs twice per step, conditional and unconditional, and combines the two with a scale, so the teacher is a guided sampler whose noised scores are not the diffusion of the guided density, and for scales above one it is not guaranteed to sample any known distribution[16][17]. The student has to pick one scale and keep it. DMD2's real score runs SDXL at guidance 8 and SD 1.5 at 1.75[1], CausVid's teacher runs at 3.5[12], Self Forcing's 14B critic at 3.0[18], rCM's at 5.0[19], and every one of these students samples at guidance 1 afterwards, which halves the cost per step before the step count falls and is why negative prompts stop working[20].

Almost every teacher worth distilling works in a VAE latent, and that moves the problem. The losses, the fake score and the discriminator all operate on latents: LCM distilled in latent space from the start[21], LADD moved ADD's discriminator from DINOv2 pixel features onto the teacher's own latent tokens[22], and DMD2's SDXL recipe trains at a latent resolution of 128 for 1024-pixel images on 500k precomputed VAE latents[23]. The decoder stays outside the loss and inside the clock. For a one-step SDXL student it costs more than the UNet call it follows, timed in Honest edges[24].

Video raises the token count until the attention pattern itself becomes the design. Wan2.1 turns 81 frames at 832x480 into 21 latent frames of 60x104, 1,560 tokens each after 2x2 patching, 32,760 tokens that the bidirectional teacher attends over in every one of its 100 passes[25][13]. A distilled video student keeps three networks resident, the student, the frozen real score and the online fake score, which is why Self Forcing's released recipe uses the 1.3B model for the student and the fake score and the 14B only as the frozen critic[13], and why SiD already needed about half again Diff-Instruct's memory per GPU on ImageNet 64[26]. The streaming students of the video section exist because they pay that attention cost per chunk instead of per clip.

Three things a student can match

Most distillation losses pick one of three targets, and Fig. 2 draws them. A trajectory method pulls the student's prediction onto a point the teacher's ODE would have reached. A distribution method noises the student's own sample, asks the teacher's score and a fake score trained on student samples how the two densities differ there, and moves the student along the difference. A discriminator method trains a classifier to tell student samples from real or teacher-made ones and drops the teacher from the loss.

TRAJECTORY noise data student teacher step loss teacher path DISTRIBUTION real fake score difference pushes fake toward real DISCRIMINATOR x student x data D real fake judge, not a target
Left, a student point is pulled onto a teacher step on the ODE path[6][14]; middle, the fake density is pushed toward the real one along the score difference[15]; right, a discriminator judges a student sample against a reference[27].
FamilyLossExtra networksNeeds real dataKeeps couplingKnown failure
Trajectory (PD, CD, LCM, reflow)Regression onto a teacher ODE pointNone beyond the teacherYes, or teacher pairsYesBlur at 1 to 2 steps; inherits teacher error
Distribution (Diff-Instruct, DMD, SiD)Score difference; reverse KL or FisherA fake score networkNoNoMode dropping; guidance scale frozen
Discriminator (ADD, LADD, Lightning, APT)Hinge or non-saturating GANA discriminator, often on teacher featuresADD and APT yes; LADD uses teacher samplesNoCollapse without regularization; lower diversity

The loss column explains how a student beats its teacher on FID. A diffusion teacher is trained by denoising score matching, which is mode-covering, since every training image must receive density. DMD, Diff-Instruct and SiD minimize reverse KL or a Fisher divergence over student samples, which is mode-seeking: the student is penalized where it puts mass and never for mass it leaves out. That is why a distilled student can post a lower FID than its teacher while covering less of the distribution. Dieleman sets up the first half as a paradox and resolves it by asking whether we "care more about fidelity or diversity"[5]; the KL framing is ours, and the numbers are in Honest edges.

What ships today follows the same split. The open few-step checkpoints from the large labs are adversarial or distribution-matching students, and the consistency family arrives through SANA-Sprint's sCM-plus-LADD hybrid. The table lists the releases whose own card, README or paper states both the method and the step count; LTX-2's distilled checkpoint names 8 steps and no method[28], so it is left out.

ModelDistillation familyStepsStated inSource
FLUX.1 [schnell]Latent adversarial diffusion distillation1 to 4Model card[29]
SD3.5 Large TurboAdversarial diffusion distillation (ADD)4Model card[30]
SD3.5 FlashDistribution matching, DMD objective reformulated for few steps4Paper[31]
SANA-SprintsCM plus LADD, from a SANA teacher1 to 4Paper[32]
Wan2.2-Lightning LoRAsPhased DMD (releases since 28 Sep 2025), no CFG at inference4README[33]
Self Forcing (Wan2.1-1.3B)DMD on the student's own rollouts4README and paper[13][18]

Matching the trajectory

Progressive distillation halves the step count repeatedly[6]. The student starts as a copy of the teacher; the teacher takes two DDIM steps from \(z_t\) to \(z_{t''}\), and the target is the clean-image prediction that makes one student DDIM step land on the same point:

\[ \begin{aligned} \tilde{x} \;&=\; \frac{z_{t''} - (\sigma_{t''}/\sigma_t)\, z_t}{\alpha_{t''} - (\sigma_{t''}/\sigma_t)\,\alpha_t}, \\ \mathcal L_\theta \;&=\; w(\lambda_t)\,\big\lVert \tilde x - \hat x_\theta(z_t)\big\rVert_2^2 \end{aligned} \tag{3} \]

\(z_{t''}\) after two teacher DDIM steps from \(z_t\), \(\hat x_\theta\) the student's clean-image prediction, \(w(\lambda_t)\) an SNR weight

PROGRESSIVE DISTILLATION zt noisy teacherDDIM step 1 teacherDDIM step 2 zt″ two steps studentone step of size 2δ ẑt″ one step lossweighted MSE same input, same t teacher frozen student starts as the teacher HALVING each student becomes the next teacher teacherN steps studentN/2 steps studentN/4 steps … student1 step log₂ N rounds
From one noisy latent \(z_t\) the frozen teacher takes two DDIM steps and the student one step of twice the size; the weighted MSE between the two endpoints trains the student, which then becomes the teacher of the next halving[6].

In posterior-mean terms, the original \(x\) "is not fully determined given" \(z_t\), so a denoiser trained on it predicts "a weighted average of possible" images, while \(\tilde x\) is "fully determined given the teacher model and starting point". On unconditional CIFAR-10, from an 8192-step DDIM teacher, the distilled sampler reaches FID 9.12 at 1 step, 4.51 at 2, 3.00 at 4 and 2.57 at 8, where undistilled DDIM needs 100 steps for 4.16[6]; ImageNet 64 is plotted only, "near optimal" up to 4 or 8 steps.

Guidance doubles the teacher's cost, and Meng et al. distilled it away first[34], regressing the two-evaluation guided prediction into one student conditioned on the scale \(w\) and then running progressive distillation on it. In their ImageNet 64 table, at \(w = 0\) FID falls with steps, 22.74 at 1 to 2.44 at 16; at \(w = 1\) the one-step student scores 7.54 against its own teacher's 9.84, and FID rises with steps to 9.99 at 16. Our reading, which the paper does not offer, is that the guided teacher is sharpened past the FID optimum and the one-step student is the least sharpened model in the table.

Guidance is part of the target. The guided score sharpens the conditional toward its modes[35], and for \(\gamma > 1\) its noised version is the diffusion of no data density: Bradley and Nakkiran prove that neither DDIM nor DDPM with CFG samples the gamma-powered distribution[16], and Karras et al. show the implied density "does not represent a valid heat diffusion", so "entire branches of the distribution are dropped"[17]. A distilled student bakes that in. LCM samples \(\omega \in [2, 14]\) through an embedding[21], LCM-LoRA fixes \(\omega = 7.5\)[36], and DMD2 uses the guided teacher as its real score[1] at the scales given in Why diffusion makes it a specific problem. Students inherit the sharpened distribution, which for a product is usually the point.

Consistency models made the trajectory itself the target[14]. Points on one probability-flow trajectory map to the same origin, \(f(x_t, t) = f(x_{t'}, t')\), with the boundary condition \(f(x_\epsilon, \epsilon) = x_\epsilon\) enforced by the parameterization. Consistency distillation noises a real image, runs one teacher ODE step to the adjacent time, and makes the student agree with an EMA copy of itself there:

\[ \begin{aligned} \mathcal L_{\text{CD}} \;&=\; \mathbb E\Big[\lambda(t_n)\, d\big(f_\theta(x_{t_{n+1}}, t_{n+1}), \\ &\qquad\qquad\ \ f_{\theta^-}(\hat x^\phi_{t_n}, t_n)\big)\Big], \\ f_\theta(x,t) \;&=\; c_{\text{skip}}(t)\,x + c_{\text{out}}(t)\,F_\theta(x,t) \end{aligned} \tag{4} \]

\(\hat x^\phi_{t_n}\) one teacher solver step from \(x_{t_{n+1}}\), \(\theta^-\) an EMA copy, \(d\) squared \(\ell_2\), \(\ell_1\) or LPIPS, \(c_{\text{skip}}(\epsilon) = 1\) and \(c_{\text{out}}(\epsilon) = 0\)

CONSISTENCY DISTILLATION xt time t teacherone ODE step, t → t′ xt′ t′ < t student fθ online target fθ⁻ EMA, stop-grad EMA consistency loss both must give the same x0 BOUNDARY AT t ≈ 0 xε student fθ at t = ε fθ(xε, ε) = xε identity by construction cskip(ε) = 1, cout(ε) = 0
One frozen teacher ODE step moves \(x_t\) to \(x_{t'}\); the online student reads \(x_t\), its EMA copy reads \(x_{t'}\), and the consistency loss pulls the two outputs together. At \(t \to 0\) the parameterization makes the student the identity, which is the boundary condition[14].

Consistency training replaces the teacher step with the unbiased score estimate from the pair \((x,\ x + t z)\), so no teacher is needed. From an EDM teacher at FID 2.04, CD with LPIPS reaches 3.55 in one step and 2.93 in two on CIFAR-10, CT gets 8.70 and 5.83, and on ImageNet 64 CD gets 6.20 and 4.70 against 2.44 at 79 evaluations[14]. iCT-deep, trained from scratch with a Pseudo-Huber distance, reaches 2.51 and 2.24 on CIFAR-10[37]. Fig. 5 draws both pictures.

CONSISTENCY xT t₃ t₂ t₁ ODE trajectory x0 f(xt, t) = x0 for every t PROGRESSIVE DISTILLATION xT x0 8 steps 4 steps 2 steps 1 step two teacher steps one student step halve, retrain, repeat
Left, one ODE trajectory with three points at different times all mapped by the consistency function to the same \(x_0\)[14]. Right, progressive distillation's halving, where two teacher DDIM steps define one student step and the schedule shrinks from 8 to 4 to 2 to 1[6].

Latent consistency models brought this to Stable Diffusion with the guided solver inside the target[21]. Four steps is the FID minimum in both of its tables, 11.10 at 512px against 35.36 at one step, the basis for the four-step default that followed; its 32 A100-hour headline covers 4,000 training steps of the 768 model, while the table runs train for 100K iterations. CTM maps any time to any earlier time[38]; Hyper-SD segments the trajectory with an adversarial distance[20].

sCM is the continuous-time version made trainable[39]. Under TrigFlow, \(x_t = \cos(t)\,x_0 + \sin(t)\,z\), the objective is the tangent \(df_{\theta^-}/dt\), and the contribution is a list of stabilizers for that tangent term. The 1.5B sCD-XXL scores 2.28 in one step and 1.88 in two on ImageNet 512 against its teacher's 1.73, a gap of 8.7 percent,‡ with under 20 percent of the teacher's training compute, and the authors still report "noticeable artifacts".

‡The abstract's "within 10%" holds on ImageNet 512 (8.7 percent above the teacher) and CIFAR-10 (2.06 against 2.01); on ImageNet 64 the two-step sCT's 1.48 sits 11 percent above EDM2's 1.33[39].

Reflow attacks the coupling instead of the solver[40]. Running a rectified flow's ODE produces a deterministic pairing with lower transport cost, and retraining on that pairing straightens the paths until "a single Euler step" works. On CIFAR-10 one Euler step on the 2-rectified flow gives FID 12.21, distilling it gives 4.85, and the full-ODE FID rises with each reflow, 2.58 to 3.36 to 3.96. Straightening buys a one-step map and pays with teacher quality. InstaFlow ran this on SD 1.5 for 199.2 A100-days[41]: FID 23.4 on COCO 2017 against the 25-step teacher's 20.1, where direct one-step distillation of SD 1.4 without reflow scored 40.9 against its 25-step teacher's 22.8. Trajectory methods keep the noise-to-image pairing; distribution methods throw it away.

InstaFlow-0.9B output in one step: a snowy mountain reflected in a lake, with a pine at the rightInstaFlow-0.9B student, 1 step, about 0.1 s
Stable Diffusion 1.5 output at 25 DPM-Solver steps: a snowy forested ridge above a clear lakeSD 1.5 teacher, 25 DPM-Solver steps, about 1.2 s
The last frame of the screen recording in the InstaFlow README[41] (MIT), cropped to the two outputs: the one-step InstaFlow-0.9B student and its teacher, Stable Diffusion 1.5 at 25 DPM-Solver steps, timed on one A100. Seeds are random and unmatched, and the color banding is the GIF's palette. The README adds that downloading each image from the server takes about 0.3 s. Prompt: "A photograph of a snowy mountain near a beautiful lake under sunshine."

Shortcut models and MeanFlow train the few-step map from scratch. A shortcut model takes the step size \(d\) as input and enforces that one step of size \(2d\) equals two steps of size \(d\)[42], reaching 10.6 in one step on ImageNet 256. MeanFlow learns the average velocity between two times through an identity that needs one JVP:

\[ u(z_t, r, t) \;=\; v(z_t, t) \;-\; (t - r)\, \frac{d}{dt}\, u(z_t, r, t) \tag{5} \]

\(u\) the average velocity from \(r\) to \(t\), \(v\) the flow-matching velocity, the total derivative taken along the flow

MeanFlow-XL/2 scores 3.43 at one evaluation on ImageNet 256 from scratch, at about 16 percent overhead over flow matching[43]. As Dieleman cautions, a distilled model shows that a few-step model exists and says nothing about whether distillation is the cheapest way to one[5].

Boffi, Albergo and Vanden-Eijnden read the whole family as one object, the flow map that carries a point between any two times along the probability-flow ODE, in a framework that unifies and extends consistency models, CTM and progressive distillation[44]. Align Your Flow trains that map with two continuous-time objectives, starting from the observation that consistency models' performance "inevitably degrades when increasing the number of steps" while flow maps "remain effective across all step counts", and reports few-step results on ImageNet 64 and 512, plus text-to-image students that, by the authors' account, beat every non-adversarially trained few-step sampler[45]. Inductive Moment Matching names a fourth target beyond the three in Fig. 2, the moments of the distribution, trains a one- or few-step sampler in a single stage, and reaches FID 1.99 at 8 steps on ImageNet 256[46].

Matching the distribution

The second family began in 3D. ProlificDreamer's variational score distillation treats the rendered image as a sample from a distribution and minimizes \(\mathrm{KL}(q_t \,\|\, p_t)\) at every noise level against a pretrained text-to-image model[47]; the gradient is the difference between the frozen teacher's noise prediction and a LoRA's fitted to the current renders. Diff-Instruct wrote the same gradient for image generators under an integral KL, reaching one-step FID 5.57 on ImageNet 64[48]. DMD added the pieces that made it work at scale[15]:

\[ \begin{aligned} &\nabla_\theta D_{\text{KL}}(p_{\text{fake}} \,\|\, p_{\text{real}}) \\ &\quad\simeq\; \mathbb E_{z,t,x,x_t}\Big[ w_t\, \alpha_t\, \Delta s(x_t, t)\, \frac{dG}{d\theta} \Big], \\ &\Delta s(x_t, t) \;=\; s_{\text{fake}}(x_t, t) - s_{\text{real}}(x_t, t) \end{aligned} \tag{6} \]

\(x = G_\theta(z)\), \(x_t\) its noised version, \(\Delta s\) the fake-minus-real score difference, \(w_t = \tfrac{\sigma_t^2}{\alpha_t}\tfrac{CS}{\lVert \mu_{\text{base}}(x_t,t) - x\rVert_1}\) a per-sample normalizer, \(s_{\text{real}}\) from the frozen guided teacher, \(s_{\text{fake}}\) from a network trained online on \(G\)'s samples

The real score comes from the teacher's guided prediction; the fake score is the teacher's architecture fine-tuned online on generator outputs; at convergence the two agree and the gradient vanishes. DMD also kept an LPIPS regression loss against the teacher's ODE output from the same noise "to ensure all modes are preserved", since without it the authors saw "training instability and a propensity for mode collapse"[15]. It scored 2.62 on ImageNet 64 in one step and 11.49 on COCO-30k from SD 1.5 against 8.78 for the 50-step teacher, as measured there.

DISTRIBUTION MATCHING LOOP z noise Ggenerator x + noiseσt xt real scoreteacher, frozen, CFG fake scoretrained on G samples sreal sfake − ∇θ ∝ (sfake − sreal) · dG/dθ gradient to G denoising loss on G samples 1 UPDATE per round 5 UPDATES per round DMD2: no ODE pairs, no regression term, teacher stays frozen DMD2: GAN HEAD ON THE FAKE SCORE real image or G sample + noiserandom t fake score networkbottleneck features classifier headD real or fake extra loss on G
Noise enters the DMD generator, the sample is re-noised, and two score networks read it: the frozen teacher with guidance and a fake score trained online on generator samples. Their difference, weighted by \(w_t \alpha_t\), is the gradient of Eq. (6); DMD2 makes five fake-score updates per generator update and adds the GAN head in the lower panel, a classifier on the fake score's features that sees real images and student samples at matched noise levels[15][1].

DMD2 removed the pairs[1]. Building them costs "about 700 A100 days" for 12 million LAION prompts and bounds the student by the teacher. Dropping the regression loss alone destabilizes training, so DMD2 updates the fake network five times per generator step and adds a GAN head on its bottleneck that sees real images and student samples at matched noise levels. On ImageNet 64 that gives 1.51 in one evaluation against 2.32 for the teacher's ODE sampler; on SD 1.5 it gives 8.35 against the teacher's 8.59.§ On SDXL the four-step model matches the 100-pass teacher on FID, 19.32 against 19.36, after about 60 hours on 64 A100s, and "outperforms its teacher in image quality for 24% of samples". The multi-step student trains on its own earlier steps, GKD's on-policy argument inside a four-step sampler.

§The SD 1.5 teacher at guidance 3 is FID 8.78 in DMD's Table 3 and 8.59 in DMD2's Table 5, same model, different sampler and run. Quote whichever paper the student row comes from[15][1].

SiD reaches the same score difference from a Fisher divergence[26], using a score-projection identity to replace the generator's unknown score with its own samples. The loss it trains on is a squared teacher-to-fake denoiser gap plus a cross term:

\[ \begin{aligned} \tilde{\mathcal L}_\theta \;=\;& (1-\alpha)\,\frac{\omega(t)}{\sigma_t^4}\,\big\lVert f_\phi(x_t,t) - f_\psi(x_t,t)\big\rVert_2^2 \\ &+\; \frac{\omega(t)}{\sigma_t^4}\,\big(f_\phi(x_t,t) - f_\psi(x_t,t)\big)^{\!\top}\big(f_\psi(x_t,t) - x_g\big) \end{aligned} \tag{7} \]

\(f_\phi\) the frozen teacher denoiser, \(f_\psi\) the fake denoiser fitted to generator samples, \(x_g\) the generator's sample and \(x_t\) its noised version, \(\omega(t)\) a weight; at \(\alpha = 1\) the first term vanishes

At \(\alpha = 1.2\) the one-step model scores 1.923 on unconditional CIFAR-10 against the EDM teacher's 1.97 and 1.524 on ImageNet 64 against 1.36, with no real data. SiD-LSG adds guidance and reaches 8.71 on COCO 2014 from SD 1.5, with the trade stated plainly: "smaller kappa values generally yield better FID metrics, while larger values enhance CLIP scores"[49]. EM Distillation shows the family is one Langevin step of an EM procedure and buys coverage back with more steps, at recall 0.59 for EMD-16 against 0.55 for EMD-1 on ImageNet 64[50].

The student can beat the teacher for three mechanical reasons. The teacher's sampler is a finite-step integrator with its own discretization error, the student is judged against the score directly on its own samples, and under reverse KL a student that drops a thin tail pays nothing for it in FID. The price is memory, 31.3 GB per GPU for SiD against 20.4 for Diff-Instruct[26], and a guidance scale frozen at distillation time.

Adding a discriminator

ADD put a GAN loss next to a score-distillation term[27]. A hinge discriminator on frozen DINOv2 features judges the output against real images; the distillation term regresses the re-noised student output onto the teacher's denoising:

\[ \begin{aligned} \mathcal L_{\text{ADD}} \;=\;& -\,\mathbb E_{s,\epsilon,x_0}\Big[\sum_k D_{\phi,k}\big(F_k(\hat x_\theta(x_s,s))\big)\Big] \\ &+\; \lambda\, \mathbb E_{t,\epsilon'}\Big[c(t)\,\big\lVert \hat x_\theta - \hat x_\psi\big(\mathrm{sg}(\hat x_{\theta,t});\,t\big)\big\rVert_2^2\Big] \end{aligned} \tag{8} \]

\(\hat x_\theta(x_s,s)\) the student's denoising of a noised real image at one of its four steps \(s\), \(F_k\) frozen DINOv2 features with hinge heads \(D_{\phi,k}\), \(\hat x_{\theta,t}\) the student output re-noised to \(t\), \(\hat x_\psi\) the teacher, \(\mathrm{sg}\) stop-gradient, \(c(t) = \alpha_t\) by default, \(\lambda = 2.5\)

ADD / LADD z noise studentone step x̂θ student sample + noiseσt teacherdenoises, with CFG x̂ψ target distill lossλ = 2.5 distillation loss, weight λ = 2.5 real imagex0 feature networkADD: DINOv2, frozenLADD: the teacher itself heads Dhinge loss real or fake per feature level adversarial loss, weight 1 LADD drops the distillation term ADD: pixels, 518² max LADD: latents, any size
Adversarial diffusion distillation as a block. The one-step student's sample is re-noised and compared with the teacher's denoising under the distillation loss at \(\lambda = 2.5\), while a discriminator on frozen DINOv2 features (ADD) or on the teacher's own token features (LADD) scores it against a real image[27][22].

On COCO 2017 FID-5k the SD 1.5 version scores 19.7 in one step against 20.1 for DPM-Solver at 25 steps, ADD-XL at four steps beats SDXL at 50 "in the majority of comparisons", and "the enhanced realism comes at the cost of slightly decreased sample diversity".

LADD moved the discriminator into latent space and onto the teacher's own features[22]. No new loss is written down; it reuses ADD's hinge loss with heads on the teacher's token sequence after each attention block and drops the distillation term because "training on synthetic data substantially outperforms training on real data rendering the distillation loss obsolete". SD3-Turbo matches SD3's image quality in four unguided steps and loses prompt alignment, with "object duplication and merging, fine-grained spatial prompting, and difficulties with negation" named as the cost. SDXL-Lightning, progressive halving with a discriminator built from the SDXL UNet encoder, scores FID 22.61 at one step against 18.49 for SDXL at 32 steps and attributes every distilled model's worse FID to "the reduction in diversity"[51].

APT is the end of this line, for video[52]. It starts from a consistency-distilled model at guidance 7.5, because direct adversarial training "leads to collapse", then trains with a pure non-saturating GAN loss on real videos, no teacher in the loss. The discriminator is the 8B diffusion transformer itself, regularized by an approximated R1; without it "training collapses rapidly", and a batch of 256 "has mode-collapsed across prompts and seeds". Trained for 300 updates on 1024 H100s, it produces 1280x720, 24 fps, 2-second video in one step. Its Table 7 is the best same-teacher comparison in video, where consistency distillation at 1 evaluation scores VBench 67.05 total and 40.15 semantic, APT 82.00 and 73.15, the 25-step base 82.15 and 73.31. The discriminator restores what trajectory matching loses in one step, at a cost the authors report as degraded "structural integrity and text alignment".

MethodFamilyStepsBenchmarkFIDSetting and teacherSource
Progressive distillationTrajectory1 / 2 / 4 / 8CIFAR-10 uncond.9.12 / 4.51 / 3.00 / 2.57From an 8192-step DDIM teacher; DDIM 100 steps 4.16[6]
Guided distillationTrajectory1 / 4 / 16ImageNet 64, \(w = 1\)7.54 / 7.95 / 9.99Deterministic sampler; guided DDIM teacher 9.84[34]
CD (LPIPS)Trajectory1 / 2CIFAR-10 uncond.3.55 / 2.93From EDM, 2.04 at 35 NFE[14]
CD (LPIPS)Trajectory1 / 2ImageNet 646.20 / 4.70From EDM, 2.44 at 79 NFE[14]
iCT-deepTrajectory, no teacher1 / 2CIFAR-102.51 / 2.24From scratch, Pseudo-Huber[37]
LCMTrajectory1 / 2 / 4 / 8LAION-Aesthetics 51235.36 / 13.31 / 11.10 / 11.84From SD 2.1-base, \(\omega = 8\)[21]
CTMTrajectory + GAN1 / 2CIFAR-101.98 / 1.87From EDM; 5.19 at 1 step without the GAN term against 2.28 with it, same settings (App. C.4)[38]
sCD-XXL (1.5B)Trajectory1 / 2ImageNet 5122.28 / 1.88From a TrigFlow teacher, 1.73 at 126 NFE[39]
InstaFlow-0.9BTrajectory (reflow)1COCO 2017 FID-5k23.4From SD 1.5; teacher 20.1 at 25 steps[41]
Shortcut, DiT-XL sizeTrajectory, no teacher1 / 4 / 128ImageNet 25610.6 / 7.8 / 3.8From scratch[42]
MeanFlow-XL/2Trajectory, no teacher1 / 2ImageNet 2563.43 / 2.93From scratch, guidance folded in[43]
Diff-InstructDistribution1ImageNet 645.57From EDM[48]
DMDDistribution1ImageNet 642.62From EDM, 2.32 at 512 NFE; regression loss on pairs[15]
DMDDistribution1COCO-30k, SD 1.5, CFG 311.49Teacher 8.78 at 50 steps, DMD's measurement[15]
DMD2Distribution + GAN1ImageNet 641.511.28 with longer training; EDM ODE 2.32, SDE 1.36[1]
DMD2Distribution + GAN1COCO-30k, SD 1.5, CFG 38.35Teacher 8.59 at 50 steps, DMD2's measurement[1]
DMD2, SDXLDistribution + GAN1 / 4COCO 2014, 10k prompts19.01 / 19.32Teacher 19.36 at 100 passes, CFG 6; Patch FID 26.98 / 20.86[1]
SiD, \(\alpha = 1.2\)Distribution1CIFAR-10 uncond.1.923From EDM, 1.97 at 35 NFE; data-free[26]
SiD, \(\alpha = 1.2\)Distribution1ImageNet 641.524From EDM, 1.36 at 511 NFE; recall 0.63 vs 0.67[26]
SiD-LSG, \(\kappa = 1.5\)Distribution1COCO 2014, SD 1.58.71Teacher 8.78 at 50 steps, CFG 3; CLIP 0.302[49]
EMD-16Distribution (EM)1ImageNet 642.20From EDM, 1.43 at 256 NFE; recall 0.59[50]
ADD-MDiscriminator + SDS1COCO 2017 FID-5k19.7SD 1.5 base; DPM-Solver 25 steps 20.1[27]
UFOGenDiscriminator1COCO 2017 FID-5k22.5SD 1.5 base[53]
SDXL-LightningDiscriminator + PD1 / 4 / 8COCO 10k, 1024px22.61 / 22.30 / 21.43From SDXL; teacher 18.49 at 32 steps in Lightning's own protocol, not comparable to the DMD2 SDXL row; DMD2's protocol gives Lightning 24.46 at 4 steps[51]

A walkthrough: SDXL to four steps

DMD2 is the clearest end-to-end image recipe in the open, with the paper, the training scripts and the checkpoints all public[1][23][54]. Fig. 9 lays out its five stages; the paragraphs below say what is frozen, what trains, and how many network evaluations each sample costs.

SDXL TO A FOUR-STEP DMD2 STUDENT SDXL teacherfrozen throughout50 steps × 2 (CFG)≈100 NFE per imagethe reference fake scorea copy of the teachertrains: denoising losson G samples only5 updates per round DMD loopG trains, from teachergrad: sfake − sreal real score frozen1 G update per round GAN headon fake-score featuresreal images enter heresmall classifier trainsextra loss on G 4-step studentbackward simulationt = 999, 749, 499, 2494 NFE, no CFGwhat ships network evaluations per image: about 100 → 4
SDXL to a four-step DMD2 student in five stages: the frozen teacher at about 100 UNet evaluations per image, the fake score trained on generator samples at five updates per round, the generator trained on the score difference at one update per round, the GAN head on the fake score's features where real images enter, and the four-step student that trains on its own earlier steps at t = 999, 749, 499 and 249[1][23].

Stage one is the teacher, and it never trains. SDXL-Base 1.0 at 50 steps with classifier-free guidance is two UNet evaluations per step, about 100 per 1024-pixel image, and DMD2 runs it at guidance 8 as the real score (--real_guidance_scale 8)[1][23]. No ODE pairs are generated for the four-step model; the script name carries noode, and the only data are LAION prompts filtered at aesthetic score 6.25 plus 500k precomputed SDXL VAE latents of LAION images for the GAN head.

Stage two is the fake score, a second copy of the SDXL UNet that starts from the teacher's weights and trains throughout, by the ordinary denoising loss, on the generator's own samples, one fake-score evaluation per re-noised sample. It is the only network that ever sees generator outputs as data, and it gets five updates for every generator update (--dfake_gen_update_ratio 5), the two-time-scale rule that let DMD2 drop DMD's regression loss without the instability its authors saw[1].

Stage three is the generator, a third copy of the teacher, trained one update per round on the score difference of Eq. (6): the fake score minus the frozen real score at a random noise level, applied to the generator's re-noised sample. Each sample in this update costs the student's own passes, two teacher calls for the guided real score at guidance 8 and one fake-score call, unguided at --fake_guidance_scale 1.0. The full-finetune run uses learning rate 5e-7 for both trainable copies; the LoRA variant raises both to 5e-5 and adds --generator_lora. Batch size is 2 per GPU at resolution 1024 and latent resolution 128, fp16 under FSDP across 64 GPUs on 8 nodes, gradient norm clipped at 10[23].

Stage four is the GAN head, a small classifier on the fake score's bottleneck features, so each real latent costs one fake-score pass to reach it. Real latents and generator samples enter it at matched noise levels (--diffusion_gan --diffusion_gan_max_timestep 1000), its loss is applied on the clean prediction (--cls_on_clean_image) and weighted 5e-3 on the generator and 1e-2 on the fake score[23]. This term lets the student pass the teacher's FID, and without it the four-step FID is 26.90[1].

Stage five turns the one-step generator into a four-step sampler. The student is evaluated at t = 999, 749, 499 and 249 (--num_denoising_step 4 --denoising_timestep 1000), and with backward simulation (--backward_simulation) the training input at each later step is the student's own output from the earlier one rather than a noised real image, which is the on-policy argument of the LM section inside a sampler; without it FID rises from 19.32 to 20.66[1]. With four evaluations and no guidance branch, about 100 UNet calls per image become 4.

The Model Zoo table gives the budgets. The full four-step model takes 19k iterations and 57 hours on 64 GPUs and reaches FID 19.32 on COCO 2014 with 10k prompts against 19.36 for the 100-pass teacher at guidance 6 (20.39 at guidance 8, the scale it distils); the LoRA four-step model takes 16k iterations and 63 hours for 19.68; the one-step model takes 24k iterations and 57 hours for 19.01, with regression pretraining on 10k ODE pairs added back[23][1]. The README also warns that FSDP for SDXL "is really slow" and that LoRA training "is actually slower than the full finetuning and takes the same amount of memory".

The pair below is the paper's own: same noise, same prompt, the teacher at 50 steps with guidance and the four-step student. Layout and subject survive; the student's texture is sharper and its lighting more saturated, the reverse-KL signature from the taxonomy section. Over four samples per prompt the student's pairwise LPIPS is 0.61 against the teacher's 0.64[1], and Gandikota and Bau found where that goes: "Distilled models commit to their final image structure almost immediately after the first timestep", where SDXL-Base needs "approximately 30% of their total inference steps" to settle the same structure[55].

SDXL teacher at 50 steps with guidance: a llama in sunglasses on a spaceship deck with the Earth behindSDXL teacher, 50 steps with CFG, about 100 UNet calls
DMD2 four-step student from the same noise and prompt: the same llama, sharper texture and more saturated lightDMD2 student, 4 steps, 4 UNet calls
SDXL at 50 steps with guidance (left) and the DMD2 four-step student (right), from the same noise and prompt: "A photo of llama wearing sunglasses standing on the deck of a spaceship with the Earth in the background." Figure 6 of Yin et al.[1], CC BY 4.0.

The adversarial route from the same teacher is SDXL-Turbo, the released ADD-XL[27]. Its student is also an SDXL copy, trained at four timesteps on noised real images with the hinge discriminator on DINOv2 features and the distillation term at \(\lambda = 2.5\) from the previous section; no fake score network and no backward simulation. The teacher comparison below is the paper's Figure 8, where the authors claim better texture and realism at four steps and admit the diversity cost; the step ladder under it is one seed at one, two and four steps, where the layout holds from the first step and detail fills in with the others[27].

SDXL-Base at 50 steps: a mouse wearing a crownSDXL-Base teacher, 50 steps
ADD-XL, released as SDXL-Turbo, at four steps: a mouse wearing a crownADD-XL (SDXL-Turbo) student, 4 steps
SDXL-Base at 50 steps (left) and ADD-XL, released as SDXL-Turbo, at four steps (right); the paper does not claim matched seeds between the two. Prompt: "A photo of a cute mouse wearing a crown." Figure 8 of Sauer et al.[27], CC BY 4.0.
SDXL-Turbo at one step: a bald eagle made of chocolate powder, mango and whipped cream1 step
SDXL-Turbo at two steps, same seed2 steps
SDXL-Turbo at four steps, same seed4 steps
ADD-XL (SDXL-Turbo) at one, two and four steps from one seed. Prompt: "A bald eagle made of chocolate powder, mango, and whipped cream". Figure 4 of Sauer et al.[27], CC BY 4.0.

At inference both students run at guidance 0. DMD2 needs its own timesteps, 999, 749, 499 and 249, in the LCM scheduler[54]; SDXL-Turbo takes one to four steps at 512x512[56]. Both code blocks are in the practical guide.

Video is a different problem

A Wan2.1 text-to-video run at its defaults is 50 steps at guidance 5.0 over 81 frames at 16 fps, 100 transformer evaluations over 21 latent frames at 832x480[25]. Self Forcing measured the 1.3B at 103 s per 5-second clip on one H100 at those defaults[18], and nothing is visible until every frame finishes.

Wan's generate.py defaults to 50 steps and guidance 5.0; the README recommends 6 for the 1.3B model. 81 frames at 16 fps is 5 seconds, 21 latent frames after 4x temporal compression[25].

The posterior-mean argument applies across time as well as space. At high noise the mean over plausible continuations blurs motion, and a loss that scores each frame alone cannot see the correlation that keeps a video steady. Self Forcing's frame-wise variant shows "worse temporal consistency" than its chunk-wise one[18], and CausVid's student "performs worse in temporal flickering and output diversity" than its bidirectional teacher[12]. T2V-Turbo added HPSv2.1 and InternVid2 rewards to latent consistency distillation of VideoCrafter2, 81.01 VBench total at four steps against the teacher's leaderboard 80.44[57]; T2V-Turbo-v2 reaches 83.52 total and 85.13 quality at 16 steps by folding a motion-guidance gradient into the solver the student copies, guidance as part of the target[58].

CausVid changed the student's architecture[12]. The teacher is a bidirectional DiT; the student has the same weights under a block-causal mask, five latent frames per chunk, four denoising steps per chunk, and a KV cache filled by one extra pass over each finished chunk. It trains with the DMD gradient after an ODE-regression warm start. For 10-second 640x352 video on one H100, including text encoder and VAE, the student shows its first frame in 1.3 s and streams at 9.4 FPS where the teacher takes 219.2 s. The paper calls this asymmetric distillation, since a causal teacher's error accumulation "would propagate from teacher to student".

Block-causal attention and the KV cache

Index the latent frames \(i, j \in \{0, \dots, N-1\}\), let every token of frame \(i\) share its row of the attention mask \(M\), let \(k\) be the frames per block, and let \(\mathrm{end}(i) = k(\lfloor i/k \rfloor + 1)\) be the first frame after the block that holds \(i\):

\[ \begin{aligned} \text{bidirectional:}\quad & M_{i,j} = 1 \\ \text{frame-causal:}\quad & M_{i,j} = \mathbb 1[\,j \le i\,] \\ \text{block-causal:}\quad & M_{i,j} = \mathbb 1\big[\lfloor j/k \rfloor \le \lfloor i/k \rfloor\big] \\ \text{window } L \text{, sink } S\text{:}\quad & M_{i,j} = \mathbb 1\big[\lfloor j/k \rfloor \le \lfloor i/k \rfloor\big] \\ & \quad \cdot\, \mathbb 1[\,j \ge \mathrm{end}(i) - L\,] \;\lor\; \mathbb 1[\,j < S\,] \end{aligned} \tag{9} \]

\(M_{i,j} = 1\) when query frame \(i\) may attend to key frame \(j\); the block-causal line is CausVid's Eq. 5

The bidirectional teacher lets query frame \(i\) see \(\{0, \dots, N-1\}\). A frame-causal student lets it see \(\{0, \dots, i\}\). CausVid's block-causal mask lets it see \(\{0, \dots, \mathrm{end}(i) - 1\}\), every earlier block plus every frame of its own block, including the frames after it in time, "bidirectional self-attention among latent frames" within a chunk and "causal attention across chunks"[12]; frame-causal is the \(k = 1\) case and bidirectional is \(k = N\). The windowed form keeps the newest \(L\) frames and the first \(S\) as a sink, which is how LongLive and Rolling Forcing run and what the Self Forcing code supports[59][60][13].

CausVid sets \(k = 5\): five latent frames per chunk, 16 pixel frames through the teacher's 3D VAE, for 12 fps video at 640x352[12]. Generating a chunk is four denoising passes at t = 999, 748, 502 and 247, each a forward pass over the chunk's own tokens plus the cache with re-noising between passes, and then a fifth pass at t = 0 on the finished clean chunk whose only job is to produce keys and values: "Compute KV pairs with a forward pass \(G_\phi(x_0^i, 0)\)" and "Append new KV pairs to cache C"[12]. The cache therefore holds the K and V of clean chunks only, never of a noisy intermediate, and once it exists the mask is redundant, "because we employ KV caching, block-wise causal attention is no longer needed at inference time". Past chunks are present only as cached K and V; the new chunk's tokens attend bidirectionally to themselves and to everything cached. CausVid's Algorithm 2 never evicts, so the cache grows with the video, and its long videos are made in 10-second segments that take "the final frames of the previous 10-second segment as context"[12].

BIDIRECTIONAL teacher: every frame sees every frame 1 1 5 5 10 10 15 15 key frame BLOCK-CAUSAL CausVid: 5-frame blocks, own and past 1 1 5 5 10 10 15 15 key frame FRAME-CAUSAL block of one frame: itself and past 1 1 5 5 10 10 15 15 key frame query frame filled: the query attends to the key; empty: masked KV CACHE DURING GENERATION Self Forcing: chunks of 3 latent frames step 1 first chunk, 4 steps, then its KV is written step 2 new chunk attends to every cached frame step 3 cache grows by one chunk per step long video evicted window keeps the newest L frames solid: chunk being generated
Attention masks over 15 latent frames, query frames down the side and key frames along the bottom: bidirectional, where every cell is filled; block-causal with CausVid's five-frame blocks, where each block sees itself and every earlier block; and frame-causal, where each frame sees itself and the past. Below, the KV cache for Self Forcing's three-frame chunks: four denoising passes on the first chunk, then its K and V are written; each new chunk attends to every cached frame; the cache grows one chunk per step; in long-video mode the oldest chunk is evicted and the window keeps the newest L frames, a layout taken from LongLive because Self Forcing states no window size[12][18][59].

What one block of cache holds is fixed by the model's shapes. Self Forcing's Wan2.1-1.3B student has 30 transformer blocks; at its training length of 21 latent frames of 60x104 with 2x2 patches, each block stores K and V for 32,760 tokens across 12 heads of width 128, which by our arithmetic from the code's shapes is about 201 MB per block in bf16 and about 6 GB for the stack, before the 512-token cross-attention cache that every block also keeps for the text[13]. Teacher forcing would have filled that cache from ground-truth frames and Diffusion Forcing from noisy ones; CausVid and Self Forcing fill it from the student's own clean outputs, and Self Forcing does so during training as well, which is the next subsection.

Self Forcing: training on your own rollouts

Exposure bias returns the moment the student is autoregressive. Teacher forcing conditions each chunk on ground-truth context in training and on the student's own output at inference, a model "trained exclusively on ground-truth context but must rely on its own imperfect predictions at inference time"[18]. Self Forcing closes the gap by rolling the student out with its KV cache during training and applying a holistic loss to the whole video[18]: DMD, with Wan2.1-14B at guidance 3.0 as the real score and the student initialized from Wan2.1-1.3B, or SiD, or a GAN, with chunk-wise DMD best at a VBench total of 84.31. Trained from prompts alone in about 1.5 hours on 64 H100s, the student runs at 17.0 FPS with a 0.69 s first frame on one H100, against 0.78 FPS and 103 s for Wan2.1-1.3B at 84.26; CausVid's recipe on the same base scores 81.20.

Algorithm 1 is CausVid's generation loop with gradients attached at one point. For chunk \(i\) and for \(j = T, \dots, 1\), the student denoises \(x^i_{t_j}\) with \(G_\theta(\cdot, t_j, \mathbf{KV})\). One step \(s\) is drawn uniformly from \([1, T]\) for each sequence at each iteration; at \(j = s\) gradients are on and the output is kept as \(\hat x^i_0\), at every other step gradients are off and the output is re-noised to \(t_{j-1}\). Then, still without gradients, the clean chunk is passed once more at t = 0 to write \(\mathbf{kv}^i\), the same cache-write pass as CausVid's. Backpropagation runs through the sampled step in each chunk, and "restricting gradient flow into KV cache embeddings" detaches every earlier chunk, so the graph of a rollout holds one step per chunk, seven for the 21-frame clip, rather than all 28 denoising passes[18]. The DMD loss then scores the whole rolled-out video against the 14B real score and the 1.3B fake score, which is what makes it holistic. The released config fixes the rest: chunks of 3 latent frames, 7 chunks for the 21-frame clip, four steps at t = 1000, 750, 500 and 250 with flow shift 5, generator learning rate 2e-6, critic 4e-7, five critic updates per generator update, batch 64, EMA 0.99 from step 200[13].

The paper's Table 2 makes the exposure-bias argument. With chunk-wise and then frame-wise context, many-step Diffusion Forcing scores 82.95 and 77.24, many-step teacher forcing 83.58 and 80.34, Diffusion Forcing plus DMD 82.76 and 80.56, teacher forcing plus DMD 82.32 and 78.12, and Self Forcing 84.31 and 84.26[18]. Generating one latent frame per step instead of three costs the forced trainers 2 to 6 VBench points and Self Forcing 0.05, because its training already runs on its own predictions.

Section 3.4 of the paper adds a rolling cache for videos longer than the training length, a "KV cache of size L frames" where, once full, "we remove the oldest KV cache entry before adding the new one". The paper never gives L a number. In the code, local_attn_size defaults to −1, which sets the attention span to 32,760 tokens, the full 21 training frames, so the released checkpoint rolls only when that flag is set; eviction then shifts the cache left, overwrites the oldest entries and writes the new keys with RoPE at their absolute frame index, and a sink_size argument, default 0, protects the first frames from eviction without forcing attention onto them[13]. Naive rolling flickers, because "every chunk always attends to the first chunk during denoising" and the first chunk is statistically special; the fix is a training-time window that keeps the final chunk from attending to the first, after which rolling runs at 16.1 FPS on 10-second videos where recomputing the cache at every shift ran at 4.6[18]. The rolling row of Fig. 13 therefore follows LongLive's layout, a 9-frame window with the first 3-frame chunk kept as a sink[59].

BIDIRECTIONAL TEACHER TO CAUSAL STUDENT bidirectional teacherall frames at once50 steps, CFG ODE pairsteacher trajectoriesnoise → video, stored causal student, initblock-causal maskregress onto the pairs student rolloutblock by block, KV cache4 steps per block noised rollout fake scoreinit from the 1.3Btrains, denoising loss real scoreSelf Forcing: Wan2.1 14BCFG 3.0, frozen − DMD gradient the 14B sits only here, as a critic CausVid: its own teacher as real score KV CACHE b1 b2 b3 b4 grows one block per step KV from the clean block
The CausVid and Self Forcing pipeline. A bidirectional teacher produces ODE pairs, the causal student is initialized by regressing onto them, then rolls out block by block with a KV cache written from each clean block; the noised rollout feeds a fake score and the real score, which is Wan2.1-14B at CFG 3.0 in Self Forcing and the model's own teacher in CausVid, and their difference returns as the DMD gradient[12][18].

rCM scales sCM to Wan2.1-14B with a DMD regularizer at weight 0.01, added because pure consistency "struggles with fine detail"; its 4-step 14B scores 84.92 at 4.5 FPS against the teacher's 83.58 at 0.18, its 1-step output is "blurry across prompts", and its DMD2 baseline "tends to produce collapsed generations"[19]. APT's Table 7 shows a discriminator restoring what a one-step consistency student loses, VBench semantic 40.15 against 73.15[52].

MethodTeacher or real scoreStepsResolutionThroughput or latencyVBench totalSource
T2V-Turbo (VC2)VideoCrafter2, CD + rewards4512x320, 16 frames"over 10 times" faster, no seconds given81.01 (own)[57]
T2V-Turbo-v2VideoCrafter2 with motion guidance16Not statedNot reported83.52 (own)[58]
CausVidOwn bidirectional DiT, DMD4 per chunk640x352, 120 frames1.3 s first frame, 9.4 FPS, one H100; teacher 219.2 s84.27 (Long)[12]
Self Forcing, chunk-wiseWan2.1-14B real score, 1.3B student4832x480, 81 frames0.69 s first frame, 17.0 FPS, one H100; Wan2.1-1.3B 103 s, 0.78 FPS84.31 (own)[18]
rCM, Wan2.1-1.3BWan2.1-1.3B, sCM + DMD term4 / 2 / 1832x480, 81 frames14.6 / 23.0 / 32.3 FPS incl. decode, one H100; teacher 0.7284.43 / 84.09 / 82.65 (own)[19]
rCM, Wan2.1-14BWan2.1-14B4 / 2 / 1832x480, 81 frames4.5 / 8.3 / 14.4 FPS; teacher 0.1884.92 / 85.05 / 83.02 (own)[19]
APTOwn 8B DiT, CD start then GAN11280x720, 49 frames, 24 fpsDiT 2.65 s, pipeline 6.03 s, one H100; 1.97 s on eight82.00 (own; base 82.15, CD 1 NFE 67.05)[52]
LTX-Video 13B distilledLTX-Video 13B8HD, two-pass"HD videos in 10 seconds" on H100, per the cardNot reported[61]

VBench totals marked own are the authors' runs of the suite on their own outputs and their baselines', and those runs differ from the model's own report by up to a point: Wan2.1-1.3B reads 83.96 in the Wan report[25], 84.26 in Self Forcing's table[18] and 83.02 in rCM's[19]. CausVid's number is VBench-Long, a different prompt set and protocol, so the column is not one scale.

A walkthrough: Wan2.1 to a streaming four-step student

Self Forcing's repository is the matching recipe for video, Wan2.1-T2V-1.3B in and a streaming four-step student out, with the ODE checkpoint, the DMD config and the training command public[13][18]. Fig. 15 draws the six stages.

WAN2.1-1.3B TO A STREAMING SELF FORCING STUDENT teacher: Wan2.1-T2V-1.3Bbidirectional, frozen50 steps × 2 (CFG)81 frames, 21 latent frames ODE pairsteacher trajectories, storedprompts only (VidProM)no video dataset causal initblock-causal mask, 3-frame chunksregress onto the ODE pairsstudent trains from 1.3B weights rollouts with KV cachechunk by chunk, 4 steps eacht = 1000, 750, 500, 250grad through one sampled step DMD vs the 14B real scorereal score: Wan2.1 14B, CFG 3.0fake score: from 1.3B, trainsloss on the whole rollout streaming student17 FPS on one H100, 4 NFE per chunkfirst frame after 0.69 srolling KV cache for long videos per 5 s clip: 50 × 2 bidirectional passes → 4 causal passes per chunk
Wan2.1-T2V-1.3B to a streaming student in six stages: the frozen bidirectional teacher at 50 steps with CFG, ODE pairs stored from prompts alone, the causal initialization with a block-causal mask over three-frame chunks, rollouts with a KV cache at t = 1000, 750, 500 and 250, DMD against the 14B real score at CFG 3.0 with a fake score trained from the 1.3B, and the streaming student at four passes per chunk[18][13].

Stage one is the teacher, Wan2.1-T2V-1.3B, a bidirectional DiT that stays frozen throughout. Its default sampler is 50 steps at guidance 5.0 over 81 frames at 832x480 and 16 fps, 21 latent frames after the VAE's 4x temporal compression, so a clip is 100 passes over all 32,760 tokens at once and takes 103 s on one H100 in Self Forcing's measurement[25][18]. Nothing is visible until the last pass finishes.

Stage two is the ODE data. Prompts come from a filtered VidProM list; for each, noise is pushed through the teacher's ODE solver and the trajectory is stored at the timesteps the student will use. No video dataset enters at any stage: "Our training algorithm (except for the GAN version) is data-free (no video data is needed)"[13]. CausVid, which defined the procedure, built 1,000 such pairs[12]; Self Forcing does not script this stage and ships its result as ode_init.pt.

Stage three is the causal initialization. The student starts as the teacher's weights with the attention mask swapped to block-causal over chunks of 3 latent frames and regresses onto the stored trajectories, in CausVid's case for 3,000 iterations at AdamW 5e-6[12]. The stage exists because "Directly training the causal student model using the DMD loss can be unstable due to architectural differences"[12]. What trains is the student; the teacher only generated the data.

Stage four is the rollout. The student generates its 21 frames chunk by chunk, 7 chunks of 3, each chunk in four passes at t = 1000, 750, 500 and 250 with flow shift 5 against the KV cache, plus the t = 0 pass that writes the chunk's keys and values. Per clip that is 7 x (4 + 1) = 35 passes of the 1.3B model, each over one chunk's 4,680 tokens (3 x 1,560, by our arithmetic) plus the cache, in place of the teacher's 100 passes over 32,760. Gradients flow through one sampled step of each chunk, which is why the rollout fits at batch 1 per GPU[18][13].

Stage five is the DMD loss on the rollout. The real score is the frozen Wan2.1-T2V-14B at guidance 3.0, the fake score a second 1.3B copy that trains on the rollouts at learning rate 4e-7 with five updates per generator update, the generator at 2e-6; batch 64 across 64 GPUs, EMA 0.99 from step 200. The README's run is 600 iterations "in under 2 hours using 64 H100 GPUs", and the authors expect "less than 16 hours using 8 H100 GPUs" with gradient accumulation[13]. The 14B sits in the loop only as a critic; it never generates a frame.

Stage six is what ships. The student runs four passes per chunk at guidance 1, streams its first chunk after 0.69 s and the rest at 17.0 FPS on one H100, scores VBench 84.31 against 84.26 for the teacher in the same run, and needs a 24 GB GPU for inference[18][13]. Per 5-second clip the teacher's 100 bidirectional passes become 35 causal ones, and the viewer sees frames while the rest are still being made.

The clips below are the project page's own comparison on one prompt: the Wan2.1-1.3B teacher, CausVid-1.3B and the Self Forcing student. In these clips the CausVid column looks over-saturated next to the teacher; our reading, which neither page offers, is that this is the mode-seeking of reverse KL from the taxonomy section. The Self Forcing column keeps the teacher's palette. The pair after it is CausVid's own teacher comparison, where the page notes slightly more flicker and less diversity in the student[12]. The rCM pair is the other branch, a bidirectional four-step Wan student that runs at 14.6 FPS and cannot stream, and its step ladder shows the one-step blur the paper describes[19].

Wan2.1-1.3B teacher, many-step bidirectional sampler
CausVid-1.3B, 4-step autoregressive student
Self Forcing 1.3B, 4-step student trained on its own rollouts
One prompt, three samplers, from the Self Forcing project page[18]: the Wan2.1-1.3B teacher, CausVid-1.3B at four autoregressive steps, and Self Forcing 1.3B at four steps trained on its own rollouts; the page does not state whether the seeds match. Prompt: "A petri dish with a bamboo forest growing within it that has tiny red pandas running around." Clips trimmed to four seconds. CC BY-NC-SA 4.0, shown for research use.
CausVid bidirectional teacher, 50 steps
CausVid causal student, 4 steps per chunk
CausVid's bidirectional teacher at 50 steps (left) and its four-step causal student (right), seconds 3 to 7 of the ten-second clips on the project page[12]. Prompt, from MovieGenBench: "Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk..." CC BY-NC-SA 4.0, shown for research use.
Wan2.1 T2V 1.3B teacher
rCM student, 4 steps, bidirectional
The Wan2.1 T2V 1.3B teacher (left) and the rCM student at four steps (right), one cell of the sample grids in the rCM README[19], Apache 2.0; the README lists no prompt and claims no shared seed, and the teacher clip was re-timed from the README's 24 fps container rate to Wan2.1's 16 fps so the pair has the same length.
1 step
2 steps
4 steps
The same rCM student at one, two and four sampling steps, from the rCM README[19], Apache 2.0; the README names no prompt, and the three clips share layout and motion, which suggests one noise sample.

Open problems and current efforts

Diversity is the first open problem, and it now has a diagnosis. DMD2's four-step SDXL student trails its teacher on pairwise LPIPS[1]; CausVid's student shows "reduced output diversity", which its paper calls "characteristic of reverse KL-based distribution matching"[12]; rCM's DMD2 baseline produced "collapsed generations, where objects converge to similar positions"[19]. Gandikota and Bau compared SDXL-Base with SDXL-Turbo, Lightning, LCM and DMD2 and located the loss at the first step: distilled models fix their layout "almost immediately after the first timestep", where the base model needs about 30 percent of its steps[55]. Their fix runs the base model for the first step and the distilled model for the rest: pairwise DreamSim diversity 0.337 for the base at 50 steps, 0.264 for DMD2 at four, 0.350 for the hybrid, with COCO-30k FID of 12.74, 15.52 and 10.79 at 9.22 s, 0.64 s and 0.64 s per image, paid for by holding both models in memory.

Fixes inside the objective followed. DP-DMD trains the first step on a teacher-derived velocity target and the remaining steps with DMD, and lifts DINOv3 feature diversity over nine seeds on SD3.5-Medium from 0.137 to 0.179 on Pick-a-Pic prompts and from 0.210 to 0.250 on COCO, against the 60-evaluation teacher's 0.240 and 0.288[62]. Decoupled DMD argues that "the primary driver of few-step distillation is not distribution matching" but the guidance term, with the matching term there for stability[63], and 1.x-Distill reports that DMD "suffers from diversity collapse and fidelity degradation when reduced to two steps or fewer" and rebuilds the first step around that[64]. For video, Data-Forcing reports "a substantial drop in sample diversity, and visibly over-saturated outputs" from DMD and pulls the student toward real data in 100 to 300 fine-tuning steps on Wan2.1-1.3B[65], and DUET pairs an sCM expert at the high-noise step with a DMD expert at the low-noise step to keep "nearly all" of the consistency model's structural diversity at two steps[66]. The measurement is still not standardized: pairwise LPIPS, DreamSim, DINO feature spread and recall each appear once, and VBench has no diversity dimension.

Long rollouts drift, and the measurements come from the follow-ups. Rolling Forcing defines drift as the imaging-quality gap between the first and last 5 seconds of a 30-second video and measures Self Forcing at 1.66, CausVid at 2.18 and its own method at 0.01, with its no-sink ablation at 4.63 and no rolling at inference at 5.53[60]; at 30 seconds on VBench-Long, LongLive scores 83.52 at 20.7 FPS against Self Forcing's 81.59 at 17.0 FPS and SkyReels-V2's 75.29 at 0.49 FPS[59]. The mechanism is the one the LM section predicted, "small prediction errors accumulate and the context inside the window becomes progressively noisier" in LongLive's words, and the fixes are a sink and a longer training horizon. LongLive keeps the first 3-frame chunk permanently in every block's cache, attends over a 9-frame local window, and tunes on its own 60-second rollouts with the 14B as DMD teacher on each new 5-second clip, about 12 hours on 64 H100s, the sink cutting compute by 28 percent and peak memory by 17 percent[59]. Rolling Forcing denoises a window of five chunks at five noise levels jointly, so "a clean frame is generated, and pure Gaussian noise is appended as the next frame", with the first three frames cached as a global sink[60].

The live tension is that sinks cut drift and freeze motion. Reward Forcing reports that with a pinned sink "models develop a strong bias toward the starting frame", folds evicted entries into an exponential moving average instead, and at 60 seconds measures dynamic degree 66.95 against LongLive's 35.54 at matched drift, 2.505 against 2.531[67]; Deep Forcing finds that "naively applying StreamingLLM-style attention sinks to video diffusion leads to fidelity degradation"[68]; Self Forcing++ drops sinks, "relies solely on historical KV cache without sink frames", samples teacher-length windows from its own long rollouts for the DMD loss, and reports rollouts "up to 20x" the teacher's horizon, and up to 4 minutes 15 seconds, 99.9 percent of the span its base model's position embedding supports[69]. Nobody has published a quality-versus-length curve that is both minutes long and measured on the standard VBench dimensions; every group defines its own drift metric.

Reward has moved inside distillation. T2V-Turbo put HPSv2.1 and InternVid2 rewards on one-step outputs during consistency distillation[57]; Reward Forcing weights the DMD gradient by \(\exp(r/\beta)\) with VideoAlign's motion reward on the decoded output, \(\beta = 1/2\), no backpropagation through the reward model, and runs at 23.1 FPS[67]. DMDR makes the case against both sequential orders on SDXL, SD3-Medium and SD3.5-Large. Distill then RL leaves "an impoverished anchor, leading to rapid overfitting and mode collapse", because "without the strong distributional anchor of the teacher, the few-step model quickly drifts into narrow, high-reward modes"; RL then distill reopens the distillation gap. Trained jointly, with the reward-tilted matching term and a ReFL term on CLIP and HPSv2.1, the four-step SDXL student's HPS rises from 28.57 to 32.98 and DPG-Bench from the teacher's 74.65 to 76.48, from 84.08 to 85.30 on SD3-Medium and from 84.12 to 85.33 on SD3.5-Large, and the authors still write that DMD "cannot completely remove" the diversity cost[70]. AdvDMD reuses DMD2's discriminator as the reward model[71] and RTDMD adds a GRPO-style term on the stochastic steps[72]. No paper in this set shows plain RL on a frozen distilled student succeeding, and DMDR's Figure 4 shows it hacking the reward.

Every distribution-matching video student trains with guidance baked in and samples at 1 afterwards: Self Forcing puts 3.0 into its real score, rCM's Wan teachers run at 5.0, and CausVid reports a guidance scale of 3.5[12][18][19]; Decoupled DMD's reading is that this baked-in guidance term is the engine of the method[63]. The untried target is autoguidance. Karras et al. guide with "a smaller, less-trained version of the model itself rather than an unconditional model" and report "disentangled control over image quality without compromising the amount of variation", FID 1.01 on ImageNet 64 and 1.25 on ImageNet 512[17]; a student distilled from an autoguided teacher would inherit the sharpening without the branch dropping that CFG's power-law density implies, and as of our search no distilled model targets one. The other route removes guidance upstream: Guidance-Free Training "matches the performance of CFG while reducing sampling to a single model" and "enables training directly from scratch"[73].

Teacher-free few-step trainers stop short of video. sCM tops out at ImageNet 512[39], MeanFlow at ImageNet 256 with 3.43 in one evaluation[43], IMM at 1.99 with eight steps on ImageNet 256[46]. For text-to-image, Self-E claims a from-scratch model that "does not require a pretrained teacher" and is "competitive with state-of-the-art Flow Matching models at 50 steps"[74]; TwinFlow "bypasses the need of fixed pretrained teacher models" yet starts from the pretrained Qwen-Image-20B and trains it end to end into a one-evaluation generator with GenEval 0.83[75]; FACM distills LightningDiT to 1.32 at two steps on ImageNet and scales the same objective to a 14B Wan 2.2 text-to-image model[76]. No teacher-free few-step video model exists at production scale; the nearest, DUET's sCM expert on Wan2.1-1.3B, starts from the pretrained weights[66].

Every one-step video result at production resolution uses a discriminator. APT reaches 1280x720 at 24 fps for two seconds in one step, and its authors "were only able to train the model to generate videos for up to two seconds"[52]. APT2 makes the one-step model autoregressive, "a latent frame at a time using a single neural function evaluation", 24 fps at 736x416 on one H100 and 1280x720 on eight, up to one minute, trained "in a student-forcing manner"[77]. V-PAE does the same for image-to-video on Wan2.1-I2V-14B[78], and One-Forcing adds "an auxiliary GAN loss" to DMD for a one-step frame-wise causal student at VBench 83.76, noting that "DMD-based methods, such as Self-Forcing, tend to generate blurry frames" at one step[79]; rCM's own one-step Wan 1.3B scores 82.65 and "appear[s] blurry across prompts"[19]. Pure distribution matching and pure consistency stop at two to four steps.

LoRA distillation costs quality, and SDXL-Lightning states the cost in words: it ships both forms and says that "fully trained models have better structures and details", that the gap "is less noticeable on 8-step models, but more observable on 2-step models", and that it provides no LoRA for one step[51]. DMD2's README gives the only numbers, on SDXL, and they agree: the LoRA four-step run reaches FID 19.68 in 63 hours where the full run reaches 19.32 in 57, and the README adds that the LoRA run "takes the same amount of memory"[23]. LCM-LoRA publishes no comparison against a full student[36]. Wan2.2-Lightning ships only rank-64 LoRAs for the two Wan2.2 experts, "on par with the base model in most scenarios", with artifacts "when the video contains elements with extremely large motion", and no video release quantifies the LoRA-versus-full gap[80].

Evaluation is unsettled too. VBench scores 16 dimensions, seven into a quality score and nine into a semantic score, weighted 0.8 to 0.2 in the total[81]; the same model lands at different points in different papers, Self Forcing at 84.31 and 83.80[18][67] and Wan2.1-1.3B at the three totals under the table in Video is a different problem; standard clips are about 2 seconds long[82]; drift is invisible to the total, which is why Rolling Forcing, Reward Forcing and LongLive each add their own; there is no diversity dimension; and DynamicEval finds that VBench and EvalCrafter metrics "often produce incorrect results in the presence of camera motion", gaining 0.32 in Pearson correlation with human preference on 45k annotated pairs once corrected[83]. Latency columns, as Honest edges notes, do not say whether the text encoder and VAE are in the clock.

A practical guide

Everything below was read from the live model cards, READMEs and configs on November 2, 2026, and the snippets are copied and trimmed rather than retyped; diffusers main was 0.42.0.dev0 that day, and from_pretrained accepts both the torch_dtype= the cards use and the newer dtype=. The licenses differ more than the code does, so each block carries its own.

Use a distilled model

SDXL-Turbo runs the ADD-XL student at one to four steps with guidance off; its card says "A single step is enough" and that the model "does not make use of guidance_scale or negative_prompt", at a native 512x512[56].

pip install diffusers transformers accelerate --upgrade
from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16")
pipe.to("cuda")

prompt = "A cinematic shot of a baby racoon wearing an intricate italian priest robe."

image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images[0]

Model card stabilityai/sdxl-turbo[56], license sai-nc-community, Stability's non-commercial research license, with commercial use referred to stability.ai. Hardware is not stated.

LCM-LoRA is a 197M-parameter adapter on SDXL-Base that swaps in the LCM scheduler and runs at 2 to 8 steps; the card asks for guidance off or between 1.0 and 2.0[84].

import torch
from diffusers import LCMScheduler, AutoPipelineForText2Image

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "latent-consistency/lcm-lora-sdxl"

pipe = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")

# load and fuse lcm lora
pipe.load_lora_weights(adapter_id)
pipe.fuse_lora()

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# disable guidance_scale by passing 0
image = pipe(prompt=prompt, num_inference_steps=4, guidance_scale=0).images[0]

Model card latent-consistency/lcm-lora-sdxl[84], license openrail++, diffusers 0.23.0 or later plus peft.

DMD2's four-step SDXL student loads as a LoRA on SDXL-Base and needs its own timesteps, because "LCMScheduler's default timesteps are different from the one we used for training"[54].

import torch
from diffusers import DiffusionPipeline, UNet2DConditionModel, LCMScheduler
from huggingface_hub import hf_hub_download
base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
repo_name = "tianweiy/DMD2"
ckpt_name = "dmd2_sdxl_4step_lora_fp16.safetensors"
# Load model.
pipe = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.load_lora_weights(hf_hub_download(repo_name, ckpt_name))
pipe.fuse_lora(lora_scale=1.0)  # we might want to make the scale smaller for community models

pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
prompt="a photo of a cat"

# LCMScheduler's default timesteps are different from the one we used for training
image=pipe(prompt=prompt, num_inference_steps=4, guidance_scale=0, timesteps=[999, 749, 499, 249]).images[0]

Model card tianweiy/DMD2[54]; the card's front matter says cc-by-nc-4.0 and its license section and the GitHub LICENSE say CC BY-NC-SA 4.0, non-commercial either way. The one-step UNet is dmd2_sdxl_1step_unet_fp16.bin loaded into UNet2DConditionModel with no LoRA, num_inference_steps=1 and timesteps=[399].

FLUX.1 [schnell] is the latent adversarial diffusion distillation student of FLUX.1 at 1 to 4 steps and the one image model here released under Apache 2.0, so it is cleared for commercial use; guidance must be 0 and max_sequence_length at most 256[29].

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    guidance_scale=0.0,
    num_inference_steps=4,
    max_sequence_length=256,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save("flux-schnell.png")

Model card black-forest-labs/FLUX.1-schnell[29], Apache 2.0, gated with automatic approval; the CPU-offload line is the card's own memory advice.

Wan2.2-Lightning is a pair of rank-64 LoRAs, one per Wan2.2-T2V-A14B expert, that run the 14B mixture-of-experts model in four steps without guidance; the route the card documents is the Wan repository, not diffusers[80][33].

pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.2-T2V-A14B --local-dir ./Wan2.2-T2V-A14B
huggingface-cli download lightx2v/Wan2.2-Lightning --local-dir ./Wan2.2-Lightning
python generate.py  --task t2v-A14B --size "1280*720" --ckpt_dir ./Wan2.2-T2V-A14B --lora_dir ./Wan2.2-Lightning/Wan2.2-T2V-A14B-4steps-lora-rank64-Seko-V1 --offload_model True --base_seed 42 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

Model card lightx2v/Wan2.2-Lightning[80] and the ModelTC repository[33], Apache 2.0. The card's settings are Euler with time shift 5, timesteps 1000, 937.5, 833.3 and 625, and cfg 1.0 "so that NFE = 4 but not 8"; the single-GPU command needs "at least 80GB VRAM", with --offload_model True, --convert_model_dtype and --t5_cpu for less, or eight GPUs through torchrun with --dit_fsdp --t5_fsdp --ulysses_size 8. The ComfyUI workflow in the same folder loads the two fp16 UNets with their LoRAs at strength 1.0, shift 5, and two four-step samplers at cfg 1 that hand over at step 2. The card carries a diffusers tag but no diffusers code.

Self Forcing's streaming student needs the Wan2.1-T2V-1.3B base and one checkpoint; the README's requirement is an "Nvidia GPU with at least 24 GB memory (RTX 4090, A100, and H100 are tested)", Linux and 64 GB of RAM[13].

huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir-use-symlinks False --local-dir wan_models/Wan2.1-T2V-1.3B
huggingface-cli download gdhe17/Self-Forcing checkpoints/self_forcing_dmd.pt --local-dir .

python inference.py \
    --config_path configs/self_forcing_dmd.yaml \
    --output_folder videos/self_forcing_dmd \
    --checkpoint_path checkpoints/self_forcing_dmd.pt \
    --data_path prompts/MovieGenVideoBench_extended.txt \
    --use_ema

Repository guandeh17/Self-Forcing[13] and checkpoints gdhe17/Self-Forcing[85], Apache 2.0, installed with Python 3.10, pip install -r requirements.txt, flash-attn and python setup.py develop. The config behind the command is where the chunk size and step count live: num_frame_per_block: 3, denoising_step_list: [1000, 750, 500, 250], timestep_shift: 5.0, 480x832 at 81 frames. python demo.py opens a GUI, and the README says the model "works better with long, detailed prompts".

Distill one yourself

The cheapest start is the diffusers consistency-distillation example, which trains an LCM-LoRA on SD 1.5 or SDXL from webdataset shards of image-caption pairs; the SDXL command from its README, trimmed[86]:

export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export OUTPUT_DIR="path/to/saved/model"

accelerate launch train_lcm_distill_lora_sdxl_wds.py \
    --pretrained_teacher_model=$MODEL_DIR \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision=fp16 \
    --resolution=1024 \
    --lora_rank=64 \
    --learning_rate=1e-4 --loss_type="huber" --use_fix_crop_and_size --adam_weight_decay=0.0 \
    --max_train_steps=1000 \
    --max_train_samples=4000000 \
    --train_shards_path_or_url="pipe:curl -L -s https://huggingface.co/datasets/laion/conceptual-captions-12m-webdataset/resolve/main/data/{00000..01099}.tar?download=true" \
    --train_batch_size=12 \
    --gradient_checkpointing --enable_xformers_memory_efficient_attention \
    --use_8bit_adam \
    --push_to_hub

Scripts and README in examples/consistency_distillation of diffusers[86], Apache 2.0; setup is a clone, pip install -e ., the example folder's requirements and accelerate config. Two things to know before running it: the README exports MODEL_NAME and then passes $MODEL_DIR, which is undefined, so set both; and the CC12M shards are "for illustrative purposes only", with the authors pointing at LAION-scale data. The peft variant, train_lcm_distill_lora_sdxl.py, takes any Hugging Face image-caption dataset, does not load two UNets, which its README says "reduces the memory requirements quite a bit", and its example runs batch 24, rank 64 and 3,000 steps at 1024px. Neither README states hardware or time.

The DMD2 SDXL recipe is the walkthrough above as shell: 64 GPUs over 8 nodes, launched once per node with the FSDP configs that create_sdxl_fsdp_configs.py writes[23].

bash experiments/sdxl/sdxl_cond999_8node_lr5e-7_denoising4step_diffusion1000_gan5e-3_guidance8_noinit_noode_backsim_scratch.sh $CHECKPOINT_PATH $WANDB_ENTITY $WANDB_PROJECT fsdp_configs/EXP_NAME NODE_RANK_ID

Inside that script, the LoRA variant runs the following, with the flags that map to the paper: five fake-score updates per generator update, the GAN branch, four steps with backward simulation, and the teacher at guidance 8.

accelerate launch --config_file $FSDP_DIR/config_rank$RANK.yaml main/train_sd.py  \
    --generator_lr 5e-5  --guidance_lr 5e-5 \
    --batch_size 2 --resolution 1024 --latent_resolution 128 --seed 10 \
    --real_guidance_scale 8 --fake_guidance_scale 1.0 --max_grad_norm 10.0 \
    --model_id "stabilityai/stable-diffusion-xl-base-1.0" \
    --dfake_gen_update_ratio 5 \
    --fsdp --sdxl --use_fp16 --max_step_percent 0.98 \
    --cls_on_clean_image --gen_cls_loss --gen_cls_loss_weight 5e-3 --guidance_cls_loss_weight 1e-2 \
    --diffusion_gan --diffusion_gan_max_timestep 1000 \
    --denoising --num_denoising_step 4 --denoising_timestep 1000 --backward_simulation \
    --train_prompt_path $CHECKPOINT_PATH/captions_laion_score6.25.pkl \
    --real_image_path $CHECKPOINT_PATH/sdxl_vae_latents_laion_500k_lmdb/ \
    --generator_lora

Repository tianweiy/DMD2[23], CC BY-NC-SA 4.0. Setup is conda with Python 3.8, pip install -r requirements.txt, python setup.py develop, then bash scripts/download_sdxl.sh $CHECKPOINT_PATH; the LAION prompts, the 500k precomputed VAE latents and COCO-10k come from the data/ folder on Hugging Face. Budget from the Model Zoo: the full four-step run is 19k iterations and 57 hours, the LoRA four-step run 16k and 63 hours, the one-step run 24k and 57 hours, all on 64 GPUs (A100s in the paper; the Model Zoo does not name the GPU); extract_lora_module.py turns a LoRA run's checkpoint into a LoRA file.

Self Forcing's training has two stages, and the repository scripts only the second. Stage one, the ODE initialization, is "identical to the process described in the CausVid repo" and is shipped as a checkpoint; stage two is DMD on the student's own rollouts, launched over 8 nodes of 8 GPUs[13]:

huggingface-cli download gdhe17/Self-Forcing checkpoints/ode_init.pt --local-dir .
huggingface-cli download gdhe17/Self-Forcing vidprom_filtered_extended.txt --local-dir prompts

torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train.py \
  --config_path configs/self_forcing_dmd.yaml \
  --logdir logs/self_forcing_dmd \
  --disable-wandb

Repository[13], Apache 2.0. The data is the prompt file alone. configs/self_forcing_dmd.yaml sets real_name: Wan2.1-T2V-14B, lr: 2.0e-06, lr_critic: 4.0e-07, dfake_gen_update_ratio: 5, total_batch_size: 64 and 21 latent frames at 60x104; the README's run is 600 iterations "in under 2 hours using 64 H100 GPUs", with "less than 16 hours using 8 H100 GPUs" expected under gradient accumulation. CausVid's own repository scripts the ODE stage, train_ode.py on 1.5K pairs from generate_ode_pairs.py and then train_distillation.py, every launch at 64 GPUs, but it needs a video dataset, MixKit's 6K clips in its example, generates at three steps rather than four, and calls itself "a work in progress"[87].

Honest edges

Of the core methods, two put a number on lost diversity. SiD's ImageNet 64 table puts its one-step student at recall 0.63 against the teacher's 0.67, level with CD, iCT and progressive distillation at 0.62 to 0.63 in the same column[26]; DMD2's Table 6 gives pairwise LPIPS over four images per prompt of 0.61 for its 4-step SDXL student against the teacher's 0.64[1]. ADD and SDXL-Lightning admit the loss in one sentence each[27][51]; LADD, Hyper-SD, UFOGen and APT say nothing. Recall[88] and coverage[89] exist for this, as does per-prompt LPIPS across seeds, and rCM's weight sweep shows the trade, more score weight for better quality and less diversity[19].

Prompt adherence pays for the baked-in guidance: LADD names duplication, spatial prompting and negation[22], Hyper-SD cannot use negative prompts[20], SiD-LSG trades FID for CLIP along its guidance scale[49]. Trajectory methods hit a ceiling instead, the one Matching the trajectory measured for reflow[40] and for one-step SD 1.4[41]. Discriminators collapse without regularization and at small batch[52], and LADD's grid search found LCM the volatile one in their hands[22], so instability runs both ways.

The metrics disagree with each other. In SD3.5-Flash's comparison table, SDXL-DMD2 at four steps has the best FID, 16.64, and a GenEval of 0.56, tied for last with SDXL-HyperSD[31][90]. DMD2 beats its teacher on FID, trails it on LPIPS diversity, and wins 24 percent of quality comparisons, three verdicts from one paper[1]. Small text goes first, with rCM reporting consistency alone failing on "small text rendering"[19] and ADD reporting that more steps improve "spelling capabilities"[27].

Latency numbers quietly drop the text encoder and VAE. SDXS measured SDXL's UNet at 56 ms per call and its decoder at 73 ms, so at one evaluation the decoder costs more than the network[24]. APT's "two seconds" is the DiT; the one-GPU pipeline is 6.03 s with 3.10 s in the VAE[52]. CausVid's 1.3 s includes everything[12], rCM's FPS includes decode[19], and Self Forcing's 0.69 s does not say[18].

What to try first

  • For an image model, start from DMD2's recipe[1]: generator and fake score initialized from the teacher, five fake updates per generator update, a GAN head on the fake network's bottleneck, four steps with the student trained on its own earlier steps, about 60 hours on 64 A100s for SDXL. LADD's hinge loss on the teacher's token features is the alternative when you want no fake score network[22]. Decide the guidance scale first; the student will not let you change it.
  • Log what the papers skip: for 200 prompts and four seeds, mean pairwise LPIPS per prompt and recall against the teacher's samples[88], every few thousand iterations. Falling FID with falling LPIPS is mode dropping.
  • For a video model, run Self Forcing on a Wan-class teacher[18]: chunks of three latent frames, four steps, the 14B model at guidance 3.0 as the real score, a 5:1 critic ratio, about 600 iterations on 64 H100s. Use the rolling cache with the training-time window that stops the last chunk attending to the first, and measure first-frame latency and drift past the training length.
  • Compare against the teacher at matched wall-clock time. Run the teacher at 8 and 16 steps with a good solver too; a four-step student that only beats the 50-step teacher has beaten the wrong baseline.
  • Measure end to end, with the text encoder and VAE in the clock[52], at the resolution you will ship, with negative prompts and counting prompts in the eval set[90].

References

  1. Yin, Gharbi, Park, Zhang, Shechtman, Durand, Freeman, Improved Distribution Matching Distillation for Fast Image Synthesis, NeurIPS 2024.
  2. Blau, Michaeli, The Perception-Distortion Tradeoff, CVPR 2018.
  3. Robbins, An Empirical Bayes Approach to Statistics, Proc. Third Berkeley Symposium, 1956.
  4. Efron, Tweedie's Formula and Selection Bias, JASA 106(496), 2011.
  5. Dieleman, The Paradox of Diffusion Distillation, sander.ai, 2024.
  6. Salimans, Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022.
  7. Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network, 2015.
  8. Kim, Rush, Sequence-Level Knowledge Distillation, EMNLP 2016.
  9. Gu, Dong, Wei, Huang, MiniLLM: Knowledge Distillation of Large Language Models, ICLR 2024.
  10. Agarwal, Vieillard, Zhou, Stanczyk, Ramos, Geist, Bachem, On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024.
  11. Lu and Thinking Machines Lab, On-Policy Distillation, 2025.
  12. Yin, Zhang, Zhang, Freeman, Durand, Shechtman, Huang, From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, CVPR 2025.
  13. Huang, Li, He, Zhou, Shechtman, Self Forcing code and checkpoints, GitHub, 2025.
  14. Song, Dhariwal, Chen, Sutskever, Consistency Models, ICML 2023.
  15. Yin, Gharbi, Zhang, Shechtman, Durand, Freeman, Park, One-step Diffusion with Distribution Matching Distillation, CVPR 2024.
  16. Bradley, Nakkiran, Classifier-Free Guidance is a Predictor-Corrector, 2024.
  17. Karras, Aittala, Kynkäänniemi, Lehtinen, Aila, Laine, Guiding a Diffusion Model with a Bad Version of Itself, NeurIPS 2024.
  18. Huang, Li, He, Zhou, Shechtman, Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, NeurIPS 2025.
  19. Zheng, Wang, Ma, Chen, Zhang, Balaji, Chen, Liu, Zhu, Zhang, Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency, ICLR 2026.
  20. Ren, Xia, Lu, Zhang, Wu, Xie, Wang, Xiao, Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, NeurIPS 2024.
  21. Luo, Tan, Huang, Li, Zhao, Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, 2023.
  22. Sauer, Boesel, Dockhorn, Blattmann, Esser, Rombach, Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation, 2024.
  23. Yin et al., DMD2 code, training scripts and Model Zoo, GitHub, 2024.
  24. Song, Sun, Yin, SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions, 2024.
  25. Team Wan, Wan: Open and Advanced Large-Scale Video Generative Models, 2025.
  26. Zhou, Zheng, Wang, Yin, Huang, Score Identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation, ICML 2024.
  27. Sauer, Lorenz, Blattmann, Rombach, Adversarial Diffusion Distillation, 2023.
  28. Lightricks, LTX-2 model card, Hugging Face, 2026.
  29. Black Forest Labs, FLUX.1 [schnell] model card, Hugging Face, 2024.
  30. Stability AI, Stable Diffusion 3.5 Large Turbo model card, Hugging Face, 2024.
  31. Bandyopadhyay et al., SD3.5-Flash, 2025.
  32. Chen, Xue, Zhao, Yu, Paul, Chen, Cai, Han, Xie, SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, 2025.
  33. ModelTC, Wan2.2-Lightning, GitHub README, 2025.
  34. Meng, Rombach, Gao, Kingma, Ermon, Ho, Salimans, On Distillation of Guided Diffusion Models, CVPR 2023.
  35. Dieleman, Guidance: A Cheat Code for Diffusion Models, sander.ai, 2022.
  36. Luo, Tan, Patil, Gu, von Platen, Passos, Huang, Li, Zhao, LCM-LoRA: A Universal Stable-Diffusion Acceleration Module, 2023.
  37. Song, Dhariwal, Improved Techniques for Training Consistency Models, ICLR 2024.
  38. Kim, Lai, Liao, Murata, Takida, Uesaka, He, Mitsufuji, Ermon, Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion, ICLR 2024.
  39. Lu, Song, Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, ICLR 2025.
  40. Liu, Gong, Liu, Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, 2022.
  41. Liu, Zhang, Ma, Peng, Liu, InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, ICLR 2024.
  42. Frans, Hafner, Levine, Abbeel, One Step Diffusion via Shortcut Models, ICLR 2025.
  43. Geng, Deng, Bai, Kolter, He, Mean Flows for One-step Generative Modeling, NeurIPS 2025.
  44. Boffi, Albergo, Vanden-Eijnden, Flow Map Matching with Stochastic Interpolants: A Mathematical Framework for Consistency Models, 2024.
  45. Sabour, Fidler, Kreis, Align Your Flow: Scaling Continuous-Time Flow Map Distillation, 2025.
  46. Zhou, Ermon, Song, Inductive Moment Matching, 2025.
  47. Wang, Lu, Wang, Bao, Li, Su, Zhu, ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, NeurIPS 2023.
  48. Luo, Hu, Zhang, Sun, Li, Zhang, Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models, NeurIPS 2023.
  49. Zhou, Wang, Zheng, Huang, Guided Score Identity Distillation for Data-Free One-Step Text-to-Image Generation, ICLR 2025.
  50. Xie, Xiao, Kingma, Hou, Wu, Murphy, Salimans, Poole, Gao, EM Distillation for One-step Diffusion Models, NeurIPS 2024.
  51. Lin, Wang, Yang, SDXL-Lightning: Progressive Adversarial Diffusion Distillation, 2024.
  52. Lin, Xia, Ren, Yang, Xiao, Jiang, Diffusion Adversarial Post-Training for One-Step Video Generation, ICML 2025.
  53. Xu, Zhao, Xiao, Hou, UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs, 2023.
  54. Yin et al., DMD2 model card, Hugging Face, 2024.
  55. Gandikota, Bau, Distilling Diversity and Control in Diffusion Models, 2025.
  56. Stability AI, SDXL-Turbo model card, Hugging Face, 2023.
  57. Li, Feng, Fu, Wang, Basu, Chen, Wang, T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback, NeurIPS 2024.
  58. Li, Long, Zheng, Gao, Piramuthu, Chen, Wang, T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design, ICLR 2025.
  59. Yang et al., LongLive: Real-time Interactive Long Video Generation, 2025.
  60. Liu, Hu, Xu, Shan, Lu, Rolling Forcing: Autoregressive Long Video Diffusion in Real Time, ICLR 2026.
  61. Lightricks, LTX-Video README and model card (ltxv-13b-0.9.7-distilled), 2025.
  62. Wu, Li, Zhang, Ma, DP-DMD, 2026.
  63. Liu et al., Decoupled DMD, 2025.
  64. Li et al., 1.x-Distill, 2026.
  65. Chen et al., Data-Forcing Distillation, 2026.
  66. Li et al., DUET, 2026.
  67. Lu et al., Reward Forcing, 2025.
  68. Yi et al., Deep Forcing, 2025.
  69. Cui et al., Self Forcing++: Towards Minute-Scale High-Quality Video Generation, 2025.
  70. Jiang et al., Distribution Matching Distillation Meets Reinforcement Learning, 2025.
  71. Wang et al., AdvDMD, 2026.
  72. Huang et al., RTDMD, 2026.
  73. Chen et al., Visual Generation Without Guidance (Guidance-Free Training), ICML 2025.
  74. Yu et al., Self-E, 2025.
  75. Cheng et al., TwinFlow, ICLR 2026.
  76. Peng et al., FACM, 2025.
  77. Lin et al., Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation, NeurIPS 2025.
  78. Cheng et al., Phased One-Step Adversarial Equilibrium for Video Diffusion Models, AAAI 2026.
  79. Feng, Cui, Ban, Hsieh, One-Forcing, 2026.
  80. lightx2v, Wan2.2-Lightning model card, Hugging Face, 2025.
  81. Huang et al., VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.
  82. Huang et al., VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models, 2024.
  83. Babu et al., DynamicEval, 2025.
  84. Latent Consistency, LCM-LoRA SDXL model card, Hugging Face, 2023.
  85. He et al., Self Forcing checkpoints, Hugging Face, 2025.
  86. Hugging Face, diffusers consistency distillation training examples, GitHub.
  87. Yin et al., CausVid code, GitHub, 2024.
  88. Kynkäänniemi, Karras, Laine, Lehtinen, Aila, Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019.
  89. Naeem, Oh, Uh, Choi, Yoo, Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020.
  90. Ghosh, Hajishirzi, Schmidt, GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, NeurIPS 2023.

Citation

@misc{saini2026diffusiondistillation,
  author       = {Saini, Shreshth},
  title        = {Distilling Diffusion Models},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://shreshthsaini.github.io/blogs/diffusion-distillation.html}},
  note         = {Blog post}
}