Distilling Diffusion Models

  1. Distilling Diffusion Models
  2. What we will cover today
  3. Part 1: Why diffusion is slow
  4. Sampling takes many small steps
  5. From 100 network passes down to 4
  6. Why one big step gives a blur
  7. One step lands on the average image
  8. Part 2: What distillation is
  9. Distillation in one picture
  10. Language models went through this first
  11. In diffusion, the student learns to skip steps
  12. Why diffusion is harder to distill
  13. Who sets the target, and on what samples
  14. Learning to drive, three ways
  15. Forward KL spreads, reverse KL picks a peak
  16. Part 3: Three things a student can copy
  17. Three ways to learn to draw
  18. Every method copies one of three things
  19. Part 4: Copy the teacher's path
  20. Learning to skip stairs
  21. Progressive distillation halves the steps each round
  22. Guidance distillation folds two passes into one
  23. The idea behind consistency models
  24. A consistency model jumps straight to the image
  25. Consistency distillation makes neighbours agree
  26. Consistency training drops the teacher
  27. LCM brought four steps to Stable Diffusion
  28. Reflow straightens the path
  29. Flow maps learn whole jumps at once
  30. The trouble with copying the path
  31. One step on a curved path blurs
  32. One-step scores kept getting better
  33. Part 5: Match the teacher's distribution
  34. Distribution matching in one picture
  35. DMD moves samples by a score difference
  36. DMD also needed teacher pairs
  37. DMD2 drops the pairs and adds a GAN
  38. Four DMD2 steps next to fifty SDXL steps
  39. Moment matching in one picture
  40. Moment matching compares average predictions
  41. A student can beat its teacher's score
  42. LADD judges images in the teacher's features
  43. SDXL-Turbo at one, two and four steps
  44. What few-step images still get wrong
  45. Why distilled models lose variety
  46. New seed, same picture
  47. Newer methods keep more of the variety
  48. What the big labs ship for images
  49. Part 6: Distilling video models
  50. Frame by frame, the video flickers
  51. A causal mask lets the student stream
  52. CausVid learns to stream from a bidirectional teacher
  53. Four streaming steps next to fifty teacher steps
  54. rCM keeps Wan2.1 sharp down to two steps
  55. Why small mistakes pile up in video
  56. Exposure bias returns once the student streams
  57. Self Forcing trains on its own rollouts
  58. CausVid oversaturates, Self Forcing does not
  59. Self Forcing drifts after five seconds
  60. Video students lose variety too
  61. What the big labs ship for video
  62. Distillation from 2015 to 2026
  63. Part 7: What still breaks
  64. Problems nobody has solved yet
  65. Where I would start
  66. Using a distilled model takes a few lines
  67. Which kind of distillation to pick
  68. References
  69. Thanks

← → or swipe to move, click the slide to advance, F for full screen. Add #12 to the address to link to slide 12.