Fleetcraft: GPU Infrastructure Notes from Video Generation Projects

01The projects, one infrastructure problem

We learned these lessons across more than 500 GPU-days of work in three large video-generation projects. The research itself is not the subject here; this post distills what those projects taught us about infrastructure. The workloads looked unrelated at first. One project ran training-free transformer caching and test-time search across more than 31,000 scored rollouts; we call it the caching project. Another combined long-horizon inference with six-rank FSDP distillation; we call it the distillation project. The third coupled rollout producers and trainers across a heterogeneous fleet that peaked at 47 GPUs; we call it the RL project. (release, launch, and coordination records)

The common failures were rarely a bad matrix multiplication. Jobs died because a VAE moved the peak to decode, a Python object kept an old KV cache alive, a supervisor trusted a stale crash line, a per-GPU worker had the wrong physical identity, or a storage path delivered bytes too slowly. We repeatedly had GPUs allocated, models loaded, and dashboards showing activity while little useful science advanced.

Our thesis is simple. The GPU kernel is almost never the thing that kills a video project. Kernel efficiency matters, but memory placement, orchestration, and measurement discipline decide whether the kernel gets a valid workload at all. The right unit of optimization is the complete cycle of loading, denoising, decoding, scoring, training, checkpointing, and publishing evidence.

02The hardware we actually ran

The projects crossed four operationally different GPU classes. LS6 full nodes exposed three A100-40 GPUs, while its H100 nodes exposed two H100-80 GPUs. Vista exposed one GH200-96 GPU per Grace Hopper node and four Blackwell-class GPUs per GB200 node, each with 185 GiB of HBM. (project case studies) The distinction was larger than capacity. It changed host links, CPU architecture, dependency availability, and the ratio between Tensor Core throughput and memory bandwidth.

These are vendor decimal units, except where the project itself reported GiB. That distinction matters near a limit. It also matters that product tables often lead with structured-sparsity throughput. We use dense rates for roofline reasoning because ordinary dense video training does not receive the doubled sparse number. The resulting dense BF16 ridge points are about 201 FLOP per HBM byte for A100-40, 295 for H100, 248 for GH200-96, and 313 for a GB200 NVL72 GPU, computed from the fact-sheet peaks and bandwidths.[5]

Grace changes offload economics without turning host memory into HBM. The GH200 CPU and GPU share a coherent address space over a 900 GB/s bidirectional C2C link. Grace LPDDR5X reaches up to 512 GB/s, far above an ordinary host path but still far below the GPU's 4 TB/s HBM. Oversubscription can make a model fit while slowing the hot path or moving the failure into host memory. On our systems the other practical change was aarch64. The ISA was standard Arm Linux, but CUDA wheel availability lagged x86.[6]

03Memory hierarchy and the roofline

A GPU is a ladder of capacity and reuse. Registers sit inside each streaming multiprocessor and feed arithmetic with the least movement. Shared memory and L1 provide software-managed or cached on-chip reuse. L2 is shared across the GPU. HBM is large and fast by host standards, but it is distant from the arithmetic units. Host memory sits beyond PCIe or a coherent C2C link. Each descent buys capacity and pays latency, bandwidth, and energy.[1][2]

The roofline turns that hierarchy into a performance test. Arithmetic intensity is the number of operations performed per byte moved. If a kernel's intensity falls below the GPU's compute-to-bandwidth ratio, memory traffic sets the ceiling. If it rises above that ridge point, arithmetic sets the ceiling. The model is intentionally simple. It tells us which resource deserves investigation before we interpret an optimization.[5]

Horace He's first-principles performance guide gives a useful test for compute-bound, memory-bandwidth-bound, and overhead-bound work. Aleksa Gordić's matmul deep dive follows the path from GPU fundamentals to Hopper kernels, while Simon Boehm's CUDA matmul worklog shows how successive memory and tiling changes approach library performance. The Modal GPU Glossary is a compact reference when the hardware and software terms blur together.

Naive attention is a canonical bandwidth problem because it materializes an N by N score matrix in HBM, then reads it for softmax and the value product. Video tokens multiply across time, height, and width, so both the quadratic arithmetic and the round trips grow quickly. VAE decode stresses a different edge. Convolutions and transformations operate on near-pixel-resolution activations, then stage large frame tensors for host transfer. Video encode and filesystem writes contribute no Tensor Core work at all. They consume copy engines, CPU, memory bandwidth, and storage service time.

This is why a single utilization label is inadequate. A denoiser can be compute-saturated while the subsequent decode is capacity-limited and the final write is storage-limited. The project throughput is the reciprocal of their sum, plus every synchronization gap between them. We therefore calibrated one full cycle instead of extrapolating from a warm transformer forward.

04FlashAttention, SDPA, and the aarch64 reality

FlashAttention is best understood as a memory-traffic algorithm, not an approximation. It divides queries, keys, and values into tiles that fit on-chip, streams those tiles from HBM, and maintains the softmax normalization online. The algorithm never writes the full N by N score matrix to HBM. Backward recomputes selected intermediates instead of storing that matrix. Attention remains exact, while its HBM footprint becomes linear rather than quadratic in sequence length.[7]

FlashAttention-2 spends less time on non-matmul rescaling, parallelizes a single head across the sequence dimension, and partitions work between warps with less shared-memory traffic. Those changes matter for the long sequences and small batches common in video. The paper reports roughly twice the speed of the first implementation and between 50 and 73 percent of A100 peak FLOP throughput for the kernel.[8]

FlashAttention-3 specializes the same IO-aware idea for Hopper. Producer warpgroups issue asynchronous TMA loads while consumer warpgroups run WGMMA Tensor Core operations. Ping-pong scheduling overlaps one group's softmax with another group's matrix products. The reported H100 result reaches 740 FP16 TFLOPS, about 75 percent of peak, and runs 1.5 to 2.0 times faster than FlashAttention-2 on the tested shapes.[9]

The implementation twist on Grace was packaging. Official prebuilt flash-attn wheels were not available for aarch64, and building from source required a long, carefully bounded compile. PyTorch's scaled-dot-product attention gave us one API with fused FlashAttention-2, memory-efficient, cuDNN, and math backends selected according to inputs and build support. The preferred sdpa_kernel() control let us constrain that dispatch when a protocol needed it.[10][11]

The caching and distillation projects therefore used PyTorch SDPA, with FlexAttention where the causal mask required it, and installed no external attention package. That choice was slower than a tuned Hopper-specific stack could be, but it was scientifically clean. Cached and full-compute arms shared the same backend, so a paired latency ratio canceled backend-level overhead to first order. The distillation project fixed SDPA and FlexAttention across every allocation arm for the same reason. Absolute latency remained specific to GH200 and the pinned stack, while within-stack comparisons remained interpretable. (caching-project environment control; distillation-project record)

Backend pinning still requires semantic parity. One vendored fallback dropped the text key-padding mask at five call sites. Routing those calls through a corrected dispatcher fixed the result. A kernel can expose the same function signature and still change the experiment if masks, dtypes, or strides take a different path. (distillation-project recipe audit record)

05Where memory actually goes in video generation

The model weights are only the floor. In our runs, Wan2.1-1.3B occupied about 30 GB on GH200 at batch one, CogVideoX-5B used about 25 GB in bf16, HunyuanVideo-13B used about 43 GB in weights, and the VideoScore verifier alone used about 17 GB. On A100-40, all Hunyuan weights occupied 38.55 GB before useful activations, which forced roughly 14 GB of Llama text-encoder state off the device. (caching-project release and evaluator records; RL-project smoke record)

Activations then scale with batch, resolution, token count, layer structure, and whether autograd is enabled. Optimizer states appear later and can move the peak after the first apparently successful update. KV and feature caches scale with horizon and can outlive the phase that created them. VAE decode expands compact latents toward pixels. The useful inventory is therefore ownership over time, not one allocator snapshot.

Accidental autograd can consume the whole card

A direct call into vendored Self-Forcing inference omitted no-grad. The resulting graph retained about 80 GB for 21 latent frames and four denoising steps on a 1.3B model, enough to OOM a 95 GB-class GH200. The repair was not a smaller video. We placed the adapter calls under no-grad or inference mode, preserving inference semantics while deleting a graph that should never have existed. (distillation-project vendored-inference record)

Caches have owners, not just bytes

Horizon changed cache geometry by model family. Windowed Self-Forcing, Rolling, and LongLive paths used about 4 GB per sample. Causal Forcing variants used about 7 GB. rCM retained full history and reached about 70 GB per sample at 240 latents, which made batch one correct even on GH200. These were algorithmic differences, not allocator quirks. (distillation-project batch-calibration record)

A more deceptive incident came from the measurement probe. The probe generated chunks at batch four, then training resumed at batch two. Its 30-layer KV and cross-attention cache kept about 24 GiB resident per GPU, so training tried to allocate a new cache beside the old one. Calling empty_cache() did nothing because the tensors were live. Releasing the objects that owned those tensors restored the intended headroom. (distillation-project memory-ownership record)

The same distinction explains why nvidia-smi, memory_allocated(), and memory_reserved() disagree. The first sees the process and non-PyTorch allocations, the second sees tensor bytes, and the third includes PyTorch's cached pool. An allocator release can return only unused blocks. It cannot invalidate a Python reference.[12]

Decode moves the peak, sometimes into host RAM

During 60-second long-horizon generation in the distillation project, batch four occupied 48 to 54 GB through denoising and then peaked at 91 GB during VAE decode. That shape delivered about three to four times the old batch-one throughput, but decode set the maximum. At 240 seconds, each sample staged about 18.5 GB of fp32 pixels in host memory. Batch four implied about 74 GB before copies and triggered the Slurm cgroup OOM killer. Batch two was safe. (distillation-project utilization doctrine record)

This gave us a fast diagnostic split. A traceback ending in torch.cuda.OutOfMemoryError identifies device allocation pressure. A bare Killed with no Python traceback identified host or cgroup pressure in these incidents. LongLive added a third pattern. It decoded the full extended video on GPU, created a second normalized fp32 copy of about 17 to 18 GB, and failed after three successful videos because sequential fragmentation accumulated. Chunking and offloading every sample fixed it. (distillation-project batch-calibration record)

The first optimizer step can lie

A Wan RL run at 33 frames completed its first update, then OOMed after optimizer-state materialization on the next update. The transactional recovery reduced policy microbatch from four to two. A separate critic incident showed why configuration inheritance must be inspected rather than assumed. One critic silently inherited gradient checkpointing from a deep-copied policy, while a separately constructed critic did not. For a roughly 14,000-token transition, adding activation checkpointing reduced peak memory from 38.91 to 19.32 GiB. (RL-project handover and smoke records)

Our calibration loop now includes model load, at least one complete denoise and decode, backward, optimizer materialization, probes, and checkpoint save. It records the owner and phase for each peak. Anything shorter is a smoke test, not a capacity decision.

06Utilization doctrine and calibration before scale

We used a strict order. First, run one complete task for each workload family at trial batch sizes. Second, observe denoise, decode, scoring, backward, and write phases. Third, freeze a per-family batch table. Only then enqueue the project. For the ordinary case, the target was about 70 to 85 percent VRAM with roughly 10 to 15 percent headroom. (distillation-project utilization doctrine record)

Every GPU host also started the shared logger as gpu_telemetry.sh <project-scratch>/telemetry 30 &, which sampled memory, utilization, and power every 30 seconds. Project monitors joined those traces to spool and run state instead of interpreting the device stream alone.

The target was a heuristic, not a quota. The caching project's Wan path used about 30 GB on GH200, yet full throughput slipped from 0.92 to 0.90 videos per minute under batching. Cached throughput rose only 8 percent, and batch eight OOMed. The 32K-token path was compute-saturated at batch one. More resident samples did not create more arithmetic throughput. (caching-project batching sweep)

GB200 supplied the opposite-looking counterexample with the same conclusion. An 81-frame Wan rollout peaked at 78,752 MiB by nvidia-smi, about 42 percent of the 185 GiB device, and produced 104.9 rollouts per hour. The implementation exposed no safe batching route, and the path was compute-bound. During live asynchronous RL the trainer used 55,865 MiB while seven rollout GPUs used about 35,447 MiB each at 85 to 100 percent reported utilization. Low HBM fraction was not idle compute. (RL-project Vista calibration and smoke records)

Protect the timed pass

Bulk generation and reported latency are different measurements. Every reported latency or FPS value in the distillation project came from a dedicated, exclusive batch-one pass. The caching project marked generation and gate tasks as timed, ran one per node, and allowed co-location only for untimed scoring and analysis. This prevented another process from stealing compute, memory bandwidth, or storage service during the measured interval. (distillation-project utilization doctrine record; caching-project worker policy)

Batched execution can also change numerics. The caching project saw about 0.6 of reduction-order drift, and its adaptive cache averaged the skip signal across a batch, producing one shared schedule. The distillation project gave each sample its own generator so initial noise matched single-sample execution, while still accepting ordinary batched-kernel differences. We never mixed batched and unbatched results inside one scientific arm. (caching-project release record; distillation-project utilization doctrine record)

What GPU-Util measures

NVML's GPU utilization is the percentage of a recent sample period during which at least one kernel executed. The underlying sample spans between one sixth of a second and one second. It is not SM occupancy, Tensor Core activity, FLOP efficiency, or useful scientific throughput. A small kernel running continuously can report 100 percent. A healthy video pipeline can report zero between denoise, decode, score, and I/O phases.[13]

For live watching during operations, per-GPU, per-node, and fleet-level utilization views came from all-smi, Lablup's open-source cross-platform accelerator monitor that extends the nvidia-smi view across many nodes at once.[14] The archived traces behind the figures in this post came from the plain nvidia-smi CSV logger, which is what the repository ships for durable records. One caveat worth knowing: all-smi does not discover a Slurm allocation by itself, and strict SSH host-key checking can silently drop recycled nodes from the view. The exact node-discovery and host-key settings we run are in the playbook's telemetry skill.

all-smi - 2026-07-22 15:39:11                                                                                                                                                                              v0.23.0
Cluster Overview
│ Nodes       │ Total RAM   │ GPU Cores   │ Total VRAM  │ Avg. Temp   │ Total Power │  ⊗○
│ 1/1         │ 0.0GB       │ 1           │ 96GB        │ 73°C        │ 0.9kW       │
│ CPU Cores   │ Used RAM    │ GPU Util    │ Used VRAM   │ Temp. Stdev │ Avg. Power  │
│ 0           │ 0.0GB       │ 100.0%      │ 1GB         │ ±0.0°C      │ 851.1W      │
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Live Statistics
GPU Util. ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿ 100.0% CPU Util.                                                                                        N/A
GPU Mem.  ⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀ 1.3%   Host Mem.                                                                                        N/A
GPU Temp. ⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶  72°C  CPU Temp.                                                                                        N/A

Tabs:  All  Users  Topology  ssh://saini_2@c642-082.vista.tacc.utexas.edu·native
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
GPU  0 120GB   NVIDI @ u   c642- Util:100.0% VRAM:   1.2/96GB Temp:  73°C Freq:1.86GHz Pwr:851/900W Drv:590.48.01 CUDA:13.1
      Slowdown:89°C Shutdown:95°C MaxOp:87°C P-State:P0
      HW NUMA:1 GSP:default v590.48.01
     Util : [▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬  100.0%]  Mem  : [▬─────────────────────────────────────────────────────────────────────────────────   1.2GB]

































h:Help q:Exit f:Filter ←→:Scroll ↑↓:Scroll p:PID m:Memory g:GPU-Mem [Sort:Default]

Point probes remain useful for attribution. They tell us which GPU is computing now, which one holds project memory between bursts, and which one is truly free. Fleet health needs a rolling view. The RL project summarized five-minute windows using active-sample fraction, defined as the fraction of samples at or above 5 percent utilization, plus mean utilization and mean memory fraction. Each summary tied itself to the source telemetry by hash. (RL-project telemetry summary)

The distillation-project trace makes the phase structure visible. Calibration began at low mean utilization, the measured batching migration raised the next regime, later training arrived in distinct waves, and the final reporters entered a quiet tail. Gaps mean no archived host CSV reported in that interval; the plot does not bridge them with invented measurements.

The RL project's controller trace shows a different failure mode. The working-GPU percentage fell through the documented 23.6 percent active-sample audit, then recovered when frozen evaluator evidence became visible to the queue; the allocation denominator alone could not explain either transition.

The first audit also measured 23.45 percent mean utilization and 20.1 percent mean memory occupancy. Recovery froze 24 exact panels, bound 1,536 source videos by hash, and published 48 independent reruns. An earlier verified full-shape window measured 91.8 percent active samples, 90.3 percent mean utilization, and 70.7 percent mean memory occupancy. (RL-project utilization learning record)

The caching project predates the telemetry-everywhere doctrine, so its evidence is the per-candidate append-only manifest rather than fleet CSVs. We do not manufacture a fleet trace for a project that did not record one.

The first audit covered a 47-GPU allocation but found only 12 working and 35 genuinely free. Both task spools had drained. Recovery published already-declared evaluator repeatability work rather than inventing a new efficacy endpoint. A later full-shape trace reached 92.4 percent active samples, 91.8 percent mean utilization, and 79.3 percent memory occupancy over 474 physical-GPU samples. Even there, a trainer could hold 76.8 percent memory without kernels while waiting for a complete 64-record transaction. Queue context explained what the dashboard alone could not. (RL-project utilization learning record)

MFU for dense training loops

Model FLOPs Utilization, or MFU, is achieved model FLOPs per second divided by the accelerator's theoretical peak. The PaLM paper introduced the metric using the model's forward and backward FLOPs rather than counting rematerialization as useful work. It is different from nvidia-smi utilization, which asks whether any kernel ran during a sample, and from DCGM SM activity, which exposes how active the streaming multiprocessors were.

These projects tracked rolling active-sample fraction, VRAM, and per-family throughput rather than MFU. That was the honest project-level view for heterogeneous inference and evaluation fleets whose cycles included decode, scoring, and I/O. MFU is the better headline number for a dense training loop because its denominator and useful arithmetic are well defined, while active-sample fraction answers whether a mixed fleet is advancing work at all.

Post hoc, the archived records are sufficient to estimate MFU analytically. Following the PaLM convention, ESTIMATED MFU divides achieved model FLOPs per second by peak dense BF16 FLOPs; inference counts forward model FLOPs, while training uses the standard 2N forward plus 4N backward approximation without rematerialization credit. Each forward includes 2 × active parameters × video tokens plus 4 × layers × sequence length² × hidden width, so the attention term is included; text encoders and VAEs are excluded, while their wall time remains in the denominator. The denominators are 312 TFLOP/s for A100, 989.4 TFLOP/s for H100, 990 TFLOP/s for GH200, and 2.25 PFLOP/s for GB200-class devices, from the GPU architecture fact sheet; these are post-hoc analytic estimates, not profiler measurements.

estimated_mfu = (model_flops_per_step * steps_per_second) / peak_dense_flops_per_second

Across the caching project, full inference estimates span 37.1 to 53.8 percent, while cached execution spans 36.1 to 53.4 percent. Cached execution reduces both transformer work and wall time, so estimated MFU remains close to the corresponding full run even when end-to-end latency falls. RL rollouts span 16.5 to 47.3 percent because their denominators retain decode and scoring phases. The three-A100 trainer update reaches 26.0 percent; it does not exceed every rollout in this post-hoc estimate.

07Fleet orchestration, where queues do the driving

On Vista, the user could submit 40 jobs, run about 20, and occupy up to 96 nodes. Job slots were scarcer than nodes. The scalable abstraction was one multi-node allocation with one worker per node, backed by a shared file spool. The caching project used a 16-node, 12-hour fleet in one slot. Its separate 64-node, five-hour VBench sweep was also one scheduler job. (caching-project progress record; account operations record)

We designed the queueing and fleet shape before launch, then started one project session with a submission such as sbatch -N N slurm/fleet.sbatch. Automation published work with commands such as bin/enqueue.sh 'command' task-name, and workers drained the shared queue on the fly. Workers changed a task's state with atomic renames among pending, running, done, and failed.

A task began as a small executable record under pending. A worker claimed it by atomic rename into running, executed it, and moved the task and log into done or failed. Stable IDs made retries idempotent. A stale-claim reaper checked whether the owning Slurm job still existed before returning a task to pending. Four caching-project claims still stranded at a walltime boundary, which taught us that controller-side recovery must exist even when workers also reap. (caching-project worker and recovery records)

Sequence the work, then exit cleanly

Queue latency should not separate every scientific unit. The spool sequences successive waves automatically. Dependent scoring and the next declared wave enter the queue as their inputs appear, so one project session flows from generation to scoring without a new submission for each task. A worker that hits an error keeps its device only for the brief recovery window, then the task is requeued automatically. The controller submits a follow-on job only when queued work justifies the exact shape. One RL-project 12-node A100 fleet needed independent eight-node and four-node lanes because the larger queued shape could not replace the smaller one under the node cap. (distillation-project scheduling record; RL-project utilization learning record)

The project ends with a short bounded tail. Workers that find no eligible work idle only until their timeout, then exit on their own so unused allocation returns to the scheduler. Every submission checks the live allocation balance, and submissions stop before the allocation nears exhaustion, with a hard headroom threshold. The queueing layer exists to move useful work efficiently, not to transform a reservation into a utilization theater. (distillation-project budget record; account operations record)

Why the second spool was necessary

The RL project's outer spool owned whole nodes. That was correct for trainers but wasteful for independent evaluations on a three-GPU A100 node. A packed outer claim solved the ownership problem. It kept the node exclusive, launched one slot worker per physical GPU, and let those slots atomically drain untimed tasks from an inner queue. Full-node work could not collide with a pack because both competed at the outer level. (RL-project runbook)

The first bridge divided free GPUs by three, assuming every node was a full A100 shape. It underfilled the mixed fleet. Counting one potential bridge per observed free GPU restored 35 of 45 working A100-class GPUs while models were still loading. Surplus bridge claims exited after one idle minute, and the controller capped them at 12. Slots kept claiming after short tasks finished so a long scorer could not strand its siblings. Every task ran in its own process group, and the pack reaped every child before giving up the node. (RL-project utilization learning record)

The qualifying word is useful. We filled free devices only with frozen panels, declared repeatability checks, pending scores, or other protocol-valid work. We did not invent endpoints, hyperparameters, or dummy kernels to improve occupancy. When no compatible work existed, workers reached the bounded idle timeout, exited, and let project closeout return unused allocation to the scheduler.

08Multi-node training pathology

Multi-node training fails as a distributed system before it fails as a model. The distillation project used six one-GPU GH200 nodes under torchrun, c10d rendezvous, and full FSDP sharding. The validated world size was part of the experimental protocol. The week that made this reliable was dominated by launch identity, stale logs, cache races, and save boundaries rather than loss functions. (distillation-project launch record)

The Hugging Face Ultra-Scale Playbook gives the broader map of data, tensor, pipeline, context, and expert parallelism, plus the memory and communication tradeoffs behind them. Our narrower lesson was to validate the topology that the launcher actually constructed on one-GPU nodes.

The crash signature missed the crash

A run at allocation fraction u = 0.29 died with RendezvousConnectionError and DistNetworkError. The supervisor recognized only ChildFailedError and unhandled CUDA failures, so its sequencer waited for progress that could never arrive. We added rendezvous and distributed-network signatures. (distillation-project supervisor record)

Broader grep was not enough. A transient ChildFailedError could remain in a log after torchrun recovered, and the next poll would count the old line as a new crash. A recently modified log plus a lingering process also caused ghost re-adoption of a dead run. We made signatures relative to a launch or adoption epoch and archived old rank logs at adoption. A log is an event stream, not current state. (distillation-project supervisor record)

Two supervisors both believed they were in charge

Pattern-based cleanup caused a separate class of damage. A pattern could match the shell running the kill, or identify the generic fleet worker rather than the child training group. Killing worker_loop ended two six-node fleets because that worker was itself the Slurm step. We switched to anchored process searches, exact PIDs, dedicated pidfiles, and process groups. An OOM that left three ranks inside a collective was recovered by cancelling only the exact affected Slurm step with scancel jobid.step, preserving the parent allocation for the controller. (distillation-project fleet-safety record)

Compiler caches became distributed mutable state

Concurrent nodes shared generated Triton sources on the filesystem. The resulting symptoms looked like malformed source and eventually collapsed rendezvous. Generated-source caches became node, run, and retry specific. Content-addressed TorchInductor graph artifacts were reused only sequentially on the same node, and training and evaluation received separate cache roots. (distillation-project compiler-cache record)

This distinction matters. A content-addressed immutable artifact can be reused after its producer finishes. A generated source tree being mutated by concurrent compilers is a shared protocol with no transaction boundary. Putting both under a variable called “cache” does not make their safety properties equal.

The sharding name concealed replication

The upstream Self-Forcing launcher used hybrid full sharding under an eight-GPU-per-node assumption. Vista exposed one GH200 per node. The intra-node shard group therefore had one member, which reduced the intended sharding to replication and drove memory to 93 GB. Full FSDP sharding across the six nodes restored the intended distribution. (distillation-project launch record)

We later received ten nodes for a registered six-rank trajectory. Adding four training ranks would have changed global batch and the optimization path. Instead, the six training nodes kept the validated world size while four surplus nodes ran dependency-safe evaluation in a disjoint Slurm step. Evaluation waited behind a readiness barrier, consumed only validated outputs, and exited after both a producer-done marker and an empty queue. Temporary idle time was cheaper than invalidating the registered comparison. (distillation-project training-topology record)

When DDP could not express the graph

The RL project hit a different distributed boundary. Classifier-free guidance used a double forward, activation checkpointing required static_graph, and chunked gradient accumulation needed no_sync. The combination asserted in reducer.cpp:1660 on the first backward. Rather than weaken the graph, we removed DDP and manually all-reduced about 2.6 GB of bf16 policy gradients once per step after local GRPO accumulation. (RL-project smoke record)

Rank zero claimed whole GRPO groups and computed DanceGRPO advantages before sharding, so the distributed backward preserved group semantics. For single-rollout optimization, 64 transitions did not divide evenly across three ranks, so chunk losses were weighted by global transition count and gradients were summed. The three-GPU GRPO path reduced a step from 1,893 to 639 seconds, a 2.96x speedup. The world-one hooks remained inert so the validated H100 control path stayed byte-identical. (RL-project data-parallel trainer record)

A checkpoint is a completed save boundary

A file appearing on disk did not prove that an FSDP checkpoint was durable. Gathers had to complete, the save routine had to confirm success, the training loop had to acknowledge that boundary, and the next step had to demonstrate continuation. Killing at the wrong moment could leave a readable partial file. (distillation-project checkpoint record)

We distinguish weight evidence from exact resume. Exact resume includes optimizer moments, per-rank RNG, sampler position, global step, branch history, probe state, and the compute ledger. The RL project likewise paired the newest policy file with atomically written trainer state containing critic, reward normalization, counters, and optimizer state. “Weights loaded” and “trajectory resumed” are different claims. (distillation-project checkpoint record; RL-project checkpoint record)

09Async RL is a queueing protocol

Asynchronous post-training couples two rates. Producers sample with a recent policy, record per-step log probabilities, decode and score the video, then publish a trajectory. The trainer claims trajectories, recomputes current-policy probabilities, applies lag correction and a hard mask, updates policy and critic, and checkpoints. Between those processes sits a queue whose contents become less useful as the policy advances. (RL-project design record)

Async was the enabling choice because rollout time varied by minutes across prompts and model families. A synchronous PPO-style batch would hold every producer at the slowest rollout, while the trainer waited for a matched barrier. Async decoupled producer and consumer rates, turning straggler time into queue management with lag caps, version guards, and backpressure.

Every trajectory first landed in a temporary file, then became visible through atomic rename into trajq/pending. The executable task queue was separate from the trajectory queue. Their rules differ. One recovers scheduler work, while the other defines optimization data and transaction boundaries. Mixing them would let an operational retry duplicate a scientific record.

Lag caps and backpressure

Wan capped its backlog at 160 records and refilled below 80. A rollout node released its claim at high water, and the controller added capacity only below low water. The trainer accepted at most eight policy versions of lag, while the DIS hard mask handled records produced during shorter gaps. One calibrated three-worker A100 node produced about 154 records per hour, while one trainer consumed about 96. Supply therefore followed queue deficit, not a symmetric count of producer and trainer nodes. (RL-project rollout-capacity and runbook records)

Heartbeats reported starting, running, or throttled. Throttled was healthy backpressure, not death. Replacement logic required a heartbeat newer than three minutes and a hardware-compatible claim. A stale worker on a one-GPU node could not suppress a required three-GPU replacement merely because the run name matched. (RL-project trajectory-queue record)

The duplicate-stream incident

Per-GPU packing introduced a subtle identity bug. Two workers on different physical GPUs both saw their process-local device as cuda:0. The original worker identity combined host and logical device, so the pair collided and replayed the deterministic prompt stream for seed 6. Three duplicate pairs entered the first 64-record batch. Three additional pending records duplicated records already consumed. (RL-project utilization learning record)

We replaced one overloaded identifier with a three-layer contract. Physical slot identity records where the worker ran. Logical CUDA identity tells the process which visible device to use. Worker stream identity seeds the deterministic prompt sequence and is stored in every trajectory and heartbeat. These values are related but never interchangeable.

Recovery happened before policy movement. The trainer quarantined six duplicate records, retained 90 unique records, and built version one from exactly 64 audited version-zero records. It rejected repeated deterministic trajectory identities before forming later batches. If rejection made a batch short, valid claims returned to pending while the duplicate stayed excluded. The failed allocation's 9,309 GPU-seconds remained in accounting even though no invalid update committed. (RL-project utilization learning record)

Make the update transactional

A trainer update was not “whatever records were available.” It was a validated 64-record transaction. Claim, identity audit, group construction, advantage computation, policy and critic update, checkpoint, and accounting commit belonged to one boundary. A short or invalid pre-update claim rolled back valid records to pending. A failure after policy movement required checkpoint-aware recovery rather than silent replay. (RL-project transactional-batch record)

Completion used the same standard. Wan seed 3204 had a done outer task and exact version-30 state, but its declared terminal version 37 did not exist. The controller restored exactly one resume claim. A done wrapper was operational evidence, not trainer completion. (RL-project utilization learning record)

ABORT is a protocol state

An ABORT marker asked rollout workers to finish the current video and release normally. The trainer finished its current step, saved policy and full state, then exited with code 3. It did not cancel the fleet because unrelated work could still be valid. Run shutdown and allocation shutdown were separate operations. (RL-project runbook)

Cleanup used bounded TERM, a grace-period poll, then KILL for survivors. The earlier unbounded wait on a child that ignored signals held a node at 24 GB and zero percent utilization. Single-instance trainer locks used exit-on-lock rather than warm standby. Stale takeover added a nonce, jitter, and recheck so two contenders could not both pass the same stale observation. (RL-project smoke and trainer-lock records)

10Storage, caches, and video I/O

Our three-filesystem discipline was part of availability. Home had a 23 GB quota and held configuration plus small launchers. Work supplied 1 TB of persistent storage for code, small runtime libraries, and the compact evidence needed to regenerate paper results. Scratch held datasets, weights, checkpoints, queues, logs, videos, telemetry, and caches, with purge risk after roughly ten idle days. (Account storage record)

“Paper-critical” did not mean copying every video to persistent storage. The caching project archived 515 measured-result files totaling 39,376,860 bytes and byte-verified them, while 89 GB of reproducible videos stayed on scratch. The archive contained the per-candidate records and manifests needed to rederive the figures. Stable seeds on a fixed stack could regenerate pixels. (caching-project storage record)

Cache precedence can corrupt a model

Hugging Face cache variables have precedence. One caching-project worker lacked the profile-level cache setting, re-downloaded a Wan-14B shard into the project cache, and corrupted shard 00005. We quarantined the shard and pinned every 14B worker to the known cache. Every job wrapper now declares the effective cache paths explicitly rather than assuming that HF_HOME wins. (caching-project storage record)

Compiler caches need equally explicit scope. Generated Triton sources belong to a node, run, and retry. Safe sequential graph reuse can persist node-locally. Evaluation model caches should be mirrored completely when upstream URLs are brittle, then jobs run offline. A partial on-demand download across many nodes is both a performance risk and a reproducibility risk.

Job wrappers declared HF_HOME, HF_HUB_CACHE, HF_DATASETS_CACHE, TRANSFORMERS_CACHE, TRITON_CACHE_DIR, and TORCHINDUCTOR_CACHE_DIR explicitly, with heavy caches rooted on scratch. Generated compiler paths also included node, run, and retry identity so concurrent jobs never mutated one source tree.

Slow resume is not necessarily a hang

The login node is a control surface

Vista login nodes exposed a measured per-user process ceiling of 100, shared across every session. Before fan-out we ran bin/login_guard.sh; we stopped launching at 40, limited login-touching SSH to three concurrent connections, never used SSH once per task in a loop, and waited at least ten seconds between repeated scheduler polls. Builds, downloads, encoding, model loads, data processing, and long agent sessions moved to compute nodes. (Account login-protection record)

Encoding also changes evaluation. An A/B test encoded the same pixel data for 500 videos with imageio quality 8 and FFmpeg CRF 23. VBench aesthetic quality moved from 0.7803 to 0.7506, about three points, while imaging quality moved 0.005 and subject consistency moved 0.0001. We fixed one encoder across comparison arms and treated codec settings as protocol, not presentation. (distillation-project scorer-calibration record)

Video dtype has similar force. Wan returned float32 frames in the zero-to-one interval. Direct conversion to uint8 produced nearly black output, so the writer scaled and clipped first. Torchcodec on the project stack silently flattened HDR and 10-bit inputs to 8-bit, so readers inspected HDR metadata before decode. A representation bug can look like a model-quality result if the infrastructure does not fail loudly. (caching-project video-I/O record; account video-I/O record)

11Standing on open infrastructure

These projects were possible because the open stack already solved most of the model substrate. We built directly on Hugging Face transformers, diffusers, and accelerate. For language-model RL, TRL, verl, and OpenRLHF were the right established starting points to check first. We did not adopt them for these projects because the video-diffusion policy class and our cluster did not fit their common scalable RL-for-LLM path.

• Language-model RL. TRL, verl, and OpenRLHF are the first tools to evaluate when the policy and infrastructure match language-model post-training.
• Video training and serving. FastVideo, xDiT, VideoSys, and Open-Sora cover complementary inference, parallelism, serving, and end-to-end training needs.
• Video RL. DanceGRPO is the open implementation we looked to for GRPO on visual generation.
• Open weights. Alibaba's Wan, Tencent's HunyuanVideo, and the LTX and CogVideoX releases supplied the model layer that made these projects possible.

Our asynchronous single-rollout design is our own. We wrote thin fleet, spool, and trainer layers only where the use case demanded them, including file-spool fleets on Slurm and deterministic paired evaluation. Check the established projects first. Most teams should not write their own trainer.

12Honest edges and limitations

This reflects one team's experience on one center's systems. There are certainly things we mis-measured or could have done better. The caching project predates the all-host telemetry policy, and the fleet traces here are descriptive records rather than causal estimates. We are still learning, and the playbook keeps evolving as new projects expose new failure modes.

The largest limitation deserves plain words. Even after every repair described here, our utilization is still not the best that the hardware allows, and we do not present it as optimum. The rolling activity we reached is respectable for an academic fleet, yet calibration overhead, queue handoffs, sequential decode phases, and recovery windows all still spend GPU time that a tighter system would not. Everything in this post is work in progress toward infrastructure that sustains the highest throughput and the best resource utilization we can defensibly measure.

It is worth knowing what the bar looks like elsewhere. Most large companies run internal platforms that treat utilization and throughput as launch criteria, and a job that cannot demonstrate a high projected efficiency often is not permitted to start at all. The open ecosystem has excellent pieces, and we cite many of them in the previous section, but nothing yet ties scheduling, calibration, batching, telemetry, evaluation, and provenance together as one end-to-end open stack. The goal behind this playbook is exactly that, a genuinely strong open-source infrastructure for these workloads, and we hope others build on it and past it.

Fleetcraft is very much a work in progress. I am still evolving this infrastructure to make it better, more generalizable, and easier to use, and the repository will keep changing as new projects teach new lessons. If any of this is useful to you, or you see a better way to do a piece of it, feel free to contribute through the repository. Issues, corrections, and pull requests are all welcome.

13Ten rules we carry forward

These projects used different models, clusters, and objectives. The same operational rules survived all three.

Calibrate complete cycles. Include load, denoise, decode, score, backward, optimizer, probe, save, and write. Track memory owners. An allocator cannot free a tensor that a live object still owns. Freeze batch tables. Tune one representative task per family, then stop changing the protocol. Keep timing exclusive. Bulk throughput and reportable batch-one latency are different measurements. Share fleet queues. Spend scheduler slots on a few multi-node project sessions, not individual scientific units.

Preserve node ownership. A per-GPU inner queue must never collide with a full-node trainer. Identify every actor. Separate launch, host, physical slot, logical device, and deterministic stream. Validate completion claims. Check full state, payloads, hashes, and the completed save boundary behind checkpoints and done markers. Join queue telemetry. A utilization percentage without work visibility cannot explain fleet health. Release finished capacity. Scientifically useful occupancy outranks a full dashboard once valid work ends.

14How we ran it

The caching project ran on one-GPU Vista GH200 nodes. A 16-node, 12-hour generation fleet and a separate 64-node VBench sweep each occupied one Slurm job slot; atomic pending, running, done, and failed queues fed one worker per node, with timed generation kept exclusive. Reproducible video bulk stayed on scratch, while 515 compact records and manifests totaling 39,376,860 bytes moved to persistent work storage. Enqueueing, stale-claim recovery, dependent scoring, fixed-stack recommit, and evidence extraction were automated.

The distillation project used the validated six-node, six-rank GH200 topology for full-shard FSDP training. One outer queue held nodes through generation, scoring, and training; when four surplus nodes were available, a separate Slurm step consumed evaluation tasks only after a ready marker without changing the training world size. Heavy checkpoints, videos, logs, and raw telemetry lived on scratch, while hash-bound manifests and the compact evidence crux lived on work storage. Supervisors automated launch adoption, exact-step cleanup, bounded stage retries, all-host telemetry, WandB tracking, and fail-closed release checks.

The RL project spanned the heterogeneous Lonestar6 pool, peaking at 47 A100 and H100 GPUs across 18 nodes, and used four-device GB200 nodes on Vista for larger rollouts. A whole-node outer spool protected trainers, a packed per-GPU inner spool filled independent evaluation slots, and a transactional trajectory queue connected producers to serial trainers. Videos, checkpoints, and rolling telemetry remained on scratch; compact trajectory records, hashes, terminal manifests, and paper-critical summaries were preserved on work storage. Watermarks, heartbeat checks, version-lag guards, duplicate audits, exact resume, and bounded abort behavior were automated.

The companion repository, Fleetcraft, is arranged as a runnable map of that operating model. README.md, AGENTS.md, and LICENSE sit at the root; bin/ holds fleet and spool shell tools; skills/ holds task-routed operational guides; docs/ holds the compendium, case studies, hardware notes, and sources including anonymized telemetry; and infra/python/ and infra/slurm/ hold analysis scripts and job templates. The telemetry figures regenerate from archived inputs with infra/python/plot_blog_telemetry.py, and provenance notes beside empirical claims identify the record behind each number.

Using the repository. This open resource contains the skill files, agent guides, Python utilities, spool and fleet scripts, and Slurm templates. Adapt paths, partitions, and policies to your site, and ALWAYS get permission from your infrastructure provider, university, or cluster operators before running fleet automation, telemetry, or login-node tooling on shared systems.

The prescription for multi-node video generation, training, post-training, and evaluation is to design queueing and calibration before launch, as emphasized by the Ultra-Scale Playbook, then adapt that design to the operating realities in the Vista and Lonestar6 guides. Build on Hugging Face transformers, diffusers, and accelerate; batch each workload to a frozen table while keeping timed measurements exclusive; stream telemetry from every host and judge rolling active fraction, or MFU for dense training; let automation claim, retry, and exit; and require every reported number to resolve to an artifact. High compute utilization is the result of that full recipe, not a setting applied after nodes arrive.

Acknowledgments

These infrastructure learnings grew out of my PhD work at The University of Texas at Austin. Following the official TACC acknowledgment guidance: The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing computational resources that have contributed to the research results reported within this paper. URL: http://www.tacc.utexas.edu. The projects used Vista and Lonestar6, with operational details documented in the Vista user guide and Lonestar6 user guide. Access was provided through the Institute for Foundations of Machine Learning, and Claude helped compile and refine this blog post.

References and further reading

GPU architecture and performance

1. NVIDIA Ampere Architecture In-Depth and NVIDIA A100 specifications.
2. NVIDIA Hopper Architecture In-Depth and NVIDIA H100 specifications.
3. NVIDIA GH200 Grace Hopper Superchip datasheet.
4. NVIDIA GB200 NVL72 specifications and NVIDIA NVLink specifications.
5. NVIDIA Deep Learning Performance Guide: GPU background.
6. NVIDIA Grace Hopper Superchip Architecture In-Depth.
7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
8. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.
9. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision and the PyTorch implementation note.
10. PyTorch scaled dot product attention documentation.
11. PyTorch and vLLM aarch64 ecosystem note and the FlashAttention aarch64 wheel record.
12. PyTorch CUDA memory management semantics.
13. NVIDIA System Management Interface documentation and DCGM feature overview.
14. Lablup, all-smi: a cross-platform command-line monitor for GPU and NPU utilization, memory, temperature, and power across many nodes. Apache License 2.0.
15. Horace He, Making Deep Learning Go Brrrr From First Principles.
16. Aleksa Gordić, Inside NVIDIA GPUs: Anatomy of High Performance Matmul Kernels.
17. Simon Boehm, How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance.
18. Modal GPU Glossary.
19. Chowdhery et al., PaLM: Scaling Language Modeling with Pathways.

Parallelism and training at scale

1. Hugging Face, The Ultra-Scale Playbook: Training LLMs on GPU Clusters.

Open-source infrastructure

1. Hugging Face transformers, diffusers, accelerate, and TRL.
2. verl and OpenRLHF.
3. FastVideo, xDiT, VideoSys, and Open-Sora.
4. DanceGRPO.
5. Wan2.1 and HunyuanVideo.
6. Shreshth Saini, Fleetcraft: an operational playbook for GPU video generation on shared HPC, 2026.

TACC and IFML

1. Texas Advanced Computing Center, Citing TACC.
2. TACC Vista system page and Vista user guide.
3. TACC Lonestar6 system page and Lonestar6 user guide.
4. Institute for Foundations of Machine Learning.

Citation

@misc{saini2026gpuinfraprojects,
  author = {Shreshth Saini},
  title = {Fleetcraft: GPU Infrastructure Notes from Video Generation Projects},
  year = {2026},
  howpublished = {\url{https://shreshthsaini.github.io/blogs/gpu-infra-for-video-gen.html}},
  note = {Blog post}
}