Candidate & Committee Members

Shreshth Saini

Candidate

UT-Austin

Prof. Alan C. Bovik (Advisor)

Chair/Member

UT-Austin

Prof. Joydeep Ghosh

Member — UT-Austin

Prof. Diana Marculescu

Member — UT-Austin

Dr. Yan Ye

Member — Alibaba Group US

Dr. Balu Adsumilli

Member — Google Inc.

Dissertation Roadmap

Review   Beyond8Bits & BrightRate

The largest HDR-UGC video quality dataset & first HDR-aware blind VQA model

44,276HDR-UGC video clips

1.5M+Crowdsourced ratings

BrightRate: Multi-Stream VQA

• HDR features via expansive non-linearity on extreme luminance
• UGC features (CONTRIQUE) + CLIP semantic features
• Temporal difference module for flicker/quality fluctuations
• SROCC 0.889 vs. prior SOTA 0.853

4-Stage Pipeline

Video Sourcing (Vimeo+Users) → Bitrate Ladder Encoding → AMT Study (35 ratings/video) → SUREAL MOS Aggregation

HDR vs SDR gap, Beyond8Bits diversity, and HDR-Q improvements over baselines

Review   LGDM: Latent Guidance for Quality Assessment

Zero-shot NR-IQA by exploiting generative priors — no task-specific training

Core Insight

Pretrained diffusion models encode a perceptual manifold. PMG steers sampling toward perceptually consistent regions — quality assessment without IQA labels.

Key Innovation: Diffusion Hyperfeatures

• Multi-timestep, multi-layer features from U-Net
• Theoretically grounded guidance on tangent spaces
• Works with off-the-shelf SD v1.5 — no fine-tuning

z*_t-1 = z_t-1 + ζ₂ · ∇_z Φ(z_0|t)

PMG: guide denoising toward perceptually consistent manifold regions

Perceptual Manifold Guidance — on-manifold quality-aware sampling

LGDM achieves SOTA median SRCC across authentic distortion benchmarks

Dissertation: Five Novel Contributions

Each project introduces new methods, datasets, or frameworks — together they form a complete pipeline from perception to generation for HDR visual media.

Unifying theme: Generative models as both evaluators and enhancers of visual quality

HDR Is Not "Better SDR" — It's a Different Signal Space

Every modern phone captures 10-bit HDR by default.

YouTube, Instagram, TikTok — billions of daily HDR uploads. HDR10 supports 10-bit depth, BT.2020 wide color gamut, PQ perceptual quantizer. Peak luminance ≥1000 nits vs SDR ~100 nits.

New perceptual phenomena SDR models cannot see:

Highlight clipping · Near-black banding · Color blooming · PQ quantization artifacts · Exposure flicker · Wide-gamut chroma shifts

Core argument:

Standard vision encoders (SigLIP, CLIP) process images in 8-bit sRGB. They are structurally blind to HDR-specific distortions. Not a training gap — a representation gap.

HDR preserves luminance and color detail that SDR collapses

Where Current Methods Break

No existing method combines HDR-aware perception with quality reasoning

Three Fundamental Obstacles

Each requires a dedicated solution — together they define the HDR-Q architecture

Beyond8Bits: The Foundation

First large-scale HDR-UGC quality dataset — the training ground for HDR-Q

44,276HDR-UGC video clips

6,861Unique source videos

1.5M+Human quality ratings

~35Ratings per video

Format: 10-bit HEVC, PQ transfer, BT.2020 · 360p–1080p · 0.2–5 Mbps bitrate ladder

Sources: 2,253 crowdsourced (diverse UGC) + 4,608 Vimeo CC (nature, outdoor, low-light)

Quality control: HDR10 display verification · Qualification quiz · Golden set (SROCC 0.85) · SUREAL MOS aggregation · Inter-subject SROCC 0.90

Dataset diversity, HDR vs SDR characteristics, and HDR-Q performance gains

HDR-Q: Architecture Overview

HDR-Aware Vision Encoder

Contrastive HDR/SDR discrimination

L_ctr = max(0, δ − D(E(x_HDR), E(c)) + D(E(x_SDR), E(c)))

Pull HDR close to caption, push SDR away

Full encoder loss

L_enc = L_Sigmoid + λ_ctr · L_ctr

Semantic alignment + HDR discrimination

Design justifications

Why SigLIP-2? Strong semantic priors, multimodal-compatible.
Why contrastive? HDR info = what's in HDR but absent in SDR.
Why 10-bit PQ input? Tone-mapping destroys the signal we need.
Captions by Qwen2.5-VL-72B for quality-aware descriptions.

SigLIP-2 contrastive finetuning with matched HDR-SDR pairs and quality-aware captions

The Modality Neglect Problem

The most surprising finding — and the core motivation for HAPO

HAPO — Core Mechanism: HDR-SDR Contrastive KL

The HAPO Objective

Training Pipeline & Reward Design

Main Results: Beyond8Bits Benchmark

Method	SROCC↑	PLCC↑	RMSE↓
BRISQUE	0.410	0.469	11.70
CONTRIQUE	0.625	0.605	15.02
CONVIQT	0.799	0.810	8.48
HIDRO-VQA	0.851	0.878	6.09
Q-Insight (best MLLM)	0.517	0.562	20.78
HDR-Q (SDR only)	0.891	0.890	7.42
HDR-Q (Full)	0.921	0.912	5.16

0.921Beyond8Bits

0.908LIVE-HDR
(zero-shot)

0.725SFV+HDR
(zero-shot)

Zero-shot generalization without fine-tuning on target datasets

Qualitative: HDR-Q vs Baseline Reasoning

HDR-Q identifies true HDR artifacts (highlight preservation, banding, color fidelity). Ovis2.5 baseline hallucinates non-existent issues.

Ablation: Every Component Is Justified

Training Dynamics: HAPO vs GRPO

Token entropy during training — GRPO collapses, HAPO maintains healthy exploration

GRPO failure mode

Entropy drops → deterministic → ignores visual modality → text-context shortcuts

HAPO stabilization

Dual entropy maintains H ≈ 0.33 at quality-critical positions. Healthy exploration preserved.

Reasoning efficiency

CoT: 168 → 137 tokens (−18%). More concise, more focused. Boilerplate removed, quality observations retained.

What We Learn & Remaining Challenges

Key Takeaways

What Happens When HDR Becomes SDR

The task lifts display-referred SDR into a wider HDR signal space

Luminance Compression

PQ: 0 to 10,000 cd/m². Gamma: 100 cd/m². SDR clips most of the luminance range.

Color Gamut Compression

The target uses BT.2020, while the SDR input uses BT.709.

What Gets Lost: 10-bit HDR vs 8-bit SDR

Illustrative split views show why clipped SDR cannot uniquely determine the original HDR signal

From SDR Input Space to HDR Output Space

LumaFlux maps BT.709 SDR inputs to display-referred PQ BT.2020 HDR outputs.

SDR input: an 8-bit BT.709 signal may contain clipped highlights and compressed color.

HDR target: a 10-bit PQ BT.2020 signal represents the display-referred output.

Why ITM Is Fundamentally Ill-Posed

Clipping and gamut compression make exact inversion impossible when source detail is absent

LumaFlux in the ITM Landscape

The release evaluates analytical, CNN, LUT, and diffusion alternatives under explicit protocols

Analytical Baselines

BT.2446c inverse and Reinhard inverse are evaluated on the synthetic Luma-Eval track under the same inputs, encoding, and metrics.

CNN Baselines

HDRTVNet++, FMNet, and KUNet are included in Luma-Eval. Published HDRTV1K baseline numbers retain their authors' protocols and are not directly rankable against LumaFlux.

Generative Baselines

LEDiff and X2HDR form the like-for-like generative comparison. LumaFlux uses fewer steps and fewer trainable parameters than both.

LumaFlux (Ours)

Frozen FLUX.1-dev MM-DiT with PGA, PCM, an HDR Residual Coupler, and an RQS tone-field decoder. 71.3M trainable parameters, 0.57% of the backbone.

CNN comparison on the Luma-Eval protocol

Generative comparison on the native track

Like-for-Like Generative Comparison

Luma-Eval: Controlled Comparison Protocol

Every method receives identical inputs, output encoding, and metric implementations

Background   Flow Matching in 4 Equations

The generation framework behind Flux and LumaFlux

Color: data x₀ · noise x₁ · interpolant xₜ · time t · network vθ

Background   Rectified Flow: Straight Paths vs. Noisy Diffusion

LumaFlux uses a short rectified-flow bridge from encoded SDR to HDR

From Liu et al. (ICLR '23): (a) Linear interpolation has crossing paths. (b) Rectified flow straightens them via reflow. (c-d) After rectification, paths are straight ≈ optimal transport. Figure from official Rectified Flow repo.

Background   Flux MM-DiT & Why Standard LoRA Isn't Enough

Architecture Evolution: CNN → MMDiT → Luma-MMDiT

Left: CNN stack (local receptive field). Center: Standard Flux MMDiT (global attention but no physical priors). Right: LumaFlux (PGA + PCM + Coupler + RQS).

LumaFlux: Full Pipeline

Standard DiT Block → Luma-MMDiT Block

Targeted adaptation with 71.3M trainable parameters, 0.57% of the frozen backbone

Left: Standard MMDiT. Right: Luma-MMDiT with PGA on value projection, PCM after norm, Coupler at output.

// Standard MMDiT block

z → LN(z)

Q = zW_Q, K = zW_K

V = zW_V

h = Attn(Q,K,V)

z = z + h + MLP(LN(z))

// + Luma-MMDiT:

V = z(W_V⁰ + R_v^t,ℓ) ← PGA

h = γ^t,ℓ⊙LN(h) + ζ^t,ℓ ← PCM

z += λ(W_pT_phys + W_cC(T_perc)) ← Coupler

HDR Residual Coupler

Fuses physical and perceptual paths under timestep-and-layer modulation.

PGA: Physically-Guided Adaptation, Step by Step

PCM (Perceptual) & RQS (Decoder)

Training: Data & Configuration

Results: Best on Every Luma-Eval Metric

Visual Result 1: Church Interior SDR/HDR Split View

SDR Input (left half)

The model receives an 8-bit BT.709 SDR frame.

LumaFlux HDR Output (right half)

The model produces display-referred PQ BT.2020 HDR from the SDR input without a text prompt.

Why this matters

The frozen VAE is not an HDR codec. The tone head cannot invent detail absent from heavily clipped SDR.

Visual Result 2: Shanghai Cityscape SDR/HDR Split View

SDR input (left) vs LumaFlux HDR output (right), with zoomed crop below

SDR input (left half)

The model receives an 8-bit BT.709 SDR frame.

LumaFlux output (right half)

The output is a display-referred PQ BT.2020 reconstruction from the 8-bit BT.709 input.

How LumaFlux handles this

PGA supplies physical and spectral cues. PCM supplies frozen SigLIP conditioning. RQS calibrates the output into HDR luminance.

Ablation: The Learned Output Mapping Matters

Separate ablation protocol; these values are not the headline Luma-Eval scores

Video: Training-Free Temporal Stabilization

The same image model is applied to video with shared bridge noise and an EMA on RQS parameters

What We Learn & Remaining Challenges

Key Takeaways

Why Do We Need Guidance At All?

Without guidance, flow models generate plausible but generic images that loosely match the prompt

The problem with unconditional generation

Models trained with conditioning dropout learn both p(x|y) and p(x). Without guidance, outputs come from a mixture of conditional and unconditional — resulting in low prompt adherence, muted details, and bland compositions.

What guidance should do

Amplify the conditional signal — push generation toward what the prompt describes. Sharper details, more vivid colors, better text-image alignment. This is what CFG achieves in diffusion models (DDPM, SDXL).

What actually happens on flow models

CFG was designed for stochastic SDE samplers. Flow models use deterministic ODE. The extrapolation that works in SDEs becomes catastrophic in ODEs — artifacts, color blow-out, structural distortion.

Flux-dev: Same prompt, same seed

No guidance

CFG ω=3.5

Ours λ=3.5

No guidance: dark, muddy, poor prompt adherence. CFG: oversaturated, unnatural. Ours: clean, natural, faithful.

No guidance

CFG ω=3.5

Ours λ=3

SD3.5: Same pattern — CFG oversaturates, ours stays natural.

CFG Artifacts on Flow Models: A Gallery of Failures

All images generated with standard CFG on flow models. These are not cherry-picked — this is what CFG does to every prompt.

Flux-dev with CFG ω=3 (from paper Figure 6)

Plastic / cartoon

Oversaturated

Flat / clipart

Blown highlights

Washed out

SD3 with CFG ω=3–3.5

Oversaturated

Text garbled

Misspelled

Color blowout

Color distortion

Root Cause: Why CFG Breaks on Flows but Works on Diffusion

The same extrapolation trick, but fundamentally different samplers

Obviously broken: CFG on SD3 & Flux

SD3: "CyberCore Cafe"

SD3: "Entropy and the Soul"

Flux: plastic/cartoon

Garbled text ("Cyberre Cidie"), misspellings ("Entφy amd"), plastic cartoonification — all from standard CFG.

SDE Sampler (Diffusion) vs ODE Sampler (Flows)

Diffusion SDEs: built-in safety net

Each step adds Gaussian noise — the renoising step. This noise accidentally provides error correction: even when CFG pushes off-manifold, the stochastic noise partially pulls the sample back toward the learned distribution.

dx = [f(x,t) − g(t)²·∇log p_t(x)]dt + g(t)·dw̄   ← stochastic noise term

Discrete (DDPM): x_t-1 = μ_θ(x_t, t) + σ_t·z,   z ~ N(0, I). The σ_t·z term adds fresh noise at every step — pulling samples back toward the learned distribution.

Flow ODEs: no correction mechanism

Rectified flows use a purely deterministic ODE. Once CFG pushes the trajectory off the manifold at step t, there is no mechanism to return. The off-manifold point becomes the starting point for step t+1.

dx = v_θ(x, t)·dt          ← purely deterministic, no noise term!

Compounding error

At each step, CFG evaluates guidance at an off-manifold point — making the velocity direction even more wrong. Errors accumulate monotonically across all N sampling steps. The more steps, the worse it gets.

This is why we need a fundamentally different approach — not a fix for CFG, but a new guidance paradigm designed specifically for deterministic ODE samplers.

What Would an Ideal Guidance Method Look Like?

Desiderata for guidance on rectified flow models

The Geometric Insight: Why CFG Fails and How We Fix It

Three panels: (Left) Ideal flow on manifold. (Center) CFG extrapolates off-manifold. (Right) Rect-CFG++ stays on-manifold via predictor-corrector. Let's walk through each.

Geometric Insight (1/3): The Ideal Flow on the Manifold

What you're seeing

Three manifolds: M_t (current time, blue/gold band), M_t-1 (next step), M₀ (data, orange curve at bottom). The red dots are the ideal positions at each timestep.

The ideal trajectory

The ideal sample flows along the manifold surface from M_t → M_t-1 → M₀. At each step, the velocity lies in the tangent space of the manifold — never leaving it.

Three flows shown

Blue = Flux conditional (v^c), Red = CFG (extrapolated), Green = Rectified-CFG++ (ours). Notice how green and blue stay near the manifold while red diverges.

Key question

How do we guide the generation (improve text adherence) while staying on the manifold? Let's see what goes wrong first...

Geometric Insight (2/3): CFG Extrapolates Off-Manifold

What CFG does

Starting from the black dot on M_t, CFG computes the guided velocity v̂_ω at the current point. With ω>1, this velocity extrapolates past the conditional velocity v^c.

v̂_ω = v^u + ω(v^c − v^u)    ω > 1 → overshoots!

The tangent space violation

The guided velocity v̂_ω does not belong to the tangent space T_xtM_t. The resulting Euler step lands the sample off M_t-1. In a deterministic ODE, there is no mechanism to return.

The compounding effect

At the next step, CFG evaluates guidance at the off-manifold point — making the direction even worse. Errors accumulate at every step. By the final image: oversaturation, color blow-out, structural distortion.

Geometric Insight (3/3): Our Predictor-Corrector Stays On-Manifold

Step 1: Predict midpoint with v^c only

Take a half-step using only the conditional velocity: x̃_mid = x_t + (Δt/2)·v^c_t. Since v^c lies in the tangent space, x̃_mid lands on or near M_t-Δt/2.

Step 2: Evaluate guidance at the predicted midpoint

Compute v^c_mid and v^u_mid at x̃_mid. The guidance direction Δv_mid = v^c_mid − v^u_mid is evaluated on the manifold, reflecting the geometry of the target manifold M_t-Δt.

Step 3: Apply bounded interpolative correction

v̂ = v^c_t + α(t)(v^c_mid − v^u_mid)

α(t) = λ_max(1-t)^γ. Base velocity is always v^c (on manifold). Correction is additive and bounded. α(t)→0 near data → fine details preserved.

Result: trajectory stays in bounded tubular neighbourhood of M_t

The Core Distinction: Extrapolation vs. Interpolation

This single difference explains why CFG fails and Rect-CFG++ works

Algorithm: CFG vs. Rectified-CFG++ (Side-by-Side)

Color highlights show exactly what changes — green = our additions

Guidance Methods Compared: Equations & Properties

Four approaches to the same problem — only ours avoids extrapolation entirely

Theoretical Guarantees — First Provable Bounds for Flow Guidance

Four mild assumptions → three rigorous guarantees. No prior guidance method has these.

Why These Guarantees Matter: CFG vs. Ours

Intermediate Denoising: Watching Artifacts Grow vs. Stay Clean

Each strip shows 7 denoising steps from noise (left) to final image (right)

Standard CFG (ω=2.5) — artifacts compound

Rectified-CFG++ (λ=0.5) — clean throughout

2D Toy: CFG Drifts Off-Support, Ours Stays On-Manifold

200 trajectories on 2D mixed Gaussian. Top: CFG (blue) drifts off support. Bottom: Ours (green) smooth on-manifold.

CFG (top)

Trajectories initially overshoot, leaving the learned transport manifold. Sharp late-stage corrections pull samples back — but damage is done. Final distribution is noisy and off-center.

Rectified-CFG++ (bottom)

Smooth convergence throughout. Predictor anchors each step on the flow field. Corrector applies bounded guidance. Samples arrive at target with tight concentration — no drift, no sharp corrections.

Flux: Baseline vs. CFG vs. Rectified-CFG++ (3 Prompts)

Top: Flux baseline (no guidance). Middle: CFG — oversaturated, cartoonish, structural distortion. Bottom: Rect-CFG++ — sharp details, faithful colors, correct structure.

Flux: Baseline vs. CFG vs. Rectified-CFG++ (Same Prompt, Same Seed)

Same model (Flux-dev), same prompt, same seed — three guidance strategies. Left: no guidance (baseline). Center: CFG (ω=3.5). Right: Rectified-CFG++ (λ=0.5). Ours: sharper details, faithful colors, no rainbow artifacts.

Diverse Prompt Gallery: Rectified-CFG++ Handles Everything

All images generated with Rectified-CFG++ on Flux-dev. No cherry-picking — random prompts from MS-COCO and Pick-a-Pic.

SD3.5: 4-Way Guidance Comparison (Same Prompt, Same Seed)

CFG ω=3.5

CFG-Zero*

APG

Ours λ=6

CFG

CFG

Ours

Ours

Flux-dev: CFG vs. Ours (8 Comparisons)

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

More Visual Comparisons: SD3 & Lumina-Next

SD3 CFG

SD3 Ours

SD3 CFG

SD3 Ours

Lumina CFG

Lumina Ours

Lumina CFG

Lumina Ours

Drop-in replacement across all rectified flow models — no retraining, no arch changes.

Text Rendering: The Stop Sign Test

Text-heavy prompts are especially sensitive to off-manifold drift at late denoising steps

CFG

Rectified-CFG++

Prompt: "A stop sign with 'ALL WAY' written below it."

Text Rendering: Neon Street Sign

CFG

Rectified-CFG++

Prompt: "A neon street sign that says 'CyberCore Cafe', glowing in magenta and blue."

Text Rendering: Headlines & Inscriptions

CFG

Ours

CFG

Ours

Text Rendering: Presentations & Artistic Text

CFG

Ours

CFG

Ours

Text Rendering: Carved & Printed Text

CFG

Ours

CFG

Ours

Text Legibility: Full Gallery (SD3.5)

Each column: CFG left → Rectified-CFG++ right

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

Consistent text legibility across all prompts — the adaptive schedule α(t) is the key mechanism.

Quantitative: MS-COCO 10K — Consistent Improvements

User Study: 30 Experts × 32 Prompts × 4 Models = 15,360 Responses

4-way forced choice: CFG (dark blue), APG (medium blue), CFG-Zero* (light blue), Rectified-CFG++ (green). 4 criteria × 4 models. Green bar is tallest in every single comparison.

Ablation 1: Guidance Scale — CFG Crashes, Ours Stays Stable

X-axis: guidance scale (λ for ours | ω for CFG). Y-axis: FID↓, CLIP↑, ImageReward↑, Aesthetic↑. Blue = CFG, Green = Ours.

CFG (blue curves)

FID explodes: 77 → 148 at ω=10. ImageReward collapses: 1.0 → -1.5. CLIP drops 30%. Aesthetic drops 35%. Catastrophic failure at high guidance.

Rectified-CFG++ (green curves)

FID: 75 → 75 (flat!). ImageReward: 1.0 → 0.95. CLIP stable. Aesthetic stable. Graceful degradation — never crashes.

Why this matters

CFG requires careful per-model tuning of ω to avoid collapse. Rect-CFG++ is robust across a 50× range of λ — no hyperparameter sensitivity. One setting works everywhere.

Ablation 2: Sampling Steps — Better Quality with Fewer Steps

X-axis: sampling steps. Blue = CFG, Green = Ours. Our method converges faster across all metrics.

Ours at 5 steps vs CFG at 5 steps

FID: 71 vs 178 (2.5× better). ImageReward: 1.0 vs -1.5. At ultra-low NFE, CFG produces garbage; ours produces usable images.

Ours at 15 steps ≈ CFG at 28 steps

Same FID, same CLIP, same ImageReward — but ~2× faster inference. This is because our on-manifold trajectories converge more efficiently.

Component ablation (SD3.5)

Config	FID	CLIP	HPSv2
Unconditional only	91.12	0.144	0.187
w/o Predictor	73.70	0.341	0.297
w/o Corrector	74.65	0.341	0.298
Full	72.97	0.345	0.300

Both predictor and corrector essential.

Key Takeaways

Summary of Contributions

Connecting the Contributions

Understanding perception enables guiding generation, which enhances perception.