Candidate & Committee Members

Shreshth Saini

Candidate

UT-Austin

Prof. Alan C. Bovik (Advisor)

Chair/Member

UT-Austin

Prof. Joydeep Ghosh

Member — UT-Austin

Prof. Diana Marculescu

Member — UT-Austin

Dr. Yan Ye

Member — Alibaba Group US

Dr. Balu Adsumilli

Member — Google Inc.

Dissertation Roadmap

Review   Beyond8Bits & BrightRate

The largest HDR-UGC video quality dataset & first HDR-aware blind VQA model

44,276HDR-UGC video clips

1.5M+Crowdsourced ratings

BrightRate: Multi-Stream VQA

• HDR features via expansive non-linearity on extreme luminance
• UGC features (CONTRIQUE) + CLIP semantic features
• Temporal difference module for flicker/quality fluctuations
• SROCC 0.889 vs. prior SOTA 0.853

4-Stage Pipeline

Video Sourcing (Vimeo+Users) → Bitrate Ladder Encoding → AMT Study (35 ratings/video) → SUREAL MOS Aggregation

HDR vs SDR gap, Beyond8Bits diversity, and HDR-Q improvements over baselines

Review   LGDM: Latent Guidance for Quality Assessment

Zero-shot NR-IQA by exploiting generative priors — no task-specific training

Core Insight

Pretrained diffusion models encode a perceptual manifold. PMG steers sampling toward perceptually consistent regions — quality assessment without IQA labels.

Key Innovation: Diffusion Hyperfeatures

• Multi-timestep, multi-layer features from U-Net
• Theoretically grounded guidance on tangent spaces
• Works with off-the-shelf SD v1.5 — no fine-tuning

z*_t-1 = z_t-1 + ζ₂ · ∇_z Φ(z_0|t)

PMG: guide denoising toward perceptually consistent manifold regions

Perceptual Manifold Guidance — on-manifold quality-aware sampling

LGDM achieves SOTA median SRCC across authentic distortion benchmarks

Dissertation: Five Novel Contributions

Each project introduces new methods, datasets, or frameworks — together they form a complete pipeline from perception to generation for HDR visual media.

Unifying theme: Generative models as both evaluators and enhancers of visual quality

HDR Is Not "Better SDR" — It's a Different Signal Space

Every modern phone captures 10-bit HDR by default.

YouTube, Instagram, TikTok — billions of daily HDR uploads. HDR10 supports 10-bit depth, BT.2020 wide color gamut, PQ perceptual quantizer. Peak luminance ≥1000 nits vs SDR ~100 nits.

New perceptual phenomena SDR models cannot see:

Highlight clipping · Near-black banding · Color blooming · PQ quantization artifacts · Exposure flicker · Wide-gamut chroma shifts

Core argument:

Standard vision encoders (SigLIP, CLIP) process images in 8-bit sRGB. They are structurally blind to HDR-specific distortions. Not a training gap — a representation gap.

HDR preserves luminance and color detail that SDR collapses

Where Current Methods Break

No existing method combines HDR-aware perception with quality reasoning

Three Fundamental Obstacles

Each requires a dedicated solution — together they define the HDR-Q architecture

Beyond8Bits: The Foundation

First large-scale HDR-UGC quality dataset — the training ground for HDR-Q

44,276HDR-UGC video clips

6,861Unique source videos

1.5M+Human quality ratings

~35Ratings per video

Format: 10-bit HEVC, PQ transfer, BT.2020 · 360p–1080p · 0.2–5 Mbps bitrate ladder

Sources: 2,253 crowdsourced (diverse UGC) + 4,608 Vimeo CC (nature, outdoor, low-light)

Quality control: HDR10 display verification · Qualification quiz · Golden set (SROCC 0.85) · SUREAL MOS aggregation · Inter-subject SROCC 0.90

Dataset diversity, HDR vs SDR characteristics, and HDR-Q performance gains

HDR-Q: Architecture Overview

HDR-Aware Vision Encoder

Contrastive HDR/SDR discrimination

L_ctr = max(0, δ − D(E(x_HDR), E(c)) + D(E(x_SDR), E(c)))

Pull HDR close to caption, push SDR away

Full encoder loss

L_enc = L_Sigmoid + λ_ctr · L_ctr

Semantic alignment + HDR discrimination

Design justifications

Why SigLIP-2? Strong semantic priors, multimodal-compatible.
Why contrastive? HDR info = what's in HDR but absent in SDR.
Why 10-bit PQ input? Tone-mapping destroys the signal we need.
Captions by Qwen2.5-VL-72B for quality-aware descriptions.

SigLIP-2 contrastive finetuning with matched HDR-SDR pairs and quality-aware captions

The Modality Neglect Problem

The most surprising finding — and the core motivation for HAPO

HAPO — Core Mechanism: HDR-SDR Contrastive KL

The HAPO Objective

Training Pipeline & Reward Design

Main Results: Beyond8Bits Benchmark

Method	SROCC↑	PLCC↑	RMSE↓
BRISQUE	0.410	0.469	11.70
CONTRIQUE	0.625	0.605	15.02
CONVIQT	0.799	0.810	8.48
HIDRO-VQA	0.851	0.878	6.09
Q-Insight (best MLLM)	0.517	0.562	20.78
HDR-Q (SDR only)	0.891	0.890	7.42
HDR-Q (Full)	0.921	0.912	5.16

0.921Beyond8Bits

0.908LIVE-HDR
(zero-shot)

0.725SFV+HDR
(zero-shot)

Zero-shot generalization without fine-tuning on target datasets

Qualitative: HDR-Q vs Baseline Reasoning

HDR-Q identifies true HDR artifacts (highlight preservation, banding, color fidelity). Ovis2.5 baseline hallucinates non-existent issues.

Ablation: Every Component Is Justified

Training Dynamics: HAPO vs GRPO

Token entropy during training — GRPO collapses, HAPO maintains healthy exploration

GRPO failure mode

Entropy drops → deterministic → ignores visual modality → text-context shortcuts

HAPO stabilization

Dual entropy maintains H ≈ 0.33 at quality-critical positions. Healthy exploration preserved.

Reasoning efficiency

CoT: 168 → 137 tokens (−18%). More concise, more focused. Boilerplate removed, quality observations retained.

What We Learn & Remaining Challenges

Key Takeaways

What Happens When HDR Becomes SDR

Every step is lossy — inverting it is ill-posed

Luminance Compression

PQ: 0–10,000 cd/m². Gamma: 100 cd/m². SDR clips 99% of luminance.

Color Gamut Compression

BT.2020: 75.8% of CIE. BT.709: 35.9%. Over half the color volume discarded.

What Gets Lost: 10-bit HDR vs 8-bit SDR (Beyond8Bits Dataset)

Split-view from our Beyond8Bits dataset — HDR (upper-left) vs SDR (lower-right) of the same frame

Real HDR Videos in Yxy Space: What SDR Clips

Computed from real 10-bit PQ BT.2020 videos (Beyond8Bits). Red prism = Rec.709 SDR volume (≤100 nits). Orange dots = pixels outside SDR — lost in tone mapping.

Sunset scene: 15% pixels above SDR ceiling. Sun flare + specular highlights on grass reach ~1,300 nits.

Indoor/night scene: 27% pixels above SDR ceiling. Bright artificial lights reach ~2,600 nits — massive clipping.

Why ITM Is Fundamentally Ill-Posed

Three irreversible losses make exact inversion impossible — any method must hallucinate missing information

Prior ITM Methods: Three Generations of Architectures

From hand-crafted curves to CNNs to diffusion — each generation has fundamental limits

Gen 1: Classical Tone Curves

BT.2446, Reinhard, Huo et al. — Fixed parametric mapping from SDR→HDR using inverse tone curves. Content-blind: the same curve is applied to sunsets, interiors, and neon signs. Cannot recover scene-specific highlights or local contrast.

Gen 2: CNN-Based Learning

HDRTVNet++ (ICCV'21): 3-branch modulation net (global, local, condition). HDCFM (MM'22): Hierarchical feature modulation with dynamic context. HDRTVDM (CVPR'23): Dynamic context transformation. Deep SR-ITM (ICCV'19): Joint super-resolution + ITM.

Limitation: Local receptive fields → cannot model global illumination. Overfit to specific TMOs used in training data.

Gen 3: Diffusion-Based

LEDiff: Latent diffusion conditioned on SDR. PromptIR: Text-guided restoration. FlashVSR: Video super-resolution with diffusion prior.

Limitation: Hue shifts, over-saturated tones, hallucinated details. No physical grounding — the model has no concept of luminance or color space.

Gen 4: LumaFlux (Ours)

Physically & perceptually guided Diffusion Transformer. Frozen 12B Flux backbone + lightweight PGA/PCM modules. First to combine physical priors + perceptual guidance + DiT scale.

HDRTVNet++ — CNN, local ops only

LEDiff — latent fusion, no physical priors

Where Prior ITM Methods Break

Prior ITM Methods: Visual Failures

Visual comparison on real HDR dataset — from FastHDRNet (arXiv 2404.04483)

Desert sunset (top) + evening cityscape (bottom). Ground Truth vs 7 methods: Ada-3DLUT, HDRTVNet, JSI-GAN, CSRNet, HDRNet, HuoPhyEO, FastHDRNet.

Background   Flow Matching in 4 Equations

The generation framework behind Flux and LumaFlux

Color: data x₀ · noise x₁ · interpolant xₜ · time t · network vθ

Background   Rectified Flow: Straight Paths vs. Noisy Diffusion

Why LumaFlux uses rectified flow (Flux) instead of standard diffusion

From Liu et al. (ICLR '23): (a) Linear interpolation has crossing paths. (b) Rectified flow straightens them via reflow. (c-d) After rectification, paths are straight ≈ optimal transport. Figure from official Rectified Flow repo.

Background   Flux MM-DiT & Why Standard LoRA Isn't Enough

Architecture Evolution: CNN → MMDiT → Luma-MMDiT

Left: CNN stack (local receptive field). Center: Standard Flux MMDiT (global attention but no physical priors). Right: LumaFlux (PGA + PCM + Coupler + RQS).

LumaFlux: Full Pipeline

Standard DiT Block → Luma-MMDiT Block

Three targeted modifications — only colored parts are trainable (~17M / 12B)

Left: Standard MMDiT. Right: Luma-MMDiT with PGA on value projection, PCM after norm, Coupler at output.

// Standard MMDiT block

z → LN(z)

Q = zW_Q, K = zW_K

V = zW_V

h = Attn(Q,K,V)

z = z + h + MLP(LN(z))

// + Luma-MMDiT:

V = z(W_V⁰ + R_v^t,ℓ) ← PGA

h = γ^t,ℓ⊙LN(h) + ζ^t,ℓ ← PCM

z += λ(W_pT_phys + W_cC(T_perc)) ← Coupler

TLAM: Ψ(t, ℓ) → 6 scalars

[α_pga, β_pga, α_pcm, β_pcm, n_spec, λ]
Early steps → global tone. Late steps → highlight detail.

PGA: Physically-Guided Adaptation — Step by Step

PCM (Perceptual) & RQS (Decoder)

Training: Data & Configuration

Results: State-of-the-Art on All Benchmarks

Visual Result 1: Church Interior — Highlight Recovery

SDR Input (left half)

Stained-glass window clips to flat white. Gold ceiling mosaics lose all texture. 8-bit gamma collapses the entire highlight range.

LumaFlux HDR Output (right half)

Window highlights show gradual luminance roll-off. Gold leaf textures fully resolved. Zoomed inset reveals mosaic detail invisible in SDR.

Why this matters

~3 stops of highlight info destroyed by SDR clipping. PGA selectively expands highlights while PCM preserves warm color temperature of the gold mosaic.

Visual Result 2: Shanghai Cityscape — Shadow & Neon Recovery

SDR input (left) vs LumaFlux HDR output (right) — split view with zoomed crop below

SDR input (left half)

Pudong waterfront is near-black. Neon signs washed out. Zoomed crop shows boat and tower as indistinguishable dark blobs.

LumaFlux output (right half)

Neon signs vivid with accurate colors. Water reflections reveal full skyline. Zoomed crop shows sharp boat details, tower structure, and signage.

How LumaFlux handles this

PGA spectral gating amplifies shadow signal without artifacts. RQS decoder expands shadows/highlights via per-pixel monotone splines. PCM maps neon colors to correct BT.2020 gamut.

Ablation: Every Component Is Justified

Progressive addition of components on Luma-Eval benchmark (100K iterations)

User Study: Expert Perceptual Evaluation

10 experts, 10 HDR clips, 5-point MOS on calibrated HDR display (LG OLED, 1000 nits)

What We Learn & Remaining Challenges

Key Takeaways

Why Do We Need Guidance At All?

Without guidance, flow models generate plausible but generic images that loosely match the prompt

The problem with unconditional generation

Models trained with conditioning dropout learn both p(x|y) and p(x). Without guidance, outputs come from a mixture of conditional and unconditional — resulting in low prompt adherence, muted details, and bland compositions.

What guidance should do

Amplify the conditional signal — push generation toward what the prompt describes. Sharper details, more vivid colors, better text-image alignment. This is what CFG achieves in diffusion models (DDPM, SDXL).

What actually happens on flow models

CFG was designed for stochastic SDE samplers. Flow models use deterministic ODE. The extrapolation that works in SDEs becomes catastrophic in ODEs — artifacts, color blow-out, structural distortion.

Flux-dev: Same prompt, same seed

No guidance

CFG ω=3.5

Ours λ=3.5

No guidance: dark, muddy, poor prompt adherence. CFG: oversaturated, unnatural. Ours: clean, natural, faithful.

No guidance

CFG ω=3.5

Ours λ=3

SD3.5: Same pattern — CFG oversaturates, ours stays natural.

CFG Artifacts on Flow Models: A Gallery of Failures

All images generated with standard CFG on flow models. These are not cherry-picked — this is what CFG does to every prompt.

Flux-dev with CFG ω=3 (from paper Figure 6)

Plastic / cartoon

Oversaturated

Flat / clipart

Blown highlights

Washed out

SD3 with CFG ω=3–3.5

Oversaturated

Text garbled

Misspelled

Color blowout

Color distortion

Root Cause: Why CFG Breaks on Flows but Works on Diffusion

The same extrapolation trick, but fundamentally different samplers

Obviously broken: CFG on SD3 & Flux

SD3: "CyberCore Cafe"

SD3: "Entropy and the Soul"

Flux: plastic/cartoon

Garbled text ("Cyberre Cidie"), misspellings ("Entφy amd"), plastic cartoonification — all from standard CFG.

SDE Sampler (Diffusion) vs ODE Sampler (Flows)

Diffusion SDEs: built-in safety net

Each step adds Gaussian noise — the renoising step. This noise accidentally provides error correction: even when CFG pushes off-manifold, the stochastic noise partially pulls the sample back toward the learned distribution.

dx = [f(x,t) − g(t)²·∇log p_t(x)]dt + g(t)·dw̄   ← stochastic noise term

Discrete (DDPM): x_t-1 = μ_θ(x_t, t) + σ_t·z,   z ~ N(0, I). The σ_t·z term adds fresh noise at every step — pulling samples back toward the learned distribution.

Flow ODEs: no correction mechanism

Rectified flows use a purely deterministic ODE. Once CFG pushes the trajectory off the manifold at step t, there is no mechanism to return. The off-manifold point becomes the starting point for step t+1.

dx = v_θ(x, t)·dt          ← purely deterministic, no noise term!

Compounding error

At each step, CFG evaluates guidance at an off-manifold point — making the velocity direction even more wrong. Errors accumulate monotonically across all N sampling steps. The more steps, the worse it gets.

This is why we need a fundamentally different approach — not a fix for CFG, but a new guidance paradigm designed specifically for deterministic ODE samplers.

What Would an Ideal Guidance Method Look Like?

Desiderata for guidance on rectified flow models

The Geometric Insight: Why CFG Fails and How We Fix It

Three panels: (Left) Ideal flow on manifold. (Center) CFG extrapolates off-manifold. (Right) Rect-CFG++ stays on-manifold via predictor-corrector. Let's walk through each.

Geometric Insight (1/3): The Ideal Flow on the Manifold

What you're seeing

Three manifolds: M_t (current time, blue/gold band), M_t-1 (next step), M₀ (data, orange curve at bottom). The red dots are the ideal positions at each timestep.

The ideal trajectory

The ideal sample flows along the manifold surface from M_t → M_t-1 → M₀. At each step, the velocity lies in the tangent space of the manifold — never leaving it.

Three flows shown

Blue = Flux conditional (v^c), Red = CFG (extrapolated), Green = Rectified-CFG++ (ours). Notice how green and blue stay near the manifold while red diverges.

Key question

How do we guide the generation (improve text adherence) while staying on the manifold? Let's see what goes wrong first...

Geometric Insight (2/3): CFG Extrapolates Off-Manifold

What CFG does

Starting from the black dot on M_t, CFG computes the guided velocity v̂_ω at the current point. With ω>1, this velocity extrapolates past the conditional velocity v^c.

v̂_ω = v^u + ω(v^c − v^u)    ω > 1 → overshoots!

The tangent space violation

The guided velocity v̂_ω does not belong to the tangent space T_xtM_t. The resulting Euler step lands the sample off M_t-1. In a deterministic ODE, there is no mechanism to return.

The compounding effect

At the next step, CFG evaluates guidance at the off-manifold point — making the direction even worse. Errors accumulate at every step. By the final image: oversaturation, color blow-out, structural distortion.

Geometric Insight (3/3): Our Predictor-Corrector Stays On-Manifold

Step 1: Predict midpoint with v^c only

Take a half-step using only the conditional velocity: x̃_mid = x_t + (Δt/2)·v^c_t. Since v^c lies in the tangent space, x̃_mid lands on or near M_t-Δt/2.

Step 2: Evaluate guidance at the predicted midpoint

Compute v^c_mid and v^u_mid at x̃_mid. The guidance direction Δv_mid = v^c_mid − v^u_mid is evaluated on the manifold, reflecting the geometry of the target manifold M_t-Δt.

Step 3: Apply bounded interpolative correction

v̂ = v^c_t + α(t)(v^c_mid − v^u_mid)

α(t) = λ_max(1-t)^γ. Base velocity is always v^c (on manifold). Correction is additive and bounded. α(t)→0 near data → fine details preserved.

Result: trajectory stays in bounded tubular neighbourhood of M_t

The Core Distinction: Extrapolation vs. Interpolation

This single difference explains why CFG fails and Rect-CFG++ works

Algorithm: CFG vs. Rectified-CFG++ (Side-by-Side)

Color highlights show exactly what changes — green = our additions

Guidance Methods Compared: Equations & Properties

Four approaches to the same problem — only ours avoids extrapolation entirely

Theoretical Guarantees — First Provable Bounds for Flow Guidance

Four mild assumptions → three rigorous guarantees. No prior guidance method has these.

Why These Guarantees Matter: CFG vs. Ours

Intermediate Denoising: Watching Artifacts Grow vs. Stay Clean

Each strip shows 7 denoising steps from noise (left) to final image (right)

Standard CFG (ω=2.5) — artifacts compound

Rectified-CFG++ (λ=0.5) — clean throughout

2D Toy: CFG Drifts Off-Support, Ours Stays On-Manifold

200 trajectories on 2D mixed Gaussian. Top: CFG (blue) drifts off support. Bottom: Ours (green) smooth on-manifold.

CFG (top)

Trajectories initially overshoot, leaving the learned transport manifold. Sharp late-stage corrections pull samples back — but damage is done. Final distribution is noisy and off-center.

Rectified-CFG++ (bottom)

Smooth convergence throughout. Predictor anchors each step on the flow field. Corrector applies bounded guidance. Samples arrive at target with tight concentration — no drift, no sharp corrections.

Flux: Baseline vs. CFG vs. Rectified-CFG++ (3 Prompts)

Top: Flux baseline (no guidance). Middle: CFG — oversaturated, cartoonish, structural distortion. Bottom: Rect-CFG++ — sharp details, faithful colors, correct structure.

Flux: Baseline vs. CFG vs. Rectified-CFG++ (Same Prompt, Same Seed)

Same model (Flux-dev), same prompt, same seed — three guidance strategies. Left: no guidance (baseline). Center: CFG (ω=3.5). Right: Rectified-CFG++ (λ=0.5). Ours: sharper details, faithful colors, no rainbow artifacts.

Diverse Prompt Gallery: Rectified-CFG++ Handles Everything

All images generated with Rectified-CFG++ on Flux-dev. No cherry-picking — random prompts from MS-COCO and Pick-a-Pic.

SD3.5: 4-Way Guidance Comparison (Same Prompt, Same Seed)

CFG ω=3.5

CFG-Zero*

APG

Ours λ=6

CFG

CFG

Ours

Ours

Flux-dev: CFG vs. Ours (8 Comparisons)

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

More Visual Comparisons: SD3 & Lumina-Next

SD3 CFG

SD3 Ours

SD3 CFG

SD3 Ours

Lumina CFG

Lumina Ours

Lumina CFG

Lumina Ours

Drop-in replacement across all rectified flow models — no retraining, no arch changes.

Text Rendering: The Stop Sign Test

Text-heavy prompts are especially sensitive to off-manifold drift at late denoising steps

CFG

Rectified-CFG++

Prompt: "A stop sign with 'ALL WAY' written below it."

Text Rendering: Neon Street Sign

CFG

Rectified-CFG++

Prompt: "A neon street sign that says 'CyberCore Cafe', glowing in magenta and blue."

Text Rendering: Headlines & Inscriptions

CFG

Ours

CFG

Ours

Text Rendering: Presentations & Artistic Text

CFG

Ours

CFG

Ours

Text Rendering: Carved & Printed Text

CFG

Ours

CFG

Ours

Text Legibility: Full Gallery (SD3.5)

Each column: CFG left → Rectified-CFG++ right

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

CFG

Ours

Consistent text legibility across all prompts — the adaptive schedule α(t) is the key mechanism.

Quantitative: MS-COCO 10K — Consistent Improvements

User Study: 30 Experts × 32 Prompts × 4 Models = 15,360 Responses

4-way forced choice: CFG (dark blue), APG (medium blue), CFG-Zero* (light blue), Rectified-CFG++ (green). 4 criteria × 4 models. Green bar is tallest in every single comparison.

Ablation 1: Guidance Scale — CFG Crashes, Ours Stays Stable

X-axis: guidance scale (λ for ours | ω for CFG). Y-axis: FID↓, CLIP↑, ImageReward↑, Aesthetic↑. Blue = CFG, Green = Ours.

CFG (blue curves)

FID explodes: 77 → 148 at ω=10. ImageReward collapses: 1.0 → -1.5. CLIP drops 30%. Aesthetic drops 35%. Catastrophic failure at high guidance.

Rectified-CFG++ (green curves)

FID: 75 → 75 (flat!). ImageReward: 1.0 → 0.95. CLIP stable. Aesthetic stable. Graceful degradation — never crashes.

Why this matters

CFG requires careful per-model tuning of ω to avoid collapse. Rect-CFG++ is robust across a 50× range of λ — no hyperparameter sensitivity. One setting works everywhere.

Ablation 2: Sampling Steps — Better Quality with Fewer Steps

X-axis: sampling steps. Blue = CFG, Green = Ours. Our method converges faster across all metrics.

Ours at 5 steps vs CFG at 5 steps

FID: 71 vs 178 (2.5× better). ImageReward: 1.0 vs -1.5. At ultra-low NFE, CFG produces garbage; ours produces usable images.

Ours at 15 steps ≈ CFG at 28 steps

Same FID, same CLIP, same ImageReward — but ~2× faster inference. This is because our on-manifold trajectories converge more efficiently.

Component ablation (SD3.5)

Config	FID	CLIP	HPSv2
Unconditional only	91.12	0.144	0.187
w/o Predictor	73.70	0.341	0.297
w/o Corrector	74.65	0.341	0.298
Full	72.97	0.345	0.300

Both predictor and corrector essential.

Key Takeaways

Summary of Contributions

Connecting the Contributions

Understanding perception enables guiding generation, which enhances perception.