Validate splat tuning decisions against a perceptual quality metric

[hummat]

Problem or Motivation

Every splat tuning decision in this pipeline is currently settled with PSNR, SSIM
and LPIPS from ns-eval. Four subjective studies published in 2024 and 2025
measured how well those metrics reproduce human quality rankings on Gaussian
splats, and all three land near 0.5 Spearman correlation or below:

Metric	3DGS-VBench	MUGSQA	3DGS-QA
PSNR	0.50	0.52	not reported
SSIM	0.51	0.37	0.31
LPIPS	0.51	0.41	0.34
DOVER	0.94	not reported	not reported
VSFA	0.94	not reported	not reported
FAST-VQA	0.93	not reported	0.29

Converting the reported Kendall coefficients, an LPIPS win predicts a human
preference about two times in three. That is not enough to settle the decisions
we have been using it for:

• cap_max: on gaudi_fountain, PSNR and SSIM were flat across 100k to 1M
  Gaussians while LPIPS moved 0.451 to 0.282 and file size moved 23.8 MB to
  232.9 MB. We do not know how much of that LPIPS movement a viewer sees.
• Cleanup threshold: filtering at opacity 0.05 removes 4.9% to 62% of
  Gaussians depending on scene. 3DGS-QA measured content-blind random pruning at
  25/50/75% but nothing published covers attribute-based pruning.
• Container format: SPZ against SOG is quantisation at fixed geometry.
  Nothing published covers it.

Full write-up with sources in docs/evaluation.md.

Proposed Solution

Adopt a no-reference video quality metric scored on the orbit renders we already
produce, and validate it against runs already on disk before trusting it.

1. Transfer check. A ladder on one of our scenes with an ordering nobody
   disputes (training truncated at 2k/5k/10k/30k, or pruning at 25/50/75%).
   Require DOVER to reproduce it. Failing kills the metric; passing only shows
   the published 0.94 survives our content.
2. The contested pairs. cap_max 250k vs 500k, cleanup 0.05 vs raw on a scene
   where the filter takes 60%, 212 vs 424 training views. Orbits side by side,
   unlabelled, randomised order, forced choice. One rater is thin, and it is
   direct evidence about the decisions we make, which a published correlation on
   someone else's stimuli is not.
3. Adopt or fall back. Agreement means wiring DOVER in after export
   alongside ns-eval. Disagreement sends us to DISTS (0.73) or CW-SSIM (0.74).
   If nothing agrees, these calls are not metric-decidable and file size decides.

Separation alone is not the acceptance criterion. A metric that moves when the
configuration changes has shown sensitivity, and LPIPS already has that; the
question is agreement with a viewer, which needs stimuli whose ordering is
established first. (Thanks to the Codex review on #35 for catching that the
original version of this plan had the same circularity the doc criticises.)

Alternatives Considered

• Build our own synthetic benchmark. MUGSQA already renders 55 Sketchfab
  meshes in Blender with exact poses and varies view count, resolution, camera
  distance and point cloud quality across 1,970 samples. Data and code are
  released. Duplicating it is not worth the effort; the gap that remains is
  attribute-based pruning and container quantisation at fixed training.
• Switch to DISTS or CW-SSIM. Both beat PSNR/SSIM/LPIPS (0.73 and 0.74) and
  are cheap full-reference metrics, so they are a reasonable fallback if the
  video metrics do not hold up on our data.
• Fine-tuned predictors (DBCNN 0.88, GSOQA 0.77). Both are trained on the
  dataset they score. Not drop-in.

Area

Export/texturing

Open question

3DGS-QA reports FAST-VQA at 0.29 while 3DGS-VBench reports 0.93. Neither paper
explains the other's result. Step 1 is what settles it for our content.

Contribution

• I would be willing to submit a PR for this feature

[hummat]

Scope decision after #35 merged.

The plan written into docs/evaluation.md is sized for the metric question: can DOVER or a similar
no-reference video model stand in for human judgment on these splats. That needs the transfer check, the
15-scene paired set, three showings per pair, and a chance floor estimated from the rater's own marginals.
It stays in the doc for whenever someone wants to make that claim.

It is the wrong size for the decision actually waiting on it, which is what opacity threshold to export at
for a blog post. That one needs no metric at all:

1. Pick three scenes spanning the cut range, since 4.9% and 62% behave differently.
2. Render orbits at unpruned, 0.05 and 0.1, PNG (--orbit-image-format png).
3. Shuffle which side is which, keep the mapping in a file, look, pick a threshold.

Blinding is the one part worth keeping. I have spent weeks arguing pruning is cheap and I will know which
file is smaller.

Worth recording alongside it: a full-resolution A/B at rest is a stricter viewing condition than the
delivery, which is a phone spinning a splat in a small canvas through a viewer that already quantises. A
comparison run that way will reject pruning nobody would notice in the post.

Leaving this open. The metric question is unanswered, and the answer would be worth having if splat exports
ever become a pipeline with a quality contract rather than a handful of blog assets.