Evaluate SfM and preprocessing settings against known poses

[hummat]

Problem

Every measurement in docs/evaluation.md and docs/methods_and_models.md §6.7 compared two configs on the
same SfM output. That pairing is what made small effects visible, and it also means SfM quality cancelled out
of every result. We have no evidence the current COLMAP settings are good, and none that they are bad.

scripts/sfm.sh has never had any of its knobs varied in an experiment: --matcher
(exhaustive/sequential/vocab_tree), --extra (affine shape estimation, domain size pooling, guided matching,
CPU only), the GLOMAP path, and camera model.

What is already answered elsewhere

MUGSQA (2511.06830) varies the input axes on 1,970 samples from 55
Sketchfab meshes: view count (72/36/9), resolution (1080/720/480), camera distance (5 m/2 m/1 m), and point
cloud initialisation quality. docs/evaluation.md calls it "the synthetic ground-truth benchmark we sketched
out for ourselves. It exists, and the code and data are released."

Read their per-axis numbers before re-running our own view-count experiment. That experiment is void (the
ceil bug) and would cost about 40 minutes a config to redo.

The part worth building, which is small

MUGSQA renders in Blender with exact poses in NeRF-synthetic format, so COLMAP is never in their loop. Their
initialisation axis degrades a point cloud synthetically rather than testing whether a matcher produced a
good one.

That also means their release is an SfM benchmark we get for free: rendered views plus known extrinsics is
exactly what scores COLMAP. Run each matcher variant on their images, compare against their poses, read
rotation and translation error. No rendering, no training run, minutes per config.

Caveat that decides how far to trust it: Blender renders of Sketchfab objects have no motion blur, no rolling
shutter, no textureless plaster, no auto-exposure wander. Use it to rank candidates cheaply, then confirm the
winner with one paired A/B on a real scene, everything downstream held fixed.

Coverage, so this does not get re-derived again

Measured on our own data:

Axis	Where
Opacity pruning ladder	docs/evaluation.md
Container (PLY/SPZ/SOG) sizes	docs/methods_and_models.md §6.7
max_gs_num / cap_max	§6.7
Training steps, seeding, MCMC vs ADC	§6.7
Appearance embeddings vs exposure drift, 55 scenes	§6.7
Transients and person masking	§6.7, docs/evaluation.md
Synthetic occluders (person cutouts over R2D2, 49 of 154 frames)	not written up
JPEG encoding contamination	docs/evaluation.md
Eval-frame autocorrelation	docs/evaluation.md

Not measured by us, and not by MUGSQA either:

• Motion blur, sensor noise, low light. These appear in docs/troubleshooting.md as capture advice ("prevent
  motion blur, avoid fast motion or low light") with nothing behind them.
• Rolling shutter.
• Matcher, feature extraction and SfM backend, per above.

3DGS-QA degrades exported assets with positional noise and SH perturbation, which is a different thing from
capture-side noise and does not substitute for it.

Plan

1. Read MUGSQA's view-count and initialisation results. Decide from those whether the frame-count rerun is
   worth 40 minutes a config.
2. If a matcher question survives that, score COLMAP variants against MUGSQA's poses.
3. Confirm any winner with one paired A/B on gaudi_fountain, everything after SfM held fixed.
4. Seed gandalf, legolas, lizard_landscape and lizard_portrait, which lack sparse_pc.ply. Check
   whether the COLMAP sparse model survived first; if it did, this is a processing re-run and not an SfM one.

Do not build a synthetic benchmark for this. Step 2 uses someone else's.

Related

• Evaluate VGGT / MASt3R-SfM as future SfM backends #18 tracks VGGT and MASt3R-SfM as future backends. This issue is about evaluating what we already run, and
  step 2 gives Evaluate VGGT / MASt3R-SfM as future SfM backends #18 a way to be answered.
• feat(pipeline): End-to-end validation with ground truth meshes #5 specs end-to-end validation against GT meshes. Different measurement: it scores the end of the pipeline,
  which folds SfM error, training and mesh extraction into one number.
• Validate splat tuning decisions against a perceptual quality metric #34 holds the perceptual metric question.

[hummat]

Existing benchmarks for capture degradations

Follow-up on the coverage table above: the gaps I listed as unmeasured are mostly covered by released data,
so the answer is download rather than build.

Motion blur, rolling shutter, noisy poses. SpectacularAI/3dgs-deblur
(ECCV 2024, Gaussian Splatting on the Move) re-rendered the Deblur-NeRF synthetic dataset into variants
carrying motion blur, rolling shutter and noisy pose initialisation. Hosted on Zenodo under CC BY-SA 4.0 and
CC BY 4.0, fetched with python download_data.py --dataset synthetic, and already shaped for a nerfstudio
fork. This is the "post-process an existing benchmark" idea, executed and released.

Exact GT for a reconstruction pipeline, object-centric.
IVL-SYNTHSFM-v2: 4000 synthetic images
of five models from varied viewpoints under varied lighting, with intrinsics, extrinsics and GT geometry.
Small and old, but the closest published match to a mini-mesh capture.

Recent, evaluates SfM and reconstruction accuracy together.
A 3D Reconstruction Benchmark for Asset Inspection: GT depth, poses and
meshes for synthetic scenes with simulated inspection trajectories, scoring both traditional SfM/MVS and
end-to-end transformer methods. No capture degradations, and the domain is inspection rather than object
capture.

Input axes plus human opinion scores. MUGSQA, as above.

Delivery and compression plus human opinion scores. 3DGS-VBench.

On building one benchmark for the whole pipeline

Tempting and mostly redundant. Between 3dgs-deblur, MUGSQA and IVL-SYNTHSFM-v2 the axes are covered, and what
remains after downloading those is narrow: low light and sensor noise. Those are a post-process on any
synthetic render with known poses, not a benchmark project. Motion blur is rendered sub-frames averaged in
linear RGB, which is how Deblur-NeRF made theirs; rolling shutter is row-wise pose interpolation; noise is a
sensor model applied in linear space.

The limit worth stating plainly: perfect ground truth answers the SfM and geometry questions completely and
the perceptual question not at all. It cannot tell you whether a viewer would notice pruning at 0.05, because
that needs opinion scores rather than exact references. #34 owns that half, and it is the half actually
blocking a decision.

The one real argument for building our own is capture style. Everything above is either object-centric
renders or someone else's smartphone footage, and none of it is a handheld walkthrough of a cluttered outdoor
facade with tourists in it. That transfer gap keeps recurring. It is still not a reason to build before
downloading what exists.

Revised step 1

Download the 3dgs-deblur synthetic set and read MUGSQA's per-axis numbers before running anything.