Skip to content

Fix alpha-masked background supervision for RGBA datasets - #1338

Open
Running-Turtle1 wants to merge 1 commit into
graphdeco-inria:mainfrom
Running-Turtle1:fix-alpha-background-supervision
Open

Fix alpha-masked background supervision for RGBA datasets#1338
Running-Turtle1 wants to merge 1 commit into
graphdeco-inria:mainfrom
Running-Turtle1:fix-alpha-background-supervision

Conversation

@Running-Turtle1

Copy link
Copy Markdown

Summary

This fixes missing background supervision for RGBA datasets, most visibly the synthetic Lego scene with --white_background.

  • Keep the renderer output intact when computing the photometric loss.
  • Alpha-composite the ground-truth RGB over the exact background used for the current iteration.
  • Use the same compositing path for training reports and offline-rendered ground truth.
  • Add CPU unit coverage for transparent, semi-transparent, opaque, and missing-alpha cases.

Fixes #1038.

Problem

Transparent pixels in the NeRF synthetic PNGs store black RGB values and a zero alpha value. The current training path multiplies the rendered image by the alpha mask before L1/SSIM:

image *= alpha_mask

That operation removes both the configured renderer background and every Gaussian contribution outside the foreground silhouette before the loss is evaluated. Since the stored RGB is also black there, transparent pixels produce no useful gradient. Synthetic scenes start from 100,000 random points in the scene volume, so unconstrained Gaussians can survive training and become visible as floaters when render.py renders the complete point cloud without applying the mask.

The target should instead be composited over the same background as the renderer:

gt = rgb * alpha + background * (1 - alpha)

This also supports --random_background, because the current iteration's sampled background is used for both rendering and target composition.

Visualization

Before and after comparison

The top row shows the final 30k render before and after the fix. The bottom row shows why the previous metric output was inconsistent: raw transparent RGB was saved as black, while the renderer used white background.

Quantitative result

Both models were evaluated on all 200 Lego test views against the corrected white-background ground truth.

Training path PSNR SSIM
Before 14.5612 dB 0.7785
After 35.9335 dB 0.9833

Training command:

python train.py \
  -s /path/to/nerf_synthetic/lego \
  -m output/lego_fixed \
  -w --eval --disable_viewer \
  --iterations 30000

Validation

python -m py_compile train.py render.py utils/image_utils.py tests/test_image_utils.py
python -m unittest tests.test_image_utils

A two-iteration CUDA smoke training run also completed and saved its point cloud successfully. The full comparison used a fresh 30,000-iteration training run; no checkpoint from the affected model was reused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Why we use White background for NeRF Synthetic Dataset?

1 participant