Add BEAST3D model: ERayZer with DINOv3 tokenizer, GT cameras, frustum…#56
Merged
Merged
Conversation
… 3DGS Port multi-view BEAST3D into the beast repo as a full-parity ERayZer subclass, following docs/developer_guide.md. Shared, opt-in changes (erayzer behavior unchanged, verified): - gaussians_renderer: add camera_frustum_constraint + opt-in frustum_constraint kwarg; capture render_alphas (was discarded) so mask loss has real alphas; accept bg_color=None. - erayzer_model: add _tokenize_images / _sample_background / _prepare_target subclass hooks to forward; thread frustum_constraint/backgrounds into Renderer; expose render_alphas/pixel_mask outputs. BEAST3D (beast/models/beast3d/): - frozen DINOv3 tokenizer with ImageNet normalization (the DinoV3 wrapper omits it) - GT cameras via _resolve_cameras; pose branch disabled with encoder_n_layer=0 - frustum-constrained render, random-background compositing, masked alpha loss - registry + config dispatch (Beast3DBeastConfig) + train.py MultiViewDataModule routing with use_mask=True Config: configs/multiview/beast3d.yaml (reference template). The machine-specific cheese3d fine-tune config (data_dir + init_checkpoint) is kept local and gitignored, alongside erayzer_cheese3d_ft.yaml. Tests: tests/models/beast3d/ (19 CPU unit tests + 1 gsplat-gated integration test); renderer + base-ERayZer regression tests. ruff + pyright clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #56 +/- ##
==========================================
+ Coverage 86.03% 86.51% +0.48%
==========================================
Files 51 55 +4
Lines 4890 5022 +132
==========================================
+ Hits 4207 4345 +138
+ Misses 683 677 -6
🚀 New features to boost your workflow:
|
The lint job runs pyright over beast + tests; three type errors in test_beast3d_model.py failed it: - _resolve_cameras(None, ...) — img_tokens is typed Tensor; the GT-camera override ignores it, so pass a throwaway torch.empty(0) instead of None. - mask.shape on the Tensor|None return of _prepare_target — assert mask is not None first to narrow the type. Test-only change; behavior unchanged. ruff + full-project pyright now clean, 19 beast3d unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
themattinthehatt
requested changes
Jun 25, 2026
themattinthehatt
left a comment
themattinthehatt
left a comment
Contributor
There was a problem hiding this comment.
@PPWangyc great PR! I left a couple minor comments, after you fix those I'll merge
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| class TestBeast3DModelConfig: |
Contributor
There was a problem hiding this comment.
minor: can you move this to tests/test_config.py? (I'll probably change the organization of these config files at some point, but for now I want to keep them all together)
Per review on PR paninski-lab#56: - Move TestBeast3DModelConfig from tests/models/beast3d/test_beast3d_model.py to tests/test_config.py, keeping all config-schema tests together. - Add a get_beast_config_class('beast3d') dispatcher test alongside the erayzer one while the config tests are consolidated. The integration test was already gated with @requires_gsplat_cuda on the test_integration_basic method. Test-only change; ruff + pyright clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
themattinthehatt
approved these changes
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… 3DGS
Port multi-view BEAST3D into the beast repo as a full-parity ERayZer subclass, following docs/developer_guide.md.
Shared, opt-in changes (erayzer behavior unchanged, verified):
BEAST3D (beast/models/beast3d/):
Config: configs/multiview/beast3d.yaml (reference template). The machine-specific cheese3d fine-tune config (data_dir + init_checkpoint) is kept local and gitignored, alongside erayzer_cheese3d_ft.yaml.
Tests: tests/models/beast3d/ (19 CPU unit tests + 1 gsplat-gated integration test); renderer + base-ERayZer regression tests. ruff + pyright clean.