Skip to content

VOXELFORMAT: add streaming OBJ export for large models - #805

Open
Tudi wants to merge 2 commits into
vengi-voxel:masterfrom
Tudi:improvement/streamobj
Open

VOXELFORMAT: add streaming OBJ export for large models#805
Tudi wants to merge 2 commits into
vengi-voxel:masterfrom
Tudi:improvement/streamobj

Conversation

@Tudi

@Tudi Tudi commented Mar 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a low-memory streaming OBJ export path that processes scene graph nodes one at a time with chunked 256^3 surface extraction
  • Bypasses the merge-all-nodes path (sceneGraph.merge()) that allocates the full bounding box as a single RawVolume - prohibitive for large scenes (e.g. 1800+ nodes, ~7GB loaded would require ~160GB merge allocation)
  • Enabled via voxformat_streamprocessobj cvar (default false, preserves existing behavior)
  • Memory usage per chunk: ~25MB for extractor arrays plus mesh output, freed each iteration
  • Writes directly to disk via SeekableWriteStream - no in-memory buffering of the full OBJ

Test plan

  • Export a multi-node scene with voxformat_streamprocessobj false (default) - verify identical output to current behavior
  • Export with voxformat_streamprocessobj true - verify correct OBJ output with all nodes properly positioned
  • Export a large scene (1000+ nodes) with streaming enabled - verify memory stays bounded
  • Verify voxformat_transform_mesh true correctly applies world transforms in streaming mode

@mgerhardy

Copy link
Copy Markdown
Collaborator

i would really prefer general solutions here and not per-format solutions were every other mesh based format could benefit from, too

maybe a generic streaming interface for the MeshFormat (not sure how that would work, because it would need some pre-production to get position locality to stream the triangles in the appropriate order (like the vertex cache on the gpu)

@Tudi

Tudi commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

voxformat OGT library accepts full scenes as input. Just one example. I hated the patch so much I canceled the PR myself. Not sure how many formats can work with stream processing/writing.
Maybe you have an idea ?

@mgerhardy

Copy link
Copy Markdown
Collaborator

for ogt_vox we have an issue here: jpaver/opengametools#80

Tudi added 2 commits April 1, 2026 12:31
Adds a low-memory streaming OBJ export path that processes nodes
one at a time with chunked surface extraction (256^3 chunks).
Bypasses the merge-all-nodes path that can require massive memory
allocations for large scenes. Enabled via voxformat_streamprocessobj cvar.
@Tudi
Tudi force-pushed the improvement/streamobj branch from 55a5aa0 to c9371d6 Compare April 1, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants