Skip to content

Three buffer overruns: compound tree stacks, hull clip polygons, mesh cast stacks - #102

Merged
nicolas-maman merged 2 commits into
mainfrom
fix/scratch-overruns
Oct 4, 2026
Merged

nicolas-maman merged 2 commits into
mainfrom
fix/scratch-overruns

Conversation

@nicolas-maman

Copy link
Copy Markdown
Collaborator

I audited every fixed-size scratch block and buffer in the library against the real limits of what gets written into it. Three can be overrun.

  1. Compound tree stack (most severe). A baked compound tree carried one traversal stack inside the compound block, 2 KB. Tree queries index a tree's stack per worker, at 12 KB per worker, so a query on any worker past the first wrote 12 KB × worker index beyond it, into the heap. The parallel continuous pass reaches this whenever bullets are swept against a compound.
    • Baked trees now point at the compound module's own per-worker stacks (query and distance), allocated with the module's scratch. The block no longer holds a stack.
    • dynamic_tree exports the per-worker sizes (query_stack_bytes, query_dist_bytes).
  2. Hull clip polygons. They held 64 vertices, but an incident face of up to 64 sides clipped against a reference face of up to 64 can reach 128. Two 40-sided cylinders cap to cap at a twist give an 80-gon, and the last worker's buffer spilled past the allocation. They now hold 128, as triangle_manifold's already did.
  3. Mesh casts. The ray and shape casts pushed the far child at every level with no guard, so a tree leaning right deeper than the 256-entry stack overran it. Every push in every mesh traversal is guarded now; a pathologically deep tree drops the far subtree instead of writing past the stack.

Tests, each run as a high worker index so an overrun lands past the allocation:

  • test_compound_workers: compound ray cast as workers 1, 7, 31 and 63. Main crashes with an access violation.
  • test_wide_faces: 40-sided cylinders collided as worker 63.
  • test_deep_tree_cast: a ray cast through a 300-deep right spine as worker 63.

On main, the last two overrun into memory that happens to be writable, so they don't fail there; they guard the result going forward.

test_compound's permanent allocation count grows by the two new stacks. All 28 test programs pass locally on aether 0.772.0, with no warnings.

…lock

- compound: a baked tree carried one traversal stack in the compound's
  block (2 KB), but tree queries take a worker's share of a tree's stack
  (12 KB a worker), so a query on any worker past the first wrote 12 KB a
  worker beyond it into the heap -- reached from the parallel continuous
  pass, bullets swept against a compound. Baked trees point at the
  module's own per-worker stacks now (query and distance), made with the
  module's scratch; the block holds none. dynamic_tree exports the share
  sizes (query_stack_bytes, query_dist_bytes).
- manifold: the hull clip polygons held 64 vertices, but an incident face
  of up to 64 sides clipped against a reference face of up to 64 can reach
  128 (two 40-sided cylinders cap to cap at a twist give an 80-gon); the
  last worker's spilled past the allocation. They hold 128 now.
- mesh: the ray and shape casts pushed the far child at every level
  unguarded, so a tree leaning right deeper than the 256-entry stack
  overran it; every mesh traversal's push is guarded now (a pathological
  tree drops the far subtree rather than write past the stack).

Tests: test_compound_workers casts as workers 1, 7, 31 and 63 (main
crashes); test_wide_faces collides 40-sided cylinders as the last worker;
test_deep_tree_cast casts through a 300-deep right spine as the last
worker. test_compound's permanent allocations grow by the two stacks.
…on count

The baked trees' per-worker stacks are the compound module's scratch now,
made on first use; test_compound_pairs made them inside the counted region.
@nicolas-maman
nicolas-maman merged commit 6f0544d into main Oct 4, 2026
1 check passed
@nicolas-maman
nicolas-maman deleted the fix/scratch-overruns branch October 6, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant