feat(data): DrivAerStar EnSight -> chunked Zarr preprocessing#238
Open
HennerM wants to merge 4 commits into
Open
feat(data): DrivAerStar EnSight -> chunked Zarr preprocessing#238HennerM wants to merge 4 commits into
HennerM wants to merge 4 commits into
Conversation
…ding Add a Zarr-v3-based data format and dataloader that subsamples by reading random pre-shuffled chunks, so training at e.g. 16k of ~1M points only transfers what it needs (byte-range reads against sharded arrays) instead of loading full samples from per-field .pt files. Format (src/noether/data/zarr_store/): - writer/layout/manifest: one array per field, positions float32 / values float16 (per-field dtype overrides with float16 overflow guard), per-domain pre-shuffled points (seeded), shared chunk grid, configurable shard cap, consolidated metadata, chunk manifest (manifest.json). - convert: generic CLI for any FileMap dataset; reads from a dataset root or any fsspec source (oci://, s3://, ...); ProcessPool-parallel; skips broken samples with a warning. - reader: ZarrChunkReader samples random chunks per epoch; serial single oindex read or thread-parallel per-chunk reads to hide object-storage latency; field-selective reads. - benchmark: equivalence and read-amplification validation vs .pt. Datasets: - ZarrAeroDataset(+Config): config-driven chunk subsampling (num_surface/volume_points), optional geometry_position emission for AB-UPT, per-sample read cache shared across getitem_* calls; local or fsspec store roots. - ZarrShapeNetCarDataset and ZarrDrivAerNetDataset (splits/blacklists read from the store root via fsspec, category filtering). Pipeline/integration: - PointSampling and AnchorPointSampling skip (rename-only) when the requested count >= available points, so dataset-side subsampling makes them inert without config surgery. - AeroCFDPipeline: geometry_position_from_dataset flag to consume the dataset-provided geometry draw. - PropertySubsetWrapper now calls pre_getitem/post_getitem like Dataset.__getitem__. - Recipe configs: shapenet/drivaernet zarr dataset configs and AB-UPT experiment files (incl. OCI-hosted DrivAerNet store). Docs in docs/source/noether/efficient_zarr_store.rst; unit tests cover round-trip, alignment, determinism, parallel reads, fsspec sources, overflow guard and conversion error handling.
Two additions to the chunked Zarr store, both on top of the existing format.
Statistics:
- noether.data.zarr_store.statistics: stream a store (local or fsspec URL)
in one pass and accumulate per-field running moments via RunningStats,
yielding {field}_mean/std/min/max, logscale moments, and global
raw_pos_min/raw_pos_max position bounds. CLI mirrors
calculate_statistics.py but reads the store directly (no dataset class);
--split-file restricts to listed ids (missing ids skipped with a warning),
thread-parallel sample reads. Adds a public RunningStats.count property.
Optional obstore backend:
- stores.make_store() transparently uses the Rust-backed
zarr.storage.ObjectStore for s3:// URLs when the optional `obstore`
package is installed (credentials/region/endpoint from standard AWS_*
env vars), falling back to FsspecStore otherwise or for non-S3 URLs.
Benchmarked ~2x faster on the warm per-sample read path against OCI's
S3-compatible endpoint; ~1.1-2.7x depending on concurrency/workload.
Added as an opt-in extra: `uv sync --extra obstore`.
Docs and unit tests updated for both.
Add a preprocessing/conversion script for the DrivAerStar dataset, the DrivAerStar analogue of caeml/preprocessing.py but writing directly into the chunked/sharded Zarr store (ZarrStoreWriter) instead of per-field .pt files. - Reads raw EnSight Gold cases (case_<T>/<id>/case/<T>_<id>.case) from a local path or any fsspec URL (oci://emmi-drivaer-star, ...). - Surface/volume split matches Emmi-AI/proprioceptive's match_experiment_probes: volume = the `domain` block; surface = WallShearStress-bearing blocks that are not the domain and not wind-tunnel walls (Block.*), combined + surface-extracted, fields taken at cell centres. - Stores surface position/pressure/friction/normals and volume position/pressure/velocity; keeps all volume cells (subsample optional). - Optional volume_sdf (distance to surface via scipy cKDTree, the caeml method) behind --sdf, OFF by default; the FileMap drops the field when disabled so the writer does not expect it. - Each case is read/extracted in a child process so a C-level crash in VTK's EnSight reader skips just that case (non-zero exit) instead of aborting the run; cases convert concurrently (--workers) and the manifest is saved at the end. Validated end-to-end on local case_E samples: full + subsampled reads via ZarrChunkReader, field alignment, unit-length normals, float16 range OK, and the default (no-SDF) store correctly omits volume_sdf.
HennerM
requested review from
MauritsBleeker,
Ndles and
kinggongzilla
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
Contributor
|
Ndles
approved these changes
Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a preprocessing/conversion script for the DrivAerStar dataset — the DrivAerStar analogue of
caeml/preprocessing.py, but writing directly into the chunked/sharded Zarr store (ZarrStoreWriter) introduced in #237 instead of per-field.ptfiles.What it does
case_<T>/<id>/case/<T>_<id>.case+.geo+ per-cellPressure/Velocity/WallShearStress) from a local path or any fsspec URL (oci://emmi-drivaer-star, …;ocifsneedsOCIFS_IAM_TYPE=api_key).Emmi-AI/proprioceptive'smatch_experiment_probes.py: volume = thedomainblock; surface =WallShearStress-bearing blocks that are not the domain and not wind-tunnel walls (Block.*), combined + surface-extracted, fields sampled at cell centres.surface_position/pressure/friction/normalsandvolume_position/pressure/velocity; keeps all volume cells (optional--subsample-factor).volume_sdf(distance-to-surface via scipycKDTree, the same method caeml uses) behind--sdf, off by default — when disabled the FileMap drops the field so the writer doesn't expect it.--workers); the manifest is saved at the end.Why a child process / KD-tree SDF
vtkEnSightGoldBinaryReaderat the C level (uncatchable from Python) — hence per-case subprocess extraction.cKDTreegives the same nearest-surface distance in ~1.3 s, so the SDF (when enabled) uses the KD-tree.Usage
OCIFS_IAM_TYPE=api_key uv run python -m noether.data.datasets.cfd.drivaerstar.preprocessing \ --source oci://emmi-drivaer-star@frwnorq7ern2 \ --output oci://emmi-drivaer-star@frwnorq7ern2/zarr_store \ --case-types E --workers 16 # add --sdf to also store volume_sdfTest plan
case_Esamples: 5/5 converted, full + subsampled reads viaZarrChunkReader, all fields finite and point-aligned, surface normals unit length, float16 range OK.--sdf) run: store correctly omitsvolume_sdfand the reader returns only the present fields.ruff+mypyclean.