perf(ingest): streaming beamtime ingest + maintainability refactor - #37

Draft

HarlanHeilman wants to merge 22 commits into

mainfrom

perf/ingest-streaming-and-maintainability

Commits on Apr 16, 2026

docs: plan for ingest streaming and maintainability
HarlanHeilman
committed
ac4e952
perf(ingest): bulk-read FITS pixels in read_image_i32
HarlanHeilman
committed
47666a6
perf(ingest): drop duplicate /processed zarr array in write_frame_raw
HarlanHeilman
committed
726cf24
perf(ingest): use RETURNING instead of SELECT-after-INSERT in catalog phase
HarlanHeilman
committed
8216275
perf(ingest): batch catalog transactions with small-scan coalescing
HarlanHeilman
committed
4f5f0fa
refactor(ingest): clean up T4 review findings
HarlanHeilman
committed
5470a34
perf(ingest): stream zarr phase with bounded-parallel read->write
HarlanHeilman
committed
778d836
perf(ingest): parallelize headers phase per-file instead of per-scan
HarlanHeilman
committed
b582a26
refactor(io): unify bulk BITPIX=16 pixel reader in src/io/raw_pixels.rs
HarlanHeilman
committed
196d443
refactor(catalog): move flatten_fits_error to catalog/mod.rs and document mmap safety
HarlanHeilman
committed
b44b104
refactor(ingest): replace stringly-typed phase labels with IngestPhase enum
HarlanHeilman
committed
b64e1ba
refactor(ingest): idiomatic IngestPhase::as_str + stronger wire-format assertions
HarlanHeilman
committed
d3378bf
refactor(ingest): split ingest_beamtime_inner into phase functions with IngestContext
HarlanHeilman
committed
93501c6
refactor(ingest): extract catalog-phase helpers and add is_cancelled helper
HarlanHeilman
committed
e2c1926
test(ingest): synthetic-beamtime test harness + bench script (T10)
HarlanHeilman
committed
db88dbd
fix(bench): remove --use-default-paths escape hatch per T10 spec
HarlanHeilman
committed
fc92dda
refactor(t10): tighten synthetic beamtime fidelity and type hints
HarlanHeilman
committed
83d0bfe
test(catalog): drop per-phase-label assertions from progress callback test
HarlanHeilman
committed
85b066d

Commits on Apr 17, 2026

feat(cli): add pyref Typer app and ingest watcher integration
HarlanHeilman
committed
d7dc39e

Commits on Apr 20, 2026

feat(catalog): per-scan Zarr uint16 stacks, migration tool, and config catalog path
HarlanHeilman
committed
d0a03a5

Commits on Apr 22, 2026

perf: streaming ingest, catalog schema (header_values, pruned frames), bench CLI
HarlanHeilman
committed
cb366e6

Commits on Apr 23, 2026

docs: organize prototyping API and workflow specs
HarlanHeilman
committed
148ee4c