From 6230e7d4bef36de1270ca7ddae7aa525afac8e2b Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:41:53 +0800 Subject: [PATCH 01/20] Add docs/ARCHITECTURE.md: system overview, data model, module map, artifact layout Co-Authored-By: Claude Fable 5 --- docs/ARCHITECTURE.md | 262 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 262 insertions(+) create mode 100644 docs/ARCHITECTURE.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..f6361ba --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,262 @@ +# Architecture + +This document is the system-level map of Aetherscan: what the pipeline computes, how the +modules fit together, the process/thread topology, and where every artifact lands on disk. +Per-surface deep dives live in the sibling documents indexed in [`README.md`](README.md); this +one is the place to start when you need to orient yourself in the codebase. + +## TL;DR + +Aetherscan is a two-stage ML pipeline for SETI anomaly detection in radio spectrograms: + +1. A **Beta-VAE** ([`src/aetherscan/models/vae.py`](../src/aetherscan/models/vae.py)) compresses + each observation spectrogram into an 8-dimensional latent vector, trained with a composite + loss whose clustering terms teach the latent space to separate ON/OFF-source structure + (see [`MODELS.md`](MODELS.md)). +2. A **Random Forest** ([`src/aetherscan/models/random_forest.py`](../src/aetherscan/models/random_forest.py)) + classifies whole cadences: the 6 per-observation latents are concatenated into a 48-feature + vector, and cadences whose P(true) clears `config.inference.classification_threshold` + become candidates. + +Training data is synthetic (setigen signal injection over real observed backgrounds, +[`src/aetherscan/data_generation.py`](../src/aetherscan/data_generation.py)); inference data is +real filterbank `.h5` observations reduced by an energy-detection preprocessing stage +([`src/aetherscan/preprocessing.py`](../src/aetherscan/preprocessing.py)). Both commands run +single-node multi-GPU via `tf.distribute.MirroredStrategy` with NCCL all-reduce (falling back +to `HierarchicalCopyAllReduce`), set up in +[`src/aetherscan/main.py`](../src/aetherscan/main.py)`:setup_gpu_strategy()`. + +`src/aetherscan/main.py` is the **sole entry point**: `python -m aetherscan.main +{train|inference}`. + +## Data model: observations, cadences, stamps, snippets + +| Term | Shape (defaults) | Meaning | +| --- | --- | --- | +| **Observation** | `(16, 4096)` raw → `(16, 512)` model-ready | One spectrogram: `time_bins` × frequency bins. The model input is downsampled ×8 along frequency (`data.width_bin // data.downsample_factor`) and log-normalized into [0, 1]. | +| **Cadence** | `(6, 16, 512)` | 6 observations of the same sky position in ABACAD order: positions 0/2/4 are ON-source ("A"), 1/3/5 are OFF-source. A technosignature should appear in the ONs and vanish in the OFFs; RFI persists in both. This is the unit both models reason about. | +| **Stamp** | `(6, 16, stamp_width // downsample_factor)` stored | Inference-side: a `stamp_width` (4096-bin) frequency window cut around one energy-detection hit, extracted from **all 6** observations. Stored downsampled by default (`inference.store_downsampled_stamps`). | +| **Snippet** | `(6, 16, 512)` | A stamp after loading (log-normalized, model-ready) — one row of a per-cadence `.npy`. "Stamp" and "snippet" index the same objects; *stamp* emphasizes the on-disk extraction, *snippet* the model input. One inference cadence typically yields many snippets (one per deduplicated hit, ×3 with overlap search). | + +Physical constants ride along in `DataConfig`: `freq_resolution` ≈ 2.79 Hz/bin, +`time_resolution` ≈ 18.25 s/bin (GBT high-frequency-resolution products). + +## Module map + +| Module | Role | +| --- | --- | +| [`main.py`](../src/aetherscan/main.py) | Entry point. Initialization order, GPU strategy setup (NCCL warmup + fallback), `train_command()` / `inference_command()` retry loops, streaming per-cadence inference driver, final `manager.cleanup_all()`. | +| [`cli.py`](../src/aetherscan/cli.py) | Argparse for both subcommands, semantic + cross-replica validation (`collect_validation_errors`), fix proposer, `apply_saved_config()` / `apply_args_to_config()`. See [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md). | +| [`config.py`](../src/aetherscan/config.py) | Dataclass-of-dataclasses `Config` singleton; every runtime parameter with a default; `to_dict()` serialization to `config_{tag}.json`. | +| [`train.py`](../src/aetherscan/train.py) | `TrainingPipeline`: curriculum rounds, distributed datasets, gradient accumulation, adaptive LR, checkpointing, the run-state stage machine, and all training diagnostics/plots. See [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md). | +| [`round_data.py`](../src/aetherscan/round_data.py) | Disk-backed (memmap) per-round datasets: `RoundDataPaths`, the atomic `.done` manifest protocol, and the `RoundDataProducer` background-generation process. | +| [`run_state.py`](../src/aetherscan/run_state.py) | Persisted `TrainingRunState` manifest (`run_state_{tag}.json`) that drives stage-aware training resume. | +| [`data_generation.py`](../src/aetherscan/data_generation.py) | setigen signal injection: `create_false` / `create_true_single` / `create_true_double`, batched memmap generation (`generate_round_to_memmap`), injection statistics. See [`PREPROCESSING.md`](PREPROCESSING.md). | +| [`preprocessing.py`](../src/aetherscan/preprocessing.py) | Training background loading; inference energy detection (fused per-coarse-channel workers, vectorized D'Agostino-Pearson test, spline/PFB bandpass flattening, stamp extraction). See [`PREPROCESSING.md`](PREPROCESSING.md). | +| [`pfb.py`](../src/aetherscan/pfb.py) | Polyphase-filterbank static passband response (native NumPy port of the bliss reference) used by the default bandpass-flattening method. | +| [`inference.py`](../src/aetherscan/inference.py) | `InferencePipeline`: distributed encoding of snippets, RF classification, positives-only result writes. See [`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md). | +| [`inference_viz.py`](../src/aetherscan/inference_viz.py) | End-of-run inference visualization suite (ED distributions, galleries, latent projection, summary card). | +| [`models/vae.py`](../src/aetherscan/models/vae.py), [`models/random_forest.py`](../src/aetherscan/models/random_forest.py) | Model definitions. See [`MODELS.md`](MODELS.md). | +| [`db/db.py`](../src/aetherscan/db/db.py) | Thread-safe SQLite singleton with a single background writer thread, schema migrations, and supersede semantics. See [`DATABASE.md`](DATABASE.md). | +| [`logger/`](../src/aetherscan/logger), [`manager/`](../src/aetherscan/manager), [`monitor/`](../src/aetherscan/monitor) | Queue-based logging (+ Slack), resource lifecycle management (pools/SHM/processes/signals), 1 Hz resource monitoring. See [`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md). | + +## Data flow + +```mermaid +flowchart TB + subgraph TRAIN["train"] + BG[".npy background plates
{data_path}/training/"] --> LOAD["DataPreprocessor.load_train_data()
mmap + parallel downsample"] + LOAD --> SHM["background plates in shared memory
(DataGenerator)"] + SHM --> PROD["RoundDataProducer (spawned process)
setigen injection via worker pool"] + PROD --> RD["round_data/{tag}/round_XX/
{main,true,false,labels}.npy memmaps"] + RD --> DS["prepare_distributed_train_dataset()
batched tf.data over memmaps"] + DS --> VAE["Beta-VAE rounds
(MirroredStrategy, grad accumulation)"] + VAE --> CKPT["checkpoints: vae_{encoder,decoder}_round_XX.keras"] + VAE --> RF["train_random_forest()
encode latents → fit RF"] + RF --> ART["vae_encoder_{tag}.keras · random_forest_{tag}.joblib
config_{tag}.json · rf_eval_artifacts_{tag}.joblib"] + VAE -. "training_stats / injection_stats / latent_snapshots" .-> DB[("SQLite
{output_path}/db/aetherscan.db")] + DB --> PLOTS["training plots
{output_path}/plots/"] + end + + subgraph INF["inference"] + CSV["CSV catalog(s)
{data_path}/inference/"] --> GROUP["group_observations_from_csv()
rows → 6-obs cadences"] + H5[".h5 filterbank files"] --> ED + GROUP --> ED["energy detection per cadence
DC spike → bandpass flatten → k² threshold"] + ED --> NPY["per-cadence stamp .npy + metadata .json
preprocessed/<csv_stem>_<tag>/"] + NPY --> ENC["InferencePipeline.run_inference()
encoder → latents → RF P(true)"] + ART -. "trained models" .-> ENC + ENC -. "inference_results (positives) + inference_cadences manifest" .-> DB + ENC --> VIZ["inference visualization suite
plots/inference/{tag}/"] + end +``` + +The train and inference halves share the models, the config/DB/logging infrastructure, and the +preprocessing conventions (downsample ×8 + log-norm) — a snippet at inference time is shaped +exactly like a training cadence. + +## Process & thread topology + +The main process owns TensorFlow (and therefore the GPUs). Everything CPU-heavy is pushed into +worker processes; everything I/O-ish runs on background threads of the main process: + +- **Main process threads**: the TF runtime (its own thread pool), the DB writer thread + ([`DATABASE.md`](DATABASE.md)), the `QueueListener` logging thread + ([`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md)), the 1 Hz resource-monitor thread, the + round-data drainer thread (training), and a 1-worker preprocessing prefetch thread + (streaming inference). +- **Worker pools** (fork-started, plain `multiprocessing.Pool`): background + downsampling (training load), energy detection + stamp extraction (one persistent pool per + inference run), and signal injection (owned by the producer, below). +- **`RoundDataProducer`** (training only): a **spawn**-started process that owns its own + injection worker pool and generates round *k+1* while round *k* trains. Spawn, not fork — + the TF-laden parent holds locks a forked child could inherit mid-acquisition + (see the `_MP_CONTEXT` note in [`round_data.py`](../src/aetherscan/round_data.py)). +- **Shared memory** carries the background plates (training) and load-time chunks; memmapped + `.npy` files carry everything bigger. Only the creator ever unlinks shared memory; the + ResourceManager tracks and cleans up all of it. + +The DB writer queue is a *thread* queue — worker **processes** never write to the DB directly. +Stats generated in workers travel back over multiprocessing queues/IPC and are written by +main-process threads. + +## Initialization order (`main.py:main()`) + +The order is load-bearing — each step depends on the previous: + +1. `load_dotenv(find_dotenv())` — `.env` into `os.environ` before any module reads it + (workers inherit the environment later). +2. `init_config()` — the `Config` singleton; everything else reads it. +3. `init_logger()` — queue-based logging up before anything wants to log. +4. `init_manager()` — the ResourceManager registers its `atexit` + `SIGINT`/`SIGTERM` handlers. +5. `register_logger()` — hands the logger to the manager so it is stopped **last** during + cleanup (you can log during teardown of everything else). +6. `setup_argument_parser()` / `parse_args()`. +7. Inference only: `apply_saved_config(args.config_path)` — layers a saved training-run JSON + under CLI flags *before* validation, so `validate_args` checks the values inference will + actually use. The saved `checkpoint` section is skipped (a training run's `save_tag` must + not leak into an inference run). +8. `validate_args(args)` — semantic + cross-replica divisibility checks; no mutation. +9. `apply_args_to_config(args)` — CLI overrides land on the singleton. +10. `init_db()` — schema creation + migration, writer thread starts. +11. `init_monitor()` — 1 Hz sampling into `system_resources`. +12. Dispatch to `train_command()` / `inference_command()`; a `finally` block calls + `manager.cleanup_all()` so non-daemon threads can't block exit. + +Priority order for any parameter: `runtime defaults < loaded config < CLI args` +(see [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md)). + +## The singleton pattern + +`Config`, `Database`, `Logger`, `ResourceManager`, and `ResourceMonitor` are all thread-safe +singletons (double-checked locking in `__new__`, `_initialized` guard in `__init__`, a +`_reset()` teardown hook used by the test suite). The rationale: + +- **One authoritative instance per process.** Config values, the DB writer queue, the log + queue, and the resource registry must be process-global — two `Database` instances would + mean two writer threads racing on one SQLite file. +- **Import-order freedom.** Any module calls `get_config()` / `get_db()` / `get_manager()` + at *use* time instead of threading instances through every constructor. +- **Deterministic teardown.** The manager holds references to the others and closes them in + a strict order (processes → pools → shared memory → monitor → DB → logger). + +Rules that follow (also in [`CLAUDE.md`](../CLAUDE.md)): always use the accessors, never +instantiate directly, and never mutate the config post-init from worker threads — reads are +unsynchronized by design, safe only because startup is the single writer. + +Worker processes get a **copy** of the parent's singletons via fork (or none at all via +spawn); they must never touch the DB singleton and must route logging through +`init_worker_logging()`. + +## Tag conventions + +Every run is identified by `config.checkpoint.save_tag` — the **tag** — which stamps every +artifact filename and every DB row. Accepted formats +(`cli.py:_TAG_PATTERN`): + +| Format | Example | Use | +| --- | --- | --- | +| `YYYYMMDD_HHMMSS` | `20260712_143000` | Default (import-time timestamp) — every untagged run gets a unique one. | +| `final_vX` | `final_v1` | Release-grade training runs. | +| `round_XX` | `round_05` | Reserved for per-round checkpoints (written by the pipeline, not passed by users). | +| `test_vX` | `test_v17` | Smoke/test runs. | + +`train.py:get_latest_tag()` ranks the families `final_vX > round_XX > timestamp > test_vX` +when hunting for the newest checkpoint pair. Same-tag **retries** are first-class: the +run-state manifest (training) and the `inference_cadences` manifest (inference) make re-running +the identical command resume rather than collide, with stale rows from dead attempts flagged +`superseded` in the DB ([`DATABASE.md`](DATABASE.md)). + +## Directory layout & artifact map + +Three roots, set by `AETHERSCAN_{DATA,MODEL,OUTPUT}_PATH` (defaults under +`/datax/scratch/zachy/...`; the container binds them 1:1 so absolute paths stay valid): + +``` +{data_path}/ +├── training/ # background plate .npy files (config.data.train_files) +├── testing/ # preprocessed test .npy files (config.data.test_files) +└── inference/ # CSV catalogs (config.data.inference_files) + └── preprocessed/_/ # per-cadence stamp .npy + metadata .json (tag-scoped) + +{model_path}/ +├── vae_encoder_{tag}.keras # final encoder (the inference model) +├── vae_decoder_{tag}.keras # final decoder (needed for latent traversal) +├── random_forest_{tag}.joblib # final RF classifier +├── rf_eval_artifacts_{tag}.joblib # val features/labels/probas consumed by all RF plots +├── rf_shap_values_{tag}.joblib # cached SHAP values (summary/interaction/log-loss) +├── umap_{obs,cadence}_nn{n}_md{m}_{tag}.joblib # persisted UMAP projections +└── checkpoints/ # per-round vae_{encoder,decoder}_round_XX.keras + └── archive// # previous runs' checkpoints, moved aside at startup + +{output_path}/ +├── config_{tag}.json # resolved config snapshot (written by final_save / inference) +├── run_state_{tag}.json # training run manifest (stage machine + completed rounds) +├── db/aetherscan.db # SQLite (WAL) — all stats/results tables +├── logs/aetherscan.log # current run's log (mode="w": overwritten each run) +├── pfb_cache/pfb_response_*.npy # content-addressed PFB passband responses +├── round_data/{tag}/round_XX/ # per-round training memmaps (deleted after the round trains) +│ └── rf/ # plus the RF training dataset +└── plots/ + ├── *_{tag}.png # end-of-training diagnostics + resource_utilization plot + ├── checkpoints/*_round_XX.png # per-round diagnostics (archived like model checkpoints) + └── inference/{tag}/*.png # inference visualization suite +``` + +Startup hygiene: `train.py:archive_directory()` moves (fresh run) or copies (resume) existing +checkpoints/plots into `archive//` and deletes `round_XX`-stamped files at or above +the resume round; `round_data.py:prepare_round_data_dir()` applies the same policy to round +data, except it deletes rather than archives (a round is ~295 GB) and keeps completed rounds +only when their `.done` manifest validates. + +## Fault-tolerance model (summary) + +Both commands wrap their pipeline in a bounded retry loop (`max_retries`/`retry_delay`, +Pattern C flags — the two modes have independent settings). What makes retries *safe* is +persistent state, not the loop: + +- **Training**: `run_state_{tag}.json` records completed rounds and pipeline stages + (`vae_rounds → vae_plots → rf_train → rf_plots → final_save`); a rebuilt pipeline skips + finished work, resumes mid-round-loop from the last checkpoint, and marks stale DB rows + superseded. Details in [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md). +- **Inference**: preprocessing resumes off each cadence's on-disk `.npy`; the inference stage + resumes off live `status='inferred'` rows in the `inference_cadences` DB manifest. One bad + cadence never aborts the catalog. Details in + [`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md). + +`KeyboardInterrupt` is never retried — it propagates so the ResourceManager's signal handling +can run cleanup exactly once (double Ctrl-C force-quits; +see [`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md)). + +## Where to go next + +- Run/build/debug the container and GPUs: [`GPU_RUNTIME_GUIDE.md`](GPU_RUNTIME_GUIDE.md) +- Add a flag or config field: [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md) +- Training internals: [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md) +- Inference internals: [`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md) +- Energy detection & signal injection: [`PREPROCESSING.md`](PREPROCESSING.md) +- Model math: [`MODELS.md`](MODELS.md) +- Storage & schema: [`DATABASE.md`](DATABASE.md) +- Logging/cleanup/monitoring: [`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md) +- Tests: [`TESTING.md`](TESTING.md) +- CI/CD & assistant workflows: [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md) +- Releases: [`RELEASE.md`](RELEASE.md) From cce33c1da157233ec0e590fc0c7eb46a4a3c5281 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:44:46 +0800 Subject: [PATCH 02/20] Add docs/TRAINING_PIPELINE.md: rounds, round data, retries, every training plot Co-Authored-By: Claude Fable 5 --- docs/TRAINING_PIPELINE.md | 384 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 docs/TRAINING_PIPELINE.md diff --git a/docs/TRAINING_PIPELINE.md b/docs/TRAINING_PIPELINE.md new file mode 100644 index 0000000..322912c --- /dev/null +++ b/docs/TRAINING_PIPELINE.md @@ -0,0 +1,384 @@ +# Training Pipeline + +This document covers the `train` command end to end: the curriculum round lifecycle, the +disk-backed round-data pipeline with background generation, the distributed training loop, +checkpointing, the run-state manifest that makes retries seamless, and every diagnostic plot +the pipeline produces (and what to look for in each). The orchestration lives in +[`src/aetherscan/train.py`](../src/aetherscan/train.py) with two supporting modules: +[`round_data.py`](../src/aetherscan/round_data.py) (memmap datasets + producer process) and +[`run_state.py`](../src/aetherscan/run_state.py) (persisted run manifest). For the model math +see [`MODELS.md`](MODELS.md); for signal injection internals see +[`PREPROCESSING.md`](PREPROCESSING.md). + +## TL;DR + +``` +train_command() (main.py) +└── retry loop (training.max_retries, training.retry_delay) + └── run_training_pipeline() → _execute_training_stages(): + 1. vae_rounds — for each round: get data → epochs → per-round plots → checkpoint + 2. vae_plots — final loss/stability/injection plots, latent GIF, latent traversal + 3. rf_train — generate RF data → encode latents → fit RF → persist model+artifacts + 4. rf_plots — the ten RF diagnostics + 5. final_save — final models + config_{tag}.json +``` + +Stages 1/3/5 are critical (failures raise → retry); 2/4 are non-critical (failures are +recorded in the manifest and retried on the next run, but never cost a data regeneration). +Every stage is skipped if the persisted manifest (`run_state_{tag}.json`) already records it +as done, so re-running the identical command resumes exactly where the last attempt died. + +## Round lifecycle + +`TrainingPipeline.train_beta_vae()` runs `training.num_training_rounds` rounds (default 20) of +`training.epochs_per_round` epochs (default 100). Each round (`train_round()`): + +1. **Obtain data** — reuse a validated on-disk round dataset if one exists, else wait on the + background producer (or generate in-process when overlap is disabled). +2. **Queue the next round** — generation of round *k+1* is requested immediately, so it runs + in the producer process while round *k* trains. +3. **Build datasets** — `prepare_distributed_train_dataset()` over the round's memmaps + (stratified 80/20 train/val split on the labels array). +4. **Prepare the latent-viz batch** (first round only) — 240 held-out val cadences per signal + type, persisted across rounds so latent-space snapshots aren't confounded by curriculum + distribution shift. +5. **Epoch loop** — `_train_epoch()` + `_validate_epoch()`, ~21 `training_stats` rows per + epoch (losses, gradient norms, LR, durations, SNR range), adaptive LR update. +6. **Per-round plots** — loss curves, training stability, injection stats (tagged + `round_XX`, saved under `plots/checkpoints/`), plus the latent traversal when + `--latent-traversal-every-round` is set. +7. **Checkpoint** — `save_models(tag="round_XX", dir="checkpoints")`, then the round is + recorded in the run manifest (`completed_rounds`). +8. **Cleanup** — holders cleared, `tf.keras.backend.clear_session()`, pools reset, and the + round's data directory deleted (unless `--keep-round-data`). + +At the start of each round the learning rate resets to `training.base_learning_rate` and the +adaptive-LR state (`best_val_loss`, `patience_counter`) is dropped — each curriculum stage is +a fresh optimization problem. Adam moments are curriculum-stage-local by the same reasoning. + +### Curriculum schedules + +`_calculate_curriculum_snr(round_idx)` narrows the injection SNR range from +`initial_snr_range` (40) down to `final_snr_range` (10) above `snr_base` (10) across the +rounds — early rounds see bright, easy signals; late rounds see predominantly faint ones. +Three schedules (`--curriculum-schedule`): + +| Schedule | Behavior | Knobs | +| --- | --- | --- | +| `linear` | Uniform narrowing: `range = initial - progress·(initial - final)` with `progress = round_idx / (total - 1)`. | — | +| `exponential` (default) | Fast-then-slow decay, normalized so progress 0 and 1 hit the exact endpoints: `range = final + (initial - final) · (e^{r·p} - e^{r}) / (1 - e^{r})`. More negative `r` = fewer easy rounds. | `exponential_decay_rate` (must be < 0; default −3.0) | +| `step` | `initial_snr_range` for the first `step_easy_rounds`, then `final_snr_range` for `step_hard_rounds`. The two must sum to `num_training_rounds`. | `step_easy_rounds`, `step_hard_rounds` | + +Each injected signal draws `snr = snr_base + U(0,1) · snr_range`, so the *floor* stays fixed +while the ceiling tightens. The per-round floor/ceiling is written to both `training_stats` +and `injection_stats` and shows up as background shading on the training plots. + +## Round data: memmaps + background producer + +A full-scale round is three arrays (`main`, `true`, `false`) of shape +`(499200, 6, 16, 512)` float32 ≈ 98 GB each — ~294 GB per round. Holding that in RAM is what +used to OOM-kill 503 GB training nodes; instead each round lives on disk under +`{round_data_dir}/{save_tag}/round_{k:02d}/` (default root `{output_path}/round_data`): + +``` +round_02/ +├── main.npy true.npy false.npy # (n, 6, 16, 512) float32 memmaps +├── main_lognorm.npy true_lognorm.npy false_lognorm.npy # (n, 6, 2) log-norm params +├── labels.npy # (n,) signal-type strings +└── round_02.done # atomic JSON manifest +``` + +Key properties (all in [`round_data.py`](../src/aetherscan/round_data.py) / +[`data_generation.py`](../src/aetherscan/data_generation.py)): + +- **Workers write straight into the memmaps.** `generate_round_to_memmap()` dispatches + batched tasks (`training.data_gen_task_size` cadences each, default 256) covering disjoint + row ranges; each worker opens the `.npy` in `r+` mode, writes its rows in place, and + returns only small stats dicts. No per-sample IPC pickling, one `pool.map` barrier per + chunk. +- **The `.done` manifest is the completion contract.** Written atomically + (`.tmp` → `os.replace`) only after every chunk finishes; it records shapes, SNR params, and + cheap sampled checksums. `validate_done_manifest()` re-checks all of it — a directory + without a valid manifest is garbage and gets regenerated. +- **Page-cache-backed reads.** Training opens the arrays with `np.load(mmap_mode="r")`; + after the first epoch the OS caches the round in otherwise-free RAM, so steady-state reads + run at RAM speed — but under memory pressure the kernel evicts pages instead of OOM-killing + the process. +- **Disk budget.** ~295 GB per round at defaults, ~2 rounds on disk at once with overlap + (~590 GB peak). `cli.py:collect_validation_errors` checks free space at startup + (`_estimate_round_data_nbytes`: 2.2× one round with overlap, 1.1× without) and hard-fails + with the computed numbers. Round *k*'s directory is deleted as soon as round *k* finishes + training (`--keep-round-data` retains it for debugging). + +### The producer process + +`RoundDataProducer` generates round *k+1* while round *k* trains, and isolates generation from +the trainer's GIL (TF's prefetch/callback threads used to make round-2+ generation far slower +than round 1's): + +- A **spawn**-started `multiprocessing.Process` (never fork — the TF/NCCL/CUDA-laden parent + holds locks a forked child can inherit mid-acquisition and deadlock on). The producer owns + a private fork-started worker pool whose workers attach to the background-plate shared + memory created by the main process. +- Protocol over two spawn-context queues: main sends `("generate", round_idx, snr_base, + snr_range)` / `("shutdown",)`; the producer streams back `stats` (per class-segment + injection statistics), `progress`, and terminal `done`/`error` messages. +- **DB writes stay in the main process**: a drainer thread consumes the `stats` messages and + calls `data_generation.write_segment_stats()` — the DB writer queue is a thread + `queue.Queue`, not process-safe. The drainer runs while the GPUs compute, so injection-stat + writes are off the training critical path. +- The producer logs into its own spawn-context queue, relayed into the main process's + handlers by a `QueueListener`; `CUDA_VISIBLE_DEVICES` is blanked during the spawn so the + child's TF import can never initialize CUDA. +- Registered with the ResourceManager (`ManagedProcess`), so cleanup escalates + terminate → join → kill. + +`--no-overlap-data-generation` falls back to sequential in-process generation (the debugging +path, also used automatically when `manager.n_processes == 1`). + +## Distributed training + +### Datasets + +`prepare_distributed_train_dataset()` builds infinite generator-backed `tf.data` datasets that +yield **whole global batches** (`per_replica_batch_size × num_replicas` rows gathered from the +memmaps by fancy indexing), with a leading batch dimension in the output signature and no +`.batch()` call — cutting per-sample Python boundary crossings by the global batch size, +which is what fixed the historical 0–14 % GPU utilization. Randomness lives at the epoch +level (train indices reshuffled per pass); within a batch, indices are sorted for memmap read +locality (the model is order-invariant within a batch). + +The train/val split is **stratified** over the four signal types (generation lays labels out +contiguously per chunk, so a positional split would skew val), then trimmed to exact multiples +of `effective_batch_size` (train) and the global val batch size. With `shuffle=False` +(used by RF training) the yield order is pinned to the returned `train_indices`/`val_indices` +— the alignment contract that lets encoded latents be matched back to labels. + +### Gradient accumulation + +Each training step accumulates `accumulation_steps = effective_batch_size / +(per_replica_batch_size × num_replicas)` micro-batch gradients before applying +(`_train_epoch()` → `_distributed_train_step()` → `_apply_gradients()`), giving an effective +batch of 3072 at defaults regardless of per-GPU memory. Guards along the way: all-None +gradient micro-batches are skipped, accumulated gradients are averaged over successful +micro-steps, NaN/Inf gradients raise immediately, and the global gradient norm is clipped at +1.0 with the pre-clip norm recorded per step (that's the `clipping_rate` statistic). + +The divisibility preconditions (`effective_batch_size % (per_replica × replicas) == 0`, sample +counts divisible by batch sizes, etc.) are validated up front by +`cli.py:collect_validation_errors` — see [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md) for the +cross-replica constraint system and the fix proposer. + +### Adaptive learning rate + +`_update_learning_rate()` tracks validation total loss: if it fails to improve by +`min_pct_improvement` (0.1 %) for `patience_threshold` (3) consecutive epochs, the LR is +scaled by `1 − reduction_factor` (×0.8), floored at `min_learning_rate` (1e-6). The LR resets +to `base_learning_rate` (1e-3) at each round start. Rule of thumb from the docstring: the LR +can only bottom out within a round if +`base_learning_rate · (1 − reduction_factor)^(epochs_per_round / patience_threshold)` +reaches `min_learning_rate`. + +## Checkpointing, the run manifest, and retries + +### What gets saved when + +| Artifact | When | Where | +| --- | --- | --- | +| `vae_{encoder,decoder}_round_XX.keras` | End of every round | `{model_path}/checkpoints/` | +| `random_forest_{tag}.joblib` + `rf_eval_artifacts_{tag}.joblib` | End of `rf_train` | `{model_path}/` | +| `vae_{encoder,decoder}_{tag}.keras`, `random_forest_{tag}.joblib` | `final_save` stage | `{model_path}/` | +| `config_{tag}.json` (resolved config snapshot) | `final_save` stage | `{output_path}/` | +| `run_state_{tag}.json` | Updated after every stage/round transition | `{output_path}/` | + +### The run-state manifest + +[`run_state.py`](../src/aetherscan/run_state.py)`:TrainingRunState` persists (atomically, +`.tmp` → `os.replace`): + +- `run_start_time` — wall clock of **attempt 1**. `TrainingPipeline.__init__` seeds + `self.start_time` from it, so every DB query and plot spans the whole run, not just the + current attempt (earlier attempts' epochs stay in the loss curves). +- `attempt` — incremented per pipeline rebuild. +- `completed_rounds` — rounds whose checkpoint landed; the round loop resumes at + `max(completed_rounds) + 1`, reloading `round_{k:02d}` weights from `checkpoints/`. +- `stages_done` / `stages_failed` — drive the stage machine + (`train.py:_execute_training_stages`). + +On resume, `_init_run_state()` also calls +`db.mark_superseded(table, tag, round_ge=resume_round)` for `training_stats`, +`injection_stats`, and `latent_snapshots`: partial rows written by the dead attempt are +flagged so default queries (and therefore the plots) ignore them — otherwise re-run epochs +would appear twice and corrupt every curve. Rows from completed rounds stay live; they are +valid history. See [`DATABASE.md`](DATABASE.md) for the supersede mechanics. + +Explicit checkpoint flags (`--load-tag` / `--load-dir` [+ `--start-round`]) are the escape +hatch: they override the manifest, trim `completed_rounds` below the forced start round, and +clear `stages_done` so downstream stages re-run against the re-trained rounds. + +### Retry semantics + +`main.py:train_command` retries up to `training.max_retries` times with +`training.retry_delay` between attempts. Each attempt rebuilds the `TrainingPipeline` from +scratch (no corrupted in-memory state survives); the manifest tells the new pipeline where to +resume. Background plates are loaded once in `train_command` and reused across attempts. +Because the manifest is on disk, a **full process relaunch of the identical command resumes +identically** — the in-process loop and a crash-and-relaunch are the same code path. + +Non-critical plot stages (`vae_plots`, `rf_plots`) never trigger a retry: each plot in the +group is attempted even if a sibling fails (`_run_plot_group`), failures are recorded in +`stages_failed`, and `train_command` exits **nonzero at the very end** if any recorded failure +never recovered — artifacts can be lost loudly, but a broken plot can't cost a +data-regeneration cycle. `rf_train` resumes cheaply too: if the tag's RF joblib + eval +artifacts already exist from a previous attempt, they are loaded instead of regenerating +~`num_samples_rf` cadences and retraining (`try_load_rf_for_resume`). + +## Training plots — what each one shows + +Per-round copies (tagged `round_XX`) land in `{output_path}/plots/checkpoints/`; the +end-of-training set (tagged with the run tag) in `{output_path}/plots/`. Every figure is also +uploaded to the run's Slack thread. All of them query the DB with `start_time = +run_start_time`, so multi-attempt runs plot complete histories with superseded rows filtered +out. + +### `beta_vae_loss_curves_{tag}.png` + +Total loss (full-width top panel) plus reconstruction / KL / true-clustering / +false-clustering components (bottom row), train and val overlaid, epochs on the x-axis with +per-round SNR-range shading in the background. What to look for: + +- Both curves trending down within each round; **val tracking train** (a widening gap = + overfitting the current curriculum stage). +- Small upward steps at round boundaries are expected — the data gets harder; a *large* + sustained jump means the curriculum narrowed too fast (`exponential_decay_rate` too + negative). +- KL should settle to a moderate plateau: collapsing toward 0 means the posterior ignores the + input (posterior collapse — consider lowering `beta`); growing without bound means the + latent space isn't regularizing. +- True/false clustering losses should decay and stay low; if `true_loss` dominates late + rounds, the ON/OFF separation is failing on faint signals. +- Doubled/serrated series are the signature of stale rows from a failed attempt leaking in — + they should never appear now that resumes mark old rows superseded; if you see them, check + the `mark_superseded` warnings in the log. + +### `beta_vae_training_stability_{tag}.png` + +2×3 grid: gradient **clipping rate** across the top, gradient-norm mean/std/max across the +bottom, same SNR shading. What to look for: clipping rate near zero after the first epochs +(sustained clipping = LR too high for the stage); norm mean smooth and slowly decaying; +isolated max spikes are fine, but spikes that coincide with loss cliffs point at bad batches +or an injection bug. NaN/Inf gradients abort the epoch outright, so anything you see here was +at least finite. + +### Injection-stats figures (from `plot_injection_stats`, 8 PNGs) + +Bias/leakage analysis of the synthetic data itself, sourced from the `injection_stats` table +(intensity statistics captured at stage **A** = raw background, **B** = post-injection, +**C** = post-normalization; see [`PREPROCESSING.md`](PREPROCESSING.md)): + +| File | Contents | What to look for | +| --- | --- | --- | +| `injected_signal_characteristics_{tag}.png` | Distributions of realized SNR, drift rate, signal width, starting bin, slope, intercept for ETI vs RFI injections, plus background-index usage. | ETI and RFI parameter distributions should match (the classifier must not be able to tell them apart from injection parameters alone); background usage should be uniform. | +| `injection_stability_{tag}.png` | Per-round NaN/Inf sanitization rate per statistic + slope-clamping rate. | Both ≈ 0. A rising sanitization rate means numerically degenerate cadences; clamping spikes mean the drift-slope edge case is being hit unusually often. | +| `{signal_type}_global_intensity_distributions_{tag}.png` (×4) | 2×3 histograms of mean/median/std/MAD/skew/kurtosis at stages A/B/C for one signal type. | Stage C distributions should be near-identical **across** the four types — any statistic that separates the types at stage C is leakage the models could shortcut on instead of learning morphology. | +| `a_b_global_intensity_biases_{tag}.png` | A→B scatter (pre- vs post-injection) per statistic, colored by signal type, outliers always kept. | Points should hug the diagonal with a modest, SNR-consistent offset for injected classes. Big vertical excursions = injections that dominate the background (dynamic-range bug). | +| `final_global_intensity_biases_{tag}.png` | Stage-C box plots per statistic, compared across signal types. | Boxes should overlap heavily. Separated medians = the normalization didn't erase injection-strength cues. | + +### `latent_space_{obs,cadence}_nn{n}_md{m}_{tag}.gif` + +UMAP animations of latent-space evolution over training, one GIF per +(n_neighbors, min_dist) combination in the configured sweep, built from the +`latent_snapshots` table (the withheld viz batch is re-encoded every +`latent_viz_step_interval` steps): + +- **obs-level**: each point is one observation's 8-dim latent, 8 classes (4 signal types × + ON/OFF) — the VAE's view. +- **cadence-level**: each point is a cadence's 48-dim concatenated latent, 4 classes — the + RF's view. + +What to look for: over the animation, true-class ON points should drift away from OFF/false +points (the clustering loss doing its job); by the final frames the cadence-level view should +show 4 separable — not necessarily linearly — clusters. Classes collapsing back together in +late rounds mean the faint-SNR curriculum is destroying earlier structure. The fitted UMAP +models are persisted (`umap_*.joblib`) and reused by the RF decision-boundary plot and by +inference's latent-projection figure. + +### `latent_traversal_{signal_type}_{tag}.png` + `latent_traversal_spectra_{signal_type}_{tag}.png` + +Decoder-based interpretation of the latent dimensions (`plot_latent_traversal`, helpers +`build_traversal_latents` / `compute_traversal_panels` / `unpreprocess_traversal_panels`). +For each signal type: the class-mean latent `z_t` (mean encoder `z_mean` over that type's ON +observations) is nudged one dimension at a time, `z_t + s·σ_d·e_d` for steps +`s ∈ linspace(−max_sigma, +max_sigma, num_steps)` (defaults 3.0, 7; `num_steps` validated odd +so the center column is the exact unperturbed decode), and decoded: + +- The **waterfall grid** (`latent_dim × num_steps` panels, shared per-row color scale) shows + what each dimension *does*: scan a row and watch the reconstruction morph. +- The **spectra figure** (per-dim time-integrated spectra, one line per step) makes + brightness/width/position shifts quantitative at a glance. + +What to look for: each row should vary one interpretable property (signal brightness, drift, +width, position...) — that's the disentanglement `beta` buys. Rows that do nothing are dead +dimensions (latent capacity to spare); rows that change everything at once suggest an +entangled space. Display inversion is an honest approximation (stated on the figure): +downsampling is undone by ×8 nearest-neighbor repetition, and intensities are un-log-normed +only where per-observation parameters were recorded at generation time (the `*_lognorm.npy` +sidecars). Runs once at end of training (`vae_plots` stage); `--latent-traversal-every-round` +adds per-round copies. On a resumed run whose rounds all completed before the resume, the +in-memory viz batch never existed, so the plot skips with a warning. + +### RF diagnostics (`rf_plots` stage, 10 PNGs) + +All consume `rf_eval_artifacts_{tag}.joblib` (val features/labels/probas thresholded at the +**deployment** `classification_threshold`, not sklearn's 0.5 default); the five SHAP figures +share `rf_shap_values_{tag}.joblib` (computed once, cached). + +| File | Contents | What to look for | +| --- | --- | --- | +| `rf_confusion_matrices_{tag}.png` | Binary (2×2) and per-subtype (4×2) confusion matrices at the deployment threshold. | With the default 0.99 threshold expect conservative behavior: near-zero false positives at the cost of true-class recall. Check the subtype panel for *which* true class carries the misses (usually `true_eti_rfi`). | +| `rf_classification_curves_{tag}.png` | ROC + AUC, PR + AP, confidence histograms (overall and per subtype). | AUC/AP near 1 on synthetic val is normal; the interesting part is the confidence histograms — a clean bimodal split means the threshold placement is easy, mass near the threshold means candidate counts will be sensitive to it. | +| `rf_shap_summary_{tag}.png` | Beeswarm of top features driving P(true). | Features are `obs{i}_z{d}` (observation × latent dim). ON-observation features (obs 0/2/4) should dominate — that's the physics. OFF features ranking high means the RF keys on OFF-source structure (leakage or RFI shortcuts). | +| `rf_shap_dependence_{tag}.png` | Dependence panels for the top-K features, colored by the strongest interacting feature. | Smooth monotone-ish trends = healthy; vertical striping = the RF memorizing discrete latent values. | +| `rf_shap_interactions_{tag}.png` | Pairwise interaction matrix (diagonal = main effects). | Strong off-diagonal ON×OFF blocks mean the forest genuinely compares ON against OFF within a cadence (good — that's the ABACAD logic); a purely diagonal matrix means per-observation features alone are being used. | +| `rf_shap_loss_monitoring_{tag}.png` | Per-sample log-loss histogram by class + per-feature loss-increasing/decreasing decomposition. | The high-loss tail is your inspection queue; any feature whose net contribution *increases* loss is actively harmful. | +| `rf_shap_explanation_clustering_{tag}.png` | UMAP of SHAP explanation vectors, colored by subtype, markers for correct/incorrect. | Errors concentrated in one explanation cluster = a single confusable mode (fixable with targeted data); errors scattered everywhere = noise-floor performance. | +| `rf_calibration_curve_{tag}.png` | Reliability diagram (quantile-binned) + Brier/ECE + probability histogram. | With a 0.99 threshold, calibration in the top bins is what matters: if the top-bin empirical frequency is well below its predicted probability, the threshold is less conservative than it looks. | +| `rf_ensemble_accuracy_curve_{tag}.png` | Cumulative accuracy vs number of trees (val + train-subsample baseline). | Should saturate well before 1000 trees; if it's still climbing at the end, raise `rf.n_estimators`. | +| `rf_latent_decision_boundary_nn{n}_md{m}_{tag}.png` | RF P(true) contour over each persisted cadence-level UMAP plane, val points + 0.5 contour. | A coherent boundary separating the true classes; ragged islands = the forest partitioning noise. Depends on the UMAPs from `plot_latent_space_gif`, so `vae_plots` must have succeeded. | + +### `resource_utilization_{tag}.png` + +Written by the resource monitor at shutdown, not by `train.py` — see +[`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md). Read it alongside the log timeline: data +generation shows as CPU-saturated plateaus, epochs as GPU-utilization bands, and (with +overlap enabled) the two should visibly coincide from round 2 onward. + +## Random Forest training (`rf_train` stage) + +`train_random_forest()` generates a fresh dataset (`num_samples_rf`, default 99 840; SNR range += `initial_snr_range` — the wide range, so the RF sees the full difficulty spectrum) into +`round_data/{tag}/rf/` using the same memmap machinery (in-process; the producer has already +shut down). It reuses `prepare_distributed_train_dataset(shuffle=False)`, encodes train and +val cadences through the (frozen) encoder with `_distributed_encode` — note the +`train_steps × accumulation_steps` step-count correction, guarded by an exact-count assertion +— fits the RF on binary labels (`true_*` vs `false_*`), and persists the model plus the eval +artifacts immediately so a retry can skip straight to plots. A `check_encoder_trained()` +heuristic (weight-std deviation from initializer expectations) guards against accidentally +encoding with untrained weights and falls back to loading the newest checkpoint. + +## Configuration quick reference + +Training-specific fields live on `TrainingConfig` +([`config.py`](../src/aetherscan/config.py)); flag routing and validation are documented in +[`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md). The load-bearing groups: + +| Group | Fields | +| --- | --- | +| Scale | `num_training_rounds`, `epochs_per_round`, `num_samples_beta_vae`, `num_samples_rf`, `train_val_split` | +| Batching | `per_replica_batch_size`, `effective_batch_size`, `per_replica_val_batch_size` | +| Round data | `round_data_dir`, `overlap_data_generation`, `keep_round_data`, `signal_injection_chunk_size`, `data_gen_task_size` | +| Curriculum | `snr_base`, `initial_snr_range`, `final_snr_range`, `curriculum_schedule`, `exponential_decay_rate`, `step_easy_rounds`, `step_hard_rounds` | +| Adaptive LR | `base_learning_rate`, `min_learning_rate`, `min_pct_improvement`, `patience_threshold`, `reduction_factor` | +| Latent viz / traversal | `latent_viz_*`, `latent_traversal_every_round`, `latent_traversal_num_steps`, `latent_traversal_max_sigma` | +| RF plots | `shap_max_samples_*`, `shap_top_k_features_dependence`, `rf_decision_boundary_*` | +| Fault tolerance | `max_retries`, `retry_delay` | From e67ffb4e260e645967b60bda30239f15143273bd Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:46:28 +0800 Subject: [PATCH 03/20] Add docs/INFERENCE_PIPELINE.md: catalog grouping, streaming flow, manifest retries, viz suite Co-Authored-By: Claude Fable 5 --- docs/INFERENCE_PIPELINE.md | 232 +++++++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 docs/INFERENCE_PIPELINE.md diff --git a/docs/INFERENCE_PIPELINE.md b/docs/INFERENCE_PIPELINE.md new file mode 100644 index 0000000..7562e8c --- /dev/null +++ b/docs/INFERENCE_PIPELINE.md @@ -0,0 +1,232 @@ +# Inference Pipeline + +This document covers the `inference` command: the CSV catalog format and cadence grouping, +the per-cadence streaming architecture, model loading, classification-threshold semantics, +the per-cadence run manifest that makes retries seamless, and every artifact/figure an +inference run produces. Orchestration lives in +[`src/aetherscan/main.py`](../src/aetherscan/main.py)`:inference_command()` / +`_run_streaming_csv_inference()`, the GPU stage in +[`src/aetherscan/inference.py`](../src/aetherscan/inference.py), preprocessing in +[`src/aetherscan/preprocessing.py`](../src/aetherscan/preprocessing.py) (see +[`PREPROCESSING.md`](PREPROCESSING.md)), and the visualization suite in +[`src/aetherscan/inference_viz.py`](../src/aetherscan/inference_viz.py). + +## TL;DR + +``` +inference_command() (main.py) +└── retry loop (inference.max_retries, inference.retry_delay) + └── _run_streaming_csv_inference(): + units = plan_cadences() # CSV rows → cadence work units, no work yet + skip units with a live 'inferred' manifest row for this tag + pipeline = InferencePipeline(strategy) # encoder + RF loaded once + for each pending cadence (prefetch depth 1): + [prefetch thread] preprocess cadence i+1 (energy detection → stamp .npy) + [main thread] load cadence i's stamps → encode on GPUs → RF → + write inference_results (positives) + inference_cadences row + render_inference_visualizations() # on a fully successful pass +``` + +Peak memory is one cadence's stamps plus the next cadence's in-flight preprocessing — +**independent of catalog size**. Two input modes: + +- `--inference-files ` — the production path described here: raw `.h5` + observations grouped into cadences, energy-detection preprocessing, streaming inference. +- `--test-files ` — legacy path for already-preprocessed arrays: one + `run_inference_pipeline()` shot over the whole file, no manifest/streaming machinery. + +## Input: CSV catalogs and cadence grouping + +Each entry in `config.data.inference_files` is a CSV (resolved via +`config.get_inference_file_path`, i.e. `{data_path}/inference/`) whose rows describe +individual `.h5` observations. `preprocessing.group_observations_from_csv()` groups rows by +the joint value of `inference.cadence_group_by_cols` (default +`["Target", "Session", "Band", "Cadence ID", "Frequency"]`) and reads each row's file path +from `inference.cadence_h5_path_col` (default `".h5 path"`): + +- Rows are assumed **already ordered** within each group (ABACAD order in the source CSV). +- Groups with exactly `inference.cadence_expected_obs` (6) rows become valid + `CadenceGroup`s; any other count is **flagged and skipped** with a warning — a cadence with + missing or extra observations can't be scored. +- The function is column-agnostic beyond the configured names; a missing required column + raises `KeyError` for that CSV (logged, and the CSV is skipped). + +`DataPreprocessor.plan_cadences()` turns the valid groups into `PendingCadence` work units, +each paired with a deterministic output path +`{output_dir}/{csv_stem}_{sanitized_group_key}.npy`. The output directory default is +**tag-scoped per CSV** — `{data_path}/inference/preprocessed/_/` — so a +same-tag retry resumes off its own stamps while a fresh run (new datetime tag) starts clean +and can never pick up a dead run's partial output. Pass `--preprocess-output-dir` explicitly +to share/reuse one directory across runs (the deliberate trade of tag isolation for +cross-run caching). + +## Model loading + + + +`InferencePipeline.__init__` → `init_models()` loads the two models from explicit paths: + +- `--encoder-path` → `tf.keras.models.load_model` **inside `strategy.scope()`** (the hard + rule: all TF model creation/loading happens in scope so variables are mirrored across + replicas). The `Sampling` layer is registered serializable, so no `custom_objects` needed. +- `--rf-path` → `RandomForestModel.load()` (joblib). +- `--config-path` → the training run's `config_{tag}.json`, layered onto the singleton by + `cli.apply_saved_config()` **before validation** so shape-critical fields + (`width_bin`, `stamp_width`, `latent_dim`, `dense_layer_size`, ...) match what the encoder + was trained with. The saved `checkpoint` section is deliberately skipped — most damagingly + `save_tag`: without the skip, an inference run would masquerade under the training run's + tag, corrupting DB provenance and output paths. The CLI `--save-tag` (or the default + timestamp) stays authoritative. + +All three paths are required by `collect_validation_errors` and must exist on disk. The three +artifacts should carry the same training tag; nothing enforces it yet (`# TODO` in +`inference_command`), so mismatched encoder/RF pairs are on the operator. + +## The streaming loop + +`_run_streaming_csv_inference()` (main.py) is the production driver: + +1. **Resume filter.** `db.flush()` (so rows queued by an in-process retry are visible), then + every unit with a live `status='inferred'` row in `inference_cadences` for + `(tag, npy_path)` is skipped outright; its stored aggregates (`n_stamps`, + `n_candidates`, confidence summary) fold into the run totals and the viz collector. +2. **Models load once** (`InferencePipeline`), reused by every cadence. The distributed + encode step is a lazily-built, cached `tf.function` — repeated `run_inference` calls reuse + one trace instead of retracing per cadence. +3. **Persistent worker pool.** `preprocessor.start_energy_detection_pool()` starts the single + pool that serves energy detection *and* stamp extraction for the whole run — started from + the main thread before the prefetch thread exists (forking after background threads spin + up risks children inheriting mid-operation locks). +4. **Prefetch depth 1.** A one-worker `ThreadPoolExecutor` runs + `process_pending_cadence(unit i+1)` (CPU-bound work happens in the pool's child processes; + the thread mostly waits) while the main thread loads, encodes, and classifies cadence *i* + on the GPUs. +5. **Per-cadence inference** (`main.py:_infer_cadence`): + - provenance derived from the group key + metadata JSON + (`preprocessing.derive_cadence_provenance`): target, session, band, cadence id, header + `tstart`, first `.h5` path, and the per-stamp center frequencies; + - stamps loaded via `load_inference_data(override_filepaths=[npy_path])` (log-norm only + for downsampled stamps; legacy full-width files also get downsampled — see + [`PREPROCESSING.md`](PREPROCESSING.md)); + - `mark_superseded("inference_results", tag, npy_path=...)` — partial positives from a + dead attempt are retired *before* fresh rows land; + - `run_inference()` (below), then a superseding `inference_cadences` row with + `status='inferred'` carrying the aggregate stats. +6. **Failure containment.** A cadence whose inference stage throws is logged, recorded as + `status='failed'` in the manifest, and the loop moves on — one bad cadence never aborts + the catalog. After the loop, the pass raises so the retry loop re-attempts, and the + manifest skip means **only the failed cadences re-run**. Permanent conditions (no work + units from the CSVs, no cadence produced stamps) raise `NonRetryableInferenceError`, which + fails fast instead of burning retries. +7. **Visualization.** On a fully successful pass (and `inference.inference_viz_enabled`), + `render_inference_visualizations()` draws the whole suite, every figure individually + exception-guarded. + +The retry loop wraps all of it: transient failures (I/O hiccups, GPU errors) retry up to +`inference.max_retries` with `inference.retry_delay` between passes; `KeyboardInterrupt` +propagates; state-based resume (stamp `.npy` for preprocessing, manifest rows for inference) +makes an in-process retry and a full relaunch of the identical command behave identically. + +## The GPU stage: `InferencePipeline.run_inference()` + +For one cadence's snippet array `(n, 6, 16, 512)`: + +1. `prepare_distributed_inf_dataset()` pads the array with duplicate rows (cycled from the + front) up to the next global-batch multiple (`per_replica_batch_size × num_replicas`, + default 2048 per replica), so the final partial batch is **encoded rather than silently + dropped** — with per-cadence batches, a cadence smaller than one global batch would + otherwise process nothing at all. Order is preserved (no shuffle). +2. `_distributed_encode()` runs the cached encode step over the distributed dataset; each + snippet's 6 observations go through the encoder as independent `(16, 512, 1)` inputs and + come back as sampled latents `z`. Per-replica results are gathered with + `experimental_local_results` + `np.concatenate` (cheaper than an NCCL gather for the tiny + latent payload), then truncated back to `n × 6` rows to drop the padding. +3. `RandomForestModel.predict_proba()` over the cadence-level 48-feature vectors yields + `proba_true = P(true)` per snippet. +4. Threshold semantics: `predictions = proba_true > classification_threshold` (default + **0.99** — deliberately conservative: at BL scale, false positives are expensive and true + positives are expected to be vanishingly rare). The stored `confidence` is the + probability of the **predicted** class — `proba_true` for candidates, `proba_false` for + rejections — so a confident rejection also reads as ~1.0. +5. `_write_inference_results()` writes **positives only** to `inference_results` (a deliberate + size trade — see [`DATABASE.md`](DATABASE.md)), each row carrying: snippet index (== row + in the `.npy`), confidence, the flattened 48-dim latent vector, and full provenance + (target/session/band/cadence id/per-stamp frequency/observation timestamp/`h5` path/tag). + Aggregates for *all* snippets (count, threshold, above-threshold count, mean/min/max, + quantiles p01–p99 via `summarize_confidences`) go into the cadence's manifest row, so + run-level summaries never depend on the positives-only table. + +## The run manifest: `inference_cadences` + +One row per (cadence, stage transition), superseding predecessors +(see [`DATABASE.md`](DATABASE.md) for the schema): + +| `status` | Written when | Meaning on retry | +| --- | --- | --- | +| `preprocessed` | Stamp `.npy` + metadata land on disk | Preprocessing is done; skip to inference (the `.npy`'s existence is the actual resume key). | +| `inferred` | Inference stage completes (aggregates attached) | Cadence fully done; skipped entirely, aggregates reused. | +| `failed` | Inference stage threw for this cadence | Re-attempted on the next pass. | + +Ordering makes each transition safe: within the DB's single-writer FIFO queue, the +`mark_superseded` command flushes buffered rows first, so every row queued before it gets +flagged while later writes stay live (`Database.mark_superseded`). + +## Artifacts of an inference run + +| Artifact | Where | Notes | +| --- | --- | --- | +| Stamp arrays + metadata | `{data_path}/inference/preprocessed/_/*.npy` + `.json` | The `.json` carries hit provenance: stamp starts/frequencies/statistics/p-values, ED statistic histograms, raw/merged hit lists, the `.h5` header. | +| Candidate rows | `inference_results` table | Positives only, with latents + provenance. | +| Run manifest | `inference_cadences` table | Per-cadence stages, aggregates, durations. | +| Config snapshot | `{output_path}/config_{tag}.json` | The resolved (saved-config + CLI) view this run actually used. | +| Figures | `{output_path}/plots/inference/{tag}/` | The visualization suite below; also uploaded to the run's Slack thread. | +| Resource plot | `{output_path}/plots/resource_utilization_{tag}.png` | Written by the monitor at shutdown. | +| PFB response cache | `{output_path}/pfb_cache/pfb_response_*.npy` | Content-addressed; reused across runs. | + +## The visualization suite — what each figure shows + +[`inference_viz.py`](../src/aetherscan/inference_viz.py) renders at end of run from three +sources: the bounded in-memory `InferenceVizCollector` (per-cadence aggregates + a +reservoir-sampled latent pool, memory O(#cadences), candidates always kept), the durable +per-cadence metadata JSONs, and the DB tables — so figures also cover cadences that were +skipped by the resume. Every figure is wrapped in `_viz_safe` (log-and-swallow): a plot bug +can never kill a science run. + +| File | Contents | What to look for | +| --- | --- | --- | +| `ed_stat_distributions_{tag}.png` | Log-log histogram of the D'Agostino–Pearson k² statistic over **all** windows (not just hits), per-ON-file overlay + total, threshold line. | The bulk should be a compact low-k² mass (noise ≈ χ², df=2) with a long RFI tail. The threshold should sit far into the tail: if the noise bulk crosses it, the stamp count explodes; if one ON file's curve is shifted, that file has a bandpass/level problem. | +| `ed_hit_spectrum_{tag}.png` | Hit density vs frequency (MHz), pre- vs post-deduplication. | Instantly shows RFI comb structure (regular spikes) and band edges. Dedup should collapse combs dramatically; a band where post-dedup density is still high dominates your stamp budget. | +| `bandpass_flattening_{tag}.png` | Raw vs flattened integrated spectrum for a few sampled coarse channels, with the removed model (scaled PFB response H or spline fit) overlaid. | The flattened spectrum should be level across the channel. Residual scalloping under PFB means the static response doesn't match the recording — check `--pfb-taps-per-channel` or fall back to `--bandpass-method spline` (the log's edge/mid-ratio warning fires on the same condition). | +| `stamp_gallery_{tag}.png` | Top-K stamps by detection statistic (`stamp_gallery_top_k`, default 12), each a 6-observation waterfall strip; overlap-offset copies collapsed first. | The cadence layout scientists actually inspect: a real technosignature shows in ONs (rows 0/2/4) and vanishes in OFFs; the top of this gallery is virtually always bright RFI present in all six — that's expected. | +| `preproc_funnel_{tag}.png` | Per-cadence bar funnel: raw hits → merged hits → stamps (incl. overlap copies) → snippets inferred, plus storage per cadence. | Where the volume goes. A weak merge step (raw ≈ merged) means hits are spread out rather than comb-like; snippets ≪ stamps indicates load-time validity rejections. | +| `confidence_distribution_{tag}.png` | P(true) histogram over all snippets inferred this pass (log-y), threshold line, per-cadence overlay when ≤ 10 cadences. | Mass should hug 0 with a thin bridge toward 1. Any mass just *below* threshold is worth manual inspection; a large mass above it usually means model/data mismatch (e.g. wrong config JSON) rather than a sky full of signals. | +| `candidate_gallery_{tag}.png` + `candidate_{i}_{tag}.png` | Gallery of top candidates by confidence + up to `max_candidate_plots` (50) per-candidate figures: 6-panel waterfall annotated with confidence, frequency, target/session/band, and the latent bar chart. Sourced from `inference_results`, so resumed cadences are included. | The human veto stage. Check the ON/OFF pattern by eye, the frequency against known RFI allocations, and whether the latent vector resembles the true-class latents from training. | +| `inference_latent_projection_{tag}.png` | This run's cadence-level latents projected through the **training run's persisted UMAP** (`umap_cadence_nn*_md*_*.joblib`, located via the training config JSON's `model_path` + tag), over the training embedding as backdrop; candidates highlighted. Skips gracefully if the UMAP is absent. | "Where does real data live relative to the synthetic classes?" Real snippets clustering onto the training false-class region = healthy. Candidates far from *any* training class are the interesting anomalies; candidates inside the false-class cloud are threshold noise. | +| `inference_summary_{tag}.png` | Table-style run card: cadence/snippet/candidate counts, per-stage durations and throughput from the manifest, per-target/band candidate counts. | The one-glance run report — read it before opening anything else. | + +## Legacy `--test-files` path + +When `--inference-files` is not set, `inference_command` falls back to loading +`config.data.test_files[0]` (a preprocessed `.npy` under `{data_path}/testing/`) in one shot +and calling `inference.run_inference_pipeline()` once. No manifest, no streaming, no viz +suite; the load repeats on retry. This path exists for model smoke-testing against curated +arrays; the CSV path is the production surface. + +## Configuration quick reference + +Inference-specific fields live on `InferenceConfig` +([`config.py`](../src/aetherscan/config.py)); routing details in +[`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md): + +| Group | Fields | +| --- | --- | +| Models | `encoder_path`, `rf_path`, `config_path` | +| Classification | `per_replica_batch_size`, `classification_threshold` | +| Cadence grouping | `cadence_group_by_cols`, `cadence_h5_path_col`, `cadence_expected_obs` | +| Energy detection | `coarse_channel_width`, `parallel_coarse_chans`, `bandpass_method`, `pfb_taps_per_channel`, `bandpass_debug_plot`, `spline_order`, `detection_window_size`, `detection_step_size`, `stat_threshold` | +| Stamps | `stamp_width`, `store_downsampled_stamps`, `overlap_search`, `overlap_fraction`, `preprocess_output_dir` | +| Visualization | `inference_viz_enabled`, `stamp_gallery_top_k`, `max_candidate_plots` | +| Fault tolerance | `max_retries`, `retry_delay` | From 6afec8eba3ce087cca81ae877b5f4c29d640d363 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:48:25 +0800 Subject: [PATCH 04/20] Add docs/PREPROCESSING.md: energy detection math, PFB vs spline, injection stages Co-Authored-By: Claude Fable 5 --- docs/PREPROCESSING.md | 283 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 docs/PREPROCESSING.md diff --git a/docs/PREPROCESSING.md b/docs/PREPROCESSING.md new file mode 100644 index 0000000..0ee330d --- /dev/null +++ b/docs/PREPROCESSING.md @@ -0,0 +1,283 @@ +# Preprocessing + +This document covers everything that happens to data before a model sees it, on both sides of +the pipeline: **energy detection** (inference: raw `.h5` filterbank files → stamp `.npy` +arrays), including the bandpass-flattening math (spline vs PFB) and the vectorized +D'Agostino–Pearson normality test, and **signal injection** (training: real backgrounds → +labeled synthetic cadences), plus the background/stamp loading paths. Code: +[`src/aetherscan/preprocessing.py`](../src/aetherscan/preprocessing.py), +[`src/aetherscan/pfb.py`](../src/aetherscan/pfb.py), +[`src/aetherscan/data_generation.py`](../src/aetherscan/data_generation.py). + +## Energy detection (inference side) + +Goal: reduce a cadence of six ~1-billion-bin observations to a few thousand +`stamp_width`-wide frequency windows worth encoding. Detection runs on the **ON-source** +observations only (ABACAD positions 0/2/4); the hits found there define the slices extracted +from all six observations, so the models see the full ON/OFF context at each hit frequency. + +Per cadence (`DataPreprocessor._process_cadence`), the chain is: + +``` +for each ON file: (one fused pool task per coarse channel) + read h5 slice → remove DC spike → flatten bandpass (pfb | spline) + → vectorized k² over sliding windows → threshold → per-channel hit list +aggregate hits → deduplicate → build stamp centers (± overlap offsets) +→ parallel memmap stamp extraction (downsample-at-extraction) → .npy + metadata .json +``` + +One **persistent worker pool** (started once per run by `start_energy_detection_pool()`) +serves every channel of every file of every cadence, and the extraction stage reuses it. Each +task returns only a small hit list plus a fixed-size statistic histogram — the bulky +`(time_bins, coarse_channel_width)` intermediates never leave the worker, so there is no +shared memory and no block-sized array in the parent at all +(`_energy_detect_channel_worker`). + +### Coarse channels and the DC spike + +Filterbank files are processed in **coarse channels** of `inference.coarse_channel_width` +(default 1 048 576) fine bins — the natural unit of the telescope backend's channelization +(`n_coarse = nchans // coarse_channel_width`; every complete coarse channel is processed). +Each coarse channel carries a 2-bin **DC spike** at its center, an FFT artifact of the +backend. `_remove_dc_spike` interpolates over it in place, pulling replacement values from +bins ±2/±3 away (the immediate neighbors are themselves contaminated; the asymmetric offsets +match the reference implementation byte-for-byte). The interpolation reaches at most ±3 bins +around the channel center, so per-channel processing touches exactly the same bins the +historical block-based implementation did. + +### Bandpass flattening — spline vs PFB + +The backend's polyphase filterbank imprints a scalloped passband shape onto every coarse +channel. Left in place, that shape — not astrophysics — would dominate the normality +statistic, so each channel is flattened first. The stage is pluggable +(`_get_bandpass_flattener()` returns a picklable callable shipped to the workers); two +implementations exist, selected by `--bandpass-method`: + +**Spline (`spline`, the historical data-driven method).** For a channel +`X ∈ ℝ^{time_bins × W}`: + +1. Time-integrate: `x̄ = mean_t X[t, :]` (shape `(W,)`). +2. Fit a cubic smoothing spline through `x̄` with interior knots every + `W // spline_order + 1` bins (`spline_order` default 16, i.e. ~16 knots across the + channel — smooth enough to follow the passband, too stiff to follow narrowband signals): + `scipy.interpolate.splrep(x, x̄, t=knots[1:])`. +3. Evaluate the spline at every bin and **subtract** it from every time row: + `X_flat = X − spl(x)`. + +Cost: one spline fit per coarse channel per file, every file. + +**PFB static equalization (`pfb`, the default).** The passband shape is a property of the +instrument's prototype filter, not of the data — so compute it once from the filter design +and **divide** it out ([`pfb.py`](../src/aetherscan/pfb.py), a native NumPy port of the bliss +reference implementation): + +1. **Prototype filter** (`firdes`): a windowed-sinc lowpass with + `N = taps_per_channel × num_coarse_channels` taps and cutoff `f_c = 1/num_coarse_channels` + (critically sampled PFB): + `h[n] = sinc(f_c · (n − (N−1)/2)) · w[n]`, `w` the Hamming window + `0.54 − 0.46·cos(2πn/(N−1))`. +2. **Response** (`gen_coarse_channel_response`): zero-pad `h` to + `num_coarse × fine_per_coarse` points; take the power spectrum + `|fftshift(FFT(h_pad))|²`; drop half a fine channel from each end so the remaining band is + an integer number of coarse-channel spans centered on channel boundaries; reshape to + `(num_coarse − 1, fine_per_coarse)` and **sum across spans** — the fold adds in + adjacent-channel leakage, which is why the method needs ≥ 2 coarse channels (files with one + fall back to spline with a warning); normalize to peak 1.0. The result `H` (shape + `(fine_per_coarse,)`) is `lru_cache`d per parameter tuple. +3. **Equalize** (`equalize_passband`): `X_flat = X / max(H, 10⁻¹⁰)` broadcast over time (the + floor guards against a pathologically sharp filter design driving edge bins to zero). + +At GBT scale the one-time FFT is an ~`n_chans`-point transform with a tens-of-GB transient, +so it runs **once in the parent**, is persisted to a content-addressed sidecar +(`{output_path}/pfb_cache/pfb_response_w{W}_c{C}_t{T}.npy`, atomic write), and workers just +read the ~8 MB file (cached per process by `_load_pfb_response`). Afterwards, flattening a +channel is a single vectorized divide — versus a fresh spline fit per channel per file. + +**Why subtract-vs-divide doesn't change detection.** The spline path subtracts its fit; the +PFB path divides by `H`, leaving each channel its DC offset and a bin-dependent scale. The +detection statistic below is built from skewness and kurtosis — *central* moments (location +cancels) normalized by powers of the variance (scale cancels) — so only the bandpass *shape* +matters, and both methods remove exactly that. `taps_per_channel` (default 12, the +GBT/Breakthrough Listen value) is **instrument-dependent** and must match the backend that +produced the `.h5`; a cheap per-file sanity check (`_warn_on_pfb_response_mismatch`) compares +the measured edge/mid power ratio of a few sampled channels against the theoretical +response's (`pfb.edge_mid_power_ratio`) and warns once per file when they disagree by > 5 % — +the cue to fix the tap count or use `--bandpass-method spline`. `--bandpass-debug-plot` saves +a raw-vs-flattened overlay for visual confirmation. + +### The detection statistic: vectorized D'Agostino–Pearson + +Narrowband signals make the flattened residuals *non-Gaussian* within a small frequency +window. Detection therefore slides a window of `detection_window_size` (256) bins with step +`detection_step_size` (128) across each flattened channel — window *j* covering columns +`[j·step, j·step + window)` — and computes D'Agostino–Pearson's k² over each window's +flattened `(time_bins × window)` sample, flagging windows with +`k² > stat_threshold` (default 2048). + +The reference statistic is `scipy.stats.normaltest`: + +``` +k² = Z₁(g₁)² + Z₂(b₂)² ~ χ²(df=2) under normality +``` + +where `g₁ = m₃/m₂^{3/2}` (sample skewness), `b₂ = m₄/m₂²` (sample kurtosis), and `Z₁`/`Z₂` +are D'Agostino's and Anscombe–Glynn's normalizing transforms. Calling `normaltest` per window +in Python — ~8 190 windows per channel × 3 ON files × hundreds of channels, several array +allocations each — was the single largest cost of inference. `_sliding_normality_k2` +computes the identical statistic in closed form: + +1. **Constant n.** Every window has `n = time_bins × window_size` samples (4 096 at + defaults), so every n-dependent constant in `Z₁`/`Z₂` is a scalar computed once. +2. **Conditioning shift.** The channel mean is subtracted up front — central moments are + shift-invariant, and the `S₂/n − mean²`-style differencing below loses precision when + `|mean| ≫ std` (flattened residuals are near-zero-mean anyway; the shift makes it + unconditional). +3. **Per-column power sums.** `p_k[c] = Σ_t X[t,c]^k` for k = 1..4, accumulated row-by-row in + float64 (temporaries stay at `(W,)`). +4. **Block → window sums.** Columns are aggregated into blocks of size `step` (fast path, + when `step | window` — the default 128 | 256) or `gcd(window, step)` (general path), via + `np.add.reduceat`; window sums `S₁..S₄` are then length-`window/block` moving sums over + the block sums, sampled every `step/block` blocks — one entry per window, no long + cumulative accumulation. +5. **Raw → central moments.** With `μ = S₁/n`: + `m₂ = S₂/n − μ²`, `m₃ = S₃/n − 3μ(S₂/n) + 2μ³`, + `m₄ = S₄/n − 4μ(S₃/n) + 6μ²(S₂/n) − 3μ⁴`. +6. **Z transforms.** Elementwise transcriptions of + `scipy.stats._stats_py::skewtest/kurtosistest` (variable names follow scipy), then + `k² = Z₁² + Z₂²`. Zero-variance windows are forced to NaN, matching scipy (and preventing + a tiny negative `m₂` from float cancellation fabricating a huge k²). + +The per-window Python/scipy overhead collapses into a handful of large ndarray operations. +Equivalence is pinned by unit tests (`tests/unit/test_preprocessing.py`) to +`rtol = 1e-9` against `scipy.stats.normaltest` across several input distributions — since the +math is exact, hit sets on identical inputs match the historical loop **exactly**. p-values +(`chi2.sf(k², 2)`) are computed only for the hit subset (they're metadata, not selection). +Each worker also folds *all* finite window statistics into a fixed 121-edge log-spaced +histogram (`ED_STAT_HIST_EDGES`, 10⁻³–10⁹) — per-channel counts add, giving the run's full +statistic distribution for `ed_stat_distributions_{tag}.png` at negligible cost. + +### Deduplication and overlap stamps + +Hits from all three ON files are pooled and **greedily merged** +(`_deduplicate_hits`): hits sorted by fine-channel index; any hit within +`stamp_width // 2` bins of the previous survivor is merged into it, keeping the higher +statistic. One drifting signal (or one RFI comb tooth) therefore yields one stamp instead of +dozens of step-offset duplicates. + +Each surviving hit becomes a stamp center; with `overlap_search` (default on), two extra +copies offset by `±overlap_fraction × stamp_width` (±2 048 bins at defaults) are added so a +signal drifting out of one stamp's frame is centered in a neighbor. Out-of-band stamps are +dropped; centers are sorted by start index before extraction (sequential reads let the HDF5 +chunk cache reuse decompressed bitshuffle chunks across adjacent stamps). + +### Stamp extraction and storage math + +Extraction writes straight into a memmap-backed `.npy` of shape +`(n_stamps, 6, time_bins, stored_width)` float32, filled by pool workers over disjoint +(observation, stamp-range) slices (`_extract_stamps_worker`), with `.tmp` → `os.replace` +atomicity — the resume path treats the `.npy`'s existence as proof of a complete write. + +**Downsample-at-extraction** (`store_downsampled_stamps`, default on): each stamp is reduced +along frequency with `skimage.transform.downscale_local_mean(stamp, (1, downsample_factor))` +*before* writing, so `stored_width = stamp_width // downsample_factor` (512 at defaults). +Per stamp: + +``` +full width: 6 obs × 16 t × 4096 f × 4 B = 1.5 MiB +downsampled: 6 obs × 16 t × 512 f × 4 B = 0.1875 MiB (÷ 8) +``` + +A cadence yielding ~10⁴–10⁵ stamps (RFI-heavy bands, with overlap ×3) thus costs ~2–20 GB +instead of ~15–150 GB, and the separate downsample pass at load time disappears. The sibling +metadata JSON records `stored_width` and `downsample_factor_applied`; the `.npy` shape +self-describes, and the loader handles both layouts (below). Disable to archive raw-resolution +stamps. + +The metadata JSON also carries the full ED provenance for the visualization suite: per-stamp +starts/frequencies/statistics/p-values, raw and merged hit frequency lists, the per-ON-file +statistic histograms, and the `.h5` header. + +## Loading paths + +**Training backgrounds** (`load_train_data`): each file in `data.train_files` +(`{data_path}/training/*.npy`, raw width 4096) is memory-mapped and processed in chunks of +`background_load_chunk_size` (15 000 cadences): the chunk is copied into a shared-memory +block, and a pool downsamples each cadence ×8 per observation (`_downsample_worker`). +Cadences containing NaN/Inf or with non-positive max are dropped. **Log-normalization is +deferred** — training-side log-norm happens per sample *after* signal injection (injection +must operate on linear intensities). Loading stops at `num_target_backgrounds` (45 000) +cadences ≈ 34 GB at model resolution, held in RAM for the whole run and shared with injection +workers via shared memory. + +**Inference snippets** (`load_inference_data`): same chunked SHM+pool pattern, branching on +the stored width: files already at `width_bin // downsample_factor` (512 — written by +downsample-at-extraction) need only per-cadence log-norm, run vectorized in the workers +(`_lognorm_worker`); legacy full-width files (4096) keep the historical +downsample-then-log-norm path. Any other width is an error (skipped with a log). The loaded +array must survive strict [0, 1] / NaN / Inf checks before inference proceeds. + +**Log-normalization** (`data_generation.log_norm`): `y = log(x + 10⁻¹⁰)` shifted by its +minimum and scaled by its range into [0, 1], per observation. With `return_params=True` it +also returns `(min_log, range_log)`, enabling the approximate inversion +`x ≈ exp(y · range_log + min_log)` used by the latent-traversal display +([`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). + +## Signal injection (training side) + +Training data is synthetic: setigen injects drifting narrowband signals into real observed +backgrounds ([`data_generation.py`](../src/aetherscan/data_generation.py)). The atom is +`new_cadence(data, snr, ...)`, which injects **one** signal into a vertically stacked cadence +(`(96, 4096)` = 6 obs × 16 time rows): + +- `starting_bin ~ U(1, width_bin−1)`; drift direction ±1 with equal probability. +- The pixel-space slope runs the signal across the full stacked height: + `slope_pixel = total_time / starting_bin` (positive drift) or + `total_time / (starting_bin − width_bin)` (negative), converted to physical units via + `time_resolution / freq_resolution` plus `U(0, 3)` noise; near-zero physical slopes are + clamped to ±10⁻⁶ (recorded via `slope_clamped` so queries can filter them) and + `drift_rate = −1 / slope_physical`. +- `signal_width = U(0, 50) + 18·|drift_rate|` Hz — the drift-proportional term keeps fast + drifters coherent across a time bin. +- setigen frame: `constant_path` (linear drift), `constant_t_profile` at + `frame.get_intensity(snr=snr)` (intensity calibrated to the requested SNR against the + actual background noise), `gaussian_f_profile(signal_width)`, flat bandpass profile. +- Realized parameters (`snr`, `drift_rate`, `signal_width`, `starting_bin`, `slope_pixel`, + `y_intercept`) are returned as `signal_info` and written to `injection_stats`. + +Signals are injected into the *stacked* array, so a drifting line naturally traverses +observation boundaries — appearing at shifted frequencies in successive observations exactly +as a real drifting source would. + +### The four signal types + +| `signal_type` | Builder | Construction | Semantics | +| --- | --- | --- | --- | +| `false_no_signal` | `create_false(inject=False)` | Background, log-normed as-is | Pure noise | +| `false_with_rfi` | `create_false(inject=True)` | One signal in **all 6** observations | RFI: persists in OFFs | +| `true_only_eti` | `create_true_single` | One signal; ONs take the injected rows, OFFs keep the original background | Technosignature: ON-only | +| `true_eti_rfi` | `create_true_double` | RFI signal everywhere + ETI signal in ONs, re-drawn until the two tracks don't intersect inside any ON region (`check_valid_intersection`) | Technosignature under RFI | + +Each builder picks a random background plate, records **intensity statistics at three +stages** — **A** (raw background, pre-injection), **B** (post-injection, pre-normalization), +**C** (post-normalization, exactly what the model sees) — six global statistics each (mean, +median, std, MAD, skew, kurtosis, float64 to survive higher-order moments), plus the +per-observation log-norm parameters. The A/B/C staging is what powers the injection-bias +plots: A→B isolates what injection adds, B→C what normalization erases, and any statistic +that still separates the classes at stage C is leakage a model could shortcut on +(see [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). + +### Round generation layout + +`generate_round_to_memmap()` fills three arrays per round — `main` (4-way balanced across the +types, the reconstruction input), `false` and `true` (2-way balanced, feeding the clustering +losses; see [`MODELS.md`](MODELS.md)) — in chunks of `signal_injection_chunk_size` (50 000, +validated divisible by 4). Each chunk decomposes into 8 contiguous class segments +(`build_chunk_segments`: 4 quarters for `main`, 2 halves each for `false`/`true`), and each +segment into batched tasks of `data_gen_task_size` cadences (`build_segment_tasks`) submitted +as **one unified `pool.map` per chunk**. Each task carries a fresh RNG seed drawn in the +parent, so results don't depend on which persistent worker picks it up. Workers write rows +directly into the round memmaps and return only stats dicts; per-segment statistics flow back +through the producer's `stats` channel and are written to `injection_stats` by the +main-process drainer ([`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). The labels array +mirrors `main`'s contiguous per-chunk layout, which is what the stratified train/val split +keys on. From b7af030af34a7597b60f6ae90abfa897f7ee0335 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:49:52 +0800 Subject: [PATCH 05/20] Add docs/MODELS.md: Beta-VAE architecture and loss math, RF features and thresholds Co-Authored-By: Claude Fable 5 --- docs/MODELS.md | 199 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/MODELS.md diff --git a/docs/MODELS.md b/docs/MODELS.md new file mode 100644 index 0000000..caf667a --- /dev/null +++ b/docs/MODELS.md @@ -0,0 +1,199 @@ +# Models + +This document specifies the two models: the **Beta-VAE** feature extractor +([`src/aetherscan/models/vae.py`](../src/aetherscan/models/vae.py)) — architecture, the +composite loss including the clustering terms, and the design rationale — and the +**Random Forest** cadence classifier +([`src/aetherscan/models/random_forest.py`](../src/aetherscan/models/random_forest.py)) — +feature layout and threshold/confidence semantics — plus the meaning of the persisted +evaluation/SHAP artifacts. How the models are *trained* is +[`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)'s territory; how they are *served* is +[`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md)'s. + +## Design overview + +The division of labor: + +- The **Beta-VAE encoder** turns each observation spectrogram `(16, 512, 1)` into an + 8-dimensional latent describing its morphology. It never sees cadence structure directly — + it processes observations independently — but the *clustering loss* (below) shapes the + latent space so that ON/OFF structure becomes linearly visible in it. +- The **Random Forest** sees a whole cadence as the concatenation of its six latents + (48 features) and decides true (ETI-like: present in ONs, absent in OFFs) vs false + (noise/RFI). Trees over a 48-dim space are cheap, interpretable (SHAP), and robust at the + extreme class-imbalance operating point inference runs at. + +Hyperparameters live on `BetaVAEConfig` / `RandomForestConfig` +([`config.py`](../src/aetherscan/config.py)): `latent_dim` 8, `dense_layer_size` 512, +`kernel_size` (3, 3), `beta` 1.5, `alpha` 10.0; RF: 1000 trees, bootstrap, `max_features` +"sqrt", seeded (11). + +## Beta-VAE + +### Encoder (`build_encoder`) + +Input `(16, 512, 1)` — one observation, "grayscale". Nine Conv2D layers (four stride-2 +downsampling, five stride-1 feature layers), then dense compression to the latent posterior: + +| Enc layer | Filters | Stride | Input shape | Output shape | +| --- | --- | --- | --- | --- | +| 1 | 16 | 2 | (16, 512, 1) | (8, 256, 16) | +| 2 | 16 | 1 | (8, 256, 16) | (8, 256, 16) | +| 3 | 32 | 2 | (8, 256, 16) | (4, 128, 32) | +| 4 | 32 | 1 | (4, 128, 32) | (4, 128, 32) | +| 5 | 32 | 1 | (4, 128, 32) | (4, 128, 32) | +| 6 | 64 | 2 | (4, 128, 32) | (2, 64, 64) | +| 7 | 64 | 1 | (2, 64, 64) | (2, 64, 64) | +| 8 | 128 | 1 | (2, 64, 64) | (2, 64, 128) | +| 9 | 256 | 2 | (2, 64, 128) | (1, 32, 256) | + +Then `Flatten` → `(8192,)` → `Dense(dense_layer_size=512)` → two parallel heads +`z_mean` and `z_log_var`, each `Dense(latent_dim=8)`, and a `Sampling` layer producing `z`. +Every layer carries the same regularization stack: HeNormal init (GlorotNormal on the latent +heads), L1(0.001) activity regularization (sparse activations), L2(0.01) kernel + bias +regularization. `z_log_var`'s bias initializes at **−3.0**, tightening the initial posterior +around the prior so early training isn't swamped by sampling noise. + +### Sampling layer (reparameterization) + +`Sampling` implements the reparameterization trick: sampling is non-differentiable, so the +randomness is isolated in `ε ~ N(0, I)` and the sample expressed as a deterministic function +of the learned parameters: + +``` +z = z_mean + exp(0.5 · z_log_var) · ε +``` + +Gradients flow through `z_mean`/`z_log_var` while `ε` carries no parameters. The layer is +registered with `keras.utils.register_keras_serializable(package="aetherscan")`, so saved +`.keras` encoders load anywhere without `custom_objects` (this is what makes cross-machine +checkpoint interop in [`GPU_RUNTIME_GUIDE.md`](GPU_RUNTIME_GUIDE.md) work). + +### Decoder (`build_decoder`) + +An exact mirror: `Dense(512)` → `Dense(8192)` → `Reshape(1, 32, 256)` → nine +`Conv2DTranspose` layers reversing the encoder table row-by-row (decoder layer *i* outputs +the *input* channel count of encoder layer *10−i*, with the same stride), ending in a +1-filter, stride-2 layer with **sigmoid** activation — the output is bounded to [0, 1] to +match the log-normalized input and license the BCE reconstruction loss. The symmetry is a +maintained invariant: any layer-structure change on one side must be mirrored on the other +(both docstrings say so, and the unit tests check output shapes). + +The decoder is not needed at inference time; it ships as `vae_decoder_{tag}.keras` because +the latent-traversal diagnostic decodes perturbed latents +([`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). + +### Composite loss (`BetaVAE.compute_total_loss`) + +Each training batch is a triplet `(main, true, false)` of cadence batches +(shape `(B, 6, 16, 512)` each; see [`PREPROCESSING.md`](PREPROCESSING.md) for how the three +arrays are generated): + +``` +total = reconstruction(main) + β · KL(main) + α · (L_true(true) + L_false(false)) +``` + +- **Reconstruction**: `main` is reshaped to `(B·6, 16, 512, 1)`, encoded, decoded, and scored + with binary cross-entropy summed over the spectrogram and averaged over the batch + (`from_logits=False`; the decoder output is already sigmoid-bounded). +- **KL divergence** against the standard-normal prior, closed form per dimension, summed over + the latent and averaged over the batch: + `KL = −½ · Σ_d (1 + log σ_d² − μ_d² − σ_d²)`. + `β = 1.5` is the Beta-VAE knob: β > 1 buys a more disentangled, better-regularized latent + space at some reconstruction cost — which is exactly the trade we want, since the latents + (not the reconstructions) are the product. +- **Clustering losses** — the SETI-specific part. Two primitives over per-observation latents + (the `z` samples of a cadence's six observations, encoded with `training=True`): + + ``` + loss_same(a, b) = mean ‖a − b‖² # minimized → pulls a and b together + loss_diff(a, b) = mean 1 / (‖a − b‖² + 1e-8) # minimized → pushes a and b apart + ``` + + With a cadence's latents labeled `a1, b, a2, c, a3, d` (ON, OFF, ON, OFF, ON, OFF): + + - **`L_true`** (`compute_clustering_loss_true`), applied to true-class cadences: the sum of + `loss_same` over all ordered ON/ON pairs (a1a2, a1a3, a2a1, a2a3, a3a1, a3a2) and all + ordered OFF/OFF pairs (bc, bd, cb, cd, db, dc), **plus** `loss_diff` over all nine ON×OFF + cross pairs. Minimizing it forces ON latents into one cluster, OFF latents into another, + and the two clusters apart — the latent-space encoding of "appears when pointed at the + target, disappears when pointed away". + - **`L_false`** (`compute_clustering_loss_false`), applied to false-class cadences: the same + pair enumeration but with `loss_same` **everywhere** (the nine cross pairs included) — in + a noise/RFI cadence, all six observations are morphologically the same thing, so all six + latents should collapse to a single cluster. + + `α = 10.0` weights the pair; the true/false batches feed only these heads (their gradients + don't flow through the reconstruction term). This is what makes the RF's job possible with + 1000 shallow trees: the class geometry is pre-built into the representation. + +Loss components are logged per epoch to `training_stats` and plotted separately +(`beta_vae_loss_curves_{tag}.png`), so a failing term is directly attributable. + +## Random Forest + +### Feature layout + +`prepare_latent_features()` reshapes `(n_cadences · 6, latent_dim)` per-observation latents +into `(n_cadences, 6 · latent_dim)` rows by concatenating each cadence's six latents +**in observation order** (row-major `.ravel()`): + +``` +feature index f = obs_idx · latent_dim + dim # obs 0..5 (ABACAD), dim 0..7 +[ obs0_z0 .. obs0_z7 | obs1_z0 .. obs1_z7 | ... | obs5_z0 .. obs5_z7 ] + ON OFF ON OFF ON OFF +``` + +The caller must keep rows `i·6 .. i·6+5` grouped as cadence *i* — both training +(`shuffle=False` datasets pin encode order to the index arrays) and inference (snippet-major +`.npy` order preserved through padding/truncation) uphold this contract. Features are indexed +`obs{i}_z{d}` in the SHAP plots; even-numbered observations are ON-source. + +Note the two models consume **different latent tensors**: the RF trains and infers on the +sampled `z`, while the latent-space visualizations use the deterministic `z_mean` — one is +the representation under the model's own noise, the other the noise-free embedding. + +### Classifier and thresholds + +`RandomForestClassifier(n_estimators=1000, bootstrap=True, max_features="sqrt", +random_state=11, n_jobs=-1)`, fit on binary labels (`true_*` subtypes = 1). Prediction +surfaces: + +- `predict_proba` → `(n, 2)` columns `[P(false), P(true)]`. With 1000 bootstrap trees the + probability resolution is fine enough for a 0.99 operating point to be meaningful. +- `predict(threshold=0.5)` / `predict_verbose(threshold=0.5)` — generic helpers; the + **pipeline never uses 0.5**. Inference thresholds at + `config.inference.classification_threshold` (default **0.99**): a snippet is a candidate iff + `P(true) > 0.99`. The RF diagnostics evaluate at this same deployment threshold so the + confusion matrix and error markers reflect production behavior, not sklearn's argmax. +- **Confidence semantics**: the `confidence` stored in `inference_results` is the probability + of the **predicted** class — `P(true)` for candidates, `P(false)` for rejections — so a + confident rejection also scores ~1.0. The full `P(true)` vector per cadence is summarized + (quantiles, above-threshold counts) into the `inference_cadences` manifest + ([`DATABASE.md`](DATABASE.md)); don't confuse the two columns. + +Why 0.99: candidates are forwarded to human review, real positives are expected to be +vanishingly rare, and the synthetic training distribution is easier than the real sky — the +threshold buys precision and the confidence-distribution / calibration plots +([`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md), [`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md)) +tell you what it costs in recall. + +## Persisted model artifacts + +| File | Producer | Contents / purpose | +| --- | --- | --- | +| `vae_encoder_{tag}.keras` | `save_models` | The inference feature extractor (outputs `z_mean, z_log_var, z`). | +| `vae_decoder_{tag}.keras` | `save_models` | Mirror decoder; needed only for latent traversal / reconstruction diagnostics. | +| `random_forest_{tag}.joblib` | `train_random_forest` / `save_models` | The fitted sklearn classifier (joblib of the bare `RandomForestClassifier`). | +| `rf_eval_artifacts_{tag}.joblib` | `train_random_forest` | Dict of train/val features, binary + subtype labels, val probabilities and threshold-consistent predictions, the threshold and SNR range used — the single source every RF plot consumes (and what lets a resumed run skip RF retraining). | +| `rf_shap_values_{tag}.joblib` | `_compute_or_load_shap_values` | Cached SHAP outputs: positive-class summary values (+ the val row indices they correspond to), pairwise interaction values, and a log-loss decomposition (`model_output="log_loss"`), normalized across shap versions by `_select_positive_class_shap`. Computing these is minutes of work; every SHAP figure reuses the cache. | +| `umap_{obs,cadence}_nn{n}_md{m}_{tag}.joblib` | `plot_latent_space_gif` | Fitted UMAP reducers per (n_neighbors, min_dist): obs-level (8-dim inputs) and cadence-level (48-dim). Reused by the RF decision-boundary plot and by inference's latent-projection figure — which is why deleting them breaks those two figures but nothing else. | + +Interpretation guidance for the SHAP/diagnostic figures lives with the plot catalog in +[`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md). The short version of what the SHAP artifacts +*mean*: summary values attribute each cadence's `P(true)` to individual `obs{i}_z{d}` +features (sign = direction, magnitude = influence); interaction values split attributions +into per-feature main effects (diagonal) and pure pairwise effects (off-diagonal — the +ON×OFF blocks are where cadence logic shows); the log-loss decomposition re-attributes *loss* +instead of probability, separating features the model uses well (loss-decreasing) from ones +it uses badly (loss-increasing). From 0bff201c8301a73331452198e21892731f402081 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:51:37 +0800 Subject: [PATCH 06/20] Add docs/DATABASE.md: schema, writer thread, flush/supersede protocols, migrations, growth Co-Authored-By: Claude Fable 5 --- docs/DATABASE.md | 261 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 261 insertions(+) create mode 100644 docs/DATABASE.md diff --git a/docs/DATABASE.md b/docs/DATABASE.md new file mode 100644 index 0000000..a482f0f --- /dev/null +++ b/docs/DATABASE.md @@ -0,0 +1,261 @@ +# Database + +This document covers Aetherscan's persistence layer +([`src/aetherscan/db/db.py`](../src/aetherscan/db/db.py)): the schema of every table, the +single-writer-thread design and its flush / mark-superseded protocols, the migration +mechanism, the query API, and how big to expect the database to get. + +## TL;DR + +One SQLite file — `{output_path}/db/aetherscan.db`, WAL mode — behind a thread-safe +`Database` singleton (`get_db()` accessor). All writes go through an in-process +`queue.Queue` drained by **one background writer thread** that batches rows and commits with +`executemany()`; reads open short-lived connections directly. Failed-attempt rows are never +deleted — they're flagged `superseded = 1`, and every query filters them out by default. +Schema evolution is a minimal `PRAGMA user_version` gate (currently version 2). + +> [!IMPORTANT] +> The write queue is a **thread** queue, not process-safe. Worker *processes* must never call +> `db.write_*` — stats produced in workers travel back over multiprocessing channels and are +> written by main-process threads (see the producer/drainer design in +> [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). + +## Writer-thread design + +**Why:** SQLite allows one writer at a time; concurrent writers from the monitor thread, the +training loop, and the round-data drainer would contend on `SQLITE_BUSY`. Serializing all +writes through a single consumer eliminates that class of failure and makes ordering +guarantees possible. + +`write_*` methods are non-blocking: they validate/sanitize their arguments and `put()` a +`(table, values_tuple)` record on the queue. The writer loop (`_writer_loop`) accumulates +records into a buffer and flushes when either the buffer reaches +`db.write_buffer_max_size` (100 records) or `db.write_interval` (5 s) elapses. A flush +(`_flush_buffer`) groups the buffer by table and bulk-inserts each group with a single +`executemany()` per table — SQL parsed once, one commit per flush; the batch is +all-or-nothing (errors are logged and the loop continues; a failed write never kills the +thread). On shutdown (`stop()`), the loop drains and performs a final flush. + +Lifecycle: `init_db()` constructs the singleton (schema init + migration) and `start()`s the +writer thread; the ResourceManager stops it during `cleanup_all()` — after the monitor (which +still writes samples) and before the logger (see +[`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md)). + +### Flush protocol + +Reads that must observe queued-but-unwritten rows (every plot function does this) call +`db.flush(timeout=...)`: a `(_FLUSH_SENTINEL, event)` tuple is queued; when the writer +dequeues it, it flushes the buffer immediately and sets the event. Queue FIFO ordering makes +the semantics exact: everything queued *before* the sentinel is on disk when `flush()` +returns True. Returns False on timeout or if shutdown began mid-wait — callers treat that as +"plot may be missing the newest rows", not as fatal. + +### Mark-superseded protocol + +`mark_superseded(table, tag, *, round_ge=None, npy_path=None)` implements the stale-data +semantics both retry systems rely on. It ships a command tuple through the same queue +(`_MARK_SUPERSEDED_SENTINEL`), so single-writer semantics are preserved and ordering is +airtight: the writer first flushes its buffer (every row queued before the command is +covered), then runs + +```sql +UPDATE {table} SET superseded = 1 +WHERE superseded = 0 AND tag = ? + [AND round_number >= ?] -- round_ge (training tables) + [AND npy_path = ?] -- npy_path (inference tables) +``` + +while rows queued *after* the command keep `superseded = 0`. The call blocks until the mark +lands (same timeout/shutdown semantics as `flush`); with no writer thread running it executes +inline. Table/filter combinations are whitelisted (`_SUPERSEDE_TABLES`): `round_ge` applies +to `training_stats` / `injection_stats` / `latent_snapshots`, `npy_path` to +`inference_results` / `inference_cadences`. + +Who calls it: + +- Training resume (`TrainingPipeline._init_run_state`): rows for `(tag, round >= resume)` in + the three training tables — a dead attempt's partial epochs would otherwise appear twice in + every curve. +- Inference retry (`main._infer_cadence`): `inference_results` rows for `(tag, npy_path)` + before fresh positives land, and the cadence's old manifest rows before the new + `status='inferred'` row. + +Rows are **never deleted** — pass `include_superseded=True` to any query to audit what a +failed attempt wrote. + +## Schema + +Six tables, created idempotently (`CREATE TABLE IF NOT EXISTS`) in `_init_database()`, each +with a composite index matched to its dominant filter pattern. All rows carry `timestamp` +(write time, `REAL` Unix seconds) and `tag` (the run's save tag — the primary provenance +key). + + + +### `system_resources` + +1 Hz samples from the resource monitor ([`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md)). + +| Column | Type | Notes | +| --- | --- | --- | +| `resource_type` | TEXT | `cpu` / `ram` / `gpu` | +| `resource_name` | TEXT | `system_total`, `process_tree`, or `{gpu_name}_{utilization,memory}` | +| `value`, `unit` | REAL, TEXT | Percentages throughout | +| `metadata` | TEXT | Optional JSON | + +Index: `(tag, timestamp)`. No `superseded` column — resource samples are attempt-agnostic +history. + +### `injection_stats` + +Signal-injection provenance, written per generated cadence by the round-data drainer +(`data_generation.write_segment_stats`). + +| Column | Type | Notes | +| --- | --- | --- | +| `stat_name`, `value` | TEXT, REAL | Intensity statistics (`global_{mean,median,std,mad,skew,kurtosis}`), signal characteristics (`eti_*` / `rfi_*`: snr, drift_rate, signal_width, starting_bin, slope_pixel, y_intercept), or segment metadata (`snr_range_floor/ceil`, `num_samples`, `inject_duration`) | +| `round_number`, `chunk_number`, `sample_index`, `background_index` | INTEGER | Position of the sample in the generation layout (`sample_index`/`background_index` NULL for segment metadata) | +| `signal_class`, `signal_type` | TEXT | `main`/`true`/`false`; `false_no_signal`/`false_with_rfi`/`true_only_eti`/`true_eti_rfi` | +| `injection_stage` | TEXT | `A` (raw background) / `B` (post-injection) / `C` (post-normalization); NULL for signal characteristics and metadata | +| `is_finite` | INTEGER | 0 when the value was NaN/Inf at write time (sanitized; filterable) | +| `slope_clamped` | INTEGER | 1 when the injection's drift slope hit the near-zero clamp | +| `superseded` | INTEGER | Default 0 | + +Index: `(tag, timestamp, stat_name, signal_type, injection_stage)`. + +### `training_stats` + +Per-epoch training telemetry (~21 rows per epoch): losses (`total_loss`, +`reconstruction_loss`, `kl_loss`, `true_loss`, `false_loss` + `val_` variants), gradient +statistics (`gradient_norm_{mean,max,std}`, `clipping_rate`), `learning_rate`, durations, +step counts, and the round's SNR floor/ceiling. + +| Column | Type | Notes | +| --- | --- | --- | +| `model_name` | TEXT | Currently always `beta_vae` | +| `stat_name`, `value` | TEXT, REAL | | +| `round_number`, `epoch_number` | INTEGER | 1-based | +| `superseded` | INTEGER | Default 0 | + +Index: `(tag, timestamp, model_name, stat_name)`. + +### `latent_snapshots` + +The withheld viz batch's latent vectors, captured every `latent_viz_step_interval` training +steps — the raw material of the latent-space GIFs. + +| Column | Type | Notes | +| --- | --- | --- | +| `round_number`, `epoch_number`, `step_number` | INTEGER | Capture coordinates | +| `cadence_index` | INTEGER | Position in the (persistent) viz batch | +| `signal_type` | TEXT | The cadence's class | +| `latent_vector` | TEXT | JSON: `(6, latent_dim)` `z_mean`, rounded to 8 decimals | +| `snr_base`, `snr_range` | INTEGER | Curriculum stage at capture time | +| `superseded` | INTEGER | Default 0 | + +Index: `(tag, timestamp, model_name, round_number, epoch_number, step_number)`. + +### `inference_results` + +**Positives only**: one row per snippet whose `P(true)` cleared the classification threshold. +This is a deliberate size trade — at threshold 0.99 the negatives are ~everything, and the +per-cadence aggregates that summaries need live in the manifest table instead. + +| Column | Type | Notes | +| --- | --- | --- | +| `npy_path`, `snippet_index` | TEXT, INTEGER | The stamp file and row — enough to reload the exact waterfall | +| `prediction`, `confidence` | INTEGER, REAL | 1; probability of the predicted class (see [`MODELS.md`](MODELS.md)) | +| `latent_vector` | TEXT | JSON, flattened `(6 · latent_dim,)` | +| `target`, `session`, `cadence_id`, `band` | TEXT/INTEGER | From the CSV group key | +| `frequency_mhz` | REAL | The snippet's stamp center frequency | +| `timestamp_observed` | REAL | The `.h5` header's `tstart` (MJD) | +| `h5_path` | TEXT | First observation of the cadence | +| `superseded` | INTEGER | Default 0 | + +Index: `(tag, timestamp, confidence, prediction)`. + +### `inference_cadences` (run manifest, schema v2) + +One row per (cadence, stage transition); the newest live row per `(tag, npy_path)` is the +cadence's current state, older rows having been superseded. Drives the stage-aware inference +resume ([`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md)). + +| Column | Type | Notes | +| --- | --- | --- | +| `csv_path`, `cadence_key` | TEXT | Source catalog; the group key as JSON | +| `npy_path` | TEXT | The cadence's stamp file (the resume key, with `tag`) | +| `status` | TEXT | `preprocessed` → `inferred` (or `failed`) | +| `n_stamps`, `n_candidates` | INTEGER | Aggregates (candidates only on `inferred` rows) | +| `confidence_summary` | TEXT | JSON from `inference.summarize_confidences`: n, threshold, above-threshold count, mean/min/max, quantiles p01–p99 | +| `duration_s` | REAL | Stage wall time | +| `superseded` | INTEGER | Default 0 | + +Index: `(tag, npy_path, status)` — the resume lookup. + +## Schema migration + +`_migrate_schema()` runs on every startup, gated on `PRAGMA user_version` +(`_SCHEMA_VERSION = 2`): + +- **v0 → v1**: `ALTER TABLE ... ADD COLUMN superseded INTEGER DEFAULT 0` on + `training_stats`, `injection_stats`, `latent_snapshots`, `inference_results` — the only + in-place change SQLite supports is additive `ADD COLUMN`, which is exactly what supersede + semantics needed. A per-table column-existence check (`PRAGMA table_info`) keeps each step + idempotent even if `user_version` was lost (a db file copied without its journal). +- **v1 → v2**: the `inference_cadences` table — no migration step needed, because + `CREATE TABLE IF NOT EXISTS` in `_init_database()` creates it for old and new databases + alike before the migration runs; only the version stamp advances. + +Fresh databases get the full current schema from the CREATE statements and are just stamped. +The pattern to follow for future changes: bump `_SCHEMA_VERSION`, add a +`if version < N:` block with additive, idempotent statements, and rely on +`CREATE TABLE IF NOT EXISTS` for whole new tables. + +## Query API + +Each table has a `query_*` method returning `list[dict]` +(`query_system_resource`, `query_injection_stat`, `query_injection_stat_stability`, +`query_training_stat`, `query_latent_snapshots`, `query_latent_snapshot_keys`, +`query_inference_result`, `query_inference_cadences`). Shared conventions: + +- **String filters** (`tag`, `stat_name`, `status`, ...) accept a single value (`=`) or a + list (`IN`); **range filters** come as `start_*`/`end_*` pairs (inclusive); + `start_time`/`end_time` bound `timestamp` — plots pass the run's `start_time` so + multi-attempt runs query their whole history. +- **`columns`** selects a subset, validated against a per-table whitelist + (`_build_select`) — column names never reach SQL unchecked, and everything is + parameter-bound. +- **`include_superseded=False`** by default on every table that has the column: stale rows + from failed attempts are invisible unless you ask. The inference resume *relies* on the + default — a superseded `inferred` row must read as "not done". +- JSON-typed columns (`latent_vector`, `cadence_key`, `confidence_summary`) come back as + strings; callers `json.loads` them. + +`get_db_stats()` returns row counts per table, the covered time range, and the database file +size — logged at startup. + +## Growth expectations + +Rules of thumb at full-scale defaults (dominant terms only): + +- **`injection_stats` is the giant.** Every generated cadence writes 18 intensity rows + (6 statistics × 3 stages) plus 0–12 signal-characteristic rows (0 / 6 / 6 / 12 by type — + average 6). A training round generates `3 × num_samples_beta_vae` cadences (main + true + + false), so at defaults: `3 × 499 200 × ~24 ≈ 36 M rows per round`, times 20 rounds plus the + RF dataset. This is why writes are batched, why the drainer runs off the training critical + path, and why the injection plots subsample (`plot_injection_subsampling_count`). If the + database size becomes a problem, this table is where the budget goes — smoke-scale runs + (`--num-samples-beta-vae 3072`) keep it trivial. +- **`training_stats`**: ~21 rows/epoch → ~42 k rows for 20 × 100 epochs. Negligible. +- **`latent_snapshots`**: one row per viz cadence per capture — 960 cadences × one capture + every `latent_viz_step_interval` steps (plus the final step) × epochs. At full scale + (130 steps/epoch → 13 captures/epoch): ~25 M rows over 20 × 100 epochs, each carrying a + 48-float JSON vector. The second heaviest table; `latent_viz_step_interval` and + `latent_viz_num_cadences_per_type` are the knobs. +- **`system_resources`**: (4 + 2 × n_GPUs) rows/second — ~1.2 M rows/day on a 6-GPU node. +- **`inference_results`**: positives only; at a 0.99 threshold this stays small by + construction. `inference_cadences`: a handful of rows per cadence. + +WAL mode keeps readers (plots, resume queries) unblocked during heavy write phases; the WAL +file is checkpointed back into the main db periodically by SQLite itself. From c18b0097523af40446bc13b57e4320344ddfd3e9 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:53:31 +0800 Subject: [PATCH 07/20] Add docs/RUNTIME_SERVICES.md: logger queue+Slack, ResourceManager lifecycle, monitor Co-Authored-By: Claude Fable 5 --- docs/RUNTIME_SERVICES.md | 181 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 docs/RUNTIME_SERVICES.md diff --git a/docs/RUNTIME_SERVICES.md b/docs/RUNTIME_SERVICES.md new file mode 100644 index 0000000..e6cb9b3 --- /dev/null +++ b/docs/RUNTIME_SERVICES.md @@ -0,0 +1,181 @@ +# Runtime Services + +This document covers the three infrastructure singletons every run stands on: the **logger** +([`src/aetherscan/logger/`](../src/aetherscan/logger)) with its queue architecture and Slack +integration, the **ResourceManager** +([`src/aetherscan/manager/manager.py`](../src/aetherscan/manager/manager.py)) that owns +pool/shared-memory/process lifecycles, signal handling, and cleanup ordering, and the +**resource monitor** +([`src/aetherscan/monitor/monitor.py`](../src/aetherscan/monitor/monitor.py)). They +initialize early (see the init order in [`ARCHITECTURE.md`](ARCHITECTURE.md)) and outlive +everything else. + +## Logger + +### Queue architecture + +Naive logging from a process tree — dozens of pool workers plus a handful of main-process +threads — interleaves output and contends on handler locks. Aetherscan funnels **every** +record through one consumer instead: + +``` +main process: logging.* → QueueHandler ─┐ +pool workers: logging.* → QueueHandler ─┼→ multiprocessing.Queue → QueueListener thread +producer tree: QueueHandler (own queue, ─┘ │ + relayed by RoundDataProducer) ├→ FileHandler logs/aetherscan.log (mode="w") + ├→ StreamHandler stdout + └→ SlackHandler (optional) +``` + +`Logger.__init__` sets the root logger to DEBUG (handlers do the filtering), attaches a +`QueueHandler` writing to a shared `multiprocessing.Queue`, and starts a `QueueListener` +thread that drains it into the real handlers, each with its own configured level +(`logger.{console,file,slack}_level`, all INFO by default). Consequences: + +- The log file is **the current run only** (`mode="w"` truncates at startup); pull it off the + cluster before relaunching if you need history. +- **Fork-started workers** inherit the queue: `init_worker_logging()` (called from every pool + initializer) resets the worker's root logger to a single `QueueHandler` — and resets + `sys.stdout`/`sys.stderr` to the real streams, because the inherited `StreamToLogger` + objects (below) would otherwise re-enqueue everything twice. +- **Spawn-started processes** (the `RoundDataProducer`) can't inherit the singleton; they log + into their own spawn-context queue which the parent relays into the same handler set via a + second `QueueListener` (see [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)). +- TF's logger is stripped of its handlers and set to propagate into the root; Python + warnings are captured (`logging.captureWarnings(True)`). + +### The stderr-redirect gotcha + +`Logger.__init__` replaces `sys.stdout` with a `StreamToLogger` at INFO and `sys.stderr` with +one at **ERROR**. That means: + +1. Bare `print()` and C-library stdout end up in the log (which is why `print()` is banned + outside `utils/` — it's not lost, but it's unformatted). +2. **Anything writing to stderr becomes an ERROR record** — and ERROR-level records are + broadcast to the main Slack channel (below). Progress bars are the classic trap: SHAP's + tqdm writes its refreshes to stderr, so every refresh would spam Slack as a separate + ERROR and eventually trip the webhook. `train.py:_silence_stderr()` exists precisely for + this — it redirects stderr to `/dev/null` around SHAP's explainer calls (exceptions still + propagate; they don't travel through the stream). Wrap any new dependency that draws + progress bars the same way, or configure the bar off. + +### Slack integration + +When `SLACK_BOT_TOKEN` (and a channel via `SLACK_CHANNEL` or `logger.slack_channel`) is set, +a `SlackHandler` ([`slack_handler.py`](../src/aetherscan/logger/slack_handler.py)) joins the +listener's handler list; otherwise Slack quietly disables itself. + +- **Per-run thread.** `start_run()` posts one summary message (host, CPU/GPU/RAM inventory, + the CLI invocation) and caches its thread timestamp; every subsequent record is posted as a + **reply in that thread**, so one channel can carry many runs without interleaving. +- **Batching.** Records buffer up to `slack_buffer_size` (100) or `slack_flush_interval` + (60 s) and go out as one combined message, color-coded by the batch's highest severity and + truncated to Slack's limits. +- **Broadcast escalation.** Records at `slack_broadcast_level`+ (ERROR) are echoed to the + main channel, not just the thread — failures surface without opening the thread. +- **Image uploads.** Every plot the pipeline saves is pushed via + `Logger.upload_image_to_slack()` → `SlackHandler.upload_file()`: the file lands in the run + thread, with a link-back message broadcast to the main channel by default. +- **Failure hygiene.** Send failures retry (`slack_retry_attempts`) and repeated failures + put the handler in a cooldown so a dead webhook can't stall the run. The handler's own + diagnostics print to the *real* stderr (`sys.__stderr__`) — logging them would recurse. + +> [!WARNING] +> Anything logged at INFO or above may reach Slack. Never log secrets, tokens, or +> internal URLs ([`SECURITY.md`](../SECURITY.md)). + +## ResourceManager + +The ResourceManager is the registry of everything that must not outlive the run: +multiprocessing pools, spawned processes, POSIX shared memory, and the other service +singletons. Modules never call `Pool(...)`/`SharedMemory(...)` directly for long-lived +resources — they go through the manager so cleanup is centralized and ordered. + +### Managed resources + +| Wrapper | Created via | Cleanup behavior | +| --- | --- | --- | +| `ManagedPool` | `create_pool(n_processes, name, initializer, initargs)` | `terminate()` → `join()` with `manager.pool_terminate_timeout`; workers still alive after the timeout are **SIGKILL-escalated** individually (`_force_kill_workers`). Termination sends SIGTERM first, which is what triggers the workers' cleanup handlers. | +| `ManagedProcess` | `register_process(process, name)` (e.g. the RoundDataProducer) | `terminate()` → `join(timeout)` → `kill()` escalation (`close_process`). | +| `ManagedSharedMemory` | `create_shared_memory(size, name)` | `close()` + `unlink()` in the creator, then a verification probe (`_check_unlinked`) that re-attaches by name to confirm the segment is really gone — leaked `/dev/shm` blocks survive process death and eat node RAM. | + +The **creator-unlinks rule** ([`CLAUDE.md`](../CLAUDE.md)): workers attaching to a shared +block only ever `close()` their own mapping (their SIGTERM handlers do exactly that — and +**never log**, because the queue handler's feeder thread needs the GIL and a blocked handler +deadlocks termination; see the canonical worker handler in +[`data_generation.py`](../src/aetherscan/data_generation.py)`:_init_worker`). Only the main +process, via the manager, unlinks. + +### Signal handling + +`_register_cleanup_handlers()` registers `cleanup_all` with `atexit` and installs a handler +for SIGINT/SIGTERM: + +- **First signal**: run `cleanup_all()` once, then `sys.exit(0)`. +- **Second signal** while cleanup is still running: reset both handlers to `SIG_IGN` and + `os._exit(130)` — a hard exit. Re-entering `cleanup_all` from the same (main) thread would + self-deadlock on the non-reentrant `_cleanup_lock`, so the force-quit is the only safe + response to an impatient double Ctrl-C. +- Signals delivered to forked workers are ignored by the handler (PID check) — workers have + their own SIGINT-ignore + SIGTERM-cleanup handlers installed by their pool initializers. + +### Cleanup order + +`cleanup_all()` (idempotent, lock-guarded) tears down in a strict order — each stage may +still need the ones after it: + +1. **Processes** — before shared memory: the producer's workers hold attachments to the + background-plate block. +2. **Pools** — same reasoning. +3. **Shared memory** — nothing is attached anymore; unlink + verify. +4. **Monitor** — stopping it triggers the final resource plot, which *queries the DB*. +5. **Database** — final buffer flush; nothing writes after this. +6. **Logger** — last, so every earlier stage could still log. Nothing can log after it stops. + +This is also why `main()` calls `manager.cleanup_all()` in its `finally` block explicitly: +the DB writer, listener, and monitor threads are non-daemon, and without an explicit cleanup +they would block interpreter exit before `atexit` ever ran. + +## Resource monitor + +A 1 Hz background thread (`monitor.monitor_interval`) sampling into the `system_resources` +table ([`DATABASE.md`](DATABASE.md)) via the normal write queue: + +| Metric (`resource_type` / `resource_name`) | Source | +| --- | --- | +| `cpu / system_total`, `ram / system_total` | `psutil.cpu_percent()`, `psutil.virtual_memory().percent` | +| `cpu / process_tree`, `ram / process_tree` | `get_process_tree_stats()` over the main process + all descendants | +| `gpu / {name}_utilization`, `gpu / {name}_memory` | `nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total` per device | + +### Why PSS, not RSS + +`get_process_tree_stats()` sums memory across the whole process tree using **PSS** +(Proportional Set Size, `psutil.Process.memory_full_info().pss`) rather than RSS. RSS counts +shared pages once *per process*: with the multi-GB background plates mapped into dozens of +injection workers, summing RSS across the tree would multiply that memory by the worker count +and happily report more than the machine has. PSS divides each shared page by the number of +processes sharing it, making the per-process values **additive** — the tree sum is the tree's +actual footprint. (PSS is Linux-specific; the corresponding tests self-skip elsewhere, and +known accounting quirks are tracked in +[`KNOWN_ISSUES.md`](../KNOWN_ISSUES.md) #6 and #11.) CPU is normalized against the system +core count, so 100 % means "all cores busy". + +### The shutdown plot + +When the monitor stops (during cleanup), `_save_plot()` queries the run's samples and renders +`plots/resource_utilization_{tag}.png` — three stacked, time-aligned panels: + +1. **CPU** — Aetherscan process tree (filled) vs system total. +2. **RAM** — same pair; the gap between them is other users/jobs on the node. +3. **GPU** — per-device utilization (solid, left axis) and memory (dashed, right axis) on a + shared twin-axis panel with a combined legend. + +The x-axis is minutes since monitor start, so the panels line up with the log timeline: data +generation reads as CPU plateaus, epochs as GPU bands, per-cadence inference as alternating +CPU/GPU activity. The figure is uploaded to the run's Slack thread like every other plot. +Missing stretches in the panels are a known symptom +([`KNOWN_ISSUES.md`](../KNOWN_ISSUES.md) #10). + + From 6daac799c2c9b98d9fde90f325fe38e139fb13b6 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:54:56 +0800 Subject: [PATCH 08/20] Add docs/TESTING.md: suite layout, markers, isolation fixtures, CI, cluster smokes Co-Authored-By: Claude Fable 5 --- docs/TESTING.md | 173 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 docs/TESTING.md diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..fd56e44 --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,173 @@ +# Testing + +This document covers the pytest suite: layout, markers, the singleton-isolation machinery in +`conftest.py`, the synthetic data factories, what CI runs, how to run the cluster-marked +integration smokes, and the checklist for adding tests. The quick-start version lives in +[`CONTRIBUTING.md`](../CONTRIBUTING.md#testing); this is the deeper reference. + +## TL;DR + +```bash +# Default selection — exactly what CI runs (no GPUs, no cluster data needed): +pytest -m "not gpu and not cluster" -q + +# One file / one test while iterating: +pytest tests/unit/test_preprocessing.py -q +pytest tests/unit/test_db.py -k supersede -q + +# Cluster integration smokes (end-to-end train + CSV inference), on a cluster, +# inside the NGC container: +./utils/run_container.sh python -m pytest tests/ -m "gpu or cluster" -q +``` + +Configuration lives in [`pyproject.toml`](../pyproject.toml) under +`[tool.pytest.ini_options]`: `pythonpath = ["src"]` (no `PYTHONPATH` prefix needed), +`testpaths = ["tests"]`, and `--strict-markers` (a typo'd marker is a collection error, not a +silently-ignored one). + +## Layout + +``` +tests/ +├── conftest.py # isolation fixture + data factories (below) +├── fixtures/ +│ └── ed_real_slice.npz # recorded real coarse-channel slice — the energy-detection +│ # equivalence gate runs against real data, not just synthetic +├── unit/ # fast, hardware-independent; the CI surface +│ ├── test_config.py # singleton semantics, to_dict round-trip +│ ├── test_cli_validation.py # tag pattern, divisibility matrix, saved-config precedence +│ ├── test_data_generation.py # log_norm, create_* shapes/labels, intersection checks, +│ │ # chunk segments / batched task partitioning +│ ├── test_round_data.py # paths/manifest protocol, reuse/delete semantics, producer +│ ├── test_run_state.py # manifest round-trip, stage machine transitions +│ ├── test_train_utils.py # get_latest_tag ladder, curriculum schedules, archiving +│ ├── test_train_datasets.py # batched generators: coverage, stratification, alignment +│ ├── test_latent_traversal.py # traversal grid math with a stub decoder +│ ├── test_models.py # feature layout, RF train/predict, encoder/decoder symmetry +│ ├── test_preprocessing.py # k² equivalence gates, dedup, grouping, DC spike, spline +│ ├── test_pfb.py # response shape/symmetry/flatness, C++ sinc cross-check +│ ├── test_inference.py # padding fix, provenance mapping, confidence summaries +│ ├── test_inference_viz.py # every figure smoke-tested against tiny synthetic inputs +│ ├── test_main.py # streaming-loop resume/containment with stubbed stages +│ ├── test_db.py # writer thread, flush/supersede sentinels, migrations, queries +│ └── test_manager.py # pool/SHM tracking and cleanup idempotence +└── integration/ # marked integration+gpu+cluster: real subprocess runs + ├── conftest.py # repo-root launcher + cluster path resolution + ├── test_train_smoke.py # known-good training smoke config, end to end + └── test_inference_smoke.py # subset CSV inference against cluster-resident .h5 data +``` + +## Markers + +| Marker | Meaning | In default selection? | +| --- | --- | --- | +| `slow` | Slower CPU tests (e.g. builds real TF graphs) | **Yes** — CI runs them | +| `gpu` | Needs one or more physical GPUs | No | +| `cluster` | Needs cluster-resident data/models (blpc3/bla0 paths) | No | +| `integration` | End-to-end subprocess runs; **skips the isolation fixture** | No (they're also gpu+cluster) | + +`--strict-markers` rejects anything not declared in `pyproject.toml` — add new markers there +first. + +## Isolation: singletons, env, and teardown + +Every singleton class (`Config`, `Database`, `Logger`, `ResourceManager`, `ResourceMonitor`) +carries a `_reset()` classmethod designed for tests. The autouse fixture +`aetherscan_isolated_env` in [`tests/conftest.py`](../tests/conftest.py) wraps every +non-integration test: + +**Setup** — point `AETHERSCAN_{DATA,MODEL,OUTPUT}_PATH` at a fresh `tmp_path` tree (with +`training/`/`testing/`/`inference/` subdirs), delete `SLACK_BOT_TOKEN`/`SLACK_CHANNEL` (tests +must never talk to Slack), snapshot the current SIGINT/SIGTERM handlers and +stdout/stderr, reset all singletons, and `init_config()`. + +**Teardown** — `_teardown_singletons()` mirrors `ResourceManager.cleanup_all()`'s ordering +(processes → pools → shared memory → monitor → DB → logger) without its logging side effects, +**unregisters the manager's `atexit` callback** (otherwise every test's manager instance +would pile up a stale cleanup hook for interpreter exit), resets all singletons again, and +restores the signal handlers and streams. + +Consequences worth knowing: + +- Constructing a `ResourceManager` in a test installs real signal handlers — the fixture + restores them, but don't spawn threads that outlive the test. +- `MPLBACKEND=Agg` and `TF_CPP_MIN_LOG_LEVEL=2` are set before any aetherscan import + (train.py imports pyplot at module level; CI runners are headless). +- Singleton imports inside the fixture are deferred so **integration runs never import + TensorFlow into the pytest parent process** — the integration tests exercise the pipeline + as a subprocess and inherit the real environment instead. + +## Synthetic data factories + +Three factory fixtures produce inputs shaped like the real thing but sized for speed: + +| Fixture | Produces | +| --- | --- | +| `make_background_npy(filename, n_cadences=8, width_bin=512)` | Positive chi-squared-ish float32 plates `(n, 6, 16, width_bin)` under the tmp `data/training/` dir. | +| `make_h5_observation(filename, n_chans=2048, time_bins=16)` | A filterbank-style `.h5` with a `data` dset `(time_bins, 1, n_chans)` + `fch1`/`foff`/`nchans` attrs (plain h5py — no bitshuffle needed for tests). | +| `make_inference_csv(filename, groups)` | A cadence-grouping CSV whose column names come from the live `InferenceConfig` (`cadence_group_by_cols` / `cadence_h5_path_col`). | + +Prefer these over hand-rolling arrays — they stay in sync with config defaults, and they're +the reason a preprocessing test can run the *real* grouping/detection code paths in +milliseconds. + +For numerical-equivalence tests, `tests/fixtures/ed_real_slice.npz` is a small recorded slice +of a real coarse channel: the vectorized D'Agostino–Pearson implementation is pinned against +`scipy.stats.normaltest` at `rtol=1e-9` on synthetic distributions *and* asserted to produce +identical hit sets on real data (see [`PREPROCESSING.md`](PREPROCESSING.md)). + +## CI + +[`.github/workflows/tests.yml`](../.github/workflows/tests.yml) runs on every PR and on +pushes to master, on Python **3.10 and 3.12** (the conda and NGC-container runtimes, +`fail-fast: false` so both report): + +``` +pip install "tensorflow-cpu==2.17.*" -r requirements-container.txt h5py hdf5plugin pandas psutil pytest +pytest -m "not gpu and not cluster" -q +``` + +`tensorflow-cpu` stands in for the container's GPU TF 2.17 build; `h5py`/`hdf5plugin`/ +`pandas`/`psutil` are installed explicitly because the NGC base image ships them (so +`requirements-container.txt` intentionally omits them). A few tests assert Linux-only +behavior (PSS memory accounting) and self-skip elsewhere — the suite is green on macOS +locally, with skips. See [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md) for how the test +workflow feeds the weekly flaky-test tracker. + +## Running the cluster smokes + +The two integration tests launch `python -m aetherscan.main ...` as a subprocess from the +repo root (their conftest prepends `/src` to `PYTHONPATH`) with the known-good smoke +configs, and assert on return codes and produced artifacts. They need real GPUs, the +cluster-resident data/models, and hours of wall time (subprocess timeout: 2 h each): + +```bash +# On the cluster, inside the container: +./utils/run_container.sh python -m pytest tests/integration -m "gpu or cluster" -q + +# AETHERSCAN_* env vars override the baked-in cluster defaults if your paths differ. +``` + +Be a good citizen: check `nvidia-smi`/`htop` for other users' jobs first, and clean up +`/dev/shm` and scratch artifacts afterwards. + +## Adding tests — checklist + +1. **Ship unit tests with new logic.** Every PR that adds or changes behavior lands tests for + it under `tests/unit/` in the matching `test_.py` (create it if the module is + new). +2. **Use the fixtures.** Never instantiate singletons directly — the autouse fixture already + gave you a clean `Config` against tmp paths; use `get_config()` and mutate it in the test + if you need non-defaults. Use the data factories for inputs. +3. **Mark honestly.** `slow` for anything that builds real TF graphs; `gpu`/`cluster` for + anything the default selection couldn't run on a laptop. Everything unmarked must pass on + a CPU-only CI runner. +4. **Test the seam, not the world.** The codebase exposes deliberate test seams — e.g. the + producer's `generate_fn` stub, duck-typed pipelines in `_execute_training_stages`, and + module-level pure functions (`_sliding_normality_k2`, `build_chunk_segments`, + `build_traversal_latents`) — prefer them over patching internals. +5. **Numerical claims get gates.** If you vectorize or port math, pin equivalence against the + reference implementation with an explicit tolerance, on synthetic *and* (where feasible) a + small recorded real fixture. +6. **Style applies.** Ruff lint+format runs on `tests/` too; same conventions as `src/` + ([`CONTRIBUTING.md`](../CONTRIBUTING.md#code-style)). From 06d09bf4016477ca6bd4b76ef1b2b1777ff2369a Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:56:41 +0800 Subject: [PATCH 09/20] Add docs/GITHUB_AUTOMATION.md: workflow catalog, dedup guards, assistant-handle rules Co-Authored-By: Claude Fable 5 --- docs/GITHUB_AUTOMATION.md | 140 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 docs/GITHUB_AUTOMATION.md diff --git a/docs/GITHUB_AUTOMATION.md b/docs/GITHUB_AUTOMATION.md new file mode 100644 index 0000000..a060873 --- /dev/null +++ b/docs/GITHUB_AUTOMATION.md @@ -0,0 +1,140 @@ +# GitHub Automation + +This document catalogs every workflow in +[`.github/workflows/`](../.github/workflows) — trigger, behavior, and the deterministic +dedup guards several of them rely on — plus the issue/PR lifecycle they collectively +implement and the rules for (not) invoking the assistant. The human-side contribution +process is [`CONTRIBUTING.md`](../CONTRIBUTING.md); this is the machine side. + +## TL;DR — the lifecycle + +``` +Discussion → Issue ──────────────► PR ─────────────► merge to master ──► weekly upkeep + │ auto-assign-author │ auto-assign │ claude-release-notes │ claude-dependency-check + │ claude-issue-triage │ sync-pr-labels │ claude-style-check │ claude-flaky-test-tracker + │ claude-contribution- │ claude-code- │ claude-update-docs + │ check │ review + │ pre-commit + tests (required checks) +``` + +Two workflow families: **deterministic** (pre-commit, tests, auto-assign, label sync) and +**assistant-driven** (`claude-*.yml` — each wraps `anthropics/claude-code-action` with a +task-specific prompt, authenticated via the `CLAUDE_CODE_OAUTH_TOKEN` secret). + +## The assistant handle — read this before writing issue/PR text + +The general assistant workflow ([`claude.yml`](../.github/workflows/claude.yml)) triggers on +a `contains(...)` match of the assistant handle — an `@` immediately followed by `claude` — +anywhere in the title or body of an issue, PR, or Discussion (or a comment/review on one). +A `contains` match has no notion of quoting or context: *any* occurrence fires a real +assistant run and, typically, a follow-up PR. + +- Write the handle **only when you intend to summon the assistant**. +- To mention it as plain text, write it as `"@ claude"` — a space after the `@`, double + quotes around it — so the substring can never match. This convention is prescribed in + [`CLAUDE.md`](../CLAUDE.md) and [`CONTRIBUTING.md`](../CONTRIBUTING.md); an accidental tag + is what spawned the spurious run around issue #83. +- The workflow also triggers on **assignment** to the `claude` user and on the `claude` + **label** — treat those as invocations too. + +## Deterministic workflows + +### `pre-commit.yml` + +**Trigger:** every push and PR to any branch, plus `workflow_dispatch`. +Runs the full [`.pre-commit-config.yaml`](../.pre-commit-config.yaml) suite (ruff lint + +format, hygiene hooks, gitleaks) on Python 3.10 with pip/ruff caches. On master it skips the +`no-commit-to-branch` hook (branch protection already restricts master to merge commits). +This is a required check — the same hooks you run locally, so a green local +`pre-commit run --all-files` predicts a green check. + +### `tests.yml` + +**Trigger:** every PR, pushes to master, `workflow_dispatch`. +Runs `pytest -m "not gpu and not cluster" -q` on Python 3.10 **and** 3.12 (the conda and NGC +container runtimes; `fail-fast: false`), with `tensorflow-cpu==2.17.*` standing in for the +container's GPU build. Details in [`TESTING.md`](TESTING.md). Its run history is the data +source for the flaky-test tracker below. + +### `auto-assign-author.yml` + +**Trigger:** issue or PR opened. +Assigns the author to their own issue/PR via `github-script` — keeps the "assignee" field +meaningful without manual bookkeeping. + +### `sync-pr-labels.yml` + +**Trigger:** PR opened/edited; issue labeled. +Copies labels from the issues a PR formally links (GraphQL `closingIssuesReferences` — i.e. +`Closes #N` / the Development sidebar) onto the PR, and re-syncs when a linked issue is +labeled later (e.g. by triage). `needs-issue`, `needs-discussion`, and `good first issue` +are excluded (they describe issues, not PRs). This is why PRs must link their issue with a +closing keyword: label sync and the contribution check both key on the formal link. + +## Assistant-driven workflows + +All of these run `anthropics/claude-code-action` (version-pinned by commit SHA) with a +scoped prompt and a `timeout-minutes` cap. Where they create issues/comments, a +**deterministic dedup guard** runs first — a plain shell step that searches for a hidden +HTML marker and skips the assistant step entirely when found, so re-runs can never +double-post. + +| Workflow | Trigger | What it does | Dedup marker | +| --- | --- | --- | --- | +| [`claude.yml`](../.github/workflows/claude.yml) | Handle mention / assignment / `claude` label on issues, PRs, Discussions, comments, reviews | The general-purpose assistant: answers, implements, opens PRs on `claude/*` branches. `allowed_bots: "claude,claude[bot]"` — deliberately, so issues *filed by* the other workflows (as the `claude` bot) can invoke it for follow-up PRs. | — | +| [`claude-code-review.yml`](../.github/workflows/claude-code-review.yml) | PR opened / ready_for_review | Automated first-pass code review with inline comments. Every PR gets one on open; address the actionable notes and resolve the conversations before human review. | — | +| [`claude-issue-triage.yml`](../.github/workflows/claude-issue-triage.yml) | Issue opened | Triage: applies labels (type/area/priority) so label sync can propagate them to the eventual PR. | — | +| [`claude-contribution-check.yml`](../.github/workflows/claude-contribution-check.yml) | Issue or PR opened | Verifies workflow compliance (issue linkage, branch-prefix conventions, template use) and comments when something's missing. **Never runs for bot authors** — see the gotcha below. | — | +| [`claude-release-notes.yml`](../.github/workflows/claude-release-notes.yml) | PR **merged** to master | Drafts a release-note entry as a PR comment — the curated raw material for release bodies (see [`RELEASE.md`](RELEASE.md)). | `` first line of the comment | +| [`claude-style-check.yml`](../.github/workflows/claude-style-check.yml) | PR merged to master | Scans the merged diff's *added* lines against the project style rules ruff can't express (docstring prose style, canonical comment markers, logging idioms); files one consolidated issue when violations exist. | `` | +| [`claude-update-docs.yml`](../.github/workflows/claude-update-docs.yml) | PR merged to master; `workflow_dispatch` with a `pr_number` (re-scan an old PR with the *current* workflow logic) | Detects doc drift caused by the merge. If `cli.py` changed, a **shell step** regenerates the README CLI Reference blocks with `utils/print_cli_help.py` (Python pinned to 3.12 — argparse help formatting changes in 3.13) and embeds the output in the issue, because the follow-up assistant run has no `python` in its tool allowlist. The filed issue contains an intentional handle mention, which triggers `claude.yml` to open the actual docs PR. | `` | +| [`claude-dependency-check.yml`](../.github/workflows/claude-dependency-check.yml) | Weekly (Mon 01:00 UTC) + `workflow_dispatch` | Audits `environment.yml` / `requirements-container.txt` / `aetherscan.def` against registries and advisories under [`SECURITY.md`](../SECURITY.md)'s version-selection policy; files a weekly report issue. | `` | +| [`claude-flaky-test-tracker.yml`](../.github/workflows/claude-flaky-test-tracker.yml) | Weekly (Mon 01:00 UTC) + `workflow_dispatch` | Reads the week's `tests.yml` runs, identifies flaky/failing tests, diagnoses the worst offender, files a weekly report issue. | `` | + +The dedup-marker pattern is a convention to preserve in any new workflow that posts content: +the guard step greps existing issues/comments for the marker (`gh ... --json body --jq` + +`grep`), and the prompt instructs the assistant to put the marker as the **first line** of +anything it posts — deterministic idempotence without trusting the model to check. + +### The `allowed_bots` gotchas + +`claude-code-action` refuses to run for non-human actors unless the triggering actor is in +its `allowed_bots` list. Three configurations coexist here, each deliberate: + +- **`claude.yml`: `allowed_bots: "claude,claude[bot]"`** — bot-authored content *can* summon + the assistant. Required for the update-docs pipeline: the doc issue is filed by the + `claude` bot and must still trigger the follow-up PR. +- **`claude-code-review.yml` / `claude-issue-triage.yml`: `allowed_bots: "claude[bot]"`** — + assistant-opened PRs get reviewed and assistant-filed issues get triaged like anyone + else's. +- **`claude-contribution-check.yml`: `allowed_bots` unset, plus a job-level `if:` that + excludes bot actors entirely.** Without the `if:`, every bot-filed issue would spin up a + runner just for the action to abort with a "non-human actor" error — a red ✗ on every + automated issue (this is the other half of the issue #83 story). Skipping the job at the + `if:` level avoids both the comment and the noisy failure. + +When adding a workflow, decide explicitly which side of this each trigger actor falls on. + +## Issue and PR conventions the automation assumes + +- **Issue templates**: [`bug_report.md`](../.github/ISSUE_TEMPLATE/bug_report.md) and + [`feature_request.md`](../.github/ISSUE_TEMPLATE/feature_request.md). Triage expects + template-shaped issues; drive-by issues may be closed per + [`CONTRIBUTING.md`](../CONTRIBUTING.md). +- **Every PR links an existing issue** (`Closes #N`) — label sync, the contribution check, + and the release-notes context all read the formal link. +- **Branch prefixes** `feature/` / `hotfix/` / `misc/` / `claude/` (the last reserved for + assistant-authored branches). +- **Required checks**: pre-commit + tests (both Python versions) must pass; commits need + verified GPG signatures; branches rebase (never merge) onto master. +- **Merges fan out**: expect a release-notes comment on your merged PR, and possibly a + style-check or update-docs issue referencing it — these are normal post-merge automation, + not review feedback. + +## Secrets and permissions + +Assistant workflows authenticate with the `CLAUDE_CODE_OAUTH_TOKEN` repository secret and +request `id-token: write` plus the minimal `contents`/`issues`/`pull-requests` permissions +each task needs; deterministic workflows use the default `GITHUB_TOKEN`. Scheduled workflows +only fire from the default branch — after editing one, remember it runs master's copy until +your change merges (use `workflow_dispatch` to test). From 4afd036c6d72d0a433c33991abf1ac9f387d35af Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:58:04 +0800 Subject: [PATCH 10/20] Add docs/RELEASE.md: version-coupling contract, CD gates, trusted publishing, runbook Co-Authored-By: Claude Fable 5 --- docs/RELEASE.md | 165 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 docs/RELEASE.md diff --git a/docs/RELEASE.md b/docs/RELEASE.md new file mode 100644 index 0000000..b5e337c --- /dev/null +++ b/docs/RELEASE.md @@ -0,0 +1,165 @@ +# Release Engineering + +This document is the release contract for Aetherscan: how one version string couples PyPI, +GitHub, and HuggingFace; what the release CD does (and refuses to do); the one-time setup a +maintainer needs; and the step-by-step runbook for cutting a release. The packaging and the +CD workflow (`.github/workflows/release.yml`) are implemented to satisfy this contract — +when they and this document disagree, fix whichever is wrong *as a PR that says so*. + +## TL;DR — the version-coupling contract + +**One version string, e.g. `v1.0.0`, names three synchronized objects:** + +1. A **git tag** `v1.0.0` on `master` — which GitHub decorates into a **GitHub Release** + (notes + source tarballs) and which the CD builds into the **PyPI release** `1.0.0`. +2. An **HF weights tag** `v1.0.0` on the model repo `zachtheyek/aetherscan`, pointing at the + blessed trained weights. + +The coupling is **by convention, enforced by verification**: CD checks that the HF tag +exists before publishing anything, but never creates weights — weights can only come from +cluster training runs (CI has neither the GPUs nor the data). The installed package pulls +its matching weights at runtime because the package version is the default HF revision: + +``` +pip install aetherscan==1.0.0 +aetherscan.main inference ... # no model-path flags + → resolves HF revision v1.0.0 → downloads the blessed weights → runs +``` + +Two invariants to internalize: + +- **PyPI versions are immutable.** A botched release means `v1.0.1`, never a re-upload. +- **Installing `aetherscan==1.0.0` *is* the tagged source** — the sdist/wheel are built from + the tag by CD. There is nothing further to "sync" between PyPI and GitHub. + +## Packaging + +- **Build system**: hatchling (`[build-system] requires = ["hatchling"]` in + [`pyproject.toml`](../pyproject.toml)), wheel target `src/aetherscan`. + `requires-python = ">=3.10,<3.13"`. +- **Dependencies**: `[project].dependencies` mirrors + [`requirements-container.txt`](../requirements-container.txt)'s ranges **plus** the + packages the NGC base image provides implicitly (`tensorflow[and-cuda]==2.17.*`, `h5py`, + `hdf5plugin`, `psutil`, `huggingface_hub`) — a pip install has no base image to lean on. + Optional extras: `dev = ["ruff", "pre-commit", "pytest"]`. Dependency *versions* follow + [`SECURITY.md`](../SECURITY.md)'s selection policy, and the documented ceilings + (`numpy<2.0`, the NGC TF 2.17 ABI) apply to the package exactly as to the container. +- **Version single-sourcing**: the static `version` in `pyproject.toml` is the only place + the version is written. `src/aetherscan/__init__.py` exposes `__version__` via + `importlib.metadata.version("aetherscan")`, falling back to `"0.0.0.dev0"` on + `PackageNotFoundError` so source-tree runs (`PYTHONPATH=src`, the container) work without + an install. +- The `Development Status` classifier is a per-release decision (Beta vs Production/Stable) + — revisit it in each release PR. + +## Weight resolution (runtime side of the contract) + +Inference resolves models in this precedence order: + +1. **Explicit local paths** (`--encoder-path` / `--rf-path` / `--config-path`) — always win; + the offline/cluster path. +2. **`--hf-revision `** — pin any HF revision (a training tag like `final_v1`, a + release tag like `v1.0.0`, or a commit). +3. **`v{__version__}`** — when running as an installed release, the package's own version is + the default revision. This is the line that makes `pip install aetherscan==1.0.0` + + bare inference pull exactly the `v1.0.0` weights. +4. **Latest `v*` semver tag** on the HF repo, then **latest `final_vX`** training tag. +5. Otherwise: error with guidance. + +Downloads happen **lazily at first inference**, never at import time (an import-time network +download would be hostile), revision-pinned and cached under the standard HF cache +(`~/.cache/huggingface`; set `HF_HOME` if home isn't writable/bound in your container +setup). Public repo — downloads need no token. + +### HF repo layout and tag families + +One model repo, **`zachtheyek/aetherscan`**, with **stable filenames** at the repo root — +`vae_encoder.keras`, `vae_decoder.keras`, `random_forest.joblib`, `config.json`, and an +auto-generated model card `README.md` — and **revisions (HF git tags) carrying the +versioning**: + +| Tag family | Created by | Points at | +| --- | --- | --- | +| Training tags (`final_v1`, `test_v17`, ...) | Training runs with `--hf-upload` (tag = the run's `save_tag`) | The commit that upload produced | +| Release tags (`v1.0.0`, ...) | The release runbook (step 3 below) | The blessed training upload's commit | + +Uploads need a write-scoped `HF_TOKEN` in the environment (`.env` on the clusters — +gitignored, forwarded into the container; never logged, never committed). Upload failure +never fails a training run — weights are already safe locally. + +## The CD workflow: `release.yml` + +`on: push: tags: ["v*"]`. Steps, in order — each gate exists to make a half-released state +impossible: + +1. **Version guard** — the pushed tag must equal `v` + the `pyproject.toml` version; + otherwise fail loudly before anything publishes. (Prevents tagging a commit whose release + PR didn't land.) +2. **Unit tests** — the same selection as `tests.yml` + (`pytest -m "not gpu and not cluster"`). A release build must not outrun a red suite. +3. **HF weights verification** — confirm the matching `v*` tag exists on + `zachtheyek/aetherscan` (public repo, no token needed). **Verify, never create**: if this + fails, you skipped the weight-blessing step; the error says so and how to fix it. +4. **Build** — `python -m build` (sdist + wheel) from the tagged source. +5. **Publish to PyPI via trusted publishing** — `pypa/gh-action-pypi-publish` with + `permissions: id-token: write` and the `pypi` environment. OIDC-based: **no long-lived + PyPI API token is stored anywhere** in the repo or its secrets. +6. **GitHub Release** — created from the tag with generated notes; curate the body from the + per-PR `claude-release-notes` comments (see + [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md)) — they are the raw material, written one + merge at a time for exactly this purpose. + +An optional `workflow_dispatch` input (`test_pypi: true`) publishing to test.pypi.org is the +recommended dry-run before the first real release. + +### One-time setup (maintainer) + +- **PyPI**: create the `aetherscan` project and add GitHub as a **trusted publisher** — + repository `zachtheyek/Aetherscan`, workflow `release.yml`, environment `pypi`. This is + what step 5 authenticates against; no token to rotate, nothing to leak. +- **HF**: confirm the model repo `zachtheyek/aetherscan` exists and that the clusters' + `.env` files carry a write-scoped `HF_TOKEN` (for training uploads only — CD never + writes to HF). + +## Release runbook + +Steps for cutting `vX.Y.Z` (maintainer + agent together): + +1. **Prereqs** — one-time setup above is done; all intended PRs are merged; `master` is + green. +2. **Train the release model** — full-scale + `train --save-tag final_vN --hf-upload` on the chosen cluster. Weights land locally and + on HF tagged `final_vN`. **Review the training artifacts** — the loss curves, injection + stats, latent diagnostics, and RF plots ([`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md)) + are the release-qualification evidence. +3. **Bless the weights** — create the release tag on HF pointing at the training upload: + `utils/hf_tag_release.py --save-tag final_vN --release vX.Y.Z` (a thin wrapper over + `HfApi.create_tag`). This is the human "these weights are the release" decision — CD + deliberately cannot make it. +4. **Release PR** — bump `version = "X.Y.Z"` in `pyproject.toml`, revisit the Development + Status classifier, draft the release notes (curate the `claude-release-notes` comments), + regenerate anything version-stamped. Merge through normal review. +5. **Tag** — from the release PR's merge commit on master: + `git tag -s vX.Y.Z && git push origin vX.Y.Z` (signed, like every commit in this repo). +6. **CD does the rest** — guard → tests → HF verify → build → PyPI → GitHub Release. Watch + the run. If the HF-verify step fails, you skipped step 3: don't touch the git tag (it's + already correct) — run step 3, then re-run the failed workflow. +7. **Smoke the release** — on a clean venv/machine: + `pip install aetherscan==X.Y.Z`, then run inference with **no model-path flags** against + a small catalog; confirm it downloads the `vX.Y.Z` weights and completes. That closes the + loop on the contract. + +## FAQ + +- *Can GitHub releases and HF tagged weights get out of sync?* Only by skipping step 3 — + and then CD refuses to publish. The same tag string on both sides, with CD enforcing + existence, is the whole synchronization mechanism. +- *Why can't CI produce the weights?* Training needs cluster GPUs, hundreds of GB of + scratch, and real background data. Hence verify-don't-create: the pipeline's most + expensive artifact is produced and reviewed by humans, and automation only checks it's + where it should be. +- *What if a release is broken?* PyPI is immutable: fix forward with `vX.Y.(Z+1)` (a new + release PR + tag). You can yank the bad version on PyPI so resolvers skip it, but the + version number is spent. +- *Do source-tree users see any of this?* No — the container/`PYTHONPATH=src` workflows are + untouched; `__version__` just reads `0.0.0.dev0` and explicit model paths keep working. From 1c9d9053bc1cd816925d5a7847be58f0bfddca46 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 14:59:17 +0800 Subject: [PATCH 11/20] Update docs index for the full suite; delete placeholder; add cross-references Co-Authored-By: Claude Fable 5 --- docs/CONFIG_AND_CLI.md | 4 ++++ docs/GPU_RUNTIME_GUIDE.md | 2 +- docs/README.md | 23 ++++++++++++++++++----- docs/placeholder | 2 -- 4 files changed, 23 insertions(+), 8 deletions(-) delete mode 100644 docs/placeholder diff --git a/docs/CONFIG_AND_CLI.md b/docs/CONFIG_AND_CLI.md index 2d17927..2d9f084 100644 --- a/docs/CONFIG_AND_CLI.md +++ b/docs/CONFIG_AND_CLI.md @@ -4,6 +4,10 @@ This document explains how Aetherscan's runtime configuration is structured, how flags map onto it, and how the `train` and `inference` subcommands stay isolated from each other. Read this before adding a new flag or config field — the patterns below exist precisely so that one mode's parameters can't silently contaminate the other. +For where config initialization sits in the overall startup sequence (and the singleton +pattern's rules), see [`ARCHITECTURE.md`](ARCHITECTURE.md); for what the individual +training/inference fields *do*, see [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md) and +[`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md). ## TL;DR diff --git a/docs/GPU_RUNTIME_GUIDE.md b/docs/GPU_RUNTIME_GUIDE.md index 8f3ee6d..7714657 100644 --- a/docs/GPU_RUNTIME_GUIDE.md +++ b/docs/GPU_RUNTIME_GUIDE.md @@ -1,6 +1,6 @@ # GPU Runtime Guide -This runbook covers running Aetherscan across GPU architectures — the Blackwell (RTX PRO 6000) workstation and the Ampere (A4000) cluster. The pipeline source tree is identical on both machines, and as of [`aetherscan.def`](../aetherscan.def) the NGC container is the canonical runtime on both clusters — one recipe builds and runs on each. The pre-container conda env is kept as an alternative install path on Ampere. +This runbook covers running Aetherscan across GPU architectures — the Blackwell (RTX PRO 6000) workstation and the Ampere (A4000) cluster. The pipeline source tree is identical on both machines, and as of [`aetherscan.def`](../aetherscan.def) the NGC container is the canonical runtime on both clusters — one recipe builds and runs on each. The pre-container conda env is kept as an alternative install path on Ampere. This document is about the *runtime*; for what the pipeline itself does once it's running, start at [`ARCHITECTURE.md`](ARCHITECTURE.md). ## TL;DR diff --git a/docs/README.md b/docs/README.md index 39cc665..8102cfe 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,21 @@ # Aetherscan docs -This directory holds long-form technical documentation that supplements the project [`README.md`](../README.md). For installation, usage examples, and the full CLI reference, start there; reach for the documents below when you need the underlying detail. +This directory holds long-form technical documentation that supplements the project [`README.md`](../README.md). For installation, usage examples, and the full CLI reference, start there; reach for the documents below when you need the underlying detail. New to the codebase? Read [`ARCHITECTURE.md`](ARCHITECTURE.md) first — it's the map the others hang off. -| Document | Summary | -| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`GPU_RUNTIME_GUIDE.md`](GPU_RUNTIME_GUIDE.md) | Runbook for building, running, and debugging Aetherscan across GPU architectures — the Blackwell (RTX PRO 6000) workstation and the Ampere (A4000) cluster — unified by the NGC container runtime. Covers container builds, image verification, the four GPU-related CLI flags, CUDA/NCCL debugging recipes, the fallback escalation ladder, and cross-machine checkpoint interop. Read this before building the container or running on a new GPU architecture (especially Blackwell/sm_120), or after any run that dies with a `CUDA_ERROR_INVALID_PTX`/NCCL/OOM failure. | -| [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md) | Deep-dive into Aetherscan's singleton-config + CLI architecture: the three flag-routing patterns (A/B/C), the four cross-mode contamination barriers, the validation + fix-proposal layer, and the checklist for adding a new flag. Read this before touching `config.py` or `cli.py`. | +| Document | Summary | +| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [`ARCHITECTURE.md`](ARCHITECTURE.md) | System-level map: the two-stage ML design, the cadence/stamp/snippet data model, module map with a data-flow diagram, process/thread topology, initialization order, the singleton pattern and its rules, tag conventions, and where every artifact lands on disk. Start here. | +| [`TRAINING_PIPELINE.md`](TRAINING_PIPELINE.md) | The `train` command end to end: curriculum schedules and the round lifecycle, disk-backed (memmap) round data with the background producer process, distributed datasets + gradient accumulation, adaptive LR, checkpointing, the run-state manifest and stage-aware retry semantics, and a catalog of every training plot with what to look for in each. | +| [`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md) | The `inference` command end to end: CSV catalog format and cadence grouping, model loading, the per-cadence streaming architecture (prefetch, per-cadence provenance, padding semantics), classification-threshold semantics, the `inference_cadences` run manifest and retry/resume flow, and a catalog of every inference artifact and figure. | +| [`PREPROCESSING.md`](PREPROCESSING.md) | Everything upstream of the models: energy detection end to end (coarse channels, DC-spike removal, spline vs PFB bandpass flattening with the math of both, the vectorized D'Agostino–Pearson derivation, deduplication, overlap stamps, downsample-at-extraction and the storage math), the training/inference loading paths, and setigen signal injection with the four signal types and A/B/C intensity staging. | +| [`MODELS.md`](MODELS.md) | The two models in full: Beta-VAE encoder/decoder architecture tables, the reparameterization trick, the composite loss math including the ON/OFF clustering terms, the Random Forest's 48-feature cadence layout, threshold/confidence semantics at the deployment operating point, and what each persisted model/eval/SHAP artifact means. | +| [`DATABASE.md`](DATABASE.md) | The persistence layer: schema of every table, the single-writer-thread design, the flush and mark-superseded queue protocols, the `PRAGMA user_version` migration mechanism, the query API with default supersede filtering, and growth expectations per table at full scale. | +| [`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md) | The infrastructure singletons: the queue-based logger and its Slack threading/batching/broadcast behavior (and the stderr-redirect gotcha), the ResourceManager's pool/shared-memory/process lifecycles, signal handling and strict cleanup ordering, and the 1 Hz resource monitor with the PSS-vs-RSS rationale and the shutdown utilization plot. | +| [`TESTING.md`](TESTING.md) | The pytest suite: layout, markers and the default CI selection, the autouse singleton/env isolation fixture and its teardown ordering, the synthetic data factories, what CI runs on which Python versions, how to run the cluster-marked integration smokes, and the adding-tests checklist. | +| [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md) | Every workflow in `.github/workflows/`: triggers and behavior for the deterministic and assistant-driven families, the hidden dedup-marker convention, the `allowed_bots` gotchas, the assistant-handle invocation rules, and the issue/PR lifecycle the automation assumes. | +| [`RELEASE.md`](RELEASE.md) | The release contract: how one version string couples the git tag, GitHub Release, PyPI package, and HuggingFace weights tag; packaging and version single-sourcing; runtime weight resolution; the CD workflow's gates (verify-don't-create); PyPI trusted-publishing setup; and the step-by-step release runbook. | +| [`GPU_RUNTIME_GUIDE.md`](GPU_RUNTIME_GUIDE.md) | Runbook for building, running, and debugging Aetherscan across GPU architectures — the Blackwell (RTX PRO 6000) workstation and the Ampere (A4000) cluster — unified by the NGC container runtime. Covers container builds, image verification, the four GPU-related CLI flags, CUDA/NCCL debugging recipes, the fallback escalation ladder, and cross-machine checkpoint interop. Read this before building the container or running on a new GPU architecture (especially Blackwell/sm_120), or after any run that dies with a `CUDA_ERROR_INVALID_PTX`/NCCL/OOM failure. | +| [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md) | Deep-dive into Aetherscan's singleton-config + CLI architecture: the three flag-routing patterns (A/B/C), the four cross-mode contamination barriers, the validation + fix-proposal layer, and the checklist for adding a new flag. Read this before touching `config.py` or `cli.py`. | + + diff --git a/docs/placeholder b/docs/placeholder deleted file mode 100644 index 8b3ffa1..0000000 --- a/docs/placeholder +++ /dev/null @@ -1,2 +0,0 @@ -# TODO: -# Document (individually) db architecture, logger architecture, resource lifecycle manager, resource monitoring, training orchestration, inference orchestration, preprocessing (including energy detection and signal injection), model architecture design (beta-VAE, RF), github actions (especially the claude integrations) From 3d9400963fd75e143928ef0f886b33e603a08075 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 15:01:25 +0800 Subject: [PATCH 12/20] Refresh CLAUDE.md, repo-context skill, and CONTRIBUTING for the docs suite - CONTRIBUTING: structure tree + module table gain round_data/run_state/pfb/inference_viz and tests/fixtures; the architecture-section TODO resolves to docs/ARCHITECTURE.md - SKILL.md: source tree updated; Reference Files list the full docs suite - CLAUDE.md: More-detail section points at the docs/README.md index Co-Authored-By: Claude Fable 5 --- .../skills/aetherscan-repo-context/SKILL.md | 18 ++++++++++++++++- CLAUDE.md | 2 +- CONTRIBUTING.md | 20 ++++++++++++++----- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/.claude/skills/aetherscan-repo-context/SKILL.md b/.claude/skills/aetherscan-repo-context/SKILL.md index 0c8fb95..826cca4 100644 --- a/.claude/skills/aetherscan-repo-context/SKILL.md +++ b/.claude/skills/aetherscan-repo-context/SKILL.md @@ -101,8 +101,12 @@ src/aetherscan/ ├── cli.py # Argument parsing, validation, config override ├── config.py # Configuration dataclasses & defaults (singleton) ├── train.py # Training orchestration, curriculum learning, checkpointing +├── round_data.py # Disk-backed (memmap) round datasets + background producer process +├── run_state.py # Persisted training-run manifest (stage-aware resume) ├── inference.py # Inference orchestration, candidate detection +├── inference_viz.py # End-of-run inference visualization suite ├── preprocessing.py # Loading / downsampling / log-normalization + energy detection +├── pfb.py # PFB static passband equalization (bandpass flattening) ├── data_generation.py # Synthetic signal injection (setigen) ├── models/{vae,random_forest}.py ├── db/db.py # Thread-safe SQLite, async queue-based writes @@ -113,7 +117,8 @@ utils/ # fetch_run_outputs.sh, find_optimal_configs.py, # get_system_info.sh, kill_pipeline.sh, print_cli_help.py, # run_container.sh, start_tmux_session.sh, # verify_train_test_files.py -docs/ # GPU_RUNTIME_GUIDE.md, CONFIG_AND_CLI.md, README.md, assets/ +docs/ # Full technical doc suite, one doc per pipeline surface — + # indexed in docs/README.md; start at docs/ARCHITECTURE.md tests/ # Pytest suite: conftest.py (singleton-reset fixtures, synthetic # data factories), unit/, integration/ (gpu+cluster-marked smokes). # Default selection: pytest -m "not gpu and not cluster" -q @@ -205,5 +210,16 @@ Paths relative to the repo root: - `SECURITY.md` — security policy, secrets management, token rotation - `KNOWN_ISSUES.md` — known bugs and workarounds - `AI_POLICY.md` — AI usage policy (read before AI-assisted contributions) +- `docs/README.md` — index of the technical documentation suite (one doc per surface) +- `docs/ARCHITECTURE.md` — system map: data model, module map, init order, artifact layout +- `docs/TRAINING_PIPELINE.md` — rounds, round data + producer, retries, every training plot +- `docs/INFERENCE_PIPELINE.md` — catalogs, streaming flow, manifest retries, inference figures +- `docs/PREPROCESSING.md` — energy detection math (PFB/spline, k² derivation), signal injection +- `docs/MODELS.md` — Beta-VAE architecture/loss math, RF features + threshold semantics +- `docs/DATABASE.md` — schema, writer thread, flush/supersede protocols, migrations +- `docs/RUNTIME_SERVICES.md` — logger/Slack, ResourceManager lifecycle, resource monitor +- `docs/TESTING.md` — suite layout, markers, isolation fixtures, CI, cluster smokes +- `docs/GITHUB_AUTOMATION.md` — every workflow, dedup guards, assistant-handle rules +- `docs/RELEASE.md` — version-coupling contract, CD gates, release runbook - `docs/GPU_RUNTIME_GUIDE.md` — container build/runtime runbook - `docs/CONFIG_AND_CLI.md` — config system deep dive diff --git a/CLAUDE.md b/CLAUDE.md index fd01916..3dbdb5c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -42,4 +42,4 @@ ruff lint+format, 100-char lines, Python 3.10 target ([`pyproject.toml`](pyproje ## More detail -On-demand deep-dive skill: [`.claude/skills/aetherscan-repo-context/SKILL.md`](.claude/skills/aetherscan-repo-context/SKILL.md) — install paths, config/CLI system, architecture patterns, full workflow & security. +On-demand deep-dive skill: [`.claude/skills/aetherscan-repo-context/SKILL.md`](.claude/skills/aetherscan-repo-context/SKILL.md) — install paths, config/CLI system, architecture patterns, full workflow & security. Per-surface technical docs (architecture, training, inference, preprocessing, models, database, runtime services, testing, automation, releases) are indexed in [docs/README.md](docs/README.md) — start at [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4ae9c14..9be8d6b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -169,8 +169,12 @@ Aetherscan/ │ ├── cli.py # Argument parsing, validation, config override │ ├── config.py # Configuration defaults │ ├── train.py # Training orchestration +│ ├── round_data.py # Disk-backed (memmap) round datasets + producer process +│ ├── run_state.py # Persisted training-run manifest (stage-aware resume) │ ├── inference.py # Inference orchestration -│ ├── preprocessing.py # Data preprocessing +│ ├── inference_viz.py # Inference visualization suite +│ ├── preprocessing.py # Data preprocessing + energy detection +│ ├── pfb.py # PFB static passband equalization │ ├── data_generation.py # Synthetic signal injection │ ├── models/ │ │ ├── __init__.py # Model exports @@ -189,9 +193,11 @@ Aetherscan/ │ └── manager/ │ ├── __init__.py # Manager exports │ └── manager.py # Resource lifecycle management -├── docs/ # Documentation (placeholder; no docs yet) +├── docs/ # Technical documentation suite, one doc per pipeline surface +│ # (indexed in docs/README.md; start at docs/ARCHITECTURE.md) ├── tests/ # Pytest suite (see Testing section below) │ ├── conftest.py # Singleton-reset fixtures, tmp paths, synthetic data factories +│ ├── fixtures/ # Small recorded data fixtures (e.g. real ED channel slice) │ ├── unit/ # Fast, hardware-independent unit tests (run in CI) │ └── integration/ # gpu+cluster-marked end-to-end smoke tests ├── utils/ # Utility scripts @@ -231,8 +237,12 @@ Aetherscan/ | `cli.py` | Argument parsing, validation, config override | | `config.py` | All configuration dataclasses and defaults | | `train.py` | Training orchestration, curriculum learning, checkpointing | +| `round_data.py` | Disk-backed round datasets, `.done` manifests, background producer | +| `run_state.py` | Persisted training-run manifest driving stage-aware resume | | `inference.py` | Model inference, candidate detection | +| `inference_viz.py` | End-of-run inference visualization suite | | `preprocessing.py` | Data loading / downsampling / log-normalization + energy detection | +| `pfb.py` | Polyphase-filterbank static passband response (bandpass flattening) | | `data_generation.py` | Synthetic signal injection using setigen | | `models/vae.py` | Beta-VAE architecture with composite clustering loss | | `models/random_forest.py` | Scikit-learn RF wrapper | @@ -241,9 +251,9 @@ Aetherscan/ | `manager/manager.py` | Resource lifecycle management (pools, shared memory) | | `logger/` | Multi-handler logging with Slack integration | -> [!WARNING] -> -> # TODO: add an architecture section? (`docs/architecture.md`?) +For how these modules fit together — data flow, process/thread topology, initialization +order, and where every artifact lands — see [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md); +the full per-surface documentation suite is indexed in [`docs/README.md`](docs/README.md). --- From 0d9578d41236e3584f9a16261996edbc25282cf8 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 15:06:14 +0800 Subject: [PATCH 13/20] Fix review findings: RF accuracy-curve filename, background RAM math, round_XX tag usage Co-Authored-By: Claude Fable 5 --- docs/ARCHITECTURE.md | 2 +- docs/PREPROCESSING.md | 4 ++-- docs/TRAINING_PIPELINE.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f6361ba..cbcee70 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -177,7 +177,7 @@ artifact filename and every DB row. Accepted formats | --- | --- | --- | | `YYYYMMDD_HHMMSS` | `20260712_143000` | Default (import-time timestamp) — every untagged run gets a unique one. | | `final_vX` | `final_v1` | Release-grade training runs. | -| `round_XX` | `round_05` | Reserved for per-round checkpoints (written by the pipeline, not passed by users). | +| `round_XX` | `round_05` | Per-round checkpoints (written by the pipeline; passed as `--load-tag` to resume from a specific round — `CheckpointConfig.infer_start_round()` derives the start round from it). | | `test_vX` | `test_v17` | Smoke/test runs. | `train.py:get_latest_tag()` ranks the families `final_vX > round_XX > timestamp > test_vX` diff --git a/docs/PREPROCESSING.md b/docs/PREPROCESSING.md index 0ee330d..586fc0c 100644 --- a/docs/PREPROCESSING.md +++ b/docs/PREPROCESSING.md @@ -206,8 +206,8 @@ block, and a pool downsamples each cadence ×8 per observation (`_downsample_wor Cadences containing NaN/Inf or with non-positive max are dropped. **Log-normalization is deferred** — training-side log-norm happens per sample *after* signal injection (injection must operate on linear intensities). Loading stops at `num_target_backgrounds` (45 000) -cadences ≈ 34 GB at model resolution, held in RAM for the whole run and shared with injection -workers via shared memory. +cadences — about 9 GB at model resolution (45 000 × 6 × 16 × 512 × 4 B) — held in RAM for +the whole run and shared with injection workers via shared memory. **Inference snippets** (`load_inference_data`): same chunked SHM+pool pattern, branching on the stored width: files already at `width_bin // downsample_factor` (512 — written by diff --git a/docs/TRAINING_PIPELINE.md b/docs/TRAINING_PIPELINE.md index 322912c..41cbba1 100644 --- a/docs/TRAINING_PIPELINE.md +++ b/docs/TRAINING_PIPELINE.md @@ -343,7 +343,7 @@ share `rf_shap_values_{tag}.joblib` (computed once, cached). | `rf_shap_loss_monitoring_{tag}.png` | Per-sample log-loss histogram by class + per-feature loss-increasing/decreasing decomposition. | The high-loss tail is your inspection queue; any feature whose net contribution *increases* loss is actively harmful. | | `rf_shap_explanation_clustering_{tag}.png` | UMAP of SHAP explanation vectors, colored by subtype, markers for correct/incorrect. | Errors concentrated in one explanation cluster = a single confusable mode (fixable with targeted data); errors scattered everywhere = noise-floor performance. | | `rf_calibration_curve_{tag}.png` | Reliability diagram (quantile-binned) + Brier/ECE + probability histogram. | With a 0.99 threshold, calibration in the top bins is what matters: if the top-bin empirical frequency is well below its predicted probability, the threshold is less conservative than it looks. | -| `rf_ensemble_accuracy_curve_{tag}.png` | Cumulative accuracy vs number of trees (val + train-subsample baseline). | Should saturate well before 1000 trees; if it's still climbing at the end, raise `rf.n_estimators`. | +| `rf_oob_accuracy_curve_{tag}.png` (from `plot_rf_ensemble_accuracy_curve`) | Cumulative accuracy vs number of trees (val + train-subsample baseline), elbow annotated. | Should saturate well before 1000 trees; if it's still climbing at the end, raise `rf.n_estimators`. | | `rf_latent_decision_boundary_nn{n}_md{m}_{tag}.png` | RF P(true) contour over each persisted cadence-level UMAP plane, val points + 0.5 contour. | A coherent boundary separating the true classes; ragged islands = the forest partitioning noise. Depends on the UMAPs from `plot_latent_space_gif`, so `vae_plots` must have succeeded. | ### `resource_utilization_{tag}.png` From 642703f954320fd6491260efc1d10b54072e3149 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 17:29:35 +0800 Subject: [PATCH 14/20] Add docs/BENCHMARKING.md: stage timers, report tool, micro-benchmarks, baselines Document the always-on stage-timing suite: the stage_timer/record_stage API and its thread-local dot-name nesting, the instrumented stage map, the report tool (stage tree, flame timeline, rules-driven suggestions), the benchmarks/ micro- benchmarks, and the current blpc3 baselines plus end-to-end speedups. Wire it into the docs index (replacing the Phase-A placeholder row). Co-Authored-By: Claude Opus 4.8 --- docs/BENCHMARKING.md | 237 +++++++++++++++++++++++++++++++++++++ docs/DATABASE.md | 66 ++++++++--- docs/INFERENCE_PIPELINE.md | 80 +++++++++++-- docs/README.md | 4 +- docs/RELEASE.md | 7 +- docs/RUNTIME_SERVICES.md | 25 +++- 6 files changed, 388 insertions(+), 31 deletions(-) create mode 100644 docs/BENCHMARKING.md diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md new file mode 100644 index 0000000..b63f525 --- /dev/null +++ b/docs/BENCHMARKING.md @@ -0,0 +1,237 @@ +# Benchmarking + +Aetherscan carries always-on stage timing plus a set of offline tools to read it. This +document covers the four pieces: the `stage_timer` instrumentation +([`src/aetherscan/benchmark.py`](../src/aetherscan/benchmark.py)), the `pipeline_stages` DB +table it writes to, the annotated resource plot the monitor renders, the report tool +([`utils/benchmark_report.py`](../utils/benchmark_report.py)), and the standalone +micro-benchmarks ([`benchmarks/`](../benchmarks/)). It closes with the current baseline +numbers and how to read the annotated resource plot. + +## TL;DR + +- Every pipeline stage is wrapped in `with stage_timer("dotted.name"):`, which queues one + `pipeline_stages` row (start, end, duration, tag) on the DB writer thread — two + `time.time()` calls and one queue put, so the timers stay on in production runs. +- Stage names are **hierarchical dot-names** and nest automatically per-thread: a + `stage_timer` entered while another is active on the same thread records its name relative + to the parent, so instrumented library code inherits whatever umbrella span the caller + opened without name plumbing. +- `python utils/benchmark_report.py --save-tag ` prints the stage tree, writes a + flame-style timeline PNG, and flags likely bottlenecks from `pipeline_stages` joined with + `system_resources`. +- The 1 Hz resource plot overlays the top-level stages as labeled bands + (`monitor.annotate_stages`), so a CPU plateau reads as "round 3 data generation" at a + glance. +- `benchmarks/*.py` time the hot CPU kernels in isolation, in seconds instead of a full run. + +## Stage timers (`benchmark.py`) + +Two entry points, both **failure-safe by contract** — benchmarking must never be able to +fail the pipeline, so a missing DB (unit tests, dev scripts) or a serialization hiccup +downgrades to a debug/warning log and the block's result is untouched: + +- **`stage_timer(stage, tag=None, metadata=None)`** — a `ContextDecorator` usable as either + `with stage_timer("train.round_02.data_generation"):` or `@stage_timer("inference.viz")`. + On `__enter__` it resolves its full name against the thread's active-stage stack and pushes + itself; on `__exit__` it pops and calls `record_stage()`. On exception the span is **still + recorded** (metadata gains `status="failed"` plus the error string) and the exception + propagates — the timer never suppresses it. +- **`record_stage(stage, start_time, end_time, tag=None, metadata=None)`** — writes a span + measured elsewhere with explicit timestamps and **no nesting resolution** (the name is used + as-is). This is the seam for work that happens in another process: the round-data producer + generates data in a separate process that can't touch the DB writer queue (a thread-only + `queue.Queue`), so it reports its `(start, end)` over its result-message channel and the + main-process drainer calls `record_stage()` (the `"timing"` branch of the `RoundDataDrainer` + thread's message handler, [`round_data.py`](../src/aetherscan/round_data.py)). + +### Automatic nesting + +The active-stage stack is **thread-local** (`_ACTIVE_STAGES`), which matters because training, +prefetch preprocessing, and the producer drainer all time stages concurrently and must not +interleave names. Within one thread, entering `stage_timer("encode")` while +`"inference.infer_cadence_001"` is active records `inference.infer_cadence_001.encode`; with +no active parent it records `encode` as-is. `current_stage()` exposes the innermost active +full name. + +`tag` defaults to `config.checkpoint.save_tag` (the run's provenance key). `metadata`, when +given, is a small JSON-serializable dict stored on the row (e.g. `{"source": "producer"}` on +the data-generation span to distinguish the overlapping-producer path from the in-process +one, which the report tool keys its first suggestion off). + +### What is instrumented + +Every span below is a real `stage_timer`/`record_stage` call site in the current code. Names +are shown at full depth; the leaf component is what the report tool and resource plot label. + +| Stage (dotted name) | Where | Notes | +| --- | --- | --- | +| `train.load_backgrounds` | `main.py` | Background plate load before the round loop | +| `train.round_{k:02d}` | `train.py` | Umbrella span for one round | +| `train.round_{k:02d}.data_generation` | `train.py` (in-process) / `round_data.py` (producer) | `metadata.source` = `in-process` or `producer` | +| `train.round_{k:02d}.epochs` | `train.py` | Round-level epoch span (`record_stage`; per-epoch detail lives in `training_stats`) | +| `train.round_{k:02d}.plots` | `train.py` | Per-round plots | +| `train.round_{k:02d}.checkpoint_save` | `train.py` | `save_models(round_XX)` | +| `train.vae_plots` | `train.py` | End-of-VAE plot stage | +| `train.rf` (`.data_generation`, `.encode`, `.fit`) | `train.py` | RF training and its sub-stages | +| `train.rf_plots` | `train.py` | RF plot stage | +| `train.final_save` | `train.py` | Final model + config save (+ HF upload) | +| `inference.infer` | `inference.py` | Legacy single-array `run_inference` span | +| `inference.preprocess_cadence_{i:03d}` (`.read_ed_on{1..3}`, `.dedup`, `.extract`) | `main.py` / `preprocessing.py` | Per-cadence energy detection | +| `inference.infer_cadence_{i:03d}` (`.load_lognorm`, `.encode`, `.rf`, `.db_write`) | `main.py` / `inference.py` | Per-cadence GPU inference + write | +| `inference.viz` | `main.py` | Visualization suite | + +## The `pipeline_stages` table (schema v3) + +`record_stage()` calls `db.write_pipeline_stage()`, which queues a +`("pipeline_stages", values)` record on the writer thread like every other write. The table +(columns `stage`, `start_time`, `end_time`, `duration_s`, `tag`, `metadata`; index +`(tag, start_time)`) is documented in full in [`DATABASE.md`](DATABASE.md#pipeline_stages-stage-timers-schema-v3). +Retried stages simply append new rows — every attempt keeps its own span, so the report tool +and resource plot see the full history. There is **no `superseded` column**: timing spans are +attempt-agnostic history, like `system_resources`. + +## Annotated resource plot (`monitor.annotate_stages`) + +When the resource monitor renders its shutdown plot, it optionally overlays this run's +top-level `pipeline_stages` spans as labeled translucent bands on the CPU panel, turning the +utilization curve into a self-explaining timeline. This is documented alongside the plot in +[`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md#stage-annotations). The essentials: + +- Config knob **`monitor.annotate_stages`** (default `True`) gates it. +- Only **depth ≤ 2** dot-names are drawn (`train.round_03`, not + `train.round_03.epochs`) — deep spans stay report-tool-only so the panel doesn't drown in + bands (`select_annotation_spans` / `_ANNOTATION_MAX_DEPTH`). +- The monitor flushes the DB first so spans recorded moments before shutdown (`final_save`, + `inference.viz`) make it onto the plot. + +## The report tool (`utils/benchmark_report.py`) + +A dev tool (it may `print` — `utils/` is exempt from the no-`print` rule) that reads the +SQLite file directly with the stdlib `sqlite3` + numpy + matplotlib only (no `aetherscan` +imports), so it also runs against a database fetched from a cluster with +`utils/fetch_run_outputs.sh`: + +```bash +# Default db path: {AETHERSCAN_OUTPUT_PATH}/db/aetherscan.db +python utils/benchmark_report.py --save-tag final_v1 +# Explicit db (e.g. a copy pulled off the cluster) +python utils/benchmark_report.py --save-tag test_v18 --db-path /path/to/aetherscan.db +``` + +It produces three things for one `--save-tag`: + +1. **Stage tree** (stdout) — the `pipeline_stages` spans folded into a tree by dot-name + component (`build_stage_tree`), each node showing its span count `n`, total duration, and + percentage of its parent. A node's **total** is its own summed span time when it has spans + (an umbrella span already covers its children's time), else the sum over children (pure + grouping nodes like a bare `train`). **Self time** is total minus child totals, floored at + 0 — children that run concurrently with a parent (producer data generation overlapping a + round) can sum past the parent's wall-clock, and the floor keeps that sane. +2. **Report PNG** — `plots/benchmark_report_{tag}.png`: a flame-style timeline with one lane + per dot-depth (bars colored by top-level family, overlaps = concurrent stages) above a + "top 10 slowest stages" table ranked by **self time** (where wall-clock actually went, + rather than umbrella spans). +3. **Suggestions** (stdout) — a rules-driven bottleneck section, sourced by joining + `pipeline_stages` with `system_resources`. + +### Suggestion rules + +| Rule | Trigger | Suggestion | +| --- | --- | --- | +| Data-gen dominates a round | `round.data_generation / round.total ≥ 0.30` | In-process (`metadata.source`) → enable `--overlap-data-generation`. Producer path → only fires when the round actually *waited* on generation (generation ended after the round started, 1 s tolerance) → raise `manager.n_processes` / `--data-gen-task-size` or lower `--num-samples-beta-vae`. Fully-overlapped generation is free and produces no suggestion. | +| Training input-bound | mean GPU % during an `epochs` span `< 40%` | Check the `tf.data` feed (batch sizes, host-side preprocessing). | +| ED dominates inference | summed `preprocess_cadence_* / inference wall ≥ 0.60` | Energy detection is the bottleneck; raise ED worker count (`manager.n_processes`) before anything GPU-side. | +| Encode input-bound | mean GPU % across `encode` spans `< 40%` | Encoding is input-bound; consider a larger inference batch (`--per-replica-batch-size`). | +| RAM pressure | peak `ram/system_total ≥ 90%` | Names the stage holding the peak; reduce chunk/sample sizes or disable `keep_round_data`. | + +Thresholds are module constants (`DATA_GEN_ROUND_FRACTION`, `GPU_UTIL_INPUT_BOUND`, +`PREPROCESS_WALL_FRACTION`, `RAM_PEAK_WARN`) — tune them there if a rule is too eager for your +hardware. + +## Micro-benchmarks (`benchmarks/`) + +Small standalone scripts that time the pipeline's hot CPU kernels in isolation, so a change to +one can be measured in seconds instead of via a full run. **Not** collected by pytest +(`testpaths = ["tests"]`); run on demand. Each prints ops/s and writes a JSON result to +`benchmarks/results/` (gitignored). See [`benchmarks/README.md`](../benchmarks/README.md) for +the maintained baseline table and per-flag detail. + +```bash +python benchmarks/bench_normality.py # sliding-window normality test (energy detection) +python benchmarks/bench_injection.py # setigen signal injection (training data gen) +python benchmarks/bench_lognorm_downsample.py # per-cadence downsample + log-norm (load path) +python benchmarks/bench_pfb_vs_spline.py # PFB static equalization vs per-channel spline fit +# On the clusters, through the container: +./utils/run_container.sh python benchmarks/bench_normality.py +``` + +| Script | Kernel it isolates | Pipeline stage it models | +| --- | --- | --- | +| `bench_normality.py` | vectorized `_sliding_normality_k2` vs the historical per-window `scipy.stats.normaltest` loop | ED thresholding (`inference.*.read_ed_on*`) | +| `bench_injection.py` | `data_generation.new_cadence` narrowband injection | `train.round_XX.data_generation` | +| `bench_lognorm_downsample.py` | per-obs `downscale_local_mean` (×8) + per-cadence `log_norm` | stamp downsample + `inference.*.load_lognorm` | +| `bench_pfb_vs_spline.py` | `pfb.equalize_passband` vs `_spline_flatten_bandpass` on one 1M-bin coarse channel | bandpass flattening inside ED | + +All numbers are single-process; the pipeline parallelizes each kernel across +`manager.n_processes` workers, so whole-stage throughput scales roughly with core count. + +## Baseline numbers + +Micro-benchmark speedups below are best-of-3 at production shapes on **blpc3 (EPYC 7313, +32 cores, NGC 25.02 container)** — the maintained table (plus a MacBook Air M3 column) is in +[`benchmarks/README.md`](../benchmarks/README.md#baseline-numbers). Higher is better; expect +~10% run-to-run noise. + +| Kernel | Vectorized / new | Baseline / old | Speedup | +| --- | --- | --- | --- | +| Sliding-window normality (`bench_normality`) | 83,667 windows/s | 679 windows/s (scipy loop) | **123×** | +| Bandpass flattening (`bench_pfb_vs_spline`) | 59.6 channels/s (PFB) | 5.0 channels/s (spline fit) | **11.9×** | + +End-to-end numbers, measured with the stage timers and resource plot on real runs: + +- **Inference (blpc3, 5× RTX PRO 6000, subset CSV run).** ~6.2× faster end-to-end after the + energy-detection overhaul. Energy detection fell from ~45 min/cadence to ~2 min/cadence + (the vectorized D'Agostino–Pearson test above is the main lever), and per-cadence stamp + storage dropped ~8× via downsample-at-extraction (the frequency axis is reduced ×8 in the + extraction worker instead of storing full-width stamps — see + [`PREPROCESSING.md`](PREPROCESSING.md)). +- **Bandpass method (blpc3, subset CSV run).** PFB and spline flattening come out at roughly + parity in **whole-pipeline** wall-clock and produce near-identical hit sets, even though the + micro-benchmark shows PFB is 11.9× cheaper on the isolated per-channel kernel. That is the + expected outcome of the report tool's finding that, once the normality test is vectorized, + **stamp extraction — not bandpass flattening or thresholding — is the inference + bottleneck**; shaving an already-small stage barely moves the total. PFB stays the default + for its determinism (no per-channel fit); spline remains available via + `--bandpass-method spline`. +- **Training (multi-GPU Ampere node, 6× RTX A4000 smoke).** The disk-backed round data + + producer redesign holds steady-state RSS at ~26% of the 503 GB node, versus the old + in-RAM design that saw-toothed to ~99% and SIGKILL-ed the run in round 2. Data-generation + wall-clock in rounds 2+ matches round 1 (the GIL-contention regression that made later + rounds slower is gone), and round *k+1*'s data is generated by the producer process while + round *k* trains (visible as a `data_generation` band overlapping the previous round's + `epochs` band on the annotated resource plot). + +## Reading the annotated resource plot + +The shutdown resource plot (`plots/resource_utilization_{tag}.png`, +[`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md#the-shutdown-plot)) has three time-aligned panels +(CPU, RAM, GPU) with the x-axis in minutes since monitor start. With `annotate_stages` on, +the top-level stage bands on the CPU panel tell you which region is which: + +- **CPU plateau under a `data_generation` band** — the worker pool is saturating cores + generating a round. If the band overlaps the previous round's `epochs`, the producer is + overlapping correctly (good); if a round's `epochs` band starts only *after* its + `data_generation` band ends, that round waited on generation — the report tool's data-gen + rule will flag it. +- **GPU band with low utilization during `epochs`** — input-bound training (feed can't keep + the GPUs busy); cross-check the report tool's GPU-util rule. +- **Alternating CPU/GPU activity across `preprocess_cadence_*` / `infer_cadence_*` bands** — + per-cadence streaming inference, CPU-heavy energy detection overlapping the previous + cadence's GPU encode. +- **RAM approaching the top of its panel** — memory pressure; the report tool's RAM rule + names the stage holding the peak. The training redesign should keep this well clear of the + node ceiling. + +Pair the plot (which region) with `benchmark_report.py` (how long, and the suggestion) to go +from "this run felt slow" to a specific stage and knob. diff --git a/docs/DATABASE.md b/docs/DATABASE.md index a482f0f..96e4b96 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -12,7 +12,7 @@ One SQLite file — `{output_path}/db/aetherscan.db`, WAL mode — behind a thre `queue.Queue` drained by **one background writer thread** that batches rows and commits with `executemany()`; reads open short-lived connections directly. Failed-attempt rows are never deleted — they're flagged `superseded = 1`, and every query filters them out by default. -Schema evolution is a minimal `PRAGMA user_version` gate (currently version 2). +Schema evolution is a minimal `PRAGMA user_version` gate (currently version 3). > [!IMPORTANT] > The write queue is a **thread** queue, not process-safe. Worker *processes* must never call @@ -85,13 +85,23 @@ failed attempt wrote. ## Schema -Six tables, created idempotently (`CREATE TABLE IF NOT EXISTS`) in `_init_database()`, each +Seven tables, created idempotently (`CREATE TABLE IF NOT EXISTS`) in `_init_database()`, each with a composite index matched to its dominant filter pattern. All rows carry `timestamp` (write time, `REAL` Unix seconds) and `tag` (the run's save tag — the primary provenance key). - +| Table | Rows | `superseded`? | Added in | +| --- | --- | --- | --- | +| `system_resources` | 1 Hz monitor samples | no (attempt-agnostic history) | v0 | +| `injection_stats` | per generated cadence | yes | v0 (+ column v1) | +| `training_stats` | per training epoch | yes | v0 (+ column v1) | +| `latent_snapshots` | per viz cadence per capture | yes | v0 (+ column v1) | +| `inference_results` | positives only | yes | v0 (+ column v1) | +| `inference_cadences` | per-cadence run manifest | yes | v2 | +| `pipeline_stages` | per timed stage span | no (attempt-agnostic history) | v3 | + +The `superseded` column and its default-filtering are what make same-tag retries safe; see +[the migration section](#schema-migration) for how the `user_version` stamp maps to these. ### `system_resources` @@ -193,19 +203,44 @@ resume ([`INFERENCE_PIPELINE.md`](INFERENCE_PIPELINE.md)). Index: `(tag, npy_path, status)` — the resume lookup. +### `pipeline_stages` (stage timers, schema v3) + +One row per timed pipeline-stage span, written by the always-on stage timers in +[`aetherscan.benchmark`](../src/aetherscan/benchmark.py) (`stage_timer` / `record_stage`) via +`db.write_pipeline_stage()`. Read back by [`utils/benchmark_report.py`](../utils/benchmark_report.py) +and the monitor's stage-band overlay. Full context in [`BENCHMARKING.md`](BENCHMARKING.md). + +| Column | Type | Notes | +| --- | --- | --- | +| `stage` | TEXT | Hierarchical dot-name (`train.round_02.data_generation`) | +| `start_time`, `end_time` | REAL | Unix timestamps bounding the span | +| `duration_s` | REAL | Derived (`end_time - start_time`) at write time so the table stays internally consistent | +| `metadata` | TEXT | Optional pre-serialized JSON (e.g. `{"source": "producer"}`, or `{"status": "failed", ...}` on a span whose block raised) | + +Index: `(tag, start_time)`. **No `superseded` column** — timing spans are attempt-agnostic +history like `system_resources`. Retried stages simply append new rows, so consumers see every +attempt, each with its own span. + ## Schema migration `_migrate_schema()` runs on every startup, gated on `PRAGMA user_version` -(`_SCHEMA_VERSION = 2`): +(`_SCHEMA_VERSION = 3`). The stamp maps to schema features as: -- **v0 → v1**: `ALTER TABLE ... ADD COLUMN superseded INTEGER DEFAULT 0` on - `training_stats`, `injection_stats`, `latent_snapshots`, `inference_results` — the only - in-place change SQLite supports is additive `ADD COLUMN`, which is exactly what supersede - semantics needed. A per-table column-existence check (`PRAGMA table_info`) keeps each step - idempotent even if `user_version` was lost (a db file copied without its journal). -- **v1 → v2**: the `inference_cadences` table — no migration step needed, because - `CREATE TABLE IF NOT EXISTS` in `_init_database()` creates it for old and new databases - alike before the migration runs; only the version stamp advances. +| `user_version` | What it added | Migration work | +| --- | --- | --- | +| v0 | pre-versioning baseline (any db with no stamp) | — | +| v1 | `superseded INTEGER DEFAULT 0` on `training_stats`, `injection_stats`, `latent_snapshots`, `inference_results` | additive `ALTER TABLE ... ADD COLUMN` | +| v2 | the `inference_cadences` run-manifest table | none (whole-table `CREATE TABLE IF NOT EXISTS`) | +| v3 | the `pipeline_stages` stage-timing table | none (whole-table `CREATE TABLE IF NOT EXISTS`) | + +- **v0 → v1**: `ALTER TABLE ... ADD COLUMN superseded INTEGER DEFAULT 0` on the four tables + above — the only in-place change SQLite supports is additive `ADD COLUMN`, which is exactly + what supersede semantics needed. A per-table column-existence check (`PRAGMA table_info`) + keeps the step idempotent even if `user_version` was lost (a db file copied without its + journal). +- **v1 → v2** (`inference_cadences`) and **v2 → v3** (`pipeline_stages`): no migration step + needed — `CREATE TABLE IF NOT EXISTS` in `_init_database()` creates each new table for old + and new databases alike *before* `_migrate_schema()` runs; only the version stamp advances. Fresh databases get the full current schema from the CREATE statements and are just stamped. The pattern to follow for future changes: bump `_SCHEMA_VERSION`, add a @@ -217,7 +252,8 @@ The pattern to follow for future changes: bump `_SCHEMA_VERSION`, add a Each table has a `query_*` method returning `list[dict]` (`query_system_resource`, `query_injection_stat`, `query_injection_stat_stability`, `query_training_stat`, `query_latent_snapshots`, `query_latent_snapshot_keys`, -`query_inference_result`, `query_inference_cadences`). Shared conventions: +`query_inference_result`, `query_inference_cadences`, `query_pipeline_stages`). Shared +conventions: - **String filters** (`tag`, `stat_name`, `status`, ...) accept a single value (`=`) or a list (`IN`); **range filters** come as `start_*`/`end_*` pairs (inclusive); @@ -256,6 +292,8 @@ Rules of thumb at full-scale defaults (dominant terms only): - **`system_resources`**: (4 + 2 × n_GPUs) rows/second — ~1.2 M rows/day on a 6-GPU node. - **`inference_results`**: positives only; at a 0.99 threshold this stays small by construction. `inference_cadences`: a handful of rows per cadence. +- **`pipeline_stages`**: one row per timed stage span — tens per training run, a few per + inference cadence. Negligible. WAL mode keeps readers (plots, resume queries) unblocked during heavy write phases; the WAL file is checkpointed back into the main db periodically by SQLite itself. diff --git a/docs/INFERENCE_PIPELINE.md b/docs/INFERENCE_PIPELINE.md index 7562e8c..be8b2e0 100644 --- a/docs/INFERENCE_PIPELINE.md +++ b/docs/INFERENCE_PIPELINE.md @@ -62,12 +62,52 @@ cross-run caching). ## Model loading - - -`InferencePipeline.__init__` → `init_models()` loads the two models from explicit paths: +### Artifact resolution: local paths or the HuggingFace Hub + +Inference needs three artifacts — the encoder, the Random Forest, and the training run's +`config.json`. They come from local disk **or** the HuggingFace Hub, resolved *before* +validation by `hf_hub.resolve_inference_artifacts(args)` (called from `main()` immediately +after argument parsing). The trio is **all-or-none** +([`src/aetherscan/hf_hub.py`](../src/aetherscan/hf_hub.py)): + +- **All three local paths given** (`--encoder-path` / `--rf-path` / `--config-path`) → used + as-is; any `--hf-revision` is ignored (logged). The offline / cluster path. +- **None given** → the artifacts are downloaded from the Hub and their cache paths are written + onto `args`, exactly as if passed on the CLI, so validation / `apply_saved_config` / model + load run unchanged. The resolved revision is also written to `config.hf.revision` for + provenance in the saved inference config. +- **A partial set (one or two paths)** → left untouched; `collect_validation_errors` reports + the missing ones. Mixing local and Hub-sourced artifacts would silently pair mismatched + models, so it is rejected rather than half-resolved. + +When downloading, the **revision** is chosen by `resolve_hf_revision` in precedence order: + +1. **`--hf-revision `** — an explicit revision (a training tag `final_v1`, a release tag + `v1.0.0`, or a commit); returned as-is, existence checked by the download itself. +2. **`v{__version__}`** — the version-coupled default. When the package is an **installed + release**, `version_default_revision()` returns `f"v{__version__}"`, so + `pip install aetherscan==1.0.0` + bare inference pulls exactly the `v1.0.0` weights. The + guard is strict — only a `vX.Y.Z` version matches (`_SEMVER_TAG_PATTERN`). Source-tree / + container runs (`PYTHONPATH=src`, the NGC image) have no installed distribution, so + `__version__` is the `"0.0.0.dev0"` fallback + ([`__init__.py`](../src/aetherscan/__init__.py)); `.dev` / `rc` / `post` pre-releases also + fail the match — all fall through to step 3. This is deliberately **not** existence-checked: + an installed release whose weights tag is missing must fail loudly (the release blessing + step was skipped), never silently pull some other version. +3. **Latest `vX.Y.Z` release tag**, else **latest `final_vX` training tag** on the repo + (`select_default_revision`; numeric comparison, so `v1.10.0 > v1.9.9`; tags in neither + family — `test_vX`, timestamps — never win). Raises `RuntimeError` with guidance when + nothing resolves. + +Downloads go through `hf_hub_download` (revision-pinned, cached under `HF_HOME` / +`~/.cache/huggingface`; repeated runs hit the cache); the public repo needs no token. The repo +defaults to `config.hf.repo_id` (`zachtheyek/aetherscan`), overridable with `--hf-repo-id`. +[`RELEASE.md`](RELEASE.md) covers how this revision couples releases to weights. + +### Loading the resolved artifacts + +`InferencePipeline.__init__` → `init_models()` loads the two models from the (now populated) +paths: - `--encoder-path` → `tf.keras.models.load_model` **inside `strategy.scope()`** (the hard rule: all TF model creation/loading happens in scope so variables are mirrored across @@ -81,9 +121,31 @@ code. --> tag, corrupting DB provenance and output paths. The CLI `--save-tag` (or the default timestamp) stays authoritative. -All three paths are required by `collect_validation_errors` and must exist on disk. The three -artifacts should carry the same training tag; nothing enforces it yet (`# TODO` in -`inference_command`), so mismatched encoder/RF pairs are on the operator. +`collect_validation_errors` enforces the trio all-or-none (a partial set is the error above); +every path that *is* set must exist on disk. The three artifacts should carry the same training +tag; nothing enforces it yet (`# TODO` in `inference_command`), so mismatched encoder/RF pairs +are on the operator. + +### Tag-dedup guards + +Immediately before command dispatch (post-validation, post-DB-init, pre-any-work), +`tag_guards.enforce_tag_guards(args)` hard-stops a run whose **explicitly-provided** +`--save-tag` collides with a previous run's state +([`src/aetherscan/tag_guards.py`](../src/aetherscan/tag_guards.py)) — the stale-artifact +confusion that used to force manual `test_vNN` incrementing. For inference the collision +markers (`find_inference_tag_collisions`) are: + +- the saved `config_{tag}.json` — written only at the very end of a successful pass, so it + marks a *completed* run and is always a collision; and +- on the **legacy `--test-files` path only** (no `inference_cadences` manifest rows), + non-superseded `inference_results` rows for the tag. + +Manifest rows are deliberately **not** a collision: they mark an in-progress streaming run that +the resume flow below consumes, so same-tag DB state there is expected. Default datetime tags +are immune by construction (a fresh second-resolution timestamp can't collide), so the guard +only fires for explicit tags; `--force-tag` consciously overrides it. (The same module also +guards training tags and, under `--hf-upload`, checks the Hub for the tag at startup rather +than after ~30 h of training.) ## The streaming loop diff --git a/docs/README.md b/docs/README.md index 8102cfe..0f438cc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,9 +13,7 @@ This directory holds long-form technical documentation that supplements the proj | [`RUNTIME_SERVICES.md`](RUNTIME_SERVICES.md) | The infrastructure singletons: the queue-based logger and its Slack threading/batching/broadcast behavior (and the stderr-redirect gotcha), the ResourceManager's pool/shared-memory/process lifecycles, signal handling and strict cleanup ordering, and the 1 Hz resource monitor with the PSS-vs-RSS rationale and the shutdown utilization plot. | | [`TESTING.md`](TESTING.md) | The pytest suite: layout, markers and the default CI selection, the autouse singleton/env isolation fixture and its teardown ordering, the synthetic data factories, what CI runs on which Python versions, how to run the cluster-marked integration smokes, and the adding-tests checklist. | | [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md) | Every workflow in `.github/workflows/`: triggers and behavior for the deterministic and assistant-driven families, the hidden dedup-marker convention, the `allowed_bots` gotchas, the assistant-handle invocation rules, and the issue/PR lifecycle the automation assumes. | +| [`BENCHMARKING.md`](BENCHMARKING.md) | The always-on performance instrumentation: the `stage_timer` API and its thread-local dot-name nesting, the `pipeline_stages` table, the annotated resource plot, the `benchmark_report.py` tool (stage tree, flame timeline, rules-driven bottleneck suggestions), the `benchmarks/` micro-benchmarks, and the current baseline/speedup numbers with how to read the annotated resource plot. | | [`RELEASE.md`](RELEASE.md) | The release contract: how one version string couples the git tag, GitHub Release, PyPI package, and HuggingFace weights tag; packaging and version single-sourcing; runtime weight resolution; the CD workflow's gates (verify-don't-create); PyPI trusted-publishing setup; and the step-by-step release runbook. | | [`GPU_RUNTIME_GUIDE.md`](GPU_RUNTIME_GUIDE.md) | Runbook for building, running, and debugging Aetherscan across GPU architectures — the Blackwell (RTX PRO 6000) workstation and the Ampere (A4000) cluster — unified by the NGC container runtime. Covers container builds, image verification, the four GPU-related CLI flags, CUDA/NCCL debugging recipes, the fallback escalation ladder, and cross-machine checkpoint interop. Read this before building the container or running on a new GPU architecture (especially Blackwell/sm_120), or after any run that dies with a `CUDA_ERROR_INVALID_PTX`/NCCL/OOM failure. | | [`CONFIG_AND_CLI.md`](CONFIG_AND_CLI.md) | Deep-dive into Aetherscan's singleton-config + CLI architecture: the three flag-routing patterns (A/B/C), the four cross-mode contamination barriers, the validation + fix-proposal layer, and the checklist for adding a new flag. Read this before touching `config.py` or `cli.py`. | - - diff --git a/docs/RELEASE.md b/docs/RELEASE.md index b5e337c..0bf4c05 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -100,11 +100,14 @@ impossible: 3. **HF weights verification** — confirm the matching `v*` tag exists on `zachtheyek/aetherscan` (public repo, no token needed). **Verify, never create**: if this fails, you skipped the weight-blessing step; the error says so and how to fix it. -4. **Build** — `python -m build` (sdist + wheel) from the tagged source. +4. **Build** — `python -m build` (sdist + wheel) from the tagged source, then **smoke the + wheel**: install it `--no-deps` and assert `aetherscan.__version__` equals the guarded + version (catches a packaging/version-single-sourcing mismatch before anything publishes). 5. **Publish to PyPI via trusted publishing** — `pypa/gh-action-pypi-publish` with `permissions: id-token: write` and the `pypi` environment. OIDC-based: **no long-lived PyPI API token is stored anywhere** in the repo or its secrets. -6. **GitHub Release** — created from the tag with generated notes; curate the body from the +6. **GitHub Release** — created from the tag (`gh release create --verify-tag + --generate-notes`) with the built sdist + wheel attached; curate the body from the per-PR `claude-release-notes` comments (see [`GITHUB_AUTOMATION.md`](GITHUB_AUTOMATION.md)) — they are the raw material, written one merge at a time for exactly this purpose. diff --git a/docs/RUNTIME_SERVICES.md b/docs/RUNTIME_SERVICES.md index e6cb9b3..797a25a 100644 --- a/docs/RUNTIME_SERVICES.md +++ b/docs/RUNTIME_SERVICES.md @@ -176,6 +176,25 @@ CPU/GPU activity. The figure is uploaded to the run's Slack thread like every ot Missing stretches in the panels are a known symptom ([`KNOWN_ISSUES.md`](../KNOWN_ISSUES.md) #10). - +### Stage annotations + +When `monitor.annotate_stages` is enabled (config default `True`), `_save_plot()` overlays the +run's pipeline-stage spans as labeled translucent bands on the **CPU panel** +(`_annotate_stage_spans`), turning the utilization curve into a self-explaining timeline: a CPU +plateau reads as `round_03` (data generation), a GPU band as `epochs`, and so on. The spans +come from the `pipeline_stages` table written by the always-on stage timers — see +[`BENCHMARKING.md`](BENCHMARKING.md) for the timers and [`DATABASE.md`](DATABASE.md#pipeline_stages-stage-timers-schema-v3) +for the table. + +Two details keep it readable and correct: + +- **Depth ≤ 2 only.** `select_annotation_spans` keeps spans whose dot-name has at most two + components (`train.round_03`, not `train.round_03.epochs`) — the deep per-ON-file and + encode/rf sub-stages stay report-tool-only so the panel doesn't drown in bands. Bands are + labeled with the leaf component (`round_03`) and cycle three face colors so adjacent stages + stay separable at low alpha. +- **Flush first.** The method calls `db.flush()` before querying, so spans recorded moments + before shutdown (`final_save`, `inference.viz`) make it onto the plot. This is safe because + the writer thread outlives the monitor in the cleanup order (monitor stops before db); a + flush timeout just means the very newest spans are missing, never an error. Annotation + failures are caught and logged — a benchmarking overlay must never break the resource plot. From 6d27f209a9308986edd9152a45ad69d746a64cf6 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 17:29:46 +0800 Subject: [PATCH 15/20] Fill Phase-B doc placeholders: pipeline_stages, HF resolution, stage annotations - DATABASE.md: document the pipeline_stages table (schema v3), add a schema- version -> feature map (v1 superseded columns, v2 inference_cadences, v3 pipeline_stages), and update the migration section and table count. - INFERENCE_PIPELINE.md: document the full HF artifact-resolution chain (explicit paths -> --hf-revision -> installed-release v{__version__} with the dev-version guard -> latest release/final tag), the all-or-none validation, and the inference-scope tag-dedup guards. - RUNTIME_SERVICES.md: document the monitor.annotate_stages resource-plot overlay (config knob, depth<=2 rule, flush-first). - RELEASE.md: cross-check gates against the implemented release.yml (wheel-smoke in build, GitHub Release artifacts). Co-Authored-By: Claude Opus 4.8 --- docs/DATABASE.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/DATABASE.md b/docs/DATABASE.md index 96e4b96..21c16d5 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -86,9 +86,10 @@ failed attempt wrote. ## Schema Seven tables, created idempotently (`CREATE TABLE IF NOT EXISTS`) in `_init_database()`, each -with a composite index matched to its dominant filter pattern. All rows carry `timestamp` -(write time, `REAL` Unix seconds) and `tag` (the run's save tag — the primary provenance -key). +with a composite index matched to its dominant filter pattern. All rows carry `tag` (the run's +save tag — the primary provenance key), and all but `pipeline_stages` carry `timestamp` +(write time, `REAL` Unix seconds); `pipeline_stages` records explicit `start_time`/`end_time` +span bounds instead. | Table | Rows | `superseded`? | Added in | | --- | --- | --- | --- | From 3b2b04425a37bec2cefb475d32901845912eced3 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Sun, 12 Jul 2026 17:37:54 +0800 Subject: [PATCH 16/20] Apply review notes: rendering + clarity polish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ARCHITECTURE.md: move setup_gpu_strategy() into clean prose so the main.py link renders without the stray trailing backtick span. - DATABASE.md: make the ~24 injection_stats rows/cadence derivation explicit (18 intensity + mean 6 signal-characteristic across the 0/6/6/12 types). - GITHUB_AUTOMATION.md: add a callout unpacking the dense claude-update-docs two-step relay (shell regen -> handle-mention issue -> assistant PR). RUNTIME_SERVICES.md KNOWN_ISSUES #6/#10/#11 verified against the current file (CPU undercounting / missing plot sections / negative memory freed) — no drift. Co-Authored-By: Claude Opus 4.8 --- docs/ARCHITECTURE.md | 4 ++-- docs/DATABASE.md | 9 +++++---- docs/GITHUB_AUTOMATION.md | 9 +++++++++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index cbcee70..0028c18 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -23,8 +23,8 @@ Training data is synthetic (setigen signal injection over real observed backgrou real filterbank `.h5` observations reduced by an energy-detection preprocessing stage ([`src/aetherscan/preprocessing.py`](../src/aetherscan/preprocessing.py)). Both commands run single-node multi-GPU via `tf.distribute.MirroredStrategy` with NCCL all-reduce (falling back -to `HierarchicalCopyAllReduce`), set up in -[`src/aetherscan/main.py`](../src/aetherscan/main.py)`:setup_gpu_strategy()`. +to `HierarchicalCopyAllReduce`), set up in `setup_gpu_strategy()` +([`src/aetherscan/main.py`](../src/aetherscan/main.py)). `src/aetherscan/main.py` is the **sole entry point**: `python -m aetherscan.main {train|inference}`. diff --git a/docs/DATABASE.md b/docs/DATABASE.md index 21c16d5..61be772 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -277,10 +277,11 @@ size — logged at startup. Rules of thumb at full-scale defaults (dominant terms only): - **`injection_stats` is the giant.** Every generated cadence writes 18 intensity rows - (6 statistics × 3 stages) plus 0–12 signal-characteristic rows (0 / 6 / 6 / 12 by type — - average 6). A training round generates `3 × num_samples_beta_vae` cadences (main + true + - false), so at defaults: `3 × 499 200 × ~24 ≈ 36 M rows per round`, times 20 rounds plus the - RF dataset. This is why writes are batched, why the drainer runs off the training critical + (6 statistics × 3 stages A/B/C) plus 0–12 signal-characteristic rows depending on its type + (0 / 6 / 6 / 12 for the four equal-weighted `main`-class types — 0 for the no-signal type, + 6 per injected signal, so their mean is 6). That is **~24 rows per cadence** (18 + 6). A + training round generates `3 × num_samples_beta_vae` cadences (main + true + false), so at + defaults: `3 × 499 200 × ~24 ≈ 36 M rows per round`, times 20 rounds plus the RF dataset. This is why writes are batched, why the drainer runs off the training critical path, and why the injection plots subsample (`plot_injection_subsampling_count`). If the database size becomes a problem, this table is where the budget goes — smoke-scale runs (`--num-samples-beta-vae 3072`) keep it trivial. diff --git a/docs/GITHUB_AUTOMATION.md b/docs/GITHUB_AUTOMATION.md index a060873..a6130eb 100644 --- a/docs/GITHUB_AUTOMATION.md +++ b/docs/GITHUB_AUTOMATION.md @@ -91,6 +91,15 @@ double-post. | [`claude-dependency-check.yml`](../.github/workflows/claude-dependency-check.yml) | Weekly (Mon 01:00 UTC) + `workflow_dispatch` | Audits `environment.yml` / `requirements-container.txt` / `aetherscan.def` against registries and advisories under [`SECURITY.md`](../SECURITY.md)'s version-selection policy; files a weekly report issue. | `` | | [`claude-flaky-test-tracker.yml`](../.github/workflows/claude-flaky-test-tracker.yml) | Weekly (Mon 01:00 UTC) + `workflow_dispatch` | Reads the week's `tests.yml` runs, identifies flaky/failing tests, diagnoses the worst offender, files a weekly report issue. | `` | +> [!NOTE] +> **`claude-update-docs` is a two-step relay**, which is why its row is dense. Step 1 is a +> deterministic shell step: when `cli.py` changed it regenerates the README CLI Reference with +> `utils/print_cli_help.py` (Python pinned to 3.12 — argparse help formatting changes in 3.13) +> and files an issue embedding that output. Step 2 is the assistant: the filed issue carries an +> intentional handle mention that triggers `claude.yml` to open the actual docs PR. The split +> exists because the follow-up assistant run has no `python` in its tool allowlist, so it +> cannot regenerate the CLI help itself — the shell step must do it first. + The dedup-marker pattern is a convention to preserve in any new workflow that posts content: the guard step greps existing issues/comments for the marker (`gh ... --json body --jq` + `grep`), and the prompt instructs the assistant to put the marker as the **first line** of From 272bef1596f6fa8ed2f6f734bf6865e5ee71fb1c Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Tue, 14 Jul 2026 22:10:16 +0800 Subject: [PATCH 17/20] docs(testing): note monitor/logger coverage gaps; complete unit-test tree Add the four unit-test files that exist on this branch but were missing from the Layout tree (test_tag_guards.py, test_hf_hub.py, test_benchmark.py, test_logger.py), and add a 'Coverage and deliberate gaps' section documenting why monitor has no dedicated unit tests and what logger is (and isn't) unit-tested for. Co-Authored-By: Claude Opus 4.8 --- docs/TESTING.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/TESTING.md b/docs/TESTING.md index fd56e44..3b8f435 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -36,6 +36,7 @@ tests/ ├── unit/ # fast, hardware-independent; the CI surface │ ├── test_config.py # singleton semantics, to_dict round-trip │ ├── test_cli_validation.py # tag pattern, divisibility matrix, saved-config precedence +│ ├── test_tag_guards.py # save-tag dedup matrix: local/HF collisions, --force-tag │ ├── test_data_generation.py # log_norm, create_* shapes/labels, intersection checks, │ │ # chunk segments / batched task partitioning │ ├── test_round_data.py # paths/manifest protocol, reuse/delete semantics, producer @@ -49,14 +50,36 @@ tests/ │ ├── test_inference.py # padding fix, provenance mapping, confidence summaries │ ├── test_inference_viz.py # every figure smoke-tested against tiny synthetic inputs │ ├── test_main.py # streaming-loop resume/containment with stubbed stages +│ ├── test_hf_hub.py # revision resolution, artifact download, upload staging, card │ ├── test_db.py # writer thread, flush/supersede sentinels, migrations, queries -│ └── test_manager.py # pool/SHM tracking and cleanup idempotence +│ ├── test_manager.py # pool/SHM tracking and cleanup idempotence +│ ├── test_benchmark.py # stage_timer nesting/failures, report tree math + suggestions +│ └── test_logger.py # StreamToLogger redirect probes: isatty/writable/readable/fileno └── integration/ # marked integration+gpu+cluster: real subprocess runs ├── conftest.py # repo-root launcher + cluster path resolution ├── test_train_smoke.py # known-good training smoke config, end to end └── test_inference_smoke.py # subset CSV inference against cluster-resident .h5 data ``` +## Coverage and deliberate gaps + +A couple of modules are unit-tested lightly or not at all, by design rather than oversight: + +- **`monitor`** has no dedicated unit-test module. It's dominated by the 1 Hz background + sampling thread (PSS process-tree stats) and the matplotlib rendering of the resource plot, + both low-value to unit-test; its behavior is exercised by the integration smokes (real runs) + and verified by manual inspection of the resource-utilization plot uploaded to Slack. (Its + one pure helper, `select_annotation_spans`, *is* covered — in `test_benchmark.py`.) +- **`logger`** is unit-tested only for the `StreamToLogger` stdout/stderr-redirect probes + (`isatty`/`writable`/`readable`/`fileno`) in `test_logger.py`; the QueueListener, + SlackHandler, and stderr-to-logger redirect are exercised by the integration smokes rather + than unit tests. + +The inference pipeline follows the suite's usual shape: its logic is unit-tested at the +function level (`test_inference.py`, `test_main.py`, `test_inference_viz.py`), with full +end-to-end behavior covered by `test_inference_smoke.py` — there is no unit-level end-to-end +inference test by nature. + ## Markers | Marker | Meaning | In default selection? | From f77cbd3bc86b3545dea9fe1078cf805ec25119d6 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Wed, 15 Jul 2026 19:49:52 +0800 Subject: [PATCH 18/20] docs: refresh stale test-suite marker in SKILL.md; note slack_handler coverage gap Co-Authored-By: Claude Opus 4.8 --- .claude/skills/aetherscan-repo-context/SKILL.md | 2 +- docs/TESTING.md | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/.claude/skills/aetherscan-repo-context/SKILL.md b/.claude/skills/aetherscan-repo-context/SKILL.md index 826cca4..2fb97c1 100644 --- a/.claude/skills/aetherscan-repo-context/SKILL.md +++ b/.claude/skills/aetherscan-repo-context/SKILL.md @@ -160,7 +160,7 @@ Enforced by **ruff** (lint + format) via pre-commit; full config in `pyproject.t | Private | \_prefix | `_init_worker` | | Config fields | snake_case | `per_replica_batch_size` | -**Grep-friendly inline comment markers** (used consistently): `# TODO:` (actionable work), `# NOTE:` (rationale/question), `# FIX:` (known issue, no time now), `# BUG:` (known bug, often with workaround), `# TEST:` (behavior to verify, no suite yet). Prefer `# NOTE:` over `# TODO:` when there's no obvious action. +**Grep-friendly inline comment markers** (used consistently): `# TODO:` (actionable work), `# NOTE:` (rationale/question), `# FIX:` (known issue, no time now), `# BUG:` (known bug, often with workaround), `# TEST:` (behavior to verify — now backed by the `tests/` suite). Prefer `# NOTE:` over `# TODO:` when there's no obvious action. --- diff --git a/docs/TESTING.md b/docs/TESTING.md index 3b8f435..9bf7aa4 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -73,7 +73,14 @@ A couple of modules are unit-tested lightly or not at all, by design rather than - **`logger`** is unit-tested only for the `StreamToLogger` stdout/stderr-redirect probes (`isatty`/`writable`/`readable`/`fileno`) in `test_logger.py`; the QueueListener, SlackHandler, and stderr-to-logger redirect are exercised by the integration smokes rather - than unit tests. + than unit tests. In particular `src/aetherscan/logger/slack_handler.py` — a `logging.Handler` + that batches records, posts them as threaded replies to a per-run summary message, color-codes + by level, retries with exponential backoff, throttles via a consecutive-failure cooldown, and + uploads images — has no unit tests at all; its batching / level-to-color coding / backoff / + throttling are only ever driven by real runs. It also carries two known `# BUG:` markers at the + top of the module (batch messages colored by the wrong priority level, and over-long batched + messages truncated to a trailing `...`), both observability-only cosmetics that do not affect + pipeline results. The inference pipeline follows the suite's usual shape: its logic is unit-tested at the function level (`test_inference.py`, `test_main.py`, `test_inference_viz.py`), with full From d9e52cc8964d692f99863a9ddcf309fe3e010908 Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Fri, 17 Jul 2026 00:36:16 +0800 Subject: [PATCH 19/20] docs: document the automated-review response loop for agents Add guidance for agents on handling the automated claude-code-review first pass: wait for it, validate each note against your own read of the code, fix genuine gaps in self-contained commits on the same PR, rebut wrong notes concretely, post one consolidated reply, and re-invoke the review via the intentional-handle case until it comes back clean or drifts off-scope. Terse subsection in CLAUDE.md; fuller paragraph in the repo-context SKILL.md, complementing the existing handle-trigger rule. Co-Authored-By: Claude Opus 4.8 --- .claude/skills/aetherscan-repo-context/SKILL.md | 2 ++ CLAUDE.md | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/.claude/skills/aetherscan-repo-context/SKILL.md b/.claude/skills/aetherscan-repo-context/SKILL.md index 2fb97c1..726e072 100644 --- a/.claude/skills/aetherscan-repo-context/SKILL.md +++ b/.claude/skills/aetherscan-repo-context/SKILL.md @@ -177,6 +177,8 @@ Enforced by **ruff** (lint + format) via pre-commit; full config in `pyproject.t **Invoking vs. mentioning the assistant.** The assistant workflow (`claude.yml`) triggers whenever the assistant handle — an `@` immediately followed by `claude` — appears in the title/body of a Discussion, issue, or PR (or a comment on one). Write it only when you actually want to summon the assistant (e.g. an auto-filed docs issue asking it to open a PR). To refer to the handle as plain text anywhere else — a PR description, issue body, commit message, review comment — write it as `"@ claude"` (a space after the `@`, double quotes on both sides) so the `contains(…, '@claude')` trigger can't match. Tagging it unintentionally spawns a spurious assistant run and follow-up PR (this is what happened around issue #83). +**Responding to the automated review.** Opening (or marking ready) a PR triggers `claude-code-review.yml`, which posts a first-pass review with inline comments (catalogued in `docs/GITHUB_AUTOMATION.md`). Treat it as input, not verdict: wait for the review to land, then work through each comment individually, weighing it against your own understanding of the codebase and the change you actually made — don't assume the reviewer is right. Where a comment exposes a genuine blind spot, fix it in a focused, self-contained commit pushed to the *same* PR; where you're convinced it's wrong, leave the code untouched and be ready to explain concretely why. Then post a single PR comment covering both halves — first the points you addressed (what you changed and the rationale), then the points you think the reviewer got wrong (with your reasoning) — and close that comment by deliberately tagging the assistant handle to kick off a second-pass review. This is precisely the "you actually want to summon it" case from the paragraph above, not a violation of the don't-tag-unintentionally rule. Then repeat the loop — wait, read, validate, address, rebut, comment, re-invoke — until the reviews either come back clean (no further notes / LGTM) or they start drifting out of scope (raising points unrelated to the PR's theme) or turn nonsensical. At that stopping point, post a comment explaining why you're stopping, and do **not** tag the assistant handle again. + **Pre-commit hooks** (`pre-commit install` to activate): `ruff` (lint, `--fix`), `ruff-format`, general `pre-commit-hooks` (large files >1 MB, case conflict, merge conflict, YAML/TOML syntax, EOF/trailing-whitespace, private-key detection, `no-commit-to-branch` on master), and `gitleaks` (secret detection). Ruff-format auto-reformats on commit — **re-run `git add` after** it modifies files, then commit again. Bypass only sparingly with `git commit --no-verify`. ```bash diff --git a/CLAUDE.md b/CLAUDE.md index 3dbdb5c..b372061 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,6 +40,14 @@ ruff lint+format, 100-char lines, Python 3.10 target ([`pyproject.toml`](pyproje - Bumping a dependency? Don't jump to the latest — target a proven version per [SECURITY.md](SECURITY.md) (the newer of ~2 minors back / latest stable ≥6 months old; a known advisory on that target overrides the lag). Never cross a documented ceiling (`numpy<2.0`, …) or the NGC TF 2.17 ABI, and keep `environment.yml` / `requirements-container.txt` / `aetherscan.def` in sync. - Security: non-critical → GitHub Discussion w/ "security" label; critical → [@zachtheyek](https://breakthroughlisten.slack.com/archives/D01SJG0L0TE) on Slack, no public issue. Rotate any leaked token immediately. +## Code review + +Every PR gets an automated `claude-code-review` first pass. **Wait for it to land**, then work each note individually — weigh it against your own read of the code, don't take it on faith. + +- Genuine gaps → fix in focused, self-contained commits on the **same** PR. Notes you think are wrong → leave the code, argue concretely why. +- Post **one** reply covering both (what you changed and why, then where you think the review erred and why), and end it by deliberately tagging the assistant handle to trigger a second pass — the sanctioned intentional case of the handle rule above, not a contradiction of it. +- Loop (wait → validate → address/rebut → comment → re-tag) until the review is clean/LGTM, or it drifts off the PR's theme or turns nonsensical — then post why you're stopping and **don't** tag again. + ## More detail On-demand deep-dive skill: [`.claude/skills/aetherscan-repo-context/SKILL.md`](.claude/skills/aetherscan-repo-context/SKILL.md) — install paths, config/CLI system, architecture patterns, full workflow & security. Per-surface technical docs (architecture, training, inference, preprocessing, models, database, runtime services, testing, automation, releases) are indexed in [docs/README.md](docs/README.md) — start at [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). From ccc1a47225bbbc4467e2193827a870f2016afc5f Mon Sep 17 00:00:00 2001 From: zachtheyek Date: Fri, 17 Jul 2026 01:09:42 +0800 Subject: [PATCH 20/20] docs(skill): add a dedicated expanded Testing section The on-demand SKILL.md is the expanded form of CLAUDE.md but had only scattered, thin testing coverage. Add a dedicated ## Testing section (unit vs gpu/cluster integration shape, the exact CI default-selection command with pythonpath=["src"], the marker table, the autouse isolation fixture summary, run-before-claiming / ship-tests-with-logic discipline, and the TF-at-collection gotcha), pointing to docs/TESTING.md for the full deep dive. Trim the tests/ blurb in the Project Structure tree to a one-liner cross-referencing the new section. Co-Authored-By: Claude Opus 4.8 --- .../skills/aetherscan-repo-context/SKILL.md | 40 +++++++++++++++++-- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/.claude/skills/aetherscan-repo-context/SKILL.md b/.claude/skills/aetherscan-repo-context/SKILL.md index 726e072..50ad89d 100644 --- a/.claude/skills/aetherscan-repo-context/SKILL.md +++ b/.claude/skills/aetherscan-repo-context/SKILL.md @@ -119,10 +119,8 @@ utils/ # fetch_run_outputs.sh, find_optimal_configs.py, # verify_train_test_files.py docs/ # Full technical doc suite, one doc per pipeline surface — # indexed in docs/README.md; start at docs/ARCHITECTURE.md -tests/ # Pytest suite: conftest.py (singleton-reset fixtures, synthetic - # data factories), unit/, integration/ (gpu+cluster-marked smokes). - # Default selection: pytest -m "not gpu and not cluster" -q - # (runs in CI via .github/workflows/tests.yml); see CONTRIBUTING.md +tests/ # Pytest suite: unit/ (CI surface) + gpu/cluster-marked + # integration/ smokes — see the "Testing" section below ``` --- @@ -164,6 +162,40 @@ Enforced by **ruff** (lint + format) via pre-commit; full config in `pyproject.t --- +## Testing + +The `tests/` suite splits along a hardware axis: + +- **`tests/unit/`** — fast, hardware-independent, one `test_.py` per source module. This is the CI surface; everything here must pass on a CPU-only runner. +- **`tests/integration/`** — `gpu`/`cluster`-marked end-to-end smokes (`test_train_smoke.py`, `test_inference_smoke.py`) that launch `python -m aetherscan.main ...` as a real subprocess on a cluster, against cluster-resident data/models. Not run in CI. + +**Default selection — exactly what CI runs** (`.github/workflows/tests.yml`, on Python 3.10 and 3.12), no GPUs or cluster data needed: + +```bash +pytest -m "not gpu and not cluster" -q +``` + +`pyproject.toml`'s `[tool.pytest.ini_options]` sets `pythonpath = ["src"]`, so **pytest needs no `PYTHONPATH=src` prefix** (unlike running `main.py` from source); it also sets `testpaths = ["tests"]` and `--strict-markers` (a typo'd marker is a collection error, not a silently-ignored one). + +**Markers** (declared in `pyproject.toml`; `--strict-markers` rejects undeclared ones): + +| Marker | Meaning | In default selection? | +| ------------- | ------------------------------------------------ | ----------------------- | +| `slow` | Slower CPU tests (e.g. builds real TF graphs) | **Yes** — CI runs them | +| `gpu` | Needs one or more physical GPUs | No | +| `cluster` | Needs cluster-resident data/models (blpc3/bla0) | No | +| `integration` | End-to-end subprocess runs; **skips isolation** | No (also `gpu`+`cluster`) | + +**Isolation.** The autouse `aetherscan_isolated_env` fixture in `tests/conftest.py` wraps every non-integration test: it points `AETHERSCAN_{DATA,MODEL,OUTPUT}_PATH` at a fresh `tmp_path` tree, deletes `SLACK_BOT_TOKEN`/`SLACK_CHANNEL` (tests must never talk to Slack), resets all five singletons (`Config`, `Database`, `Logger`, `ResourceManager`, `ResourceMonitor`) via their `_reset()` hooks, then on teardown stops any leaked background threads/pools and restores the snapshotted SIGINT/SIGTERM handlers and stdout/stderr. Net effect: tests never touch real data and can't leak state into one another. Integration tests are exempt — they inherit the real environment and run the pipeline as a subprocess. + +**Discipline.** Run the suite (or the subset you can) before claiming a change works, and **ship unit tests with new logic** — every behavior change should land tests under `tests/unit/` in the matching `test_.py` (create it if the module is new). + +**Gotcha.** Most unit modules import TensorFlow at collection time, so a bare `pytest` needs the full dependency stack (CI installs `tensorflow-cpu==2.17.*` plus the container requirements). If that stack isn't available locally, run the TF-free subset you can — e.g. `pytest tests/unit/test_config.py -q` — and **say exactly what you ran** rather than claiming the whole suite passed. + +Deep dive: `docs/TESTING.md` covers the full layout, the synthetic data factories, the coverage-and-deliberate-gaps notes (`monitor` / `logger` / `slack_handler`), CI specifics, how to run the cluster smokes, and the adding-tests checklist. + +--- + ## Contribution Workflow > **All issues are actionable, and all PRs must be tied to an existing issue.** Read `AI_POLICY.md` before doing AI-assisted work — the project has strict AI-usage rules.