Inference: seamless stage-aware retries + result/preprocessing visualization suite

[zachtheyek]

Problem

Inference fault tolerance is coarse and result inspection is nearly blind:

1. No inference-stage resume. Preprocessing resumes via per-cadence .npy existence, but the inference stage has none (# NOTE: at main.py): a retry re-encodes and re-classifies every cadence, and an attempt that dies mid-DB-write can leave duplicate/partial inference_results rows for a cadence that is then re-run.
2. apply_saved_config clobbers save_tag. Layering a saved training config under CLI flags copies its checkpoint section (including save_tag) onto the singleton, so an inference run can masquerade under the training run's tag unless --save-tag is passed explicitly.
3. Per-cadence failure containment stops at preprocessing. A single-cadence inference failure aborts the whole pass, and the retry loop re-attempts everything instead of just the failed cadences.
4. No visualization of what a science run actually did. Energy detection, dedup, bandpass flattening, stamp extraction, RF confidences, and candidates produce no figures; the plot_candidate() stub in inference.py was never implemented. Scientists have to spelunk the DB to scrutinize a run.

Proposal

Stage-aware retries

• Skip the checkpoint section entirely in apply_saved_config (CLI --save-tag/default timestamp stays authoritative); regression test.
• New DB table inference_cadences (per-cadence run manifest: status preprocessed/inferred, aggregate stats incl. n_stamps, n_candidates, confidence-quantile summary, durations), created via the PRAGMA user_version migration mechanism.
• Resume flow per cadence: inferred manifest row → skip entirely; preprocessing artifacts only → skip to inference; else preprocess. Before re-running inference on a cadence, mark_superseded(inference_results, tag, npy_path=...) so partial rows from a dead attempt can't mix in.
• Extend per-cadence failure containment to the inference stage; the retry loop then re-attempts only failed cadences, with --max-retries bounding whole-pass retries.

Inference visualization suite — new src/aetherscan/inference_viz.py, every figure saved under plots/inference/{tag}/ + Slack upload, all calls wrapped so a plot bug can never kill a science run: ED statistic distributions (all-window summary histograms accumulated in the vectorized workers), hit spectrum pre/post dedup, bandpass-flattening overlay (formalizes the PR-07 debug artifact), top-K stamp gallery, preprocessing funnel, confidence distribution, candidate gallery + per-candidate plots (implements plot_candidate(), capped), inference latent projection through the persisted training cadence-level UMAP, and a run summary card.

Config: inference_viz_enabled (default on), stamp_gallery_top_k, max_candidate_plots + CLI flags.

[claude]

Hey @zachtheyek — friendly reminder that per CONTRIBUTING.md, issues should generally be preceded by a GitHub Discussion or Slack thread to allow for community input before formalizing a proposal. I wasn't able to find a related Discussion for this issue.

If the conversation happened on Slack rather than GitHub Discussions, this reminder may be safely ignored — but please consider linking the relevant Slack thread(s) in the issue body (under "Discussion links") for traceability.