Faster Nonlocal denoise: GEMM ONNX graph, halo 40, remove Maximum

CHANGELOG.md

@@ -28,10 +28,22 @@ Changes added by this fork to RapidRAW. Upstream application changes remain in t
 - Embed and verify an sRGB ICC profile in JPEG, PNG, TIFF, and WebP desktop and CLI exports. Profile labeling is independent of capture-metadata retention and GPS removal.
 - Report unsupported capture-metadata retention requests before writing output, including TIFF inputs or outputs. Remove the ineffective RAW highlight-compression setting without changing RAW development or existing saved edits.
 
+### Nonlocal denoise speed
+
+- Cut Nonlocal model time about 4× on NVIDIA CUDA. On an RTX 5060 Ti with the shipped settings (TF32 and graph optimization off), a 32.5 MP photograph drops from an estimated 91 s to an estimated 23 s of model time (tile count × measured per-tile time; RAW decoding, noise estimation and DNG writing excluded). Two changes combine:
+  - The ONNX model is derived from the previous export by `python -m rapidraw_denoise.onnx_rewrite`. It keeps every weight and the FP32 precision, drops the all-zero convolution biases, and runs the 1×1 convolutions (20 dense, 10 grouped) as matrix multiplications, taking a 320×320 tile from 370 ms to 153 ms. It passes the same frozen fixture gates as the previous model on all 18 fixture tiles, on CUDA and on CPU.
+  - Shrink the tile halo from 64 to 40 pixels, so a 32.5 MP photograph needs 150 tiles instead of 247 (39% fewer, which also shortens CoreML runs; that was not timed end to end). Against the previous halo, 99.9% of pixels in 8-bit test renders of three photographs differ by at most 0.71 of 255 levels (worst pixel 6.8). The largest difference to a converged reference over nine test crops is 0.031 of the noise level. With the noise level halved (a research setting; Intensity does not do this), differences are larger: 99.9% within 2.2 levels.
+- Reinstall the Nonlocal model with `install_model kind=nonlocal`. This release accepts only the new model (a single pinned hash; 320 px tiles, 40 px halo, 240 px core), and a model installed by an earlier release is rejected with reinstall instructions. Cached predictions from earlier versions are not reused. Job records and the prediction cache record which model ran. The CoreML package is unchanged and uses the new halo; its per-tile time is unchanged, and full-photograph CoreML parity was last measured with the previous halo.
+- Remove the Maximum quality option. Nonlocal always denoises each tile in one pass; four-rotation averaging gained only about 0.04 to 0.06 dB in the patch study at four times the cost. Requests that pass `quality` to `start_denoise` are rejected by the MCP host's schema validation, and the research CLI and Python pipeline no longer take `--ensemble`.
+
 ### MCP mask composition
 
 - Add native parent-mask duplication with fresh IDs, optional inversion, and independent local adjustments. Generated AI subject, depth, and other supported AI components can now join an existing parent in additive, subtractive, or intersect mode. The Node MCP host exposes both operations; installed releases require an updated native bridge and host.
 
+### MCP session storage
+
+- Store each distinct embedded mask image once in a session's manifest instead of repeating it in every history snapshot. Sessions with many AI mask edits shrink from tens or hundreds of megabytes to a few, and previously saved sessions still load. A manifest written by this version cannot be read by an older native bridge.
+
 ### Preview reliability
 
 - Clear the previous photo's GPU frame, interactive patch, and mask overlay when switching photos, including cached photos with matching dimensions. Wait for the selected photo's native load before rendering its adjustments.

denoise/README.md

@@ -2,11 +2,11 @@
 
 Process Bayer RAW photographs with a noise-conditioned denoiser, and test improvements using reproducible experiments. RapidRAW's production Nonlocal path runs **in-process through native ONNX Runtime** (CPU or Linux CUDA) or **directly through CoreML.framework on macOS** from a pinned FP32 bundle — no Python, PyTorch, NumPy or SciPy in production. This optional Python package remains as research/export/benchmark tooling: model export and validation, reference inference, and the study CLI. The editor's existing NIND AI denoiser remains the lightweight option.
 
-The GPU path combines the [2026 learned nonlocal model](https://github.com/MIA-UIB/nonlocal-matchfilter) with per-channel blind noise estimation, bounded-memory tiling and optional transform averaging. Native MCP jobs produce a **Bayer DNG** for RapidRAW's normal demosaic and colour pipeline. The research CLI produces a 16-bit sRGB TIFF, review renders, a floating-point packed RAW tensor and a provenance manifest. Both preserve the original photograph. Commercial parity and state-of-the-art performance have **not** been established.
+The GPU path combines the [2026 learned nonlocal model](https://github.com/MIA-UIB/nonlocal-matchfilter) with per-channel blind noise estimation and bounded-memory tiling. Native MCP jobs produce a **Bayer DNG** for RapidRAW's normal demosaic and colour pipeline. The research CLI produces a 16-bit sRGB TIFF, review renders, a floating-point packed RAW tensor and a provenance manifest. Both preserve the original photograph. Commercial parity and state-of-the-art performance have **not** been established.
 
 ## Native runtime bundle (production)
 
-Production inference reads an ONNX bundle directory containing `model.onnx` + `manifest.json`. The accepted bundle is identified by model SHA-256 `df7f21ddbdfecd0984896a902623f75aa883d25b5f862c49d8c2b3f63a7dd2d9`, source checkpoint lineage `c16747d852b93a95908792cdbac901f89cca98e35b91ea7db6214de42fbd3cad`, tile 320 / halo 64 / FP32 / RGBG. The native backend validates this contract fail-closed before inference. Produce or re-validate bundles with this package's export tooling (`rapidraw_denoise.onnx_export`, `rapidraw_denoise.onnx_validate`); the Python environment below is only needed for that tooling, never for production inference.
+Production inference reads an ONNX bundle directory containing `model.onnx` + `manifest.json`. The accepted bundle is identified by model SHA-256 `ba788a97f247058ea03d14bbf2f289adc8f4666039f27d0d530e6f67d23dd73c`, source checkpoint lineage `c16747d852b93a95908792cdbac901f89cca98e35b91ea7db6214de42fbd3cad`, tile 320 / halo 40 / retained core 240 / FP32 / RGBG. The native backend validates this contract fail-closed before inference and rejects any other model hash or tiling. Produce or re-validate bundles with this package's export tooling (`rapidraw_denoise.onnx_export`, `rapidraw_denoise.onnx_rewrite`, `rapidraw_denoise.onnx_validate`); the Python environment below is only needed for that tooling, never for production inference.
 
 Normal setup needs no bundle path: call `install_model` with `kind: "nonlocal"` (selects the bundle for `RAPIDRAW_NONLOCAL_PROVIDER`, default `cuda` on Linux, `coreml` on macOS, and `cpu` on other platforms) and the pinned runtime downloads from the [public model repo](https://huggingface.co/sheldonxxxx/RapidRAW-Nonlocal-Denoise) at a pinned commit with every hash verified. `RAPIDRAW_NONLOCAL_BUNDLE` remains an explicit developer override when set:
 
@@ -17,6 +17,30 @@ export RAPIDRAW_NONLOCAL_PROVIDER=cpu
 
 Use `cpu`, or `cuda` on a Linux x86_64 host with the pinned NVIDIA runtime pack (recommended — see [ONNX CUDA setup](../docs/mcp/onnx-cuda.md#install-the-pinned-runtime-pack), no `ORT_DYLIB_PATH` needed) or, advanced only, a manual CUDA runtime via `ORT_DYLIB_PATH`. On macOS, `coreml` selects direct native CoreML.framework inference from a pinned `.mlpackage` bundle (see below). There is no provider fallback: `cuda` on an unsupported platform, `coreml` off macOS, or a failing execution provider fails the job explicitly, and CPU never silently substitutes. Graph optimization stays off and CUDA TF32 stays off under the accepted strict policy. Results are cached under the workspace's `nonlocal-onnx-cache`, keyed by backend generation, bundle and checkpoint hashes, provider, ORT build identity and inference configuration; legacy Python-worker cache entries are never reused.
 
+### How the ONNX bundle is derived
+
+The published ONNX model is the FP32 export of the pinned upstream checkpoint with its graph rewritten by `rapidraw_denoise.onnx_rewrite` for faster execution. The rewrite drops the all-zero convolution biases the exporter materialises and runs every dense and grouped 1×1 convolution with at least 256 input channels as a matrix multiplication. It keeps every weight and FP32 arithmetic, and the runtime keeps graph optimization and TF32 off. The rewritten model's manifest records the source export hash and rewrite counts. On an RTX 5060 Ti with the strict policy (no TF32, no graph optimization, cuDNN heuristic search), a 320×320 tile takes about 153 ms instead of 370 ms. The rewritten graph passes the same frozen fixture gates as the unrewritten export on all 18 fixture tiles, on CUDA and on CPU (elementwise |difference| ≤ 1e-4 + 1e-4·|reference|, mean absolute error ≤ 1e-5, 99th-percentile absolute error ≤ 1e-4). CPU was not timed, and CoreML uses a separate model.
+
+To reproduce the bundle, first build the unrewritten source bundle with `rapidraw_denoise.onnx_export` (it needs `--checkpoint`, `--fixtures` and `--output`; its `model.onnx` must hash to `df7f21ddbdfecd0984896a902623f75aa883d25b5f862c49d8c2b3f63a7dd2d9`) or take it from the earlier model-repository revision. Then install the two packages the rewrite needs (the model hash assumes `onnx` 1.23.1; other versions may serialise differently and give a different hash, which the runtime rejects) and derive the bundle. The tool never overwrites an existing output directory, and `--fixtures` adds a first-tile CPU check against the source graph:
+
+```sh
+uv pip install --python .venv/bin/python onnx==1.23.1 onnxruntime
+.venv/bin/python -m rapidraw_denoise.onnx_rewrite \
+  --bundle /path/to/unrewritten-bundle \
+  --output /path/to/bundle-packed-gemm-halo40-320-fp32 \
+  --fixtures /path/to/fixtures
+```
+
+Validate the output before use:
+
+```sh
+.venv/bin/python -m rapidraw_denoise.onnx_validate \
+  --bundle /path/to/bundle-packed-gemm-halo40-320-fp32 \
+  --fixtures /path/to/fixtures --provider cpu --report /path/to/validate-cpu.json
+```
+
+[`tests/test_onnx_rewrite.py`](tests/test_onnx_rewrite.py) covers the rewrite.
+
 ### Direct CoreML bundle (macOS production)
 
 On macOS, production CoreML inference is native: it uses CoreML.framework directly and does not require Python, coremltools, PyTorch or ONNX Runtime. Prefer `install_model kind="nonlocal"` with `RAPIDRAW_NONLOCAL_PROVIDER=coreml`; to use a manual bundle instead, point `RAPIDRAW_NONLOCAL_BUNDLE` at a directory containing `packed.mlpackage` + `manifest.json`:
@@ -26,7 +50,7 @@ export RAPIDRAW_NONLOCAL_BUNDLE=/path/to/bundle-packed-coreml-fp32
 export RAPIDRAW_NONLOCAL_PROVIDER=coreml
 ```
 
-The accepted bundle is identified by package SHA-256 `ecf8f41b72b8e79a7210f61b13b3a21ad2ddbc79c475039be367fc011a98f5d3`, source checkpoint lineage `c16747d852b93a95908792cdbac901f89cca98e35b91ea7db6214de42fbd3cad`, direct TorchScript conversion (`coremltools-direct-torchscript`, FLOAT32, static-packed sampler), tile 320 / halo 64 / RGBG with `raw_with_noise` [1,8,320,320] input and `denoised_raw` [1,4,320,320] output. The native backend validates this contract fail-closed — including the package digest and a runtime tree hash — before compiling the package to `.mlmodelc` and loading one `MLModel` per job (all compute units, no low-precision GPU accumulation). The Python converter (`rapidraw_denoise` tooling / `denoise/scripts/convert_coreml_direct.py`) remains developer-only artifact tooling, never a production dependency. CoreML predictions are cached under the workspace's `nonlocal-coreml-cache` (protocol 3) and cannot collide with ONNX or legacy worker caches; jobs that predate the native runtimes, and cross-family ONNX↔CoreML resumes, are rejected — start a new denoise job instead.
+The accepted bundle is identified by package SHA-256 `ecf8f41b72b8e79a7210f61b13b3a21ad2ddbc79c475039be367fc011a98f5d3`, source checkpoint lineage `c16747d852b93a95908792cdbac901f89cca98e35b91ea7db6214de42fbd3cad`, direct TorchScript conversion (`coremltools-direct-torchscript`, FLOAT32, static-packed sampler), tile 320 / halo 40 / RGBG with `raw_with_noise` [1,8,320,320] input and `denoised_raw` [1,4,320,320] output. The native backend validates this contract fail-closed — including the package digest and a runtime tree hash — before compiling the package to `.mlmodelc` and loading one `MLModel` per job (all compute units, no low-precision GPU accumulation). The Python converter (`rapidraw_denoise` tooling / `denoise/scripts/convert_coreml_direct.py`) remains developer-only artifact tooling, never a production dependency. CoreML predictions are cached under the workspace's `nonlocal-coreml-cache` (protocol 3) and cannot collide with ONNX or legacy worker caches; jobs that predate the native runtimes, and cross-family ONNX↔CoreML resumes, are rejected — start a new denoise job instead.
 
 ## Research export environment (not production)
 
@@ -68,20 +92,17 @@ Open a Bayer RAW on that host, then call:
   "session_id": "<opened-session-uuid>",
   "expected_revision": 0,
   "method": "nonlocal",
-  "intensity": 100,
-  "quality": "balanced"
+  "intensity": 100
 }
 ```
 
 Supply the current revision to `rapidraw_start_denoise`. Poll `rapidraw_get_job` with its `job_id`, then continue editing the completed `result_session_id`. The new session contains the source edits captured at submission; later parent edits do not change the job. The original session and original RAW remain intact. Nonlocal cancellation is cooperative at tile boundaries and cannot abort an active ORT or CoreML call; the editing engine remains alive. `resume_job` restarts an interrupted, cancelled or failed job from the captured input; partial tiles are not checkpoints. Cross-family ONNX↔CoreML resume is rejected; legacy worker jobs cannot resume.
 
-- **Balanced**: one inference pass; the default.
-- **Maximum**: average four rotations; roughly four inference passes. The quality benefit in the patch study was small.
 - **Intensity**: blend the full prediction with the sensor input, from 0 to 100. Zero writes an identity DNG without inference. Nonlocal defaults to 100; existing NIND/BM3D defaults remain 50.
 
 Predictions are cached under the workspace's `nonlocal-onnx-cache`, keyed by backend generation, bundle and checkpoint hashes, provider, ORT build identity and inference configuration; legacy Python-worker cache entries are never reused. macOS CoreML predictions use the separate `nonlocal-coreml-cache` (protocol 3), keyed by package/tree/checkpoint hashes, compute policy and OS identity. Changing intensity reuses the full prediction. Every cache read verifies the result manifest, dimensions and checksum. Corrupt entries fail explicitly. Completed DNG sessions are independent of this cache; verified prediction directories may be removed while no job is active to reclaim space. There is no automatic cache eviction.
 
-The native path uses fixed 320×320 packed tiles with a 64-pixel halo. Sensor planes stay in host RAM, so total RAM still scales with photograph size. Output is uncompressed float32 Bayer DNG to avoid integer requantization, approximately 130 MB for a 32.5 MP active sensor. The session and export copies, prediction cache and temporary buffers require additional storage. Jobs reserve 20 GiB plus estimated working space. Inactive sensor borders are removed with CFA/black-level/crop coordinates adjusted; the normal rendered crop and orientation are retained. Signed samples feed noise estimation, and sensor values above white remain unchanged by the prediction blend.
+The native path uses fixed 320×320 packed tiles with a 40-pixel halo (240-pixel retained core), one inference pass per tile; a 32.5 MP photograph takes 150 tiles. Sensor planes stay in host RAM, so total RAM still scales with photograph size. Output is uncompressed float32 Bayer DNG to avoid integer requantization, approximately 130 MB for a 32.5 MP active sensor. The session and export copies, prediction cache and temporary buffers require additional storage. Jobs reserve 20 GiB plus estimated working space. Inactive sensor borders are removed with CFA/black-level/crop coordinates adjusted; the normal rendered crop and orientation are retained. Signed samples feed noise estimation, and sensor values above white remain unchanged by the prediction blend.
 
 This first integration is an MCP feature on a configured host. The desktop denoise dialog continues to offer its existing methods. A Mac can open the resulting DNG; automatic desktop-to-server submission, transfer and local Metal inference are not included.
 
@@ -102,12 +123,12 @@ Run inside the server's usual headless display environment on Linux. The check e
 .venv/bin/rapidraw-denoise \
   /path/to/photo.CR3 \
   --checkpoint /path/to/model-cache/nonlocal-raw-weights.pt \
-  --output /path/to/results/photo-v1 --tile 320 --halo 64
+  --output /path/to/results/photo-v1 --tile 320 --halo 40
 ```
 
 The command works with the default reference sampler. After installing and verifying the optional extension, prefix it with `RAPIDRAW_DENOISE_SAMPLER=cuda` for the faster path. The output directory must be new. The command requires at least 20 GiB free space beyond its estimated outputs. Inference defaults to CUDA and fails if it is unavailable; CPU diagnostic runs require `--device cpu`. Full-resolution arrays stay in host RAM and tiles bound GPU memory. Select smaller tiles if other applications need VRAM. A 320-pixel tile means 640 sensor pixels across, since Bayer channels are packed at half resolution.
 
-`--ensemble 4` or `--ensemble 8` averages transformed predictions and saves their disagreement. Four rotations produced a small, consistent gain in the [expanded patch study](STUDY.md#observed-patch-study-outcome), at roughly four inference passes per photograph. Disagreement measures transform sensitivity, **not** calibrated uncertainty. `--noise-scale` scales the estimated noise standard deviation; it is not a blend with the original. `--noise-profile` accepts a JSON `NoiseProfile` with four `shot` and four `read` values in RGBG order, where `read` is variance in black-subtracted, white-normalized RAW units. Poorly identified estimates are recorded in the manifest; severely clipped inputs can require a measured profile.
+`--noise-scale` scales the estimated noise standard deviation; it is not a blend with the original. `--noise-profile` accepts a JSON `NoiseProfile` with four `shot` and four `read` values in RGBG order, where `read` is variance in black-subtracted, white-normalized RAW units. Poorly identified estimates are recorded in the manifest; severely clipped inputs can require a measured profile.
 
 Inspect `original.png` and `denoised.png` at the same native magnification. Open `denoised.tiff` as a new photograph in RapidRAW for further editing. For agent editing, use the existing `rapidraw_open_photo` workflow with the returned TIFF path. The new file has denoising and demosaicing baked in; the original RAW and sidecar remain the authoritative source. Existing RAW adjustments are not copied onto the TIFF.