diff --git a/.gitignore b/.gitignore index 4e4285201..8e1ae121d 100644 --- a/.gitignore +++ b/.gitignore @@ -45,7 +45,7 @@ scratch/ specs/ docs/specs/ -# Disposable validation output. Tracked O2 A evidence lives directly in validation/. +# Disposable validation output. Tracked evidence lives directly in validation/. validation/.cache/ validation/function_diff/ validation/latest/ diff --git a/AGENTS.md b/AGENTS.md index c64eed8b3..fc338127e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Repo Notes -- `zdisamar` is an O2 A RTM lab. Treat DISAMAR as the reference family used for validation, not as the codebase architecture. +- `zdisamar` is an O2A RTM lab. Treat DISAMAR as the reference family used for validation, not as the codebase architecture. - Keep the public flow simple: input -> RTM -> output. - Keep routines under `src/rtm/`, `src/optics/`, `src/spectrum/`, and `src/retrieval/` free of file I/O, CLI wiring, text parsing, and hidden global state. - Keep scientific assets under `data/reference_data/`; loaders and parsers live under `src/input/reference_data/`. diff --git a/DISAMAR_IMPLEMENTATION_DIFFERENCES.md b/DISAMAR_IMPLEMENTATION_DIFFERENCES.md index 0b8cac84c..0f47db3e3 100644 --- a/DISAMAR_IMPLEMENTATION_DIFFERENCES.md +++ b/DISAMAR_IMPLEMENTATION_DIFFERENCES.md @@ -9,15 +9,15 @@ zdisamar links use the current GitHub PR branch. | Area | Fortran does | zdisamar does | Why | Impact | | --- | --- | --- | --- | --- | -| Polarization | Can model scalar light and polarized light. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Uses the scalar O2 A path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L326-L452) | The current O2 A validation target is scalar. | Faster and smaller. This route does not produce polarized-light output. | +| Polarization | Can model scalar light and polarized light. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Uses the scalar path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L326-L452) | The current validation target is scalar. | Faster and smaller. This route does not produce polarized-light output. | | Scattering angle basis | Stores separate values for positive and negative viewing directions. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1065-1517) | Stores one side and derives the other when needed. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L127-L305) | In the scalar case, the two sides are mathematically linked. | Less stored data, with the same scalar result. | | Scattering matrix work | Builds full scattering matrices. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Builds the full matrix when layers need it, but builds only one row when reflectance or aerosol weighting only reads one row. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L584-L689) | Some later steps only use one row. | Avoids matrix work that would be thrown away. | | Gas and aerosol scattering | Carries a more general phase-matrix representation. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1549-1562) | Combines gas and aerosol scattering into the scalar form needed by the current layer or interface. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L454-L582) | The scalar model only needs the combined scattering shape at the point being computed. | Keeps the layer math smaller while still carrying gas and aerosol contributions. | | Path attenuation storage | Stores attenuation for many direction and level pairs. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L971-1062) | Uses either full path storage or a smaller storage layout, depending on what the route will read. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/attenuation.zig#L93-L231) | Not every route reads every path. | Saves memory on the common route, while keeping full lookup available when needed. | | Curved direct beam | Starts from a flat-layer direct beam, then corrects the top-of-atmosphere path for curvature. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1005-1025) | Uses a fast curved-path calculation for normal support grids, and a slower fallback for larger grids. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/attenuation.zig#L644-L725) | The solar beam bends through a spherical atmosphere. | Keeps the spherical correction, without letting large grids overflow fixed local storage. | | Layer doubling | Decides whether each layer needs the more expensive doubling calculation. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1855-1908) | Makes the same kind of decision using an explicit cutoff before choosing the cheap or expensive layer path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/layer_reflect_transmit.zig#L1280-L1339) | Very thin weak-scattering layers often do not need the expensive path. | Faster layer builds. The cutoff must stay covered by validation because it can slightly affect results. | -| Small matrix multiply | Uses general small-matrix routines. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3528-3720) | Adds fixed-size routines for the common O2 A matrix shape. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L384-L595) | The common O2 A shape is known before runtime. | Less loop overhead and better compiler output in the hot matrix path. | +| Small matrix multiply | Uses general small-matrix routines. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3528-3720) | Adds fixed-size routines for the common matrix shape. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L384-L595) | The common shape is known before runtime. | Less loop overhead and better compiler output in the hot matrix path. | | Multiple reflection solve | Computes the repeated-reflection correction in a general helper. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3616-3720) | Solves the main repeated-reflection part first, then fills the viewing and solar pieces around it. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L2023-L2537) | Most repeated reflection happens through the Gaussian stream block. | Makes the optimized solve match the physical block structure. | | Local source fields | Computes local radiation fields for source-function work. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/addingToolsModule.f90#L1667-1816) | Carries those fields only on routes that need source integration or aerosol Jacobians. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/scattering_orders.zig#L1394-L1518) | Direct reflectance does not use them. | Simpler routes avoid extra arrays and transport work. | -| Jacobians | Computes many weighting functions for gases, aerosols, clouds, absorption, scattering, and interface movement. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L2699-3525) | Implements the derivative outputs used by the current O2 A model. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/reflectance.zig#L460-L858) | Unsupported derivative outputs should not appear as if they are supported. | Public Jacobians are narrower, but less misleading. | +| Jacobians | Computes many weighting functions for gases, aerosols, clouds, absorption, scattering, and interface movement. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L2699-3525) | Implements the derivative outputs used by the current model. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/reflectance.zig#L460-L858) | Unsupported derivative outputs should not appear as if they are supported. | Public Jacobians are narrower, but less misleading. | | Source quadrature | Builds layer-interface source weighting inside the radiative-transfer flow. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L92-105) | Prepares the within-layer source samples before LABOS runs. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/optics/source_levels.zig#L76-L318) | Aerosol source weighting needs samples inside the layer, not only at layer edges. | LABOS gets ready-to-use scalar source data. | diff --git a/NOTICE b/NOTICE index 5066c67f9..4c7855caa 100644 --- a/NOTICE +++ b/NOTICE @@ -1,4 +1,4 @@ -zdisamar is an independent implementation of the O2 A radiative-transfer +zdisamar is an independent implementation of the O2A radiative-transfer problem used in DISAMAR-family validation studies. The Fortran DISAMAR project is a scientific reference for validation; this repository does not include DISAMAR source code. diff --git a/README.md b/README.md index 121046bda..96bb83a3c 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ spectral response all affect the measured spectrum. The Fortran DISAMAR code family is the scientific reference for this work. `zdisamar` keeps the same radiative-transfer problem and reorganizes the -repeated oxygen A-band calculations so validation cases can be run, timed, and +repeated oxygen A-band calculations so validation scenes can be run, timed, and inspected through generated spectra and timing files. The Python wrapper is demonstrated in executable notebooks under @@ -26,9 +26,9 @@ uv run --with jupyterlab --with ipykernel python -m jupyter lab scripts/demo The two demos are [`o2a_plot_bundle.ipynb`](./scripts/demo/o2a_plot_bundle.ipynb), which shows the -Python-facing O2 A output and plotting accessors, and +Python-facing output and plotting accessors, and [`optimal_estimation_demo.ipynb`](./scripts/demo/optimal_estimation_demo.ipynb), -which shows a two-state O2 A optimal-estimation flow. +which shows a two-state optimal-estimation flow. The published Python package has no third-party runtime dependencies: the RTM, optimal-estimation helpers, and notebook-facing SVG plots ship with the native @@ -53,7 +53,7 @@ clouds, so aerosol retrievals need a detailed RTM. ![Aerosol scene and oxygen A-band reflectance](./docs/assets/o2a-aerosol-spectrum-context.png) -The figure links a visible aerosol scene to the O2 A reflectance spectrum seen +The figure links a visible aerosol scene to the O2A reflectance spectrum seen by the instrument: the aerosol contribution changes both the absolute reflectance and the structure inside the absorption band. Aerosol optical thickness, aerosol vertical distribution, and surface reflection all affect the @@ -81,11 +81,11 @@ it reads a retrieval configuration, prepares atmospheric and surface inputs, calculates spectra and Jacobians, and runs inverse methods such as optimal estimation. Its strength is breadth. It supports many retrieval families, spectral ranges, configuration options, and operational/research use cases. That -breadth also makes focused O2 A benchmarking difficult: a single aerosol-height -case still passes through general setup, broad configuration handling, and +breadth also makes focused benchmarking difficult: a single aerosol-height +scene still passes through general setup, broad configuration handling, and general numerical routines built for a much wider set of retrieval problems. -Both implementations target the same O2 A retrieval RTM: line-by-line +Both implementations target the same retrieval RTM: line-by-line oxygen absorption, multiple scattering, instrument-grid convolution, and reflectance derivatives for optimal estimation. The performance improvements come from reducing repeated setup around that calculation: @@ -94,17 +94,17 @@ come from reducing repeated setup around that calculation: across repeated RTM calls; - optimal-estimation retrievals call the RTM several times while the scene, instrument grid, spectroscopy, and many optical inputs stay the same. - Each iteration reuses that O2 A calculation state in memory; + Each iteration reuses that calculation state in memory; - retrieval Jacobians are calculated for the active state-vector columns; -- common O2 A LABOS matrix shapes for the 20-stream case use specialized +- common LABOS matrix shapes for the 20-stream case use specialized implementations in the repeated layer-doubling calculations; - validation and benchmark evidence is stored under [`validation/outputs/`](./validation/outputs/). -The benchmark cases use `nstreamsSim = 20` and `nstreamsRetr = 20`. Streams are +The benchmark scenes use `nstreamsSim = 20` and `nstreamsRetr = 20`. Streams are the angular quadrature directions used by the multiple-scattering radiative-transfer solver; more streams resolve the angular radiation field more -finely, but each RTM call costs more. The production DISAMAR O2 A +finely, but each RTM call costs more. The production DISAMAR setup usually uses 16 streams, so these retained 20-stream timings are deliberately slower than a production-tuned Fortran run. @@ -122,12 +122,12 @@ optimal-estimation retrieval timing. The timings in this section were recorded on the local benchmark machine: Mac mini `Mac16,10`, Apple M4, 10 CPU cores (4 performance, 6 efficiency), 24 GB RAM. Treat the absolute seconds as machine-specific wall-clock measurements; the -linked validation artifacts are the source for the reported case counts, +linked validation artifacts are the source for the reported scene counts, retrieved-state deltas, and timing summaries. ### RTM -The RTM benchmark calculates one O2 A spectrum over 755-776 nm. The +The RTM benchmark calculates one O2A spectrum over 755-776 nm. The reported spectrum has 701 instrument-grid wavelengths, but each instrument channel is an average over sharper oxygen absorption structure at higher spectral resolution: @@ -172,8 +172,8 @@ aerosol optical depth: median +1.688e-08, mean -3.025e-07, range -3.703e-0 aerosol mid pressure [hPa]: median -0.0016, mean -0.0020, range -0.0522 to +0.0821 ``` -Fastmode retrieval is a zdisamar-only optimisation lane on the same O2 A case. -The normal case remains the full-physics reference. Enabling +Fastmode retrieval is a zdisamar-only optimisation lane on the same scene. +The normal scene remains the full-physics reference. Enabling `case.optimisation.fastmode.enabled` resolves inspectable RTM, adaptive-grid, OE, sparse fast-stage wavelength sampling, and final-correction defaults before execution. The shipped fastmode path solves the fast stage on the sparse @@ -192,7 +192,7 @@ correction wavelengths on the validation measurement grid. Median speedup versus fullmode is `+1.382 s`. The maximum fastmode-minus-fullmode deltas are `4.197e-04` aerosol optical depth and `0.551 hPa` aerosol mid pressure. These timings are wall-clock durations around the public retrieval call. They -include session/cache creation, native case load and preparation, native OE work, +include session/cache creation, native scene load and preparation, native OE work, and the sparse full-physics correction; they do not include scene construction, simulated measurement construction, CSV writing, or plot rendering. The technical note is @@ -215,13 +215,13 @@ OE 5-case fastmode sweep retrieval total 4.516 s That benchmark artifact is [`benchmark/results.json`](./benchmark/results.json). It is a shorter -production-path timing check; the retained 100-case validation sweep above +production-path timing check; the retained 100-scene validation sweep above remains the accuracy and convergence contract. -The wrapper-overhead probe used the 10-worker baseline case directly. Under that +The wrapper-overhead probe used the 10-worker baseline scene directly. Under that boundary, the current one-shot fastmode public call is `0.230 s` median, and a repeated-start loop with a caller-owned session cache is `0.179 s` median after -the first sparse-case load. That focused evidence is tracked in +the first sparse-scene load. That focused evidence is tracked in [`scaffolding/reports/performance/retrieval/fastmode-session-overhead.md`](./scaffolding/reports/performance/retrieval/fastmode-session-overhead.md). The tracked paired DISAMAR/zdisamar summary is @@ -256,26 +256,26 @@ counts are in ## Python Package Use `uv` to download the published package into an isolated environment. The -wheel includes the native RTM library and bundled O2 A reference data. The +wheel includes the native RTM library and bundled reference data. The Python package has no third-party runtime dependencies. ```bash uv run --with zdisamar python ``` -The public flow is one case object: configure the simulated O2 A scene, pass it +The public flow is one scene object: configure the simulated scene, pass it to the RTM, and pass it to optimal estimation when you want a retrieval. ```python from zdisamar import rtm -from zdisamar.inverse_method import optimal_estimation as oe +from zdisamar import optimal_estimation as oe from zdisamar.rtm import SessionCache from zdisamar.wavelength_bands import o2a ``` ### Simulated Scene -Start from the packaged O2 A reference case and set the aerosol state directly. +Start from the packaged reference scene and set the aerosol state directly. ```python case = o2a.reference_scene() @@ -285,7 +285,7 @@ case.aerosol_layer.mid_pressure_hpa = 760.0 ### Forward Model -Run the native forward model on the case. The result exposes wavelength, +Run the native forward model on the scene. The result exposes wavelength, radiance, irradiance, reflectance, optional Jacobians, and dependency-free SVG plot accessors. @@ -296,7 +296,6 @@ spectrum.plot.reflectance() jacobian_spectrum = rtm.spectrum( case, jacobian=True, - jacobian_state_names=("aerosol_optical_depth",), ) jacobian_spectrum.plot.jacobian("aerosol_optical_depth") ``` @@ -330,9 +329,9 @@ profile_spectrum = rtm.spectrum(profile_case) ### Fastmode Controls -Fastmode is a case-owned optimisation mode. The case keeps the physical scene -and exposes the RTM, adaptive-grid, OE, sparse fast-stage, and final-correction -controls in one place. +Fastmode is a scene-owned optimisation mode. The scene keeps the physical +configuration and exposes the RTM, adaptive-grid, OE, sparse fast-stage, and +final-correction controls in one place. ```python fast_case = o2a.reference_scene() @@ -359,7 +358,7 @@ fastmode.oe.final_correction.wavelength_count = 4 ### Optimal Estimation -Create a simulated measurement from a truth case. Public measurement noise is +Create a simulated measurement from a truth scene. Public measurement noise is expressed as signal-to-noise ratio; pass one scalar SNR or one value per wavelength. State-vector prior uncertainty is the one-sigma prior spread in the same units as the state value. diff --git a/benchmark/results.json b/benchmark/results.json index 1183a4229..f8330d7c1 100644 --- a/benchmark/results.json +++ b/benchmark/results.json @@ -4,22 +4,22 @@ "forward_fast_mode": { "average_active_cores": { "baseline_reference_run": { - "max": 1.9599994162596415, - "mean": 1.9599994162596415, - "median": 1.9599994162596415, - "min": 1.9599994162596415, - "p90": 1.9599994162596415, + "max": 1.9595680699588713, + "mean": 1.9595680699588713, + "median": 1.9595680699588713, + "min": 1.9595680699588713, + "p90": 1.9595680699588713, "runs": 1, - "total": 1.9599994162596415 + "total": 1.9595680699588713 }, "four_scene_fast_total": { - "max": 1.93868115789034, - "mean": 1.9380168027586784, - "median": 1.937998296293367, - "min": 1.9374139302719615, - "p90": 1.9384600105372183, + "max": 1.935765296243262, + "mean": 1.9322070917933563, + "median": 1.9349577848458255, + "min": 1.908357723847781, + "p90": 1.9354636507011336, "runs": 10, - "total": 19.380168027586784 + "total": 19.322070917933562 } }, "kind": "forward", @@ -27,22 +27,22 @@ "mode": "fast_mode", "process_cpu_s": { "baseline_reference_run_s": { - "max": 3.8705299999999987, - "mean": 3.8705299999999987, - "median": 3.8705299999999987, - "min": 3.8705299999999987, - "p90": 3.8705299999999987, + "max": 3.850176000000001, + "mean": 3.850176000000001, + "median": 3.850176000000001, + "min": 3.850176000000001, + "p90": 3.850176000000001, "runs": 1, - "total": 3.8705299999999987 + "total": 3.850176000000001 }, "four_scene_fast_total_s": { - "max": 9.388624000000007, - "mean": 9.372467599999997, - "median": 9.369382000000002, - "min": 9.358913999999992, - "p90": 9.386293900000007, + "max": 9.436226999999988, + "mean": 9.405632099999996, + "median": 9.399705000000008, + "min": 9.383469000000005, + "p90": 9.43296, "runs": 10, - "total": 93.72467599999997 + "total": 94.05632099999997 } }, "residuals": { @@ -93,46 +93,46 @@ }, "timing_s": { "baseline_reference_run_s": { - "max": 1.9747607922181487, - "mean": 1.9747607922181487, - "median": 1.9747607922181487, - "min": 1.9747607922181487, - "p90": 1.9747607922181487, + "max": 1.964808499906212, + "mean": 1.964808499906212, + "median": 1.964808499906212, + "min": 1.964808499906212, + "p90": 1.964808499906212, "runs": 1, - "total": 1.9747607922181487 + "total": 1.964808499906212 }, "four_scene_fast_total_s": { - "max": 4.844657499343157, - "mean": 4.836113303992898, - "median": 4.834621437592432, - "min": 4.828357667196542, - "p90": 4.844299411447719, + "max": 4.9427824155427516, + "mean": 4.867921541351825, + "median": 4.8603268335573375, + "min": 4.848407666664571, + "p90": 4.884749553306028, "runs": 10, - "total": 48.36113303992897 + "total": 48.67921541351825 } } }, "forward_no_session": { "average_active_cores": { - "max": 1.9238138540323793, - "mean": 1.8926037643443732, - "median": 1.9230114302458627, - "min": 1.6191924791219334, - "p90": 1.9236884695514684, + "max": 1.9238180840989547, + "mean": 1.8935161348684653, + "median": 1.9223516531992337, + "min": 1.6381322250107577, + "p90": 1.9233515444126261, "runs": 10, - "total": 18.92603764344373 + "total": 18.93516134868465 }, "kind": "forward", "label": "Forward, no session", "mode": "no_session", "process_cpu_s": { - "max": 2.005627, - "mean": 1.9307546000000002, - "median": 1.921417, - "min": 1.9193940000000005, - "p90": 1.9388002, + "max": 1.9667810000000001, + "mean": 1.9352844999999999, + "median": 1.9318954999999995, + "min": 1.9268610000000006, + "p90": 1.9420849999999998, "runs": 10, - "total": 19.307546000000002 + "total": 19.352845 }, "residuals": { "disamar_reference": { @@ -163,46 +163,46 @@ } }, "timing_s": { - "max": 1.238658791873604, - "mean": 1.0236098250374197, - "median": 0.9993099165149033, - "min": 0.997940625064075, - "p90": 1.0279582545626909, + "max": 1.2006240827031434, + "mean": 1.0246980749536305, + "median": 1.0044917918276042, + "min": 1.002010916825384, + "p90": 1.031424982845783, "runs": 10, - "total": 10.236098250374198 + "total": 10.246980749536306 } }, "forward_session": { "average_active_cores": { "cached_run": { - "max": 1.0007810148348482, - "mean": 1.000363235622375, - "median": 1.0003722666171457, - "min": 0.9998209634819565, - "p90": 1.0006844178662166, + "max": 1.0004582566986748, + "mean": 0.9999458146992998, + "median": 0.9999262971687768, + "min": 0.9995393501128464, + "p90": 1.0002942297212107, "runs": 10, - "total": 10.00363235622375 + "total": 9.999458146992998 }, - "first_cached_run": 1.9887671314652464, - "first_use_total": 1.9220800186817615, - "setup": 1.8969937289963639 + "first_cached_run": 1.9897431806058123, + "first_use_total": 1.9217841073690336, + "setup": 1.8963817434301875 }, "kind": "forward", "label": "Forward, session", "mode": "session", "process_cpu_s": { "cached_run_s": { - "max": 0.0016020000000018797, - "mean": 0.0014356000000006475, - "median": 0.0014164999999994876, - "min": 0.0013690000000003977, - "p90": 0.001504800000001083, + "max": 0.0015690000000034843, + "mean": 0.001460000000001216, + "median": 0.0014590000000005432, + "min": 0.0013590000000007763, + "p90": 0.001541100000000384, "runs": 10, - "total": 0.014356000000006475 + "total": 0.014600000000012159 }, - "first_cached_run_s": 0.5445139999999995, - "first_use_total_s": 1.9252050000000018, - "setup_s": 1.3806910000000023 + "first_cached_run_s": 0.5441159999999989, + "first_use_total_s": 1.9314899999999966, + "setup_s": 1.3873739999999977 }, "residuals": { "vs_no_session": { @@ -215,47 +215,47 @@ }, "timing_s": { "cached_run_s": { - "max": 0.0016007497906684875, - "mean": 0.0014350708574056626, - "median": 0.0014157078694552183, - "min": 0.00136879226192832, - "p90": 0.001504562422633171, + "max": 0.001569374930113554, + "mean": 0.0014600878581404685, + "median": 0.0014586669858545065, + "min": 0.0013586250133812428, + "p90": 0.001541775418445468, "runs": 10, - "total": 0.014350708574056625 + "total": 0.014600878581404686 }, - "first_cached_run_s": 0.27379475021734834, - "first_use_total_s": 1.0016258331015706, - "setup_s": 0.7278310828842223 + "first_cached_run_s": 0.27346041705459356, + "first_use_total_s": 1.0050504594109952, + "setup_s": 0.7315900423564017 } }, "retrieval_fast_mode": { "average_active_cores": { "first_use_total": { - "max": 1.9034424816525248, - "mean": 1.9017526400043814, - "median": 1.9020141999131321, - "min": 1.8987979334088607, - "p90": 1.9029651451016456, + "max": 1.9006525687748617, + "mean": 1.898689435637888, + "median": 1.8997844834127726, + "min": 1.8930785212904726, + "p90": 1.9003899937178343, "runs": 10, - "total": 19.017526400043813 + "total": 18.98689435637888 }, "retrieval": { - "max": 1.9299125433363025, - "mean": 1.9270780629085142, - "median": 1.9274034095959518, - "min": 1.9236515898568265, - "p90": 1.9293805569437859, + "max": 1.9288725327445093, + "mean": 1.9267964132755135, + "median": 1.9270069959661296, + "min": 1.9210190930313076, + "p90": 1.9288712198736575, "runs": 10, - "total": 19.27078062908514 + "total": 19.267964132755136 }, "setup": { - "max": 1.8814487512549403, - "mean": 1.8802272670767177, - "median": 1.880633672986343, - "min": 1.8774659275200618, - "p90": 1.8813761121001378, + "max": 1.878383229716903, + "mean": 1.8751509106291313, + "median": 1.87622926453257, + "min": 1.8693083737053804, + "p90": 1.878276103681468, "runs": 10, - "total": 18.802272670767177 + "total": 18.751509106291312 } }, "kind": "optimal_estimation", @@ -263,31 +263,31 @@ "mode": "fast_mode", "process_cpu_s": { "first_use_total_s": { - "max": 2.1691549999999893, - "mean": 2.1482185000000045, - "median": 2.1438130000000086, - "min": 2.140625, - "p90": 2.1594953000000006, + "max": 2.183932999999996, + "mean": 2.1637756999999964, + "median": 2.16078899999998, + "min": 2.1572409999999707, + "p90": 2.1722446999999874, "runs": 10, - "total": 21.482185000000044 + "total": 21.637756999999965 }, "retrieval_s": { - "max": 1.0197309999999788, - "mean": 1.000152, - "median": 0.9985730000000075, - "min": 0.9954809999999839, - "p90": 1.0016814999999923, + "max": 1.0145960000000116, + "mean": 1.0008668999999997, + "median": 0.9992569999999859, + "min": 0.996422999999993, + "p90": 1.005939799999996, "runs": 10, - "total": 10.00152 + "total": 10.008668999999998 }, "setup_s": { - "max": 1.1587460000000078, - "mean": 1.1480665000000045, - "median": 1.1456539999999933, - "min": 1.1440660000000094, - "p90": 1.1561558000000247, + "max": 1.1693369999999845, + "mean": 1.1629087999999967, + "median": 1.1622770000000031, + "min": 1.1587100000000135, + "p90": 1.1663048999999917, "runs": 10, - "total": 11.480665000000045 + "total": 11.629087999999967 } }, "residuals": { @@ -296,62 +296,62 @@ }, "timing_s": { "first_use_total_s": { - "max": 1.14051558310166, - "mean": 1.1296011375263333, - "median": 1.127380437683314, - "min": 1.1249205842614174, - "p90": 1.1371092338114976, + "max": 1.153641000855714, + "mean": 1.1396208001300692, + "median": 1.1371017289347947, + "min": 1.135196874383837, + "p90": 1.1438435254152863, "runs": 10, - "total": 11.296011375263333 + "total": 11.396208001300693 }, "retrieval_s": { - "max": 0.5291183330118656, - "mean": 0.5190001083072275, - "median": 0.5182693749666214, - "min": 0.5164273749105632, - "p90": 0.520501958252862, + "max": 0.5281550837680697, + "mean": 0.5194493168033659, + "median": 0.5183592080138624, + "min": 0.516583124641329, + "p90": 0.5222717840224504, "runs": 10, - "total": 5.190001083072275 + "total": 5.1944931680336595 }, "setup_s": { - "max": 0.6171861672773957, - "mean": 0.6106010292191059, - "median": 0.6092561658006161, - "min": 0.608172292355448, - "p90": 0.614657317334786, + "max": 0.6254859170876443, + "mean": 0.6201714833267034, + "median": 0.6193227500189096, + "min": 0.6172903338447213, + "p90": 0.6231065043248236, "runs": 10, - "total": 6.106010292191058 + "total": 6.201714833267033 } } }, "retrieval_session": { "average_active_cores": { "first_use_total": { - "max": 1.9476442979666493, - "mean": 1.9466221441806872, - "median": 1.947458519688773, - "min": 1.9398314162002976, - "p90": 1.9475784863191568, + "max": 1.9482760504465428, + "mean": 1.947140383715044, + "median": 1.9470003712372859, + "min": 1.9460349290914665, + "p90": 1.9479394551311018, "runs": 10, - "total": 19.466221441806873 + "total": 19.47140383715044 }, "retrieval": { - "max": 1.981263667616968, - "mean": 1.979647494970666, - "median": 1.9808250763506539, - "min": 1.9692818323543977, - "p90": 1.981146125330225, + "max": 1.9813520873214674, + "mean": 1.9805844077966452, + "median": 1.9805606878081667, + "min": 1.9798507624031743, + "p90": 1.9812075721971327, "runs": 10, - "total": 19.79647494970666 + "total": 19.80584407796645 }, "setup": { - "max": 1.8985466472960637, - "mean": 1.8971801020635914, - "median": 1.8974079505024601, - "min": 1.8953064432175761, - "p90": 1.8982462253226426, + "max": 1.9002655636015189, + "mean": 1.8976577192787272, + "median": 1.897440416529498, + "min": 1.895778570655443, + "p90": 1.8989136029953497, "runs": 10, - "total": 18.971801020635915 + "total": 18.97657719278727 } }, "kind": "optimal_estimation", @@ -359,31 +359,31 @@ "mode": "session", "process_cpu_s": { "first_use_total_s": { - "max": 3.5694339999999727, - "mean": 3.560944799999993, - "median": 3.5605064999999883, - "min": 3.555556000000024, - "p90": 3.566390199999984, + "max": 3.5642609999999877, + "mean": 3.551356899999996, + "median": 3.5502264999999937, + "min": 3.544240000000002, + "p90": 3.5592704999999767, "runs": 10, - "total": 35.60944799999993 + "total": 35.51356899999996 }, "retrieval_s": { - "max": 2.1810199999999895, - "mean": 2.1712208000000004, - "median": 2.17048250000002, - "min": 2.1672770000000128, - "p90": 2.174331199999995, + "max": 2.1677500000000123, + "mean": 2.1555191000000007, + "median": 2.154267500000003, + "min": 2.149449000000004, + "p90": 2.1605337999999934, "runs": 10, - "total": 21.712208000000004 + "total": 21.555191000000008 }, "setup_s": { - "max": 1.398081999999988, - "mean": 1.3897239999999926, - "median": 1.3884959999999893, - "min": 1.3852550000000008, - "p90": 1.3936377999999792, + "max": 1.3995789999999886, + "mean": 1.3958377999999954, + "median": 1.3954845000000091, + "min": 1.393981999999994, + "p90": 1.3973874999999965, "runs": 10, - "total": 13.897239999999925 + "total": 13.958377999999954 } }, "residuals": { @@ -393,53 +393,53 @@ }, "timing_s": { "first_use_total_s": { - "max": 1.8400743333622813, - "mean": 1.8292984292842447, - "median": 1.8283372079022229, - "min": 1.8262582505121827, - "p90": 1.8319814964663237, + "max": 1.8300513750873506, + "mean": 1.8238829955458642, + "median": 1.8236639581155032, + "min": 1.8203542078845203, + "p90": 1.8269428121857345, "runs": 10, - "total": 18.292984292842448 + "total": 18.23882995545864 }, "retrieval_s": { - "max": 1.1075205001980066, - "mean": 1.0967777250800281, - "median": 1.0958766879048198, - "min": 1.0938862077891827, - "p90": 1.0984904244542122, + "max": 1.0941647910512984, + "mean": 1.0883247330784798, + "median": 1.0877012708224356, + "min": 1.0852649160660803, + "p90": 1.0911868412513286, "runs": 10, - "total": 10.967777250800282 + "total": 10.883247330784798 }, "setup_s": { - "max": 0.7363959173671901, - "mean": 0.7325207042042166, - "median": 0.7321603330783546, - "min": 0.730292041786015, - "p90": 0.7349410670809448, + "max": 0.7366091660223901, + "mean": 0.7355582624673843, + "median": 0.7353832710068673, + "min": 0.7345859999768436, + "p90": 0.7365267413202673, "runs": 10, - "total": 7.325207042042166 + "total": 7.355582624673843 } } }, "retrieval_sweep_fast_mode": { "average_active_cores": { "first_use_total": { - "max": 1.9316602534518694, - "mean": 1.9234436970303332, - "median": 1.9231076512255552, - "min": 1.9106641883432653, - "p90": 1.9315997674218632, + "max": 1.9292465704217443, + "mean": 1.923083279928195, + "median": 1.9211929736299802, + "min": 1.9171923347523585, + "p90": 1.9290186965376586, "runs": 5, - "total": 9.617218485151666 + "total": 9.615416399640974 }, "retrieval": { - "max": 1.9619961591685882, - "mean": 1.9528505486080738, - "median": 1.954541237281785, - "min": 1.9340702789473894, - "p90": 1.9619147534710688, + "max": 1.9614083043821908, + "mean": 1.9557679195959312, + "median": 1.95418389924391, + "min": 1.950820218263699, + "p90": 1.9607246854018578, "runs": 5, - "total": 9.76425274304037 + "total": 9.778839597979657 } }, "converged_count": 5, @@ -448,24 +448,24 @@ "mode": "fast_mode", "process_cpu_s": { "first_use_total_s": { - "max": 3.172604999999976, - "mean": 2.8567433999999823, - "median": 2.7238669999999843, - "min": 2.5898210000000006, - "p90": 3.1620609999999827, + "max": 3.1942820000000154, + "mean": 2.8719246000000114, + "median": 2.7363670000000297, + "min": 2.6018459999999948, + "p90": 3.1803860000000044, "runs": 5, - "total": 14.28371699999991 + "total": 14.359623000000056 }, "retrieval_s": { - "max": 2.020212999999984, - "mean": 1.7088631999999904, - "median": 1.5774879999999882, - "min": 1.444593999999995, - "p90": 2.0124569999999893, + "max": 2.028275000000008, + "mean": 1.7105520000000012, + "median": 1.5753270000000157, + "min": 1.4429409999999905, + "p90": 2.016867400000001, "runs": 5, - "total": 8.544315999999952 + "total": 8.552760000000006 }, - "total_process_cpu_s": 14.28371699999991 + "total_process_cpu_s": 14.359623000000056 }, "residuals": { "truth_aerosol_mid_pressure_abs_error_hpa": { @@ -492,45 +492,45 @@ "run_count": 5, "timing_s": { "first_use_total_s": { - "max": 1.6425525001250207, - "mean": 1.484759433288127, - "median": 1.4163882080465555, - "min": 1.3554558753967285, - "p90": 1.6370425500907004, + "max": 1.6557147484272718, + "mean": 1.4930677577853202, + "median": 1.4243061668239534, + "min": 1.3571126656606793, + "p90": 1.6487054323777557, "runs": 5, - "total": 7.423797166440636 + "total": 7.465338788926601 }, "retrieval_s": { - "max": 1.0296722501516342, - "mean": 0.8745379417203367, - "median": 0.807088625151664, - "min": 0.7469190834090114, - "p90": 1.0257614500820638, + "max": 1.034091165754944, + "mean": 0.8743570582009852, + "median": 0.8061303752474487, + "min": 0.7396586248651147, + "p90": 1.0286307995207609, "runs": 5, - "total": 4.372689708601683 + "total": 4.371785291004926 }, - "total_wall_s": 7.423797166440636 + "total_wall_s": 7.465338788926601 } }, "retrieval_sweep_session": { "average_active_cores": { "first_use_total": { - "max": 1.9797486691413195, - "mean": 1.9727135715880764, - "median": 1.969043036589504, - "min": 1.9668675032987888, - "p90": 1.9795218960940542, + "max": 1.9792287673415523, + "mean": 1.9719103497840087, + "median": 1.9696950956809465, + "min": 1.9681633728050625, + "p90": 1.9770847753977279, "runs": 5, - "total": 9.863567857940382 + "total": 9.859551748920044 }, "retrieval": { - "max": 1.9934707785364014, - "mean": 1.991458134875399, - "median": 1.9910156146854183, - "min": 1.9897213736131625, - "p90": 1.9934115872403158, + "max": 1.9938274039832375, + "mean": 1.9910007936208967, + "median": 1.9909385235456762, + "min": 1.987433716155894, + "p90": 1.993138215758069, "runs": 5, - "total": 9.957290674376996 + "total": 9.955003968104483 } }, "converged_count": 5, @@ -539,24 +539,24 @@ "mode": "session", "process_cpu_s": { "first_use_total_s": { - "max": 10.216241000000025, - "mean": 7.690339399999994, - "median": 6.374689999999987, - "min": 5.874586999999991, - "p90": 10.062170600000012, + "max": 10.24295699999999, + "mean": 7.715566200000012, + "median": 6.368831, + "min": 5.869263000000018, + "p90": 10.124553000000002, "runs": 5, - "total": 38.45169699999997 + "total": 38.57783100000006 }, "retrieval_s": { - "max": 8.827287000000013, - "mean": 6.301708399999995, - "median": 4.986369999999994, - "min": 4.486376000000007, - "p90": 8.673810200000002, + "max": 8.839351999999991, + "mean": 6.316100400000005, + "median": 4.974997000000002, + "min": 4.474050000000005, + "p90": 8.719159599999994, "runs": 5, - "total": 31.508541999999977 + "total": 31.580502000000024 }, - "total_process_cpu_s": 38.45169699999997 + "total_process_cpu_s": 38.57783100000006 }, "residuals": { "truth_aerosol_mid_pressure_abs_error_hpa": { @@ -583,24 +583,24 @@ "run_count": 5, "timing_s": { "first_use_total_s": { - "max": 5.160372707527131, - "mean": 3.8956601083278657, - "median": 3.2379757496528327, - "min": 2.986773125361651, - "p90": 5.083118474762887, + "max": 5.175226415973157, + "mean": 3.910810708347708, + "median": 3.2334095840342343, + "min": 2.981446583289653, + "p90": 5.120861932914704, "runs": 5, - "total": 19.478300541639328 + "total": 19.55405354173854 }, "retrieval_s": { - "max": 4.428428249899298, - "mean": 3.163610566686839, - "median": 2.5060156658291817, - "min": 2.25477600004524, - "p90": 4.351307016890496, + "max": 4.433358666021377, + "mean": 3.1724465833045543, + "median": 2.4991195420734584, + "min": 2.2458912921138108, + "p90": 4.378587432857603, "runs": 5, - "total": 15.818052833434194 + "total": 15.862232916522771 }, - "total_wall_s": 19.478300541639328 + "total_wall_s": 19.55405354173854 } } }, @@ -628,35 +628,35 @@ "runs": 5 }, "first_use_total_speedup_s": { - "max": 3.531595082487911, - "max_abs": 3.531595082487911, - "mean": 2.4109006750397386, - "median": 1.8215875416062772, - "min": 1.6313172499649227, + "max": 3.5370349576696754, + "max_abs": 3.5370349576696754, + "mean": 2.4177429505623875, + "median": 1.809103417210281, + "min": 1.6243339176289737, "runs": 5 }, "retrieval_speedup_s": { - "max": 3.40853299992159, - "max_abs": 3.40853299992159, - "mean": 2.289072624966502, - "median": 1.6989270406775177, - "min": 1.5078569166362286, + "max": 3.412918415851891, + "max_abs": 3.412918415851891, + "mean": 2.298089525103569, + "median": 1.6929891668260098, + "min": 1.506232667248696, "runs": 5 }, - "total_retrieval_wall_s_saved": 12.054503375198692 + "total_retrieval_wall_s_saved": 12.088714752811939 } }, - "created_at_unix_s": 1781518266.7367098, - "finished_at_unix_s": 1781518402.0570638, + "created_at_unix_s": 1781712300.413607, + "finished_at_unix_s": 1781712436.118529, "memory": { - "peak_rss_bytes": 95158272, - "peak_rss_delta_bytes": 57016320, - "peak_rss_delta_mib": 54.375, - "peak_rss_mib": 90.75, + "peak_rss_bytes": 72712192, + "peak_rss_delta_bytes": 34504704, + "peak_rss_delta_mib": 32.90625, + "peak_rss_mib": 69.34375, "peak_rss_source": "resource.getrusage(RUSAGE_SELF).ru_maxrss", "peak_rss_supported": true, - "start_peak_rss_bytes": 38141952, - "start_peak_rss_mib": 36.375 + "start_peak_rss_bytes": 38207488, + "start_peak_rss_mib": 36.4375 }, "memory_layout": { "instrument_sampling": { @@ -1016,12 +1016,12 @@ "current_collision_complex_sample_spline_scratch_bytes": 0, "previous_collision_complex_profile_cache_bytes": 2072, "previous_collision_complex_sample_spline_scratch_bytes": 10240, - "source": "collision-complex profile cache used by benchmark O2 A optical accumulation" + "source": "collision-complex profile cache used by benchmark optical accumulation" } }, "native_binding": { "command": "zig build sync-python-package -Doptimize=ReleaseFast", - "elapsed_s": 0.24331470904871821 + "elapsed_s": 0.12126141740009189 }, "output": "benchmark/results.json", "report": { @@ -1029,27 +1029,27 @@ { "case": "Forward, no session", "residuals": "DISAMAR fixture worst interior max_abs 9.569e-14; RTM reflectance 5.390e-14", - "timing": "median 0.999s" + "timing": "median 1.004s" }, { "case": "Forward, session", "residuals": "vs no-session: radiance max_abs 0.000e+00; reflectance max_abs 0.000e+00", - "timing": "setup 0.728s; cached run 0.001s; first-use 1.002s" + "timing": "setup 0.732s; cached run 0.001s; first-use 1.005s" }, { "case": "Forward, fast mode", "residuals": "baseline max_abs 2.203e-04; 4-scene worst 4.969e-04, 1.601x noise", - "timing": "baseline reference 1.975s; 4-scene fast median 4.835s" + "timing": "baseline reference 1.965s; 4-scene fast median 4.860s" }, { "case": "OE, session", "residuals": "AOD diff 8.699e-08; mid-pressure diff 1.733e-04 hPa", - "timing": "retrieval 1.095877s; setup 0.732160s" + "timing": "retrieval 1.087701s; setup 0.735383s" }, { "case": "OE, fast mode", "residuals": "single fast-vs-session AOD delta -4.294e-06; sweep max AOD delta 2.719e-04; pressure delta 1.744e-01 hPa", - "timing": "5-case median 1.416388s; mean 1.484759s; range 1.355456-1.642553s" + "timing": "5-case median 1.424306s; mean 1.493068s; range 1.357113-1.655715s" } ], "memory_layout_rows": [ @@ -1073,11 +1073,11 @@ { "benchmark": "Full benchmark process", "process_cpu": "see top-level totals", - "process_memory": "peak RSS 90.8 MiB; delta since benchmark start 54.4 MiB" + "process_memory": "peak RSS 69.3 MiB; delta since benchmark start 32.9 MiB" }, { "benchmark": "Forward, no session", - "process_cpu": "median CPU 1.921s; median active cores 1.92", + "process_cpu": "median CPU 1.932s; median active cores 1.92", "process_memory": "included in full benchmark process peak RSS" }, { @@ -1087,44 +1087,44 @@ }, { "benchmark": "Forward, fast mode", - "process_cpu": "median CPU 9.369s; median active cores 1.94", + "process_cpu": "median CPU 9.400s; median active cores 1.93", "process_memory": "included in full benchmark process peak RSS" }, { "benchmark": "OE, no fast mode sweep", - "process_cpu": "total CPU 38.452s; median active cores 1.97", + "process_cpu": "total CPU 38.578s; median active cores 1.97", "process_memory": "included in full benchmark process peak RSS" }, { "benchmark": "OE, fast mode sweep", - "process_cpu": "total CPU 14.284s; median active cores 1.92", + "process_cpu": "total CPU 14.360s; median active cores 1.92", "process_memory": "included in full benchmark process peak RSS" } ], "total_wall_clock_rows": [ { "benchmark": "Forward, no session", - "total": "0.999s per baseline run median" + "total": "1.004s per baseline run median" }, { "benchmark": "Forward, session", - "total": "1.002s first-use total, or 0.001s cached run only" + "total": "1.005s first-use total, or 0.001s cached run only" }, { "benchmark": "Forward, fast mode", - "total": "4.835s median over 4 scenes" + "total": "4.860s median over 4 scenes" }, { "benchmark": "OE, no fast mode", - "total": "19.478s over 5 retrievals" + "total": "19.554s over 5 retrievals" }, { "benchmark": "OE, fast mode", - "total": "7.424s over 5 retrievals" + "total": "7.465s over 5 retrievals" }, { "benchmark": "OE fast-mode savings", - "total": "12.055s over 5 retrievals" + "total": "12.089s over 5 retrievals" } ] }, @@ -1141,10 +1141,10 @@ "retrieval_sweep_fast_mode_count": 5, "retrieval_sweep_session_count": 5 }, - "run_id": "3e5a4527eb754d47a4c56ebcba781bc7", + "run_id": "caf8446bd8c54385bfb9fe22f811eb48", "schema_version": 4, "scratch_db": "benchmark/.runs/benchmark.sqlite", - "total_benchmark_average_active_cores": 1.9377135211677916, - "total_benchmark_process_cpu_s": 261.154183, - "total_benchmark_wall_s": 134.7744029997848 + "total_benchmark_average_active_cores": 1.9348558685442712, + "total_benchmark_process_cpu_s": 261.736896, + "total_benchmark_wall_s": 135.27462187502533 } diff --git a/benchmark/suite/cases.py b/benchmark/suite/cases.py index b482ca7f2..24ff2c7d9 100644 --- a/benchmark/suite/cases.py +++ b/benchmark/suite/cases.py @@ -1,4 +1,4 @@ -"""Benchmark case construction.""" +"""Benchmark scene construction.""" import copy import json diff --git a/benchmark/suite/forward.py b/benchmark/suite/forward.py index 6d0ee4442..55e670f35 100644 --- a/benchmark/suite/forward.py +++ b/benchmark/suite/forward.py @@ -41,7 +41,6 @@ def run_no_session( lambda: rtm.spectrum( case, jacobian=True, - jacobian_state_names=config.FORWARD_STATE_NAMES, ) ) samples.append((timing, spectrum)) @@ -107,7 +106,6 @@ def run_session( first_timing, first_sample = timed( lambda: cache.spectrum( jacobian=True, - jacobian_state_names=config.FORWARD_STATE_NAMES, ) ) db.sample( @@ -126,7 +124,6 @@ def run_session( timing, _ = timed( lambda: cache.spectrum( jacobian=True, - jacobian_state_names=config.FORWARD_STATE_NAMES, ) ) cached.append(timing) diff --git a/benchmark/suite/layout.py b/benchmark/suite/layout.py index 39a999756..ee1575c2a 100644 --- a/benchmark/suite/layout.py +++ b/benchmark/suite/layout.py @@ -1,4 +1,4 @@ -"""Memory-layout diagnostics derived from benchmark cases.""" +"""Memory-layout diagnostics derived from benchmark scenes.""" from typing import Any @@ -109,7 +109,7 @@ def optical_accumulation_layout() -> dict[str, Any]: return { "boundary": "post-timing diagnostic; excluded from benchmark wall time and peak RSS", - "source": "collision-complex profile cache used by benchmark O2 A optical accumulation", + "source": "collision-complex profile cache used by benchmark optical accumulation", "benchmark_profile_node_count": COLLISION_COMPLEX_PROFILE_NODE_COUNT, "collision_complex_profile_cache_capacity": (COLLISION_COMPLEX_PROFILE_CACHE_CAPACITY), "previous_collision_complex_profile_cache_bytes": ( diff --git a/benchmark/suite/residuals.py b/benchmark/suite/residuals.py index 0aeeaac63..00ba17295 100644 --- a/benchmark/suite/residuals.py +++ b/benchmark/suite/residuals.py @@ -5,7 +5,7 @@ from typing import Any import numpy as np -from zdisamar.inverse_method.optimal_estimation import o2a as band_retrieval +from zdisamar.optimal_estimation import o2a as band_retrieval from zdisamar.output.spectrum import Spectrum from validation.o2a.measurement_noise import components_from_spectrum diff --git a/benchmark/suite/retrieval.py b/benchmark/suite/retrieval.py index 895ec6d15..2ef98afa4 100644 --- a/benchmark/suite/retrieval.py +++ b/benchmark/suite/retrieval.py @@ -3,7 +3,7 @@ from typing import Any from zdisamar import rtm -from zdisamar.inverse_method.optimal_estimation import o2a as band_retrieval +from zdisamar.optimal_estimation import o2a as band_retrieval from validation.o2a.measurement_noise import measurement_from_o2a_baseline_noise from validation.optimal_estimation import setup @@ -162,7 +162,8 @@ def run_once( setup_start = timing_start() - with rtm.SessionCache(case) as cache: + cache_context = rtm.SessionCache() if mode == "fast_mode" else rtm.SessionCache(case) + with cache_context as cache: setup_timing = elapsed_since(setup_start) retrieval_start = timing_start() result = band_retrieval.retrieve( diff --git a/build.zig b/build.zig index e71071726..3ab556b84 100644 --- a/build.zig +++ b/build.zig @@ -316,7 +316,7 @@ pub fn build(b: *std.Build) void { const run_c_api_retained_tests = b.addRunArtifact(c_api_retained_tests); b.step( "test-api-c-retained", - "Run retained C ABI tests with the default O2 A product case", + "Run retained C ABI tests with the default product scene", ).dependOn(&run_c_api_retained_tests.step); const cost_timing_o2a_scene_module = b.createModule(.{ @@ -342,7 +342,7 @@ pub fn build(b: *std.Build) void { const run_cost_timing_analysis = b.addRunArtifact(cost_timing_analysis); b.step( "cost-timing-forward-analysis", - "Run one enabled O2 A forward analysis and emit merged cost timing", + "Run one enabled forward analysis and emit merged cost timing", ).dependOn(&run_cost_timing_analysis.step); const run_cost_timing_fast_analysis = b.addSystemCommand(&.{ "uv", @@ -353,7 +353,7 @@ pub fn build(b: *std.Build) void { run_cost_timing_fast_analysis.addArtifactArg(cost_timing_analysis); b.step( "cost-timing-fast-analysis", - "Run fast cost-timing analysis gate for one enabled O2 A forward", + "Run fast cost-timing analysis gate for one enabled forward", ).dependOn(&run_cost_timing_fast_analysis.step); const labos_bottleneck_trace = addTraceExecutable( b, @@ -414,7 +414,7 @@ pub fn build(b: *std.Build) void { const test_retained_step = b.step( "test-retained", - "Run retained explicit-dataflow suite, including full default O2 A/OE coverage", + "Run retained explicit-dataflow suite, including full default OE coverage", ); test_retained_step.dependOn(test_fast_step); test_retained_step.dependOn(&run_unit_tests.step); diff --git a/docs/README.md b/docs/README.md index 61114d16c..1b2ed8d1d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,10 +1,10 @@ # zdisamar Docs -The retained docs describe the DISAMAR scientific context, the current O2 A +The retained docs describe the DISAMAR scientific context, the current RTM package, and the reference-data boundary. ## Current Docs 1. [DISAMAR Overview](./disamar-overview.md) -2. [O2 A RTM](./o2a-rtm.md) +2. [O2A RTM](./o2a-rtm.md) 3. [Reference Data And Bundles](./reference-data-and-bundles.md) diff --git a/docs/disamar-overview.md b/docs/disamar-overview.md index 59a0168bb..eea4cc98f 100644 --- a/docs/disamar-overview.md +++ b/docs/disamar-overview.md @@ -24,7 +24,7 @@ Several aspects of the literature matter directly for this repository. The 2022 GMD model description presents DISAMAR as both a radiative-transfer model and a retrieval system. That context matters even while this repository's -retained public runtime is centered on O2 A RTM execution. +retained public runtime is centered on O2A RTM execution. ### Operational oxygen A-band work @@ -83,11 +83,11 @@ The RTM runtime is separate from DISMAS retrieval execution. ## Why This Matters For `zdisamar` -The current implementation is organized around O2 A wavelength-band cases, RTM +The current implementation is organized around wavelength-band scenes, RTM execution, and inverse methods. - `src/input/` carries the typed scene and observation description. -- `python/zdisamar/wavelength_bands/` carries wavelength-band case surfaces. +- `python/zdisamar/wavelength_bands/` carries wavelength-band scene surfaces. - `python/zdisamar/rtm/` carries packaged RTM execution helpers. - `src/input/reference_data/` carries the scientific input surfaces needed to prepare execution without letting file I/O leak into numerical routines. - the Zig numerical tree carries optical-property construction, radiative @@ -115,5 +115,5 @@ For the present codebase, that means the implementation has to satisfy three con After this overview: -1. Read [O2 A RTM](./o2a-rtm.md) for the retained public runtime path. +1. Read [O2A RTM](./o2a-rtm.md) for the retained public runtime path. 2. Read [Reference Data And Bundles](./reference-data-and-bundles.md) for the data-loading boundary. diff --git a/docs/o2a-rtm.md b/docs/o2a-rtm.md index 8e527afd4..efb4388a0 100644 --- a/docs/o2a-rtm.md +++ b/docs/o2a-rtm.md @@ -1,4 +1,4 @@ -# O2 A RTM +# O2A RTM The Python package keeps wavelength-band scenes separate from radiative-transfer execution: @@ -22,11 +22,11 @@ with rtm.SessionCache(scene) as cache: The public Python runtime surface is: -- `zdisamar.wavelength_bands.o2a` for O2 A scene data classes and reference-scene +- `zdisamar.wavelength_bands.o2a` for scene data classes and reference-scene construction, - `zdisamar.rtm` for radiance, reflectance, atmospheric-budget, instrument response, and collision-induced-absorption execution helpers, -- `zdisamar.inverse_method.optimal_estimation` for inverse-method data classes +- `zdisamar.optimal_estimation` for inverse-method data classes and the Rodgers-style optimal-estimation implementation. The Python package loads the Zig-built binding and reference data from packaged diff --git a/docs/reference-data-and-bundles.md b/docs/reference-data-and-bundles.md index 0ed6a86b2..6310b64a6 100644 --- a/docs/reference-data-and-bundles.md +++ b/docs/reference-data-and-bundles.md @@ -60,7 +60,7 @@ These structures are not bookkeeping wrappers. They are the physical data surfac ## Runtime Preparation Path -The default O2 A execution path is: +The default execution path is: 1. `Input` records the spectral grid, geometry, surface, atmosphere, and instrument controls. 2. `ReferenceData` loads the reference datasets needed for that input. @@ -90,7 +90,7 @@ runs without turning the entire codebase into a mission-specific file reader. Reference data matters only if a result can state which dataset family it used. For that reason the current implementation records dataset identity, hashes, -case settings, radiative-transfer method, and report provenance. +scene settings, radiative-transfer method, and report provenance. The result is that a stored artifact can be tied back to both a scientific configuration and a concrete set of reference data. diff --git a/python/zdisamar/__init__.py b/python/zdisamar/__init__.py index 2a002ea71..f25c903c6 100644 --- a/python/zdisamar/__init__.py +++ b/python/zdisamar/__init__.py @@ -7,8 +7,8 @@ def __getattr__(name: str): - if name == "inverse_method": - module = import_module(".inverse_method", __name__) + if name == "optimal_estimation": + module = import_module(".optimal_estimation", __name__) globals()[name] = module return module @@ -19,5 +19,6 @@ def __getattr__(name: str): __all__ = [ "reference_data", "rtm", + "optimal_estimation", "wavelength_bands", ] diff --git a/python/zdisamar/api.py b/python/zdisamar/api.py index 789b6dbd4..dc46b0f02 100644 --- a/python/zdisamar/api.py +++ b/python/zdisamar/api.py @@ -1,5 +1,5 @@ """Public Python API module.""" -from . import reference_data, rtm, wavelength_bands +from . import optimal_estimation, reference_data, rtm, wavelength_bands -__all__ = ["reference_data", "rtm", "wavelength_bands"] +__all__ = ["reference_data", "rtm", "optimal_estimation", "wavelength_bands"] diff --git a/python/zdisamar/bindings/handles.py b/python/zdisamar/bindings/handles.py index a4b37f376..175ea5e32 100644 --- a/python/zdisamar/bindings/handles.py +++ b/python/zdisamar/bindings/handles.py @@ -32,9 +32,10 @@ COptimalEstimationBatchResult, COptimalEstimationControls, COptimalEstimationFastmodeBatchResult, + COptimalEstimationPressureSpec, COptimalEstimationRequest, COptimalEstimationResult, - COptimalEstimationStateSpec, + COptimalEstimationScalarSpec, CSpectrum, O2LineContributionsRaw, OxygenCollisionInducedAbsorptionDiagnosticsRaw, @@ -43,6 +44,7 @@ _MAX_OPTIMAL_ESTIMATION_ITERATIONS = 1000 _MAX_UINT32 = 2**32 - 1 _NATIVE_PRESSURE_STATE = "aerosol_layer_mid_pressure_hpa" +_OPTIMAL_ESTIMATION_STATE_NAMES = ("aerosol_optical_depth", _NATIVE_PRESSURE_STATE) _BATCH_RUN_STATUS_NAMES = ("pending", "ok", "failed") @@ -70,20 +72,6 @@ def channel_mask(channels: tuple[str, ...]) -> int: return mask -def jacobian_state_ids(state_names: tuple[str, ...]): - """Translate retrieval names into model Jacobian selectors.""" - - ids = [] - - for state_name in state_names: - try: - ids.append(JACOBIAN_STATE_NAMES.index(state_name)) - except ValueError as exc: - raise ValueError(f"unsupported Jacobian state: {state_name}") from exc - - return (ctypes.c_uint8 * len(ids))(*ids) - - def batch_run_status(value: int) -> str: """Translate native per-start batch status into a stable Python label.""" @@ -148,17 +136,17 @@ def __init__(self): @property def input(self) -> Scene | None: - """Return the wavelength-band case loaded into this handle.""" + """Return the wavelength-band scene loaded into this handle.""" return None if self._scene is None else copy.deepcopy(self._scene) def default_scene(self) -> Scene: - """Return the packaged O2 A reference case.""" + """Return the packaged reference scene.""" return default_o2a_scene() def load_scene(self, scene: Scene, *, copy_scene: bool = True) -> None: - """Load one O2 A wavelength-band case into the RTM handle.""" + """Load one wavelength-band scene into the RTM handle.""" rtm_scene = scene.with_rtm_optimisation_applied() resolved = rtm_scene.with_resolved_asset_resolver(reference_data.resolve_asset_path) @@ -176,11 +164,11 @@ def load_scene(self, scene: Scene, *, copy_scene: bool = True) -> None: self._solar_mu0 = rtm_scene.geometry.solar_mu0 def loaded_scene_matches(self, scene: Scene) -> bool: - """Return whether the native handle already owns this prepared case. + """Return whether the native handle already owns this prepared scene. The fingerprint uses the resolved deterministic payload passed to Zig. That lets session users avoid a duplicate prepare without retaining a - Python case copy solely for invalidation. + Python scene copy solely for invalidation. """ if self._scene_fingerprint is None: @@ -199,52 +187,27 @@ def warm_cache(self) -> None: def warm_optimal_estimation_cache(self, state_names: tuple[str, ...]) -> None: """Build reusable RTM work arrays for the OE Jacobian route.""" - state_ids = jacobian_state_ids(state_names) - self._check( - self._lib.zds_warm_o2a_optimal_estimation( - self._ctx, - state_ids, - len(state_ids), + if tuple(state_names) != _OPTIMAL_ESTIMATION_STATE_NAMES: + raise ValueError( + "optimal-estimation cache warming requires aerosol_optical_depth " + "and aerosol_layer_mid_pressure_hpa in that order" ) - ) + + self._check(self._lib.zds_warm_o2a_optimal_estimation(self._ctx)) def spectrum( self, *, jacobian: bool = False, - jacobian_state_names: tuple[str, ...] | None = None, include_scene: bool = False, ) -> Spectrum: - """Run the loaded wavelength-band case and return copied spectral arrays.""" + """Run the loaded wavelength-band scene and return copied spectral arrays.""" raw = CSpectrum() - if jacobian_state_names is not None and not jacobian: - raise ValueError("jacobian_state_names requires jacobian=True") - if jacobian and self._loaded_has_multi_layer_aerosol_profile: raise ValueError("multi-layer aerosol profile Jacobians are not supported") - if jacobian_state_names is not None: - if len(jacobian_state_names) == 0: - raise ValueError("jacobian_state_names must not be empty") - - state_ids = jacobian_state_ids(jacobian_state_names) - self._check( - self._lib.zds_run_spectrum_jacobian_for_states( - self._ctx, - ctypes.byref(raw), - state_ids, - len(state_ids), - ) - ) - - return self._copied_spectrum( - raw, - jacobian_state_names=jacobian_state_names, - include_scene=include_scene, - ) - runner = self._lib.zds_run_spectrum_jacobian if jacobian else self._lib.zds_run_spectrum self._check(runner(self._ctx, ctypes.byref(raw))) @@ -267,7 +230,7 @@ def atmospheric_budget(self, wavelengths_nm) -> AtmosphericBudget: return AtmosphericBudget(self._copied_rows(raw, self._lib.zds_atmospheric_budget_free)) def o2_line_contributions(self, wavelengths_nm, max_rows: int = 50_000) -> O2LineContributions: - """Return copied line-by-line O2 evidence rows.""" + """Return copied line-by-line evidence rows.""" wavelengths = contiguous_wavelengths(wavelengths_nm) @@ -339,7 +302,7 @@ def collision_induced_absorption( ) def optimal_estimation(self, *, measurement, state_vector, controls): - """Run native O2 A optimal estimation for the loaded case.""" + """Run native optimal estimation for the loaded scene.""" return self._run_optimal_estimation( "zds_run_o2a_optimal_estimation", @@ -349,7 +312,7 @@ def optimal_estimation(self, *, measurement, state_vector, controls): ) def optimal_estimation_correction(self, *, measurement, state_vector, controls): - """Run one native OE correction for the loaded prepared O2 A case.""" + """Run one native OE correction for the loaded prepared scene.""" return self._run_optimal_estimation( "zds_run_o2a_optimal_estimation_correction", @@ -368,7 +331,7 @@ def optimal_estimation_batch( controls, batch_workers=1, ): - """Run many native O2 A retrievals against one loaded case and measurement.""" + """Run many native retrievals against one loaded scene and measurement.""" request, buffers = self._optimal_estimation_batch_request( measurement=measurement, @@ -491,7 +454,15 @@ def _optimal_estimation_request(self, *, measurement, state_vector, controls): ) state_buffers = [] - state_specs = [] + scalar_specs = [] + pressure_interval_index_1based = 0 + pressure_thickness_hpa = 0.0 + pressure_profile_altitude = None + pressure_profile_pressure = None + pressure_profile_count = 0 + parameters = tuple(state_vector.parameters) + parameter_names = tuple(parameter.name for parameter in parameters) + raw_max_iterations = controls.max_iterations if not isinstance(raw_max_iterations, Integral) or isinstance(raw_max_iterations, bool): @@ -505,7 +476,7 @@ def _optimal_estimation_request(self, *, measurement, state_vector, controls): f"1 and {_MAX_OPTIMAL_ESTIMATION_ITERATIONS}" ) - for parameter in state_vector.parameters: + for parameter in parameters: parameter_name = parameter.name state_name = getattr(parameter, "jacobian_name", parameter_name) @@ -523,10 +494,8 @@ def _optimal_estimation_request(self, *, measurement, state_vector, controls): "native optimal estimation does not support custom jacobian_scale transforms" ) - try: - state_id = JACOBIAN_STATE_NAMES.index(state_name) - except ValueError as exc: - raise ValueError(f"unsupported optimal-estimation state: {state_name}") from exc + if state_name not in _OPTIMAL_ESTIMATION_STATE_NAMES: + raise ValueError(f"unsupported optimal-estimation state: {state_name}") profile = getattr(parameter, "pressure_altitude_profile", None) profile_altitude = None @@ -562,35 +531,45 @@ def _optimal_estimation_request(self, *, measurement, state_vector, controls): "optimal-estimation state prior_uncertainty values must be finite and positive" ) - state_specs.append( - COptimalEstimationStateSpec( - state_id=state_id, + scalar_specs.append( + COptimalEstimationScalarSpec( has_lower=0 if lower is None else 1, has_upper=0 if upper is None else 1, - interval_index_1based=interval_index_1based, initial=float(parameter.initial), prior=float(parameter.prior), variance=uncertainty * uncertainty, lower=0.0 if lower is None else float(lower), upper=0.0 if upper is None else float(upper), - thickness_hpa=float(getattr(parameter, "thickness_hpa", 0.0)), - pressure_profile_count=profile_count, - pressure_profile_altitude_km=profile_altitude, - pressure_profile_pressure_hpa=profile_pressure, ) ) - if not state_specs: - raise ValueError("state vector must contain at least one parameter") + if state_name == _NATIVE_PRESSURE_STATE: + pressure_interval_index_1based = interval_index_1based + pressure_thickness_hpa = float(getattr(parameter, "thickness_hpa", 0.0)) + pressure_profile_altitude = profile_altitude + pressure_profile_pressure = profile_pressure + pressure_profile_count = profile_count + + if parameter_names != _OPTIMAL_ESTIMATION_STATE_NAMES: + raise ValueError( + "state vector must contain aerosol_optical_depth and " + "aerosol_layer_mid_pressure_hpa in that order" + ) - state_spec_array = (COptimalEstimationStateSpec * len(state_specs))(*state_specs) request = COptimalEstimationRequest( sample_count=len(wavelength), wavelength_nm=wavelength, reflectance=reflectance, variance=measurement_covariance, - state_count=len(state_specs), - states=state_spec_array, + aerosol_optical_depth=scalar_specs[0], + aerosol_layer_pressure=COptimalEstimationPressureSpec( + scalar=scalar_specs[1], + interval_index_1based=pressure_interval_index_1based, + thickness_hpa=pressure_thickness_hpa, + pressure_profile_count=pressure_profile_count, + pressure_profile_altitude_km=pressure_profile_altitude, + pressure_profile_pressure_hpa=pressure_profile_pressure, + ), controls=COptimalEstimationControls( max_iterations=max_iterations, state_vector_convergence_threshold=float( @@ -603,7 +582,6 @@ def _optimal_estimation_request(self, *, measurement, state_vector, controls): wavelength, reflectance, measurement_covariance, - state_spec_array, *state_buffers, ) @@ -625,7 +603,7 @@ def _optimal_estimation_batch_request( state_vector=state_vector, controls=controls, ) - state_count = len(state_vector.parameters) + state_count = len(_OPTIMAL_ESTIMATION_STATE_NAMES) initial_values, run_count = flattened_state_rows( initial_states, state_count, @@ -655,8 +633,8 @@ def _optimal_estimation_batch_request( wavelength_nm=template_request.wavelength_nm, reflectance=template_request.reflectance, variance=template_request.variance, - state_count=template_request.state_count, - state_template=template_request.states, + aerosol_optical_depth=template_request.aerosol_optical_depth, + aerosol_layer_pressure=template_request.aerosol_layer_pressure, run_count=run_count, initial=initial, prior=prior, @@ -682,7 +660,6 @@ def _copied_spectrum( self, raw: CSpectrum, *, - jacobian_state_names: tuple[str, ...] | None = None, include_scene: bool = True, ) -> Spectrum: @@ -692,7 +669,7 @@ def _copied_spectrum( radiance = self._copied_double_array(raw.radiance, length) irradiance = self._copied_double_array(raw.irradiance, length) reflectance = self._copied_double_array(raw.reflectance, length) - state_names = self._jacobian_names(raw, jacobian_state_names) + state_names = self._jacobian_names(raw) radiance_jacobian = None if raw.jacobian and raw.jacobian_state_count != 0: @@ -731,21 +708,11 @@ def _copied_double_array(self, pointer, count: int) -> array: return array("d", (float(pointer[index]) for index in range(count))) - def _jacobian_names( - self, - raw: CSpectrum, - requested: tuple[str, ...] | None, - ) -> tuple[str, ...]: + def _jacobian_names(self, raw: CSpectrum) -> tuple[str, ...]: if raw.jacobian_state_count == 0: return () - if requested is not None: - if len(requested) != raw.jacobian_state_count: - raise RuntimeError("spectrum Jacobian state names do not match model output") - - return requested - return JACOBIAN_STATE_NAMES[0 : raw.jacobian_state_count] def _spectrum_report(self, raw: CSpectrum) -> DiagnosticReport: diff --git a/python/zdisamar/bindings/signatures.py b/python/zdisamar/bindings/signatures.py index 1a874d736..1cbf93c84 100644 --- a/python/zdisamar/bindings/signatures.py +++ b/python/zdisamar/bindings/signatures.py @@ -40,7 +40,7 @@ def configure(lib: ctypes.CDLL) -> ctypes.CDLL: bind( lib, "zds_warm_o2a_optimal_estimation", - [ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint8), ctypes.c_size_t], + [ctypes.c_void_p], ctypes.c_int, ) bind(lib, "zds_run_spectrum", [ctypes.c_void_p, ctypes.POINTER(CSpectrum)], ctypes.c_int) @@ -50,17 +50,6 @@ def configure(lib: ctypes.CDLL) -> ctypes.CDLL: [ctypes.c_void_p, ctypes.POINTER(CSpectrum)], ctypes.c_int, ) - bind( - lib, - "zds_run_spectrum_jacobian_for_states", - [ - ctypes.c_void_p, - ctypes.POINTER(CSpectrum), - ctypes.POINTER(ctypes.c_uint8), - ctypes.c_size_t, - ], - ctypes.c_int, - ) bind( lib, "zds_spectrum_report", diff --git a/python/zdisamar/bindings/structures.py b/python/zdisamar/bindings/structures.py index 9f01495d1..aec7463dc 100644 --- a/python/zdisamar/bindings/structures.py +++ b/python/zdisamar/bindings/structures.py @@ -160,17 +160,22 @@ class OxygenCollisionInducedAbsorptionDiagnosticsRaw(ctypes.Structure): ] -class COptimalEstimationStateSpec(ctypes.Structure): +class COptimalEstimationScalarSpec(ctypes.Structure): _fields_ = [ - ("state_id", ctypes.c_uint8), ("has_lower", ctypes.c_uint8), ("has_upper", ctypes.c_uint8), - ("interval_index_1based", ctypes.c_uint32), ("initial", ctypes.c_double), ("prior", ctypes.c_double), ("variance", ctypes.c_double), ("lower", ctypes.c_double), ("upper", ctypes.c_double), + ] + + +class COptimalEstimationPressureSpec(ctypes.Structure): + _fields_ = [ + ("scalar", COptimalEstimationScalarSpec), + ("interval_index_1based", ctypes.c_uint32), ("thickness_hpa", ctypes.c_double), ("pressure_profile_count", ctypes.c_size_t), ("pressure_profile_altitude_km", ctypes.POINTER(ctypes.c_double)), @@ -192,8 +197,8 @@ class COptimalEstimationRequest(ctypes.Structure): ("wavelength_nm", ctypes.POINTER(ctypes.c_double)), ("reflectance", ctypes.POINTER(ctypes.c_double)), ("variance", ctypes.POINTER(ctypes.c_double)), - ("state_count", ctypes.c_size_t), - ("states", ctypes.POINTER(COptimalEstimationStateSpec)), + ("aerosol_optical_depth", COptimalEstimationScalarSpec), + ("aerosol_layer_pressure", COptimalEstimationPressureSpec), ("controls", COptimalEstimationControls), ] @@ -224,8 +229,8 @@ class COptimalEstimationBatchRequest(ctypes.Structure): ("wavelength_nm", ctypes.POINTER(ctypes.c_double)), ("reflectance", ctypes.POINTER(ctypes.c_double)), ("variance", ctypes.POINTER(ctypes.c_double)), - ("state_count", ctypes.c_size_t), - ("state_template", ctypes.POINTER(COptimalEstimationStateSpec)), + ("aerosol_optical_depth", COptimalEstimationScalarSpec), + ("aerosol_layer_pressure", COptimalEstimationPressureSpec), ("run_count", ctypes.c_size_t), ("initial", ctypes.POINTER(ctypes.c_double)), ("prior", ctypes.POINTER(ctypes.c_double)), diff --git a/python/zdisamar/input/aerosol.py b/python/zdisamar/input/aerosol.py index 1b2701bf7..27ecf0dff 100644 --- a/python/zdisamar/input/aerosol.py +++ b/python/zdisamar/input/aerosol.py @@ -117,7 +117,7 @@ def validate(self) -> None: @dataclass class Aerosol: - """Aerosol optical properties for the O2 A scene.""" + """Aerosol optical properties for the scene.""" optical_depth_550_nm: float single_scatter_albedo: float diff --git a/python/zdisamar/input/assets.py b/python/zdisamar/input/assets.py index 7de57fea6..11dfc2f27 100644 --- a/python/zdisamar/input/assets.py +++ b/python/zdisamar/input/assets.py @@ -7,7 +7,7 @@ @dataclass class ReferenceAsset: - """Named reference-data file used by an O2 A scene.""" + """Named reference-data file used by a scene.""" id: str path: str @@ -39,7 +39,7 @@ def with_resolved_path(self, resolver) -> Self: @dataclass class ReferenceAssets: - """Reference-data files required to reproduce one O2 A calculation.""" + """Reference-data files required to reproduce one calculation.""" atmosphere_profile: ReferenceAsset vendor_reference_csv: ReferenceAsset diff --git a/python/zdisamar/input/atmosphere.py b/python/zdisamar/input/atmosphere.py index 2f97fd154..593a21ccd 100644 --- a/python/zdisamar/input/atmosphere.py +++ b/python/zdisamar/input/atmosphere.py @@ -58,7 +58,7 @@ def to_dict(self) -> dict[str, float | int]: @dataclass class Atmosphere: - """Layer grid and fit interval used to build the O2 A atmosphere.""" + """Layer grid and fit interval used to build the O2A atmosphere.""" layer_count: int sublayer_divisions: int diff --git a/python/zdisamar/input/geometry.py b/python/zdisamar/input/geometry.py index 4603eb582..a3a5eb550 100644 --- a/python/zdisamar/input/geometry.py +++ b/python/zdisamar/input/geometry.py @@ -8,7 +8,7 @@ @dataclass class Geometry: - """Solar and viewing geometry for the O2 A radiance calculation.""" + """Solar and viewing geometry for the radiance calculation.""" model: str solar_zenith_deg: float @@ -45,7 +45,7 @@ def to_dict(self) -> dict[str, float | str]: @dataclass class Surface: - """Lambertian surface parameters used by the current O2 A model.""" + """Lambertian surface parameters used by the current model.""" albedo: float pressure_hpa: float diff --git a/python/zdisamar/input/radiative_transfer.py b/python/zdisamar/input/radiative_transfer.py index 1fc84e829..aa0291f61 100644 --- a/python/zdisamar/input/radiative_transfer.py +++ b/python/zdisamar/input/radiative_transfer.py @@ -6,26 +6,26 @@ that another Fourier term, multiple-scattering order, layer-doubling step, or small matrix product is no longer worth evaluating. -The default O2 A thresholds are the reference-grade settings used by validation. -The `fast()` preset returns the O2 A defaults plus the current aggressive fast -override. For an existing case, prefer +The default thresholds are the reference-grade settings used by validation. +The `fast()` preset returns the defaults plus the current aggressive fast +override. For an existing scene, prefer `performance_thresholds.with_fast_mode()` so scene- or validation-family-specific thresholds are preserved while the speed/accuracy override is applied. The -higher-level O2 A fast mode also applies O2 A adaptive-reference-grid settings; +higher-level fast mode also applies adaptive-reference-grid settings; that combined preset is the user-facing speed mode. Threshold guide: - `fourier_tail_reflectance_epsilon`: stops the azimuthal Fourier expansion once a Fourier reflectance coefficient is smaller than this value after the - floor order. It must be finite and positive; the O2 A default is `3e-14`. - In the O2 A sweep over default, azimuth, aerosol-loading, and bright-surface + floor order. It must be finite and positive; the default is `3e-14`. + In the sweep over default, azimuth, aerosol-loading, and bright-surface scenes, raising it to `1e-11` produced a worst reflectance residual of about `1e-10`, but this was too conservative to provide the desired fast-mode speedup by itself. - `fourier_floor_scalar`: the earliest Fourier order where the Fourier-tail - stop may trigger. It must fit in the model's 16-bit field; the O2 A + stop may trigger. It must fit in the model's 16-bit field; the default is `2`. Keeping this above the low orders protects the dominant angular structure in the reflectance field. Lowering it would risk pruning physically important low-order azimuthal terms; we have not validated that as @@ -34,7 +34,7 @@ - `fourier_order_cap`: a hard maximum Fourier order, or `None` for the geometry-derived limit. When set, it must fit in the model's 16-bit field. It is useful for experiments because it skips whole Fourier-order - evaluations directly, but it is cruder than the tail threshold. The O2 A + evaluations directly, but it is cruder than the tail threshold. The fast preset uses `5`; in the retained four-scene spectra sweep this was the main contributor to the speedup while keeping the worst reflectance residual below `5e-4` when combined with the layer-doubling override. @@ -42,7 +42,7 @@ - `aerosol_tangent_order_cap`: a hard maximum Fourier order for aerosol AOD and aerosol-pressure tangent weighting functions, or `None` to evaluate every active order. The perturbation sweep found orders `>=12` to be near-neutral, - so the O2 A fast preset uses `11`. With the current generic Fourier cap of + so the fast preset uses `11`. With the current generic Fourier cap of `5`, this is mainly a centrally exposed custom-fast-mode knob; it becomes active when callers relax the full Fourier cap but still want to skip high-order tangent work. @@ -56,33 +56,33 @@ - `threshold_conv_first`: after the first scattering-order transport pass, LABOS returns early when the estimated contribution is below this threshold. - It must be finite and positive; the O2 A default is `1.5e-7`. In the sweep, + It must be finite and positive; the default is `1.5e-7`. In the sweep, raising it into the `5e-7` to `1.5e-5` range produced residuals from roughly `3e-9` to `4e-8` but did not produce a reliable speed win, so it is not part of the fast preset. - `threshold_conv_mult`: convergence threshold for later multiple-scattering - orders. It must be finite and positive; the O2 A default is `1.5e-9`. + orders. It must be finite and positive; the default is `1.5e-9`. Raising it stops the order loop earlier after the initial pass. In the tested range `5e-9` to `1.5e-7`, worst residuals were about `1e-9` to `8e-8`, again without a consistent timing win. - `threshold_doubl`: controls whether a layer is split to a thinner starting optical thickness and then rebuilt through doubling. It must be finite and - positive; the O2 A default is `1e-6`. Relaxing it can save layer-doubling - work, but this is a stronger approximation. The O2 A fast preset uses + positive; the default is `1e-6`. Relaxing it can save layer-doubling + work, but this is a stronger approximation. The fast preset uses `3e-5`; by itself this was not fast enough, but in combination with the Fourier-order cap it kept the retained spectra sweep below the `5e-4` reflectance-residual envelope. - `threshold_mul`: suppresses matrix products when the product of known matrix - traces is small. It must be finite and positive; the O2 A default is + traces is small. It must be finite and positive; the default is `1e-8`. This looked unsafe as a broad fast-mode control: even `3e-8` produced about `2.3e-5` worst reflectance residual in the sweep. - `phase_function_truncation_threshold`: controls how much aerosol phase-function structure is retained before the radiative-transfer solve. - It must be finite and positive; the O2 A default is `1e-8`. It affects the + It must be finite and positive; the default is `1e-8`. It affects the angular scattering physics before LABOS runs. Raising it was also too lossy for a generic fast preset: even `3e-8` produced about `1.3e-6` worst residual in the sweep. @@ -116,7 +116,7 @@ class RadiativeTransferPerformanceThresholds: @classmethod def o2a_default(cls) -> Self: - """Use the reference-grade O2 A thresholds unless fast mode is requested.""" + """Use the reference-grade thresholds unless fast mode is requested.""" return cls( num_orders_max=0, @@ -133,7 +133,7 @@ def o2a_default(cls) -> Self: @classmethod def fast(cls) -> Self: - """Return the validated O2 A speed/accuracy threshold bundle.""" + """Return the validated speed/accuracy threshold bundle.""" thresholds = cls.o2a_default() thresholds.fourier_order_cap = cls.FAST_FOURIER_ORDER_CAP @@ -146,7 +146,7 @@ def fast(cls) -> Self: def with_fast_mode(self) -> Self: """Return a copy with the validated fast-mode overrides applied. - This preserves case-specific threshold choices, such as a validation + This preserves scene-specific threshold choices, such as a validation family's phase-function truncation threshold, and changes only the broadly validated speed knobs. The current fast preset applies the Fourier-order cap, aerosol tangent cap, Fourier-tail reflectance @@ -220,7 +220,7 @@ def to_dict(self) -> dict[str, float | int | bool | None]: @dataclass class RadiativeTransferControls: - """Radiative-transfer settings for one O2 A scene.""" + """Radiative-transfer settings for one scene.""" scattering: str n_streams: int diff --git a/python/zdisamar/input/wavelength_band/o2a.py b/python/zdisamar/input/wavelength_band/o2a.py index 04dd6b634..157931276 100644 --- a/python/zdisamar/input/wavelength_band/o2a.py +++ b/python/zdisamar/input/wavelength_band/o2a.py @@ -1,4 +1,4 @@ -"""Typed O2 A wavelength-band input object and JSON conversion.""" +"""Typed wavelength-band input object and JSON conversion.""" import json import math @@ -25,7 +25,7 @@ def _object_dict(data: dict[str, object], key: str) -> dict[str, object]: @dataclass class Scene: - """Complete O2 A wavelength-band case passed to the zdisamar RTM.""" + """Complete wavelength-band scene passed to the zdisamar RTM.""" metadata: dict[str, object] plan: dict[str, object] @@ -72,7 +72,7 @@ def aerosol_layer(self) -> "AerosolLayer": # noqa: UP037 @property def measurement_wavelengths_nm(self) -> tuple[float, ...]: - """Return the nominal measurement axis for the case.""" + """Return the nominal measurement axis for the scene.""" measured_wavelengths = self.instrument_response.measured_wavelengths_nm @@ -140,7 +140,7 @@ def from_dict(cls, data: dict[str, object]) -> Self: removed_metadata = {"outputs", "validation"}.intersection(data) if removed_metadata: joined = ", ".join(sorted(removed_metadata)) - raise ValueError(f"unsupported O2 A input fields: {joined}") + raise ValueError(f"unsupported input fields: {joined}") return cls( metadata=_object_dict(data, "metadata"), @@ -168,12 +168,12 @@ def from_dict(cls, data: dict[str, object]) -> Self: @classmethod def from_json(cls, raw: str | bytes) -> Self: - """Read an O2 A scene emitted by the zdisamar model.""" + """Read a scene emitted by the zdisamar model.""" return cls.from_dict(json.loads(raw)) def to_dict(self) -> dict[str, object]: - """Return the Python O2 A case shape, including optimisation controls.""" + """Return the Python scene shape, including optimisation controls.""" payload = self.to_native_dict() payload["optimisation"] = self.optimisation.to_dict() @@ -181,7 +181,7 @@ def to_dict(self) -> dict[str, object]: return payload def to_native_dict(self) -> dict[str, object]: - """Return the O2 A scene shape expected by the native zdisamar model.""" + """Return the scene shape expected by the native zdisamar model.""" return { "metadata": self.metadata, @@ -204,7 +204,7 @@ def to_native_dict(self) -> dict[str, object]: } def to_json_bytes(self) -> bytes: - """Encode the Python case deterministically, including optimisation controls.""" + """Encode the Python scene deterministically, including optimisation controls.""" return json.dumps(json_value(self.to_dict()), sort_keys=True, separators=(",", ":")).encode( "utf-8" @@ -218,7 +218,7 @@ def to_native_json_bytes(self) -> bytes: ).encode("utf-8") def set_aerosol_profile(self, layers: object) -> None: - """Install a case-owned aerosol profile for forward simulations.""" + """Install a scene-owned aerosol profile for forward simulations.""" profile = coerce_profile_layers(layers) @@ -254,7 +254,7 @@ def set_single_aerosol_profile_layer(self, layer: AerosolProfileLayer) -> None: self.aerosol.set_profile_layers((layer,)) def with_rtm_optimisation_applied(self) -> Self: - """Return the native RTM case after applying enabled optimisation modes.""" + """Return the native RTM scene after applying enabled optimisation modes.""" resolved = deepcopy(self) diff --git a/python/zdisamar/input/wavelength_band/o2a_default.py b/python/zdisamar/input/wavelength_band/o2a_default.py index 60e2593e7..a8dd59447 100644 --- a/python/zdisamar/input/wavelength_band/o2a_default.py +++ b/python/zdisamar/input/wavelength_band/o2a_default.py @@ -1,4 +1,4 @@ -"""Packaged O2 A reference scene factory.""" +"""Packaged reference scene factory.""" from ..aerosol import Aerosol, AerosolPlacement from ..assets import ReferenceAsset, ReferenceAssets @@ -19,13 +19,13 @@ def reference_asset(id: str, path: str, format: str) -> ReferenceAsset: def default_o2a_scene() -> Scene: - """Return the packaged DISAMAR-family O2 A reference scene.""" + """Return the packaged DISAMAR-family reference scene.""" return Scene( metadata={ "id": "disamar_reference_o2a", "storage": "disamar-reference-o2a", - "description": "DISAMAR O2 A reference scene defined in Python.", + "description": "DISAMAR O2A reference scene defined in Python.", }, plan={ "derivative_mode": "none", diff --git a/python/zdisamar/input/wavelength_band/optimisation.py b/python/zdisamar/input/wavelength_band/optimisation.py index b8b35942c..8e0f12ebb 100644 --- a/python/zdisamar/input/wavelength_band/optimisation.py +++ b/python/zdisamar/input/wavelength_band/optimisation.py @@ -1,4 +1,4 @@ -"""Case-owned optimisation settings for wavelength-band inputs.""" +"""Scene-owned optimisation settings for wavelength-band inputs.""" import math from bisect import bisect_left @@ -163,7 +163,7 @@ def to_dict(self) -> dict[str, object]: def default_fast_stage_wavelength_windows() -> tuple[FastModeWavelengthWindow, ...]: - """Return the tuned sparse fast-stage O2 A row selectors used by fastmode.""" + """Return the tuned sparse fast-stage row selectors used by fastmode.""" return ( FastModeWavelengthWindow((758.0, 758.08), 2), @@ -177,9 +177,9 @@ def default_fast_stage_wavelength_windows() -> tuple[FastModeWavelengthWindow, . @dataclass class FastModeRadiativeTransfer: - """RTM work-reduction controls applied when O2 A fastmode is enabled. + """RTM work-reduction controls applied when fastmode is enabled. - These fields are a case-owned view onto the LABOS performance thresholds. + These fields are a scene-owned view onto the LABOS performance thresholds. They do not change the physical scene, measurement grid, aerosol state, or instrument model. They change how much radiative-transfer work is done before a term is treated as small enough to skip. @@ -227,7 +227,7 @@ def from_dict(cls, data: dict[str, object]) -> Self: ) def apply_to(self, thresholds: object) -> None: - """Apply the fastmode threshold overrides to a case copy.""" + """Apply the fastmode threshold overrides to a scene copy.""" for key, value in self.to_dict().items(): if not hasattr(thresholds, key): @@ -250,13 +250,13 @@ class FastModeAdaptiveReferenceGrid: """High-resolution O2 line-grid controls changed by fastmode. These knobs reduce work before convolution to the instrument grid. Lower - values produce fewer high-resolution samples around O2 A lines, so they can + values produce fewer high-resolution samples around O2A lines, so they can speed up the forward model but may under-resolve narrow line structure. `points_per_fwhm` controls the nominal high-resolution sampling density per instrument-line FWHM. `strong_line_min_divisions` and `strong_line_max_divisions` bound the extra subdivisions around strong line - cores, where most of the O2 A information lives. + cores, where most of the information lives. """ points_per_fwhm: int = 28 @@ -285,7 +285,7 @@ def from_dict(cls, data: dict[str, object]) -> Self: ) def apply_to(self, grid: dict[str, int]) -> None: - """Apply fastmode adaptive-grid overrides to a case copy.""" + """Apply fastmode adaptive-grid overrides to a scene copy.""" for key, value in self.to_dict().items(): if key not in grid: @@ -458,7 +458,7 @@ class FastModeFastStageSampling: Fastmode can trade retrieval speed against information content only through this explicit OE setting. The default keeps a blue continuum window plus - sparse samples across the O2 A P branch, then the final-correction settings + sparse samples across the O2A P branch, then the final-correction settings run one full-physics update on their own sparse wavelength set. `windows` selects evenly spaced samples from one or more measured wavelength @@ -635,12 +635,12 @@ def to_dict(self) -> dict[str, object]: @dataclass class FastModeOptimisation: - """Inspectable O2 A fastmode settings owned by the case. + """Inspectable fastmode settings owned by the scene. - Enabling this section does not mutate the physical case immediately. The - RTM and adaptive-grid overrides are applied to a copied native case at load + Enabling this section does not mutate the physical scene immediately. The + RTM and adaptive-grid overrides are applied to a copied native scene at load time, while OE controls and final-correction settings stay on the Python - case so `retrieve` can run the complete fastmode retrieval in one session. + scene so `retrieve` can run the complete fastmode retrieval in one session. """ enabled: bool = False @@ -673,7 +673,7 @@ def from_dict(cls, data: dict[str, object]) -> Self: ) def apply_to_scene(self, scene: FastModeApplicationScene) -> None: - """Apply fastmode RTM controls to a copied wavelength-band case.""" + """Apply fastmode RTM controls to a copied wavelength-band scene.""" self.radiative_transfer.apply_to(scene.radiative_transfer.performance_thresholds) self.adaptive_reference_grid.apply_to(scene.instrument_response.adaptive_reference_grid) @@ -700,9 +700,9 @@ def to_dict(self) -> dict[str, object]: @dataclass class Optimisation: - """Case-owned optimisation modes for O2 A workflows. + """Scene-owned optimisation modes for workflows. - The public flow remains one O2 A case and one OE entrypoint. Optimisation + The public flow remains one scene and one OE entrypoint. Optimisation settings live beside the physical inputs so they can be serialized and rejected if unknown fields are parsed. """ @@ -713,7 +713,7 @@ class Optimisation: def from_dict(cls, data: dict[str, object]) -> Self: allowed = {"fastmode"} - reject_unknown_fields(data, allowed, "O2 A optimisation") + reject_unknown_fields(data, allowed, "optimisation") return cls(fastmode=FastModeOptimisation.from_dict(object_dict(data.get("fastmode", {})))) diff --git a/python/zdisamar/inverse_method/__init__.py b/python/zdisamar/inverse_method/__init__.py deleted file mode 100644 index 9a05393ef..000000000 --- a/python/zdisamar/inverse_method/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Inverse-method APIs for zdisamar.""" - -from . import optimal_estimation - -__all__ = ["optimal_estimation"] diff --git a/python/zdisamar/inverse_method/optimal_estimation/__init__.py b/python/zdisamar/optimal_estimation/__init__.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/__init__.py rename to python/zdisamar/optimal_estimation/__init__.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/diagnosis.py b/python/zdisamar/optimal_estimation/diagnosis.py similarity index 98% rename from python/zdisamar/inverse_method/optimal_estimation/diagnosis.py rename to python/zdisamar/optimal_estimation/diagnosis.py index 17fd88980..f536a6e97 100644 --- a/python/zdisamar/inverse_method/optimal_estimation/diagnosis.py +++ b/python/zdisamar/optimal_estimation/diagnosis.py @@ -8,9 +8,9 @@ from dataclasses import dataclass from pathlib import Path -from ... import rtm -from ...display import PrettyMapping -from ...input.wavelength_band.o2a import Scene +from .. import rtm +from ..display import PrettyMapping +from ..input.wavelength_band.o2a import Scene from .retrieval import Measurement, RetrievalControls from .state_vector import ( AEROSOL_LAYER_MID_PRESSURE_HPA, @@ -168,6 +168,14 @@ class RetrievalDiagnosis: native_worker_limit: int | None = None elapsed_s: float = 0.0 + def __post_init__(self) -> None: + + if self.state_names != (AEROSOL_OPTICAL_DEPTH, AEROSOL_LAYER_MID_PRESSURE_HPA): + raise ValueError( + "retrieval diagnosis requires aerosol_optical_depth and " + "aerosol_layer_mid_pressure_hpa in that order" + ) + def value(self, name: StateName) -> tuple[float, ...]: """Return one retrieved state column by name.""" @@ -251,8 +259,8 @@ def plot( ): """Render the retrieval path plot as SVG.""" - from ...plot.properties import PLOT - from ...plot.retrieval_diagnosis import retrieval_diagnosis_figure + from ..plot.properties import PLOT + from ..plot.retrieval_diagnosis import retrieval_diagnosis_figure return PLOT.finish(retrieval_diagnosis_figure(self), save=save) diff --git a/python/zdisamar/inverse_method/optimal_estimation/measurement.py b/python/zdisamar/optimal_estimation/measurement.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/measurement.py rename to python/zdisamar/optimal_estimation/measurement.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/o2a.py b/python/zdisamar/optimal_estimation/o2a.py similarity index 94% rename from python/zdisamar/inverse_method/optimal_estimation/o2a.py rename to python/zdisamar/optimal_estimation/o2a.py index b462d3dc1..b8ec455e6 100644 --- a/python/zdisamar/inverse_method/optimal_estimation/o2a.py +++ b/python/zdisamar/optimal_estimation/o2a.py @@ -1,4 +1,4 @@ -"""O2 A wavelength-band helpers for optimal estimation.""" +"""Wavelength-band helpers for optimal estimation.""" import copy import math @@ -7,21 +7,22 @@ from contextlib import contextmanager from dataclasses import dataclass, replace -from ... import rtm -from ...input.instrument import SpectralGrid -from ...input.wavelength_band.o2a import Scene -from ...input.wavelength_band.optimisation import ( +from .. import rtm +from ..input.instrument import SpectralGrid +from ..input.wavelength_band.o2a import Scene +from ..input.wavelength_band.optimisation import ( FastModeFastStageSampling, measurement_indices_for_wavelengths, ) -from ...output.tables import AtmosphericBudget +from ..output.tables import AtmosphericBudget from .measurement import require_matching_wavelength_grid from .retrieval import FastCorrection, Iteration, Measurement, Result, RetrievalControls from .rtm_evaluation import RtmEvaluation -from .state_vector import StateVector +from .state_vector import AEROSOL_LAYER_MID_PRESSURE_HPA, AEROSOL_OPTICAL_DEPTH, StateVector from .state_vector.pressure_altitude_profile import PressureAltitudeProfile FULL_CORRECTION_WINDOW_NM = (762.0, 768.0) +OPTIMAL_ESTIMATION_STATE_NAMES = (AEROSOL_OPTICAL_DEPTH, AEROSOL_LAYER_MID_PRESSURE_HPA) @dataclass(frozen=True) @@ -53,7 +54,7 @@ def value(self, name: str) -> tuple[float, ...]: @dataclass(frozen=True) class RetrievalRoute: - """Resolved case/cache inputs for one O2 A retrieval handoff.""" + """Resolved scene/cache inputs for one retrieval handoff.""" scene: Scene measurement: Measurement @@ -71,7 +72,7 @@ def resolved_retrieval_route( state_vector: StateVector, cache: rtm.SessionCache | None, ) -> Iterator[RetrievalRoute]: - """Resolve the shared case/cache/template setup for single and batch OE.""" + """Resolve the shared scene/cache/template setup for single and batch OE.""" if scene.optimisation.fastmode.enabled: fast_scene, fast_measurement = fast_stage_retrieval_inputs(scene, measurement) @@ -151,7 +152,7 @@ def scene_for_state( state: Sequence[float], state_vector: StateVector, ) -> Scene: - """Create a wavelength-band case for one retrieval state.""" + """Create a wavelength-band scene for one retrieval state.""" scene = copy.copy(template) scene.aerosol = copy.copy(template.aerosol) @@ -192,7 +193,7 @@ def retrieve( controls: RetrievalControls | None = None, cache: rtm.SessionCache | None = None, ) -> Result: - """Retrieve O2 A state-vector parameters.""" + """Retrieve state-vector parameters.""" _require_aerosol_retrieval_compatible(scene) active_controls = retrieval_controls_for_scene(scene, controls) @@ -298,7 +299,7 @@ def retrieval_controls_for_scene( scene: Scene, controls: RetrievalControls | None, ) -> RetrievalControls: - """Return caller controls or the active case-owned OE defaults.""" + """Return caller controls or the active scene-owned OE defaults.""" if controls is not None: return controls @@ -327,7 +328,7 @@ def run_fastmode_oe( fast_scene_loaded: bool, preserve_fast_cache: bool = False, ) -> Result: - """Run an O2 A fastmode retrieval in one session cache.""" + """Run a fastmode retrieval in one session cache.""" fast_result = run_native_retrieval( scene=fast_scene, @@ -441,7 +442,7 @@ def fast_stage_retrieval_inputs( scene: Scene, measurement: Measurement, ) -> tuple[Scene, Measurement]: - """Return the case and measurement used by the fast OE stage.""" + """Return the scene and measurement used by the fast OE stage.""" sampling = scene.optimisation.fastmode.oe.fast_stage_sampling @@ -459,7 +460,7 @@ def fast_stage_measurement( *, sampling: FastModeFastStageSampling, ) -> Measurement: - """Apply the case-owned sparse wavelength selection to the fast OE vector.""" + """Apply the scene-owned sparse wavelength selection to the fast OE vector.""" wavelengths_nm = sampling.resolved_wavelengths(measurement.wavelength_nm) @@ -520,7 +521,7 @@ def run_full_physics_correction_in_temporary_cache( def full_physics_scene(scene: Scene) -> Scene: - """Return the same physical case with fastmode disabled.""" + """Return the same physical scene with fastmode disabled.""" full_scene = copy.deepcopy(scene) full_scene.optimisation.fastmode.enabled = False @@ -534,7 +535,7 @@ def full_correction_measurement( wavelengths_nm: Sequence[float] | None = None, uncertainty_scale: float | None = None, ) -> Measurement: - """Retain the O2 A wavelengths used by the final full-physics correction.""" + """Retain the O2A wavelengths used by the final full-physics correction.""" if wavelengths_nm is None: start_nm, end_nm = window_nm @@ -593,13 +594,13 @@ def measurement_on_wavelengths( def full_correction_scene(scene: Scene, measurement: Measurement) -> Scene: - """Use full physics on the correction window instead of the whole O2 A band.""" + """Use full physics on the correction window instead of the whole O2A band.""" return scene_on_measurement_grid(scene, measurement) def scene_on_measurement_grid(scene: Scene, measurement: Measurement) -> Scene: - """Return the same case sampled on a selected measured wavelength grid.""" + """Return the same scene sampled on a selected measured wavelength grid.""" correction_scene = copy.copy(scene) correction_scene.instrument_response = copy.copy(scene.instrument_response) @@ -689,7 +690,7 @@ def run_native_retrieval( cache: rtm.SessionCache, load_scene: bool = True, ) -> Result: - """Bind the O2 A RTM relation to the generic OE solver.""" + """Bind the RTM relation to the generic OE solver.""" _require_aerosol_retrieval_compatible(scene) resolved_state_vector = resolved_state_vector_for_scene(scene, state_vector) @@ -725,7 +726,7 @@ def run_native_retrieval_batch( load_scene: bool = True, batch_workers: int = 1, ) -> BatchResult: - """Bind one prepared O2 A relation to many native OE starts.""" + """Bind one prepared relation to many native OE starts.""" _require_aerosol_retrieval_compatible(scene) active_controls = controls or RetrievalControls.from_disamar_retrieval_specs() @@ -764,6 +765,11 @@ def batch_result_from_native( """Convert a copied native batch payload into Python diagnosis state.""" state_count = _native_int(raw, "state_count") + if state_names != OPTIMAL_ESTIMATION_STATE_NAMES: + raise RuntimeError("optimal-estimation batch state vector is not the two-state contract") + if state_count != len(OPTIMAL_ESTIMATION_STATE_NAMES): + raise RuntimeError("native optimal-estimation batch state count must be two") + state_values = _native_floats(raw, "state") states = tuple( tuple(state_values[offset : offset + state_count]) @@ -836,7 +842,7 @@ def batch_start_rows( ) -> tuple[tuple[tuple[float, ...], ...], tuple[tuple[float, ...], ...]]: """Return native initial/prior rows for operational diagnosis starts. - In the operational O2 A retrieval path, the apriori is also the solver + In the operational retrieval path, the apriori is also the solver initial state. Each diagnosis row therefore represents one complete prior/start scenario, rather than a fixed-prior optimizer basin probe. """ @@ -863,7 +869,7 @@ def resolved_state_vector_for_scene( scene: Scene, state_vector: StateVector, ) -> StateVector: - """Attach case-owned pressure metadata before native OE sees the state vector.""" + """Attach scene-owned pressure metadata before native OE sees the state vector.""" return resolved_state_vector_from_profile( scene, @@ -877,7 +883,7 @@ def resolved_state_vector_for_loaded_scene( state_vector: StateVector, cache: rtm.SessionCache, ) -> StateVector: - """Attach pressure metadata from a cache already matched to this case.""" + """Attach pressure metadata from a cache already matched to this scene.""" return resolved_state_vector_from_profile( scene, @@ -891,7 +897,7 @@ def resolved_state_vector_from_profile( state_vector: StateVector, pressure_profile: Callable[[], PressureAltitudeProfile], ) -> StateVector: - """Resolve state-vector parameters that need case-owned pressure metadata.""" + """Resolve state-vector parameters that need scene-owned pressure metadata.""" parameters = [] pressure_altitude_profile = None @@ -1020,7 +1026,6 @@ def evaluate_reflectance( scene, cache=cache, jacobian=True, - jacobian_state_names=state_names, include_scene=False, ) wavelength_nm = spectrum.wavelength_nm @@ -1036,7 +1041,7 @@ def evaluate_reflectance( ) if available_state_names != state_names: - raise ValueError("RTM Jacobian state selection did not preserve requested state order") + raise ValueError("RTM Jacobian state order does not match optimal-estimation state order") return RtmEvaluation( wavelength_nm=wavelength_nm, @@ -1077,7 +1082,7 @@ def simulate_measurement( *, signal_to_noise: float | Sequence[float], ) -> Measurement: - """Simulate a reflectance measurement from a truth case.""" + """Simulate a reflectance measurement from a truth scene.""" spectrum = rtm.spectrum(scene) @@ -1100,7 +1105,7 @@ def pressure_altitude_profile_from_loaded_cache( scene: Scene, cache: rtm.SessionCache, ) -> PressureAltitudeProfile: - """Read pressure-altitude metadata from a cache already matched to this case.""" + """Read pressure-altitude metadata from a cache already matched to this scene.""" return pressure_altitude_profile_from_budget( cache.atmospheric_budget([scene.spectral_grid.start_nm]) @@ -1210,6 +1215,12 @@ def _result_from_native( history_snr_normal = _native_ints(raw, "history_snr_normal") state_names = state_vector.names + if state_names != OPTIMAL_ESTIMATION_STATE_NAMES: + raise RuntimeError("optimal-estimation state vector is not the two-state contract") + + if state_count != len(OPTIMAL_ESTIMATION_STATE_NAMES): + raise RuntimeError("native optimal-estimation state count must be two") + if state_count != len(state_names): raise RuntimeError("native optimal-estimation state count does not match request") diff --git a/python/zdisamar/inverse_method/optimal_estimation/retrieval.py b/python/zdisamar/optimal_estimation/retrieval.py similarity index 98% rename from python/zdisamar/inverse_method/optimal_estimation/retrieval.py rename to python/zdisamar/optimal_estimation/retrieval.py index dfd83a864..27fb44c37 100644 --- a/python/zdisamar/inverse_method/optimal_estimation/retrieval.py +++ b/python/zdisamar/optimal_estimation/retrieval.py @@ -6,7 +6,7 @@ from numbers import Real from typing import Self -from ...display import PrettyMapping +from ..display import PrettyMapping from .rtm_evaluation import RtmEvaluation from .state_vector import StateName @@ -52,7 +52,7 @@ class Measurement: """Observed retrieval vector and per-sample reflectance signal-to-noise ratio. The first implementation uses reflectance as the retrieval quantity because - the current O2 A validation bundle is expressed as sun-normalized + the current validation bundle is expressed as sun-normalized radiance/reflectance. The native OE request converts signal-to-noise into reflectance sigma and then into the diagonal measurement covariance expected by the solver. @@ -335,6 +335,6 @@ def __repr__(self) -> str: def plot(self): """Import plotting only when a caller asks for retrieval figures.""" - from ...plot.optimal_estimation import OptimalEstimationPlot + from ..plot.optimal_estimation import OptimalEstimationPlot return OptimalEstimationPlot(self) diff --git a/python/zdisamar/inverse_method/optimal_estimation/rtm_evaluation.py b/python/zdisamar/optimal_estimation/rtm_evaluation.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/rtm_evaluation.py rename to python/zdisamar/optimal_estimation/rtm_evaluation.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/state_vector/__init__.py b/python/zdisamar/optimal_estimation/state_vector/__init__.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/state_vector/__init__.py rename to python/zdisamar/optimal_estimation/state_vector/__init__.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/state_vector/aerosol_layer_mid_pressure.py b/python/zdisamar/optimal_estimation/state_vector/aerosol_layer_mid_pressure.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/state_vector/aerosol_layer_mid_pressure.py rename to python/zdisamar/optimal_estimation/state_vector/aerosol_layer_mid_pressure.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/state_vector/aerosol_optical_depth.py b/python/zdisamar/optimal_estimation/state_vector/aerosol_optical_depth.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/state_vector/aerosol_optical_depth.py rename to python/zdisamar/optimal_estimation/state_vector/aerosol_optical_depth.py diff --git a/python/zdisamar/inverse_method/optimal_estimation/state_vector/parameter.py b/python/zdisamar/optimal_estimation/state_vector/parameter.py similarity index 87% rename from python/zdisamar/inverse_method/optimal_estimation/state_vector/parameter.py rename to python/zdisamar/optimal_estimation/state_vector/parameter.py index 86fd6a65b..546d14c52 100644 --- a/python/zdisamar/inverse_method/optimal_estimation/state_vector/parameter.py +++ b/python/zdisamar/optimal_estimation/state_vector/parameter.py @@ -1,11 +1,18 @@ -"""Generic state-vector composition.""" +"""Fixed two-state retrieval composition.""" import math from collections.abc import Sequence from dataclasses import dataclass from typing import Protocol +from .aerosol_layer_mid_pressure import AEROSOL_LAYER_MID_PRESSURE_HPA +from .aerosol_optical_depth import AEROSOL_OPTICAL_DEPTH + StateName = str +OPTIMAL_ESTIMATION_STATE_NAMES: tuple[StateName, ...] = ( + AEROSOL_OPTICAL_DEPTH, + AEROSOL_LAYER_MID_PRESSURE_HPA, +) class StateVectorParameter(Protocol): @@ -32,13 +39,13 @@ def __post_init__(self) -> None: parameters = tuple(self.parameters) - if not parameters: - raise ValueError("state vector must contain at least one parameter") - names = tuple(parameter.name for parameter in parameters) - if len(set(names)) != len(names): - raise ValueError("state vector parameter names must be unique") + if names != OPTIMAL_ESTIMATION_STATE_NAMES: + raise ValueError( + "state vector must contain aerosol_optical_depth and " + "aerosol_layer_mid_pressure_hpa in that order" + ) for parameter in parameters: uncertainty = float(parameter.prior_uncertainty) @@ -116,7 +123,7 @@ def clip_to_bounds(self, state: Sequence[float]) -> tuple[float, ...]: return tuple(bounded) def write_to(self, target: object, state: Sequence[float]) -> None: - """Write all retrieval variables into one O2 A scene.""" + """Write all retrieval variables into one scene.""" if len(state) != len(self.parameters): raise ValueError("state length does not match state vector") diff --git a/python/zdisamar/inverse_method/optimal_estimation/state_vector/pressure_altitude_profile.py b/python/zdisamar/optimal_estimation/state_vector/pressure_altitude_profile.py similarity index 100% rename from python/zdisamar/inverse_method/optimal_estimation/state_vector/pressure_altitude_profile.py rename to python/zdisamar/optimal_estimation/state_vector/pressure_altitude_profile.py diff --git a/python/zdisamar/output/spectrum.py b/python/zdisamar/output/spectrum.py index ff53f10d9..676b7de2b 100644 --- a/python/zdisamar/output/spectrum.py +++ b/python/zdisamar/output/spectrum.py @@ -71,7 +71,7 @@ def __repr__(self) -> str: @property def input(self) -> Scene | None: - """Return the wavelength-band case associated with this spectrum.""" + """Return the wavelength-band scene associated with this spectrum.""" return self.scene diff --git a/python/zdisamar/plot/axes.py b/python/zdisamar/plot/axes.py index 0f01f3f91..56ad0b051 100644 --- a/python/zdisamar/plot/axes.py +++ b/python/zdisamar/plot/axes.py @@ -1,4 +1,4 @@ -"""Shared axis and O2 A wavelength helpers.""" +"""Shared axis and wavelength helpers.""" import math @@ -12,7 +12,7 @@ def label(name: str) -> str: def marker_values(values) -> list[float]: - """Draw fixed O2 A reference wavelengths only inside the plotted band.""" + """Draw fixed reference wavelengths only inside the plotted band.""" finite = finite_values(values) diff --git a/python/zdisamar/plot/optimal_estimation.py b/python/zdisamar/plot/optimal_estimation.py index 1a55983b9..8034b7c8f 100644 --- a/python/zdisamar/plot/optimal_estimation.py +++ b/python/zdisamar/plot/optimal_estimation.py @@ -110,7 +110,7 @@ def state_history_rows(result) -> list[dict[str, object]]: def fit_rows(result) -> list[dict[str, float]]: - from ..inverse_method.optimal_estimation.measurement import require_matching_wavelength_grid + from ..optimal_estimation.measurement import require_matching_wavelength_grid measurement = require_measurement(result) evaluation = require_final_evaluation(result) @@ -141,7 +141,7 @@ def fit_rows(result) -> list[dict[str, float]]: def jacobian_frame(result) -> list[dict[str, object]]: """Put final reflectance Jacobians into one table for all states.""" - from ..inverse_method.optimal_estimation.measurement import require_matching_wavelength_grid + from ..optimal_estimation.measurement import require_matching_wavelength_grid measurement = require_measurement(result) evaluation = require_final_evaluation(result) diff --git a/python/zdisamar/plot/retrieval_diagnosis.py b/python/zdisamar/plot/retrieval_diagnosis.py index 1f8675d88..5aa491603 100644 --- a/python/zdisamar/plot/retrieval_diagnosis.py +++ b/python/zdisamar/plot/retrieval_diagnosis.py @@ -6,7 +6,7 @@ from html import escape from pathlib import Path -from ..inverse_method.optimal_estimation.diagnosis import ( +from ..optimal_estimation.diagnosis import ( RETRIEVAL_BASIN_CLUSTER_RADIUS, RetrievalBasin, RetrievalDiagnosis, diff --git a/python/zdisamar/plot/spectrum.py b/python/zdisamar/plot/spectrum.py index e8fc3683b..47a1185b3 100644 --- a/python/zdisamar/plot/spectrum.py +++ b/python/zdisamar/plot/spectrum.py @@ -11,7 +11,7 @@ class SpectrumPlot(PlotAccessor): - """Plots for the spectral quantities returned by one O2 A run.""" + """Plots for the spectral quantities returned by one run.""" def __init__(self, spectrum): diff --git a/python/zdisamar/rtm/run.py b/python/zdisamar/rtm/run.py index 478720244..18dfa0dd4 100644 --- a/python/zdisamar/rtm/run.py +++ b/python/zdisamar/rtm/run.py @@ -26,23 +26,20 @@ def spectrum( *, cache: SessionCache | None = None, jacobian: bool = False, - jacobian_state_names: tuple[str, ...] | None = None, include_scene: bool = False, ): - """Run radiative transfer for one wavelength-band case.""" + """Run radiative transfer for one wavelength-band scene.""" if cache is not None: return cache.spectrum( scene, jacobian=jacobian, - jacobian_state_names=jacobian_state_names, include_scene=include_scene, ) with _temporary_cache(scene, copy_scene=include_scene) as temporary: return temporary.spectrum( jacobian=jacobian, - jacobian_state_names=jacobian_state_names, include_scene=include_scene, ) @@ -108,7 +105,7 @@ def o2_line_contributions( max_rows: int = 50_000, cache: SessionCache | None = None, ): - """Return line-by-line O2 evidence rows.""" + """Return line-by-line evidence rows.""" if cache is not None: cache.load(scene) @@ -120,7 +117,7 @@ def o2_line_contributions( def nominal_wavelengths(scene: Scene): - """Recreate the nominal spectrum grid from a wavelength-band case.""" + """Recreate the nominal spectrum grid from a wavelength-band scene.""" count = int(scene.spectral_grid.sample_count) @@ -143,6 +140,6 @@ def nominal_wavelengths(scene: Scene): def reference_scene() -> Scene: - """Return the packaged O2 A reference case.""" + """Return the packaged reference scene.""" return default_o2a_scene() diff --git a/python/zdisamar/rtm/session_cache.py b/python/zdisamar/rtm/session_cache.py index 726487fd4..c6a7638d9 100644 --- a/python/zdisamar/rtm/session_cache.py +++ b/python/zdisamar/rtm/session_cache.py @@ -6,10 +6,15 @@ from ..bindings.handles import RtmHandle from ..input.wavelength_band.o2a import Scene +OPTIMAL_ESTIMATION_STATE_NAMES = ( + "aerosol_optical_depth", + "aerosol_layer_mid_pressure_hpa", +) + @dataclass(eq=False) class SessionCache: - """Cache reusable RTM storage across repeated wavelength-band cases.""" + """Cache reusable RTM storage across repeated wavelength-band scenes.""" scene: Scene | None = None _handle: RtmHandle = field(init=False, repr=False) @@ -27,17 +32,17 @@ def __post_init__(self) -> None: self.warm() def load(self, scene: Scene, *, copy_scene: bool = True) -> None: - """Load a wavelength-band case into the cached RTM storage.""" + """Load a wavelength-band scene into the cached RTM storage.""" self._handle.load_scene(scene, copy_scene=copy_scene) self._loaded = True self._oe_warm_state_names = None def warm(self) -> None: - """Prepare reusable native RTM work arrays for the loaded case.""" + """Prepare reusable native RTM work arrays for the loaded scene.""" if not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") self._handle.warm_cache() @@ -45,7 +50,13 @@ def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: """Prepare reusable native RTM work arrays for an OE Jacobian route.""" if not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") + + if tuple(state_names) != OPTIMAL_ESTIMATION_STATE_NAMES: + raise ValueError( + "optimal-estimation cache warming requires aerosol_optical_depth " + "and aerosol_layer_mid_pressure_hpa in that order" + ) if self._oe_warm_state_names == state_names: return @@ -54,7 +65,7 @@ def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: self._oe_warm_state_names = state_names def has_loaded_scene(self, scene: Scene) -> bool: - """Return whether this cache already owns the prepared native case.""" + """Return whether this cache already owns the prepared native scene.""" return self._loaded and self._handle.loaded_scene_matches(scene) @@ -63,7 +74,6 @@ def spectrum( scene: Scene | None = None, *, jacobian: bool = False, - jacobian_state_names: tuple[str, ...] | None = None, include_scene: bool = False, ): """Run the RTM using cached storage.""" @@ -71,19 +81,18 @@ def spectrum( if scene is not None and (include_scene or not self.has_loaded_scene(scene)): self.load(scene, copy_scene=include_scene) elif not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") return self._handle.spectrum( jacobian=jacobian, - jacobian_state_names=jacobian_state_names, include_scene=include_scene, ) def atmospheric_budget(self, wavelengths_nm): - """Return atmospheric optical-depth budget rows for the loaded case.""" + """Return atmospheric optical-depth budget rows for the loaded scene.""" if not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") return self._handle.atmospheric_budget(wavelengths_nm) @@ -92,18 +101,18 @@ def instrument_response( wavelengths_nm, channels: tuple[str, ...] = ("radiance", "irradiance"), ): - """Return instrument response rows for the loaded case.""" + """Return instrument response rows for the loaded scene.""" if not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") return self._handle.instrument_response_sampling(wavelengths_nm, channels=channels) def collision_induced_absorption(self, wavelengths_nm): - """Return O2-O2 collision-induced absorption rows for the loaded case.""" + """Return O2-O2 collision-induced absorption rows for the loaded scene.""" if not self._loaded: - raise RuntimeError("SessionCache has no loaded wavelength-band case") + raise RuntimeError("SessionCache has no loaded wavelength-band scene") return self._handle.collision_induced_absorption(wavelengths_nm) diff --git a/python/zdisamar/wavelength_bands/__init__.py b/python/zdisamar/wavelength_bands/__init__.py index 94e892bca..0a563fd51 100644 --- a/python/zdisamar/wavelength_bands/__init__.py +++ b/python/zdisamar/wavelength_bands/__init__.py @@ -1,4 +1,4 @@ -"""Wavelength-band case definitions.""" +"""Wavelength-band scene definitions.""" from . import o2a diff --git a/python/zdisamar/wavelength_bands/o2a.py b/python/zdisamar/wavelength_bands/o2a.py index 813fcdec6..1d29ea9ed 100644 --- a/python/zdisamar/wavelength_bands/o2a.py +++ b/python/zdisamar/wavelength_bands/o2a.py @@ -1,4 +1,4 @@ -"""O2 A wavelength-band case API.""" +"""Wavelength-band scene API.""" from ..input import ( Aerosol, @@ -31,7 +31,7 @@ def reference_scene() -> Scene: - """Return the packaged DISAMAR-family O2 A reference case.""" + """Return the packaged DISAMAR-family reference scene.""" return default_o2a_scene() diff --git a/scaffolding/README.md b/scaffolding/README.md index 087f4dbf4..b12344d4e 100644 --- a/scaffolding/README.md +++ b/scaffolding/README.md @@ -13,7 +13,7 @@ First-class gates stay in their own roots: ## Instrumentation -- `instrumentation/trace/zig/` builds trace-enabled O2 A harnesses. +- `instrumentation/trace/zig/` builds trace-enabled harnesses. - `instrumentation/trace/capture/` holds Tracy and Lauka capture scripts. - `instrumentation/trace/evidence/` holds intentionally retained compact trace summaries. - `instrumentation/telemetry/zig/` builds calculation telemetry sinks and CLIs. diff --git a/scaffolding/instrumentation/cost_timing/analysis/run_cost_timing_fast_analysis.py b/scaffolding/instrumentation/cost_timing/analysis/run_cost_timing_fast_analysis.py index d43557dc7..0e5c5b96a 100644 --- a/scaffolding/instrumentation/cost_timing/analysis/run_cost_timing_fast_analysis.py +++ b/scaffolding/instrumentation/cost_timing/analysis/run_cost_timing_fast_analysis.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Fast cost-timing analysis gate for one enabled O2 A forward run.""" +"""Fast cost-timing analysis gate for one enabled forward run.""" import re import subprocess diff --git a/scaffolding/instrumentation/telemetry/README.md b/scaffolding/instrumentation/telemetry/README.md index 30fbcc039..559807481 100644 --- a/scaffolding/instrumentation/telemetry/README.md +++ b/scaffolding/instrumentation/telemetry/README.md @@ -33,10 +33,10 @@ The script runs: zig build calculation-telemetry -Doptimize=ReleaseFast -- --output-dir out/scaffolding/telemetry/data/o2a-default ``` -The default capture is a compact O2 A window: 760-761 nm, 21 output samples, +The default capture is a compact window: 760-761 nm, 21 output samples, 0.1 nm high-resolution spacing, forward-only, and single-scattering LABOS. That keeps the starter dataset quick enough to regenerate while still exercising the -real O2 A preparation, wavelength plan, Fourier loop, order convergence, and +real preparation, wavelength plan, Fourier loop, order convergence, and reflectance assembly paths. The same script forwards larger-capture controls to the Zig harness: diff --git a/scaffolding/instrumentation/telemetry/capture/full-spectrum-sweep.md b/scaffolding/instrumentation/telemetry/capture/full-spectrum-sweep.md index e6fed2051..9ab799829 100644 --- a/scaffolding/instrumentation/telemetry/capture/full-spectrum-sweep.md +++ b/scaffolding/instrumentation/telemetry/capture/full-spectrum-sweep.md @@ -1,7 +1,7 @@ # Full-Spectrum Calculation Telemetry Sweep This sweep is the first retained data set for expression-level math analysis -across a retrieval-like O2 A window. +across a retrieval-like window. ## Capture Settings diff --git a/scaffolding/instrumentation/telemetry/schema/schema.md b/scaffolding/instrumentation/telemetry/schema/schema.md index 9bfbb1c3a..a6a4b8d4f 100644 --- a/scaffolding/instrumentation/telemetry/schema/schema.md +++ b/scaffolding/instrumentation/telemetry/schema/schema.md @@ -103,7 +103,7 @@ Primary uses: - find thresholds with very large positive or negative margins; - estimate work controlled by a branch; -- identify branches that are effectively constant for an O2 A workload. +- identify branches that are effectively constant for a workload. ## `reduction_expression_rows.parquet` diff --git a/scaffolding/instrumentation/telemetry/zig/calculation_telemetry_cli.zig b/scaffolding/instrumentation/telemetry/zig/calculation_telemetry_cli.zig index f1fb20cc0..5b6897211 100644 --- a/scaffolding/instrumentation/telemetry/zig/calculation_telemetry_cli.zig +++ b/scaffolding/instrumentation/telemetry/zig/calculation_telemetry_cli.zig @@ -17,10 +17,10 @@ const default_sample_count: u32 = 21; const default_high_resolution_step_nm: f64 = 0.1; // instrumentation: calculation telemetry harness -// captures: configured O2 A run and row counts +// captures: configured run and row counts // why: make expression data reproducible. // Configuration remains CLI-owned. The forward model only sees the resolved -// typed O2 A input and the compile-time telemetry facade. +// typed input and the compile-time telemetry facade. const Config = struct { output_dir: []const u8 = default_output_dir, scene_id: ?[]const u8 = null, diff --git a/scaffolding/instrumentation/trace/README.md b/scaffolding/instrumentation/trace/README.md index 89e81f5c7..e97046e90 100644 --- a/scaffolding/instrumentation/trace/README.md +++ b/scaffolding/instrumentation/trace/README.md @@ -1,6 +1,6 @@ # Tracy Forward-Model Tracing -This folder owns the Tracy/ztracy tracing workflow for the O2 A forward-model +This folder owns the Tracy/ztracy tracing workflow for the forward-model performance investigation. ## Capture @@ -82,7 +82,7 @@ The default counter set is restricted to counters reported as supported by fixed_cycles,fixed_instructions,arm_l1d_cache_refill,arm_l1d_cache,arm_br_mis_pred,arm_br_pred ``` -The script builds the same O2 A LABOS forward harness in `ReleaseFast`, but +The script builds the same LABOS forward harness in `ReleaseFast`, but without ztracy: ```sh diff --git a/scaffolding/instrumentation/trace/capture/record-lauka-forward-model.sh b/scaffolding/instrumentation/trace/capture/record-lauka-forward-model.sh index fdd3e665a..d4d5bb124 100755 --- a/scaffolding/instrumentation/trace/capture/record-lauka-forward-model.sh +++ b/scaffolding/instrumentation/trace/capture/record-lauka-forward-model.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -euo pipefail -# Record Apple Silicon PMU counters around the O2 A LABOS forward model. +# Record Apple Silicon PMU counters around the LABOS forward model. # The run shape is intentionally fixed: one serial pass for clean per-kernel # ratios and one threaded pass for the real forward workload. @@ -165,7 +165,7 @@ summary_path.parent.mkdir(parents=True, exist_ok=True) summary_path.write_text(json.dumps(summary, indent=2) + "\n") print() -print("================ O2 A forward PMU: serial vs threaded ================") +print("================ forward PMU: serial vs threaded ================") print(f"{'metric':24s} {'serial (1 worker)':>18s} {'threaded':>18s}") print(f"{'forward_wall_s':24s} {serial_forward['forward_wall_s']:18.3f} {threaded_forward['forward_wall_s']:18.3f}") print(f"{'fixed_cycles':24s} {serial['fixed_cycles'] / 1e9:17.2f}G {threaded['fixed_cycles'] / 1e9:17.2f}G") diff --git a/scaffolding/instrumentation/trace/evidence/o2a-jacobian-trace/zig-0.16-root-cause.md b/scaffolding/instrumentation/trace/evidence/o2a-jacobian-trace/zig-0.16-root-cause.md index f2b4b66e5..95e26f3c9 100644 --- a/scaffolding/instrumentation/trace/evidence/o2a-jacobian-trace/zig-0.16-root-cause.md +++ b/scaffolding/instrumentation/trace/evidence/o2a-jacobian-trace/zig-0.16-root-cause.md @@ -1,7 +1,7 @@ # Zig 0.16 Cached Jacobian Boundary This note captures the retained evidence for the Zig 0.16 migration draft. It -uses the benchmark-shaped O2 A Jacobian case, not the heavier default retained +uses the benchmark-shaped Jacobian case, not the heavier default retained trace case, because the fast canary regression is in the reused session/Jacobian boundary. diff --git a/scaffolding/instrumentation/trace/zig/labos_bottleneck_trace_cli.zig b/scaffolding/instrumentation/trace/zig/labos_bottleneck_trace_cli.zig index b809c6ec3..41ff2f064 100644 --- a/scaffolding/instrumentation/trace/zig/labos_bottleneck_trace_cli.zig +++ b/scaffolding/instrumentation/trace/zig/labos_bottleneck_trace_cli.zig @@ -321,7 +321,7 @@ fn runDerivativeSweep( try summary_writer.interface.flush(); std.debug.print( - "wrote O2 A Jacobian trace sweep summary to {s} (variants={})\n", + "wrote Jacobian trace sweep summary to {s} (variants={})\n", .{ output_dir, derivative_variants.len }, ); } diff --git a/scaffolding/reports/performance/README.md b/scaffolding/reports/performance/README.md index 8d253389d..f47536ee5 100644 --- a/scaffolding/reports/performance/README.md +++ b/scaffolding/reports/performance/README.md @@ -6,10 +6,10 @@ artifacts are not mixed with historical checkpoint numbers. ## Documents -- [O2 A forward performance](o2a-forward/): current forward elapsed time, historical optimization path, detailed optimisation notes, remaining LABOS bottlenecks, and rejected ideas. -- [O2 A retrieval performance](o2a-retrieval/): session reuse, state-vector Jacobians, paired DISAMAR/zdisamar validation, fastmode sampling and final correction, optimisation notes, and current retrieval elapsed time. -- [Performance cases](cases/): case provenance for original reference measurements, current baseline config, slow OE case, and paired sweep scenes. -- [O2 A calculation demo](o2a-calculation-demo.ipynb): Jupyter notebook that isolates the measured O2 A counts and the small LABOS matrix calculations behind the elapsed time. +- [Forward performance](o2a-forward/): current forward elapsed time, historical optimization path, detailed optimisation notes, remaining LABOS bottlenecks, and rejected ideas. +- [Retrieval performance](o2a-retrieval/): session reuse, state-vector Jacobians, paired DISAMAR/zdisamar validation, fastmode sampling and final correction, optimisation notes, and current retrieval elapsed time. +- [Performance scenes](cases/): scene provenance for original reference measurements, current baseline config, slow OE scene, and paired sweep scenes. +- [Calculation demo](o2a-calculation-demo.ipynb): Jupyter notebook that isolates the measured counts and the small LABOS matrix calculations behind the elapsed time. ## Run the notebook diff --git a/scaffolding/reports/performance/cases/README.md b/scaffolding/reports/performance/cases/README.md index d84e25f9c..dacfe6b48 100644 --- a/scaffolding/reports/performance/cases/README.md +++ b/scaffolding/reports/performance/cases/README.md @@ -1,11 +1,11 @@ -# Performance Cases +# Performance Scenes -This folder pins the cases behind the research measurements. Timing numbers are -only useful when the case and artifact are clear. +This folder pins the scenes behind the research measurements. Timing numbers are +only useful when the scene and artifact are clear. -Current case notes: +Current scene notes: -- [Original reference case](original-reference-case.md) +- [Original reference scene](original-reference-case.md) - [Current baseline config](current-baseline-config.md) -- [Slow OE case](slow-oe-case.md) +- [Slow OE scene](slow-oe-case.md) - [Paired sweep scenes](paired-sweep-scenes.md) diff --git a/scaffolding/reports/performance/cases/current-baseline-config.md b/scaffolding/reports/performance/cases/current-baseline-config.md index 07de09e7c..dafe28114 100644 --- a/scaffolding/reports/performance/cases/current-baseline-config.md +++ b/scaffolding/reports/performance/cases/current-baseline-config.md @@ -1,6 +1,6 @@ # Current Baseline Config -The validation baseline is the DISAMAR-style O2 A aerosol case used by the +The validation baseline is the DISAMAR-style aerosol scene used by the current scripts and tracked outputs. Rules: diff --git a/scaffolding/reports/performance/cases/original-reference-case.md b/scaffolding/reports/performance/cases/original-reference-case.md index 93c05f2c9..abb036534 100644 --- a/scaffolding/reports/performance/cases/original-reference-case.md +++ b/scaffolding/reports/performance/cases/original-reference-case.md @@ -1,11 +1,11 @@ -# Original Reference Case +# Original Reference Scene -Some older forward-performance numbers came from an earlier O2 A reference case +Some older forward-performance numbers came from an earlier reference scene or from historical commits in the checkpoint series. Those values are kept as historical optimization evidence, not as current benchmarks. This matters because a later retained trace can use a different -baseline case and still be correct. +baseline scene and still be correct. When referencing old numbers, use historical wording: diff --git a/scaffolding/reports/performance/cases/paired-sweep-scenes.md b/scaffolding/reports/performance/cases/paired-sweep-scenes.md index db73d1b89..28b0b70b7 100644 --- a/scaffolding/reports/performance/cases/paired-sweep-scenes.md +++ b/scaffolding/reports/performance/cases/paired-sweep-scenes.md @@ -9,9 +9,9 @@ RNG_SEED = 20260507 ``` The script runs the first `RUN_COUNT` scenes from the 500-scene pool. This keeps -the first 100 cases stable if the sweep is later expanded. +the first 100 scenes stable if the sweep is later expanded. -The varied scene parameters follow the ranges used in the AMT 2019 O2 A +The varied scene parameters follow the ranges used in the AMT 2019 aerosol-height retrieval study, `https://doi.org/10.5194/amt-12-6619-2019`, narrowed to sensible cloud-free aerosol-only validation ranges. diff --git a/scaffolding/reports/performance/cases/slow-oe-case.md b/scaffolding/reports/performance/cases/slow-oe-case.md index e02845233..e79426de8 100644 --- a/scaffolding/reports/performance/cases/slow-oe-case.md +++ b/scaffolding/reports/performance/cases/slow-oe-case.md @@ -1,6 +1,6 @@ -# Slow OE Case +# Slow OE Scene -The slow retained zdisamar OE case comes from paired sweep case 71. +The slow retained zdisamar OE scene comes from paired sweep case 71. Scene: @@ -38,4 +38,4 @@ iterations 3 lazy final evaluation cached true ``` -Use this case for focused latency work before scaling up broad paired sweeps. +Use this scene for focused latency work before scaling up broad paired sweeps. diff --git a/scaffolding/reports/performance/forward/README.md b/scaffolding/reports/performance/forward/README.md index f2f15d55b..c7ff5141a 100644 --- a/scaffolding/reports/performance/forward/README.md +++ b/scaffolding/reports/performance/forward/README.md @@ -1,6 +1,6 @@ -# O2 A Forward Performance +# Forward Performance -Scope: one O2 A forward spectrum on the 755-776 nm, 701-output-wavelength +Scope: one forward spectrum on the 755-776 nm, 701-output-wavelength validation route. The current retained trace is: diff --git a/scaffolding/reports/performance/forward/optimisation-notes/01-high-resolution-wavelengths-and-layer-geometry.md b/scaffolding/reports/performance/forward/optimisation-notes/01-high-resolution-wavelengths-and-layer-geometry.md index 825b66a62..fe2036bbe 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/01-high-resolution-wavelengths-and-layer-geometry.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/01-high-resolution-wavelengths-and-layer-geometry.md @@ -11,7 +11,7 @@ geometry across those samples. Source links: - DISAMAR - - [Wavelength setup](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/DISAMARModule.f90#L2716-L2732): prepares broad wavelength tables for the general executable, so the O2 A path does not start from the smallest unique radiance list. + - [Wavelength setup](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/DISAMARModule.f90#L2716-L2732): prepares broad wavelength tables for the general executable, so the path does not start from the smallest unique radiance list. - [Layer setup](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/DISAMARModule.f90#L2734-L2744): builds vertical layer state in the broad setup flow rather than exposing a narrow scene-stable geometry cache. - [LABOS layer use](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L472-L483): consumes the general layer structures inside repeated transport work. - zdisamar @@ -20,7 +20,7 @@ Source links: - [Shared geometry](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/optical_properties/state_build/shared_geometry.zig#L127-L150): prepares scene-stable vertical geometry once. - [Forward input](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/instrument_grid/grid_calculation/forward_input.zig#L21-L53): combines cached geometry with wavelength-dependent optical values. -The useful O2 A boundary is the full set of work that is stable for the scene. +The useful boundary is the full set of work that is stable for the scene. The output grid needs repeated high-resolution radiance samples, and each sample uses the same vertical geometry with wavelength-dependent optical values. diff --git a/scaffolding/reports/performance/forward/optimisation-notes/03-labos-storage.md b/scaffolding/reports/performance/forward/optimisation-notes/03-labos-storage.md index 55b2435d9..3281245eb 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/03-labos-storage.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/03-labos-storage.md @@ -10,9 +10,9 @@ Source links: - DISAMAR - [LABOS allocation region](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L268-L304): allocates general LABOS storage around repeated Fourier work. - - [LABOS cleanup region](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L421-L427): frees that storage after the loop, which is flexible but costly in repeated O2 A calls. + - [LABOS cleanup region](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L421-L427): frees that storage after the loop, which is flexible but costly in repeated calls. - zdisamar - - [LABOS workspace](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/workspace.zig#L46-L140): owns reusable arrays sized for the O2 A route and resets them across repeated work. + - [LABOS workspace](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/workspace.zig#L46-L140): owns reusable arrays sized for the route and resets them across repeated work. LABOS repeats the same storage shapes across wavelengths and Fourier terms. The arrays should live in a reusable workspace rather than being allocated and freed diff --git a/scaffolding/reports/performance/forward/optimisation-notes/04-fused-layer-doubling-updates.md b/scaffolding/reports/performance/forward/optimisation-notes/04-fused-layer-doubling-updates.md index 17d67e4b5..42e0d6ff6 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/04-fused-layer-doubling-updates.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/04-fused-layer-doubling-updates.md @@ -11,7 +11,7 @@ Source links: - DISAMAR - [Doubling recurrence](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1857-L1895): expresses the exact layer-doubling math in a general matrix-update style with repeated intermediate traffic. - zdisamar - - [Doubling loop](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/layers.zig#L228-L284): keeps the same recurrence but fuses common O2 A update shapes inside the repeated loop. + - [Doubling loop](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/layers.zig#L228-L284): keeps the same recurrence but fuses common update shapes inside the repeated loop. Layer doubling updates reflection and transmission matrices many times. The naive shape materializes intermediate arrays and then combines them in later diff --git a/scaffolding/reports/performance/forward/optimisation-notes/05-direct-matrix-calculations.md b/scaffolding/reports/performance/forward/optimisation-notes/05-direct-matrix-calculations.md index fd060187f..e4b389629 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/05-direct-matrix-calculations.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/05-direct-matrix-calculations.md @@ -3,7 +3,7 @@ Historical checkpoint: `97088cf -> 0ae1cad`, where forward elapsed time moved from `5.911137 s` to `2.460360 s`. This was the largest measured checkpoint win. -In short: replace general matrix loops with direct fixed-shape O2 A matrix +In short: replace general matrix loops with direct fixed-shape matrix operations. Source links: @@ -11,9 +11,9 @@ Source links: - DISAMAR - [General matrix shape](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1799-L1803): keeps dimensions general for many model configurations, leaving dynamic shape overhead in the hot path. - zdisamar - - [Fixed matrix helper](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/matrix.zig#L106-L144): specializes the common O2 A 12x10 and 12x12 matrix shapes so the compiler can produce direct arithmetic. + - [Fixed matrix helper](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/matrix.zig#L106-L144): specializes the common 12x10 and 12x12 matrix shapes so the compiler can produce direct arithmetic. -The O2 A route uses 20 streams, represented in LABOS as 10 Gauss directions plus +The route uses 20 streams, represented in LABOS as 10 Gauss directions plus direct solar and viewing directions. The common shapes are therefore 12x10 and 12x12. A general dynamic matrix route keeps loop bounds and shape checks in the hot path. @@ -28,7 +28,7 @@ def multiply(a, b, rows, inner, cols): out[i, j] += a[i, k] * b[k, j] return out -# Narrow route: O2 A common shape is known. +# Narrow route: common shape is known. def multiply_12x10(a, b): out = zeros(12, 12) for i in range(12): diff --git a/scaffolding/reports/performance/forward/optimisation-notes/06-skip-empty-layer-work.md b/scaffolding/reports/performance/forward/optimisation-notes/06-skip-empty-layer-work.md index d0b6de84b..7be44156d 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/06-skip-empty-layer-work.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/06-skip-empty-layer-work.md @@ -9,7 +9,7 @@ work. Source links: - DISAMAR - - [Layer loop shape](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1832-L1870): enters the general layer-building loop before the O2 A path can avoid all no-contribution work. + - [Layer loop shape](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1832-L1870): enters the general layer-building loop before the path can avoid all no-contribution work. - zdisamar - [Layer checks](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/layers.zig#L321-L344): proves cheap no-contribution cases before building phase matrices or doubled layers. - [Orders activity use](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/orders.zig#L432-L441): carries the same inactive-layer decision into later scattering-order work. diff --git a/scaffolding/reports/performance/forward/optimisation-notes/07-fourier-tail-and-basis-reuse.md b/scaffolding/reports/performance/forward/optimisation-notes/07-fourier-tail-and-basis-reuse.md index 1ff026218..be95d176f 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/07-fourier-tail-and-basis-reuse.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/07-fourier-tail-and-basis-reuse.md @@ -9,14 +9,14 @@ negligible. Source links: - DISAMAR - - [Fourier storage shape](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L268-L304): supports general Fourier storage, but the narrow O2 A path can reuse basis work more directly. + - [Fourier storage shape](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L268-L304): supports general Fourier storage, but the narrow path can reuse basis work more directly. - zdisamar - [Fourier basis workspace](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/workspace.zig#L155-L180): caches basis values across repeated terms. - [Fourier tail stop](https://github.com/bout3fiddy/zdisamar/blob/36598b67287c918b410ae25ca54319cbe63ade4b/src/forward_model/radiative_transfer/labos/execute.zig#L406-L414): exits once later Fourier terms are below the retained convergence scale. The Fourier basis depends on geometry and Fourier order. It can be reused. The series also has a numerical tail: once later terms are below the configured -scale, continuing the sum does not change the retained O2 A result. +scale, continuing the sum does not change the retained result. ```python # Broad route: rebuild basis and sum every configured term. diff --git a/scaffolding/reports/performance/forward/optimisation-notes/08-program-setup-boundary.md b/scaffolding/reports/performance/forward/optimisation-notes/08-program-setup-boundary.md index b158c7be1..974a55b17 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/08-program-setup-boundary.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/08-program-setup-boundary.md @@ -1,7 +1,7 @@ # 08. Keep Program Setup Out Of The Forward Timer This is a measurement and architecture boundary rather than one checkpoint win. -The forward timer should measure a prepared O2 A forward call, not full program +The forward timer should measure a prepared forward call, not full program startup, config parsing, or asset loading. In short: measure prepared forward calls separately from full executable setup. diff --git a/scaffolding/reports/performance/forward/optimisation-notes/13-parallel-setup-phases.md b/scaffolding/reports/performance/forward/optimisation-notes/13-parallel-setup-phases.md index d1728100b..96a9874ad 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/13-parallel-setup-phases.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/13-parallel-setup-phases.md @@ -3,17 +3,17 @@ Historical checkpoint: `cd03a91^ -> cd03a91`, where the trace harness moved from `prepare_o2a=0.177154 s` to `prepare_o2a=0.044454 s`. -In short: build independent O2 A setup products in parallel before the repeated +In short: build independent setup products in parallel before the repeated LABOS transport work begins. Source links: - DISAMAR - No direct Fortran analogue is used here. This is a zdisamar setup-boundary - optimization around the typed O2 A baseline. + optimization around the typed baseline. - zdisamar - [Wavelength sampling](https://github.com/bout3fiddy/zdisamar/blob/cd03a913b49cf1d65b8f6d6c3fed5074843122b6/src/forward_model/instrument_grid/grid_calculation/wavelength_sampling.zig): parallelizes instrument support and forward-miss preparation. - - [Absorber setup](https://github.com/bout3fiddy/zdisamar/blob/cd03a913b49cf1d65b8f6d6c3fed5074843122b6/src/forward_model/optical_properties/state_build/absorbers.zig): prepares fixed absorber state for the O2 A case. + - [Absorber setup](https://github.com/bout3fiddy/zdisamar/blob/cd03a913b49cf1d65b8f6d6c3fed5074843122b6/src/forward_model/optical_properties/state_build/absorbers.zig): prepares fixed absorber state for the scene. - [Layer accumulation](https://github.com/bout3fiddy/zdisamar/blob/cd03a913b49cf1d65b8f6d6c3fed5074843122b6/src/forward_model/optical_properties/state_build/layer_accumulation.zig): splits independent layer accumulation work across workers. The forward calculation needs several setup products before it can enter the diff --git a/scaffolding/reports/performance/forward/optimisation-notes/14-tracy-layer-reuse-and-inactive-rt.md b/scaffolding/reports/performance/forward/optimisation-notes/14-tracy-layer-reuse-and-inactive-rt.md index c9a40777c..8a07b77fa 100644 --- a/scaffolding/reports/performance/forward/optimisation-notes/14-tracy-layer-reuse-and-inactive-rt.md +++ b/scaffolding/reports/performance/forward/optimisation-notes/14-tracy-layer-reuse-and-inactive-rt.md @@ -58,5 +58,5 @@ if layer_cannot_contribute: ``` The generic helper path still writes zero matrices when no active mask is -provided. The retained workspace route is residual-clean: the O2 A validation +provided. The retained workspace route is residual-clean: the validation lanes passed, and the vendor-shaped residual stayed at numerical zero. diff --git a/scaffolding/reports/performance/forward/optimization-history.md b/scaffolding/reports/performance/forward/optimization-history.md index 54a791faf..917c5616b 100644 --- a/scaffolding/reports/performance/forward/optimization-history.md +++ b/scaffolding/reports/performance/forward/optimization-history.md @@ -1,7 +1,7 @@ # Optimization History This is historical evidence. It explains the speedup path across commits and -older cases; it is not the current forward elapsed time. +older scenes; it is not the current forward elapsed time. Retained checkpoint source: @@ -17,7 +17,7 @@ The main historical moves were: | Spectroscopy preparation | Move pressure/temperature line state work into preparation. | `35.364130 s -> 8.432518 s` forward elapsed time in the checkpoint table | | LABOS storage | Reuse LABOS arrays across repeated Fourier and wavelength work. | `8.432518 s -> 7.020602 s` | | Fused doubling math | Reduce repeated matrix traffic inside layer doubling. | `7.020602 s -> 5.911137 s` | -| Fixed-shape matrix math | Use the O2 A 20-stream shapes directly: common matrices are 12x10 and 12x12. | `5.911137 s -> 2.460360 s` | +| Fixed-shape matrix math | Use the 20-stream shapes directly: common matrices are 12x10 and 12x12. | `5.911137 s -> 2.460360 s` | | Layer and Fourier skips | Skip layers and terms that cannot contribute. | `2.460360 s -> 2.266849 s` | | Fourier tail reuse | Reuse Fourier basis work and stop tiny tails. | `2.266849 s -> 2.025331 s` | | Orders activity | Carry inactive-layer knowledge into scattering orders. | `2.025331 s -> 1.980342 s` | @@ -25,7 +25,7 @@ The main historical moves were: | Traced setup parallelism | Parallelize wavelength sampling, optical absorber setup, and layer accumulation in the trace harness. | `prepare_o2a 0.177154 s -> 0.044454 s`; trace forward `1.799918 s -> 1.538076 s` | These changes are mostly data-handling and exact arithmetic-shape changes. They -do not change the retrieval math or the O2 A physical model. +do not change the retrieval math or the O2A physical model. Detailed mechanism notes are in [optimisation-notes](optimisation-notes/). Those notes keep source links and Python-shaped explanations without copying long diff --git a/scaffolding/reports/performance/forward/remaining-bottlenecks.md b/scaffolding/reports/performance/forward/remaining-bottlenecks.md index 50b8c19f2..e6dbefe76 100644 --- a/scaffolding/reports/performance/forward/remaining-bottlenecks.md +++ b/scaffolding/reports/performance/forward/remaining-bottlenecks.md @@ -15,7 +15,7 @@ work. ## LABOS Fourier Transport The forward route still expands one spectrum into `120,390` LABOS Fourier terms. -This is expected: O2 A reflectance is azimuth dependent, so the model evaluates +This is expected: O2A reflectance is azimuth dependent, so the model evaluates many Fourier terms before the tail is small enough. ## RT-Layer Construction @@ -59,4 +59,4 @@ Phase matrix construction costs `1.040362 s`. The PLM basis cost is only and `Zmin`. The next large gain probably needs to reduce one of these counts, or introduce a -new reuse boundary that preserves the same O2 A result. +new reuse boundary that preserves the same result. diff --git a/scaffolding/reports/performance/o2a-calculation-demo.ipynb b/scaffolding/reports/performance/o2a-calculation-demo.ipynb index 2ded4a2cf..78905f5a1 100644 --- a/scaffolding/reports/performance/o2a-calculation-demo.ipynb +++ b/scaffolding/reports/performance/o2a-calculation-demo.ipynb @@ -4,21 +4,7 @@ "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": {}, - "source": [ - "# O2 A Forward-Model Bottleneck Demo\n", - "\n", - "This notebook isolates the per-spectrum bottleneck by following the actual forward-model work from output O2 A samples to the LABOS matrix calculations. The goal is not only to show that a small calculation is repeated many times. The goal is to explain why the code takes those steps, why the counts have their current values, and why individually cheap operations become expensive in aggregate.\n", - "\n", - "Code sites:\n", - "\n", - "- `src/spectrum/spectrum_run.zig`\n", - "- `src/spectrum/sampling_table.zig`\n", - "- `src/spectrum/sampling_table.zig`\n", - "- `src/rtm/solve.zig`\n", - "- `src/rtm/layer_reflect_transmit.zig`\n", - "- `src/rtm/matrix_12x10.zig`\n", - "- `scaffolding/experiments/kernels/labos_kernel_bench.zig`" - ] + "source": "# Forward-Model Bottleneck Demo\n\nThis notebook isolates the per-spectrum bottleneck by following the actual forward-model work from output samples to the LABOS matrix calculations. The goal is not only to show that a small calculation is repeated many times. The goal is to explain why the code takes those steps, why the counts have their current values, and why individually cheap operations become expensive in aggregate.\n\nCode sites:\n\n- `src/spectrum/spectrum_run.zig`\n- `src/spectrum/sampling_table.zig`\n- `src/spectrum/sampling_table.zig`\n- `src/rtm/solve.zig`\n- `src/rtm/layer_reflect_transmit.zig`\n- `src/rtm/matrix_12x10.zig`\n- `scaffolding/experiments/kernels/labos_kernel_bench.zig`" }, { "cell_type": "code", @@ -63,47 +49,13 @@ "cell_type": "markdown", "id": "9a63283cbaf04dbcab1f6479b197f3a8", "metadata": {}, - "source": [ - "## What the Per-Spectrum Path Does\n", - "\n", - "One O2 A spectrum is requested on a output grid, but the radiance is not evaluated only at those output wavelengths. The measurement is an instrument-weighted quantity. For the O2 A reference configuration, zdisamar first calculates radiance at high-resolution wavelengths, then averages those values back to the output grid.\n", - "\n", - "The core path is:\n", - "\n", - "```text\n", - "701 output wavelengths\n", - " -> build instrument-response offsets and weights for each output wavelength\n", - " -> make the high-resolution radiance wavelength list\n", - " -> calculate radiance once for each high-resolution wavelength\n", - " -> average those radiances back to the 701 output wavelengths\n", - "```\n", - "\n", - "That high-resolution wavelength step exists because O2 A absorption and solar structure vary sharply inside a single instrument channel. Evaluating only the channel center would erase the sub-channel structure that the instrument response is supposed to average." - ] + "source": "## What the Per-Spectrum Path Does\n\nOne O2A spectrum is requested on a output grid, but the radiance is not evaluated only at those output wavelengths. The measurement is an instrument-weighted quantity. For the reference configuration, zdisamar first calculates radiance at high-resolution wavelengths, then averages those values back to the output grid.\n\nThe core path is:\n\n```text\n701 output wavelengths\n -> build instrument-response offsets and weights for each output wavelength\n -> make the high-resolution radiance wavelength list\n -> calculate radiance once for each high-resolution wavelength\n -> average those radiances back to the 701 output wavelengths\n```\n\nThat high-resolution wavelength step exists because O2A absorption and solar structure vary sharply inside a single instrument channel. Evaluating only the channel center would erase the sub-channel structure that the instrument response is supposed to average." }, { "cell_type": "markdown", "id": "8dd0d8092fe74a7c96281538738b07e2", "metadata": {}, - "source": [ - "## Why 701 Output Wavelengths Becomes 3,874 High-Resolution Wavelengths\n", - "\n", - "The output grid count is direct input configuration: `start_nm=755.0`, `end_nm=776.0`, `sample_count=701`. That gives a 0.03 nm output spacing.\n", - "\n", - "The high-resolution count is not `701 * some fixed local sample count`. The DISAMAR high-resolution response builds one wavelength list over the band plus a response margin. O2 line centers split that list into intervals, and each interval receives a Gauss quadrature order. The high-resolution count is the sum of those interval orders.\n", - "\n", - "The important rules are:\n", - "\n", - "```text\n", - "global_start = start_nm - 2 * fwhm_nm\n", - "global_end = end_nm + 2 * fwhm_nm\n", - "line threshold = max(line_strength) * threshold_line_sim\n", - "intervals are split at strong O2 line centers\n", - "interval divisions are clamped between strong_line_min_divisions and strong_line_max_divisions\n", - "```\n", - "\n", - "For the O2 A reference case, this becomes:" - ] + "source": "## Why 701 Output Wavelengths Becomes 3,874 High-Resolution Wavelengths\n\nThe output grid count is direct input configuration: `start_nm=755.0`, `end_nm=776.0`, `sample_count=701`. That gives a 0.03 nm output spacing.\n\nThe high-resolution count is not `701 * some fixed local sample count`. The DISAMAR high-resolution response builds one wavelength list over the band plus a response margin. O2 line centers split that list into intervals, and each interval receives a Gauss quadrature order. The high-resolution count is the sum of those interval orders.\n\nThe important rules are:\n\n```text\nglobal_start = start_nm - 2 * fwhm_nm\nglobal_end = end_nm + 2 * fwhm_nm\nline threshold = max(line_strength) * threshold_line_sim\nintervals are split at strong O2 line centers\ninterval divisions are clamped between strong_line_min_divisions and strong_line_max_divisions\n```\n\nFor the reference scene, this becomes:" }, { "cell_type": "code", @@ -203,20 +155,7 @@ "cell_type": "markdown", "id": "10185d26023b46108eb7d9f57d49d2b3", "metadata": {}, - "source": [ - "## Measured O2 A Work Counts\n", - "\n", - "These counts are the per-spectrum workload after keeping exact repeated wavelengths once:\n", - "\n", - "```text\n", - "701 output wavelengths -> 3874 high-resolution radiance calculations\n", - "3874 calculations -> 120390 Fourier terms\n", - "RT-layer construction -> 5417550 layer visits\n", - "RT-layer doubling -> 8389666 doubling steps\n", - "```\n", - "\n", - "The forward model spends little time averaging back to 701 output values. The expensive section is the high-resolution radiance calculation stage: each high-resolution wavelength runs configured input construction and LABOS transport." - ] + "source": "## Measured Work Counts\n\nThese counts are the per-spectrum workload after keeping exact repeated wavelengths once:\n\n```text\n701 output wavelengths -> 3874 high-resolution radiance calculations\n3874 calculations -> 120390 Fourier terms\nRT-layer construction -> 5417550 layer visits\nRT-layer doubling -> 8389666 doubling steps\n```\n\nThe forward model spends little time averaging back to 701 output values. The expensive section is the high-resolution radiance calculation stage: each high-resolution wavelength runs configured input construction and LABOS transport." }, { "cell_type": "code", @@ -224,62 +163,13 @@ "id": "8763a12b2bbd4a93a75aff182afb95dc", "metadata": {}, "outputs": [], - "source": [ - "SPECTRUM_COUNTS = {\n", - " \"output_samples\": 701,\n", - " \"high_resolution_radiance_calculations\": 3874,\n", - " \"fourier_terms\": 120390,\n", - " \"labos_layers\": 5417550,\n", - " \"doubled_layers\": 1075939,\n", - " \"double_steps\": 8389666,\n", - "}\n", - "\n", - "MATRIX_CALLS = [\n", - " MatrixCall(\"Q = qseries(R * R)\", \"qseries_12x10\", 3_408_299),\n", - " MatrixCall(\"D = T + Q * diag(E) + Q * T\", \"smulAddSemul3_12\", 3_408_299),\n", - " MatrixCall(\"rd = R * D\", \"smul_12x10\", 8_389_666),\n", - " MatrixCall(\"U = R * diag(E) + rd\", \"semulAdd_12\", 8_389_666),\n", - " MatrixCall(\"tu = T * U\", \"smul_12x10\", 8_389_666),\n", - " MatrixCall(\"R_next = R + diag(E) * U + tu\", \"matAddEsmul3_12\", 8_389_666),\n", - " MatrixCall(\"td = T * D\", \"smul_12x10\", 8_389_666),\n", - " MatrixCall(\"T_next = diag(E) * D + T * diag(E) + td\", \"esmulSemulAdd_12\", 8_389_666),\n", - "]\n", - "\n", - "print(\"O2 A wavelength expansion\")\n", - "for key, value in SPECTRUM_COUNTS.items():\n", - " print(f\" {key:24s} {value:>10,d}\")\n", - "\n", - "fourier_terms_per_wavelength = (\n", - " SPECTRUM_COUNTS[\"fourier_terms\"] / SPECTRUM_COUNTS[\"high_resolution_radiance_calculations\"]\n", - ")\n", - "labos_layers_per_fourier = SPECTRUM_COUNTS[\"labos_layers\"] / SPECTRUM_COUNTS[\"fourier_terms\"]\n", - "double_steps_per_doubled_layer = SPECTRUM_COUNTS[\"double_steps\"] / SPECTRUM_COUNTS[\"doubled_layers\"]\n", - "print(f\" Fourier terms / wavelength {fourier_terms_per_wavelength:10.3f}\")\n", - "print(f\" LABOS layers / Fourier {labos_layers_per_fourier:10.3f}\")\n", - "print(f\" double steps / doubled {double_steps_per_doubled_layer:10.3f}\")" - ] + "source": "SPECTRUM_COUNTS = {\n \"output_samples\": 701,\n \"high_resolution_radiance_calculations\": 3874,\n \"fourier_terms\": 120390,\n \"labos_layers\": 5417550,\n \"doubled_layers\": 1075939,\n \"double_steps\": 8389666,\n}\n\nMATRIX_CALLS = [\n MatrixCall(\"Q = qseries(R * R)\", \"qseries_12x10\", 3_408_299),\n MatrixCall(\"D = T + Q * diag(E) + Q * T\", \"smulAddSemul3_12\", 3_408_299),\n MatrixCall(\"rd = R * D\", \"smul_12x10\", 8_389_666),\n MatrixCall(\"U = R * diag(E) + rd\", \"semulAdd_12\", 8_389_666),\n MatrixCall(\"tu = T * U\", \"smul_12x10\", 8_389_666),\n MatrixCall(\"R_next = R + diag(E) * U + tu\", \"matAddEsmul3_12\", 8_389_666),\n MatrixCall(\"td = T * D\", \"smul_12x10\", 8_389_666),\n MatrixCall(\"T_next = diag(E) * D + T * diag(E) + td\", \"esmulSemulAdd_12\", 8_389_666),\n]\n\nprint(\"wavelength expansion\")\nfor key, value in SPECTRUM_COUNTS.items():\n print(f\" {key:24s} {value:>10,d}\")\n\nfourier_terms_per_wavelength = (\n SPECTRUM_COUNTS[\"fourier_terms\"] / SPECTRUM_COUNTS[\"high_resolution_radiance_calculations\"]\n)\nlabos_layers_per_fourier = SPECTRUM_COUNTS[\"labos_layers\"] / SPECTRUM_COUNTS[\"fourier_terms\"]\ndouble_steps_per_doubled_layer = SPECTRUM_COUNTS[\"double_steps\"] / SPECTRUM_COUNTS[\"doubled_layers\"]\nprint(f\" Fourier terms / wavelength {fourier_terms_per_wavelength:10.3f}\")\nprint(f\" LABOS layers / Fourier {labos_layers_per_fourier:10.3f}\")\nprint(f\" double steps / doubled {double_steps_per_doubled_layer:10.3f}\")" }, { "cell_type": "markdown", "id": "7623eae2785240b9bd12b16a66d81610", "metadata": {}, - "source": [ - "## Why So Many Fourier Terms\n", - "\n", - "LABOS expands the azimuthal dependence of the scattering field into Fourier components. The loop in `execute.zig` runs one transport calculation per Fourier index:\n", - "\n", - "```text\n", - "for m in 0..fourier_max:\n", - " build PLM basis for m\n", - " build RT layers for m\n", - " calculate multiple scattering for m\n", - " add weighted reflectance contribution for m\n", - "```\n", - "\n", - "The O2 A case is not a near-normal geometry, so it cannot collapse to the scalar Fourier term. It uses `solar_zenith_deg=60`, `viewing_zenith_deg=30`, and `relative_azimuth_deg=120`. The aerosol phase function has `g=0.7`, so scattering is angularly structured enough that many azimuthal modes remain nonzero.\n", - "\n", - "The measured average is `120390 / 3874 = 31.076` Fourier terms per high-resolution wavelength. The non-integer average is expected: the Fourier tail is stopped per wavelength when the contribution falls below `3.0e-14` after the configured floor. Wavelength-dependent optical depth changes make the per-wavelength Fourier workload vary slightly." - ] + "source": "## Why So Many Fourier Terms\n\nLABOS expands the azimuthal dependence of the scattering field into Fourier components. The loop in `execute.zig` runs one transport calculation per Fourier index:\n\n```text\nfor m in 0..fourier_max:\n build PLM basis for m\n build RT layers for m\n calculate multiple scattering for m\n add weighted reflectance contribution for m\n```\n\nThe scene is not a near-normal geometry, so it cannot collapse to the scalar Fourier term. It uses `solar_zenith_deg=60`, `viewing_zenith_deg=30`, and `relative_azimuth_deg=120`. The aerosol phase function has `g=0.7`, so scattering is angularly structured enough that many azimuthal modes remain nonzero.\n\nThe measured average is `120390 / 3874 = 31.076` Fourier terms per high-resolution wavelength. The non-integer average is expected: the Fourier tail is stopped per wavelength when the contribution falls below `3.0e-14` after the configured floor. Wavelength-dependent optical depth changes make the per-wavelength Fourier workload vary slightly." }, { "cell_type": "code", @@ -357,13 +247,7 @@ "cell_type": "markdown", "id": "504fb2a444614c0babb325280ed9130a", "metadata": {}, - "source": [ - "## Run the Zig Matrix Benchmark\n", - "\n", - "The benchmark uses the actual Zig matrix routines from `matrix.zig`. It is intentionally narrow: it measures the small matrix calculations, then the notebook multiplies those matrix timings by the O2 A call counts.\n", - "\n", - "This keeps the demo focused on the LABOS wall by measuring the same matrix routines used by the forward model." - ] + "source": "## Run the Zig Matrix Benchmark\n\nThe benchmark uses the actual Zig matrix routines from `matrix.zig`. It is intentionally narrow: it measures the small matrix calculations, then the notebook multiplies those matrix timings by the call counts.\n\nThis keeps the demo focused on the LABOS wall by measuring the same matrix routines used by the forward model." }, { "cell_type": "code", @@ -406,25 +290,13 @@ "cell_type": "markdown", "id": "b43b363d81ae4b689946ece5c682cd59", "metadata": {}, - "source": [ - "## Why Some Matrix Calculations Are Cheap and One Is Relatively Expensive\n", - "\n", - "Most matrix calls are cheap because the matrix shape is fixed and small. The O2 A case uses `n_streams=20`, so `n_gauss=10`; with the direct solar and view directions, the LABOS matrix size is `n=12`. `matrix.zig` has specialized 12x10 and 12x12 paths for this shape. The arrays fit in cache, there is no heap allocation, and the innermost products are straight-line floating-point multiply-add chains.\n", - "\n", - "`smul_12x10` is a compact direct 12x10 product. The diagonal add/update helpers are cheaper because they mostly scale or add 12x12 arrays.\n", - "\n", - "`qseries_12x10` is more expensive because it does more than multiply two matrices. It forms `R * R`, checks the trace, builds `(I - R*R)` on the 10-stream Gauss block, performs pivoted factorization, calculates an inverse, and then applies the inverse back to the extra direct/view rows and columns. That means divisions, pivots, triangular back-substitution, and dependent floating-point operations. It is still sub-microsecond, but it is the most expensive matrix calculation in the doubling update." - ] + "source": "## Why Some Matrix Calculations Are Cheap and One Is Relatively Expensive\n\nMost matrix calls are cheap because the matrix shape is fixed and small. The scene uses `n_streams=20`, so `n_gauss=10`; with the direct solar and view directions, the LABOS matrix size is `n=12`. `matrix.zig` has specialized 12x10 and 12x12 paths for this shape. The arrays fit in cache, there is no heap allocation, and the innermost products are straight-line floating-point multiply-add chains.\n\n`smul_12x10` is a compact direct 12x10 product. The diagonal add/update helpers are cheaper because they mostly scale or add 12x12 arrays.\n\n`qseries_12x10` is more expensive because it does more than multiply two matrices. It forms `R * R`, checks the trace, builds `(I - R*R)` on the 10-stream Gauss block, performs pivoted factorization, calculates an inverse, and then applies the inverse back to the extra direct/view rows and columns. That means divisions, pivots, triangular back-substitution, and dependent floating-point operations. It is still sub-microsecond, but it is the most expensive matrix calculation in the doubling update." }, { "cell_type": "markdown", "id": "8a65eabff63a45729fe45fb5ade58bdc", "metadata": {}, - "source": [ - "## Reconstruct the Doubling Matrix Cost\n", - "\n", - "The doubling loop in `layers.zig` applies the same small set of matrix calculations millions of times. This cell multiplies isolated matrix timings by the measured O2 A call counts." - ] + "source": "## Reconstruct the Doubling Matrix Cost\n\nThe doubling loop in `layers.zig` applies the same small set of matrix calculations millions of times. This cell multiplies isolated matrix timings by the measured call counts." }, { "cell_type": "code", @@ -484,21 +356,7 @@ "cell_type": "markdown", "id": "3ed186c9a28b402fb0bc4494df01f08d", "metadata": {}, - "source": [ - "## What This Says About the Wall\n", - "\n", - "The per-spectrum wall is not a single slow line. It is a multiplicative structure:\n", - "\n", - "```text\n", - "T_spectrum ~= high_resolution_wavelengths\n", - " * Fourier_terms_per_wavelength\n", - " * active_layer_work_per_Fourier_term\n", - " * doubling_steps_per_active_layer\n", - " * small_matrix_calculation_cost\n", - "```\n", - "\n", - "The first-order optimization boundary is therefore clear. A material speedup has to reduce one of the multiplicative factors or replace a core matrix calculation while keeping the O2 A reference result inside the accepted agreement envelope. Small improvements to output-grid averaging or result collection cannot move the wall much because those steps happen after the expensive high-resolution radiance calculations have already been paid for." - ] + "source": "## What This Says About the Wall\n\nThe per-spectrum wall is not a single slow line. It is a multiplicative structure:\n\n```text\nT_spectrum ~= high_resolution_wavelengths\n * Fourier_terms_per_wavelength\n * active_layer_work_per_Fourier_term\n * doubling_steps_per_active_layer\n * small_matrix_calculation_cost\n```\n\nThe first-order optimization boundary is therefore clear. A material speedup has to reduce one of the multiplicative factors or replace a core matrix calculation while keeping the reference result inside the accepted agreement envelope. Small improvements to output-grid averaging or result collection cannot move the wall much because those steps happen after the expensive high-resolution radiance calculations have already been paid for." } ], "metadata": { diff --git a/scaffolding/reports/performance/retrieval/README.md b/scaffolding/reports/performance/retrieval/README.md index 29d3257da..3b0bbd96d 100644 --- a/scaffolding/reports/performance/retrieval/README.md +++ b/scaffolding/reports/performance/retrieval/README.md @@ -1,6 +1,6 @@ -# O2 A Retrieval Performance +# Retrieval Performance -Scope: aerosol-only O2 A optimal estimation with two state-vector dimensions: +Scope: aerosol-only optimal estimation with two state-vector dimensions: ```text aerosol optical depth @@ -19,12 +19,12 @@ zdisamar: 100/100 converged, median 3.624 s, mean 3.667 s ``` The result is not a different optimal-estimation shape. zdisamar is faster -because it runs a narrow in-process O2 A path, reuses forward-session state +because it runs a narrow in-process path, reuses forward-session state inside the retrieval, and asks only for Jacobian columns in the state vector. -Fastmode is a case-owned zdisamar optimisation lane. It runs OE with resolved +Fastmode is a scene-owned zdisamar optimisation lane. It runs OE with resolved fastmode RTM controls, sparse fast-stage wavelength sampling, and one sparse -full-physics OE update after fastmode convergence. The retained 100-case sweep +full-physics OE update after fastmode convergence. The retained 100-scene sweep reports: ```text @@ -37,7 +37,7 @@ full-physics correction wavelengths in the validation sweep. It keeps median speedup at `+1.382 s` versus fullmode while staying within `4.197e-04` AOD and `0.551 hPa` pressure maximum retrieved-state deltas. The reported fullmode and fastmode durations are wall-clock timings around the -public retrieval call, including session/cache creation, native case load and +public retrieval call, including session/cache creation, native scene load and preparation, native OE work, and the sparse full-physics correction. The retained benchmark artifact for code-speed changes is @@ -50,17 +50,17 @@ forward no-session median 1.031 s forward session cached-run median 0.001 s OE single session retrieval median 1.151 s OE single fastmode retrieval median 0.535 s -OE 5-case session sweep retrieval median 2.604 s -OE 5-case fastmode sweep retrieval median 0.840 s -OE 5-case fastmode sweep retrieval total 4.516 s +OE 5-scene session sweep retrieval median 2.604 s +OE 5-scene fastmode sweep retrieval median 0.840 s +OE 5-scene fastmode sweep retrieval total 4.516 s ``` -Use the benchmark artifact for retained local timing changes. Use the 100-case +Use the benchmark artifact for retained local timing changes. Use the 100-scene fastmode sweep above for accuracy, convergence, and wavelength-shape claims. -For the 10-worker baseline-case wrapper boundary, the current one-shot fastmode +For the 10-worker baseline-scene wrapper boundary, the current one-shot fastmode public call is `0.230 s` median. A repeated-start loop with a caller-owned -session cache is `0.179 s` median after the first sparse-case load. See +session cache is `0.179 s` median after the first sparse-scene load. See [`fastmode-session-overhead.md`](fastmode-session-overhead.md) for that focused probe. diff --git a/scaffolding/reports/performance/retrieval/current-retrieval-elapsed-time.md b/scaffolding/reports/performance/retrieval/current-retrieval-elapsed-time.md index bf9b74342..cbce126c4 100644 --- a/scaffolding/reports/performance/retrieval/current-retrieval-elapsed-time.md +++ b/scaffolding/reports/performance/retrieval/current-retrieval-elapsed-time.md @@ -32,7 +32,7 @@ zdisamar: max mid-pressure error 0.608 hPa ``` -Slow retained zdisamar case: +Slow retained zdisamar scene: ```text source case paired sweep case 71 diff --git a/scaffolding/reports/performance/retrieval/fastmode-final-correction.md b/scaffolding/reports/performance/retrieval/fastmode-final-correction.md index 17274c570..771607abf 100644 --- a/scaffolding/reports/performance/retrieval/fastmode-final-correction.md +++ b/scaffolding/reports/performance/retrieval/fastmode-final-correction.md @@ -1,15 +1,15 @@ # Fastmode Sampling And Final Correction -Fastmode is a case-owned optimisation mode, not a separate retrieval API. A -normal O2 A case remains the full-physics reference case. Fastmode is enabled -on that same case: +Fastmode is a scene-owned optimisation mode, not a separate retrieval API. A +normal scene remains the full-physics reference scene. Fastmode is enabled +on that same scene: ```python case.optimisation.fastmode.enabled = True ``` -The case keeps the optimisation settings visible. The execution path resolves -those settings before loading the native RTM case, so callers can inspect the +The scene keeps the optimisation settings visible. The execution path resolves +those settings before loading the native RTM scene, so callers can inspect the actual knobs, fast-stage wavelengths, and final-correction wavelengths: ```python @@ -30,7 +30,7 @@ The default fastmode configuration changes: ## API Knobs Fastmode is configured by editing fields under `case.optimisation.fastmode`. -The defaults are intended to be usable directly, but the fields are ordinary case +The defaults are intended to be usable directly, but the fields are ordinary scene data and can be changed before calling `disamar_oe`. ```python @@ -89,24 +89,24 @@ measurement vectors. The retained OE path uses one public `disamar_oe` call: ```text -1. The input case has fastmode enabled. +1. The input scene has fastmode enabled. 2. Fastmode resolves sparse fast-stage wavelengths on the measurement grid. -3. The native session loads that sparse case with fastmode RTM controls resolved. +3. The native session loads that sparse scene with fastmode RTM controls resolved. 4. OE runs to convergence on the sparse fast-stage measurement vector. 5. If final correction is enabled, zdisamar disables fastmode on the original - full measurement case copy. + full measurement scene copy. 6. The correction keeps explicit sparse wavelengths on the measurement grid. 7. For a one-shot `cache=None` call, the same temporary session handle loads - that sparse full-physics correction case. For a caller-supplied cache, the - correction uses a temporary handle so the warmed sparse fast-stage case + that sparse full-physics correction scene. For a caller-supplied cache, the + correction uses a temporary handle so the warmed sparse fast-stage scene remains reusable. 8. Native OE computes one full-physics forward model, Jacobian, and update. 9. The result returns the corrected state plus fast-stage correction diagnostics. ``` -There is no public correction case and no public spectral-branch object. The +There is no public correction scene and no public spectral-branch object. The default sparse selectors are just windows and counts that resolve to concrete -wavelengths on the case sampling: +wavelengths on the scene sampling: ```text fast stage: @@ -140,7 +140,7 @@ case.optimisation.fastmode.oe.final_correction.wavelengths_nm = ( ## Dimension Shape -The fast solve and the correction solve each clip the measurement and RTM case +The fast solve and the correction solve each clip the measurement and RTM scene to their own explicit wavelength grid. The dimensions therefore match inside each OE boundary: @@ -158,7 +158,7 @@ K_sparse^T S_e_sparse^-1 K_sparse n_state x n_state K_sparse^T S_e_sparse^-1 (y - F(x)) n_state ``` -The solve is still a state-vector solve. For the retained two-state O2 A +The solve is still a state-vector solve. For the retained two-state retrieval, `n_state = 2` for aerosol optical depth and aerosol layer mid pressure. Sparse wavelengths change the number of Jacobian rows, not the number of retrieved state dimensions. @@ -171,8 +171,8 @@ construction, plotting, or the lazy final-state spectrum evaluation. It does include: - the fastmode OE retrieval call; -- native case load/prepare for the sparse fast-stage case; -- native case load/prepare for the sparse full-physics correction; +- native scene load/prepare for the sparse fast-stage scene; +- native scene load/prepare for the sparse full-physics correction; - one exact full-physics forward model and Jacobian on that sparse grid; - one native OE correction solve; - copying the result back through the Python binding. @@ -183,7 +183,7 @@ and residual inspection can ask for it later, but it is not part of ## Retained Evidence -Regenerate the retained 100-case fastmode sweep with: +Regenerate the retained 100-scene fastmode sweep with: ```sh uv run validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py @@ -212,7 +212,7 @@ fastmode max mid-pressure delta vs fullmode: 0.551 hPa The sweep compares: - `reference`: normal full-physics zdisamar OE; -- `fastmode`: case-owned fastmode with sparse fast-stage sampling and the default +- `fastmode`: scene-owned fastmode with sparse fast-stage sampling and the default sparse full-physics final correction. The tracked outputs are: @@ -258,6 +258,6 @@ Open questions: - rank candidate wavelengths by Jacobian information for the retrieved state dimensions; - keep a held-out validation set so sparse defaults do not overfit the retained - 100 cases; + 100 scenes; - measure correction-only timing separately from fast-stage timing; - revisit uncertainty scaling for sparse windows. diff --git a/scaffolding/reports/performance/retrieval/fastmode-session-overhead.md b/scaffolding/reports/performance/retrieval/fastmode-session-overhead.md index 4ef542c37..0b75875b2 100644 --- a/scaffolding/reports/performance/retrieval/fastmode-session-overhead.md +++ b/scaffolding/reports/performance/retrieval/fastmode-session-overhead.md @@ -1,25 +1,25 @@ # Fastmode Session Overhead Log -Goal: reduce overhead around one 10-worker O2 A fastmode OE retrieval without +Goal: reduce overhead around one 10-worker fastmode OE retrieval without changing the retrieval result or the fastmode accuracy contract. Boundary: - public Python `o2a.retrieve(...)`; - `ZDISAMAR_WORKER_LIMIT=10`; -- baseline fastmode case; +- baseline fastmode scene; - measurement construction outside the timed block; - no lazy final-state spectrum evaluation. ## 2026-05-27 Pressure Profile Resolution Finding: resolving pressure metadata for `AerosolLayerMidPressure` used the -original full measurement case before the fast-stage case was loaded. That +original full measurement scene before the fast-stage scene was loaded. That created a temporary 301-sample native prepare even though the pressure-altitude profile is independent of the sparse wavelength subset. Change: resolve the state vector against the already loaded sparse fast-stage -case, using the session cache's atmospheric budget table. +scene, using the session cache's atmospheric budget table. Correctness check: @@ -45,10 +45,10 @@ Accepted: removes the full-grid pressure-profile prepare from the timed ## 2026-05-27 Preserve Caller-Owned Sparse Cache -Finding: a caller can preload and warm the sparse fast-stage case before timing, -but the final correction loaded its 4-sample full-physics case into that same +Finding: a caller can preload and warm the sparse fast-stage scene before timing, +but the final correction loaded its 4-sample full-physics scene into that same cache. The first call stayed fast; repeated calls with the same sparse cache -then reloaded the fast-stage case and moved warmup work back into the +then reloaded the fast-stage scene and moved warmup work back into the invocation. Scratch comparison for five repeated calls with one caller-owned sparse cache: @@ -60,7 +60,7 @@ states_equal: true ``` Change: when the caller supplies a cache to fastmode OE, keep that cache loaded -with the fast-stage case and run the final correction on a temporary native +with the fast-stage scene and run the final correction on a temporary native handle. The `cache=None` one-shot path keeps the previous single-handle shape. Timing evidence after the change: @@ -76,9 +76,9 @@ converged: true Accepted: preserves the warmed sparse fast-stage session across repeated calls without changing the returned state. -## 2026-05-27 Case-Owned Controls And Repeated Starts +## 2026-05-27 Scene-Owned Controls And Repeated Starts -The canonical repeated-start shape keeps controls in the case and passes only +The canonical repeated-start shape keeps controls in the scene and passes only the varying state vector through the public retrieval call: ```python @@ -114,13 +114,13 @@ calls reuse the same fast-stage cache; the final correction does not replace it. ## 2026-05-27 Duplicate Cache-Match Checks Finding: after `retrieve()` had already established that the supplied cache held -the sparse fast-stage case, pressure-profile resolution called the generic -`has_loaded_case` path again. That path fingerprints the case by rebuilding the +the sparse fast-stage scene, pressure-profile resolution called the generic +`has_loaded_case` path again. That path fingerprints the scene by rebuilding the native JSON payload. Change: use a loaded-cache pressure-profile path once stale-cache protection has already run at the retrieval boundary. The caller-supplied cache still performs -one `has_loaded_case(fast_case)` check per retrieval so case or measurement +one `has_loaded_case(fast_case)` check per retrieval so scene or measurement changes reload the cache correctly. Timing evidence from the repeated-start probe: @@ -149,12 +149,12 @@ load samples_4.fast_False: ``` The current correction C entrypoint consumes a fully prepared `PreparedO2A`. -`correctPreparedO2A` copies that prepared case, switches derivative mode, and +`correctPreparedO2A` copies that prepared scene, switches derivative mode, and simulates the RTM/Jacobian from `prepared.scene` and `prepared.prepared`. It does not apply the correction initial state to a base scene and rebuild state-dependent optical properties. -Rejected for this pass: simply keeping a prepared correction case alive in +Rejected for this pass: simply keeping a prepared correction scene alive in Python. That would reuse optics prepared at the wrong aerosol optical depth or layer pressure for later starts. @@ -372,7 +372,7 @@ static 3-worker iteration sums: [132, 131, 151] Change: keep the inner native forward-prefetch policy unchanged, but let the outer native OE batch workers claim one start at a time from `ChunkQueue`. -Each worker still owns its prepared case and product storage; the queue only +Each worker still owns its prepared scene and product storage; the queue only balances which start index is run next. Timing evidence on the same scene-008 100-start boundary: @@ -454,9 +454,9 @@ the 100-start scene-008 fast-stage-only boundary to `19.607 s`, but moved the retrieved state by `7.17e-4` AOD and `1.15 hPa` versus the retained fastmode default, outside the retained validation pressure gate. -Change: set the case-owned fastmode OE +Change: set the scene-owned fastmode OE `state_vector_convergence_threshold` to `10.0` and make retained validation -callers omit explicit controls so they exercise the same case-owned fastmode +callers omit explicit controls so they exercise the same scene-owned fastmode path as public `retrieve(..., controls=None)` and `Result.diagnose()`. Scene-008 fast-stage-only timing evidence: @@ -469,7 +469,7 @@ all 100 starts converged ``` Retained fastmode-vs-reference validation was regenerated with the fullmode -rows unchanged and the fastmode rows rerun under the new case-owned controls: +rows unchanged and the fastmode rows rerun under the new scene-owned controls: ```text fastmode rows: 100 @@ -499,7 +499,7 @@ iteration histogram: {2: 3, 3: 20, 4: 42, 5: 31, 6: 3, 7: 1} ``` Change: `runO2AFastmodeBatch` now gives each native batch worker both the -fast-stage prepared case and the sparse correction prepared case. Each worker +fast-stage prepared scene and the sparse correction prepared scene. Each worker runs its assigned starts end-to-end, avoiding the intermediate `BatchResult` state table and the second batch/thread handoff between fast-stage and correction solves. @@ -527,7 +527,7 @@ model speedup. ## 2026-05-28 Prefetch-Mode Boundary Check The retained `~175 ms` figure comes from the steady repeated-start public API -path: one caller-owned empty `SessionCache()`, fixed case and measurement, +path: one caller-owned empty `SessionCache()`, fixed scene and measurement, `ZDISAMAR_WORKER_LIMIT=10`, omitted controls, and only the starting state vector changing after an initial warm call. It is a valid prefetch-session number, but it should not be multiplied across the scene-008 basin grid without rechecking @@ -629,12 +629,12 @@ measured boundary. ## 2026-05-28 Fast-Stage Convergence Retune Finding: the sparse full-physics correction absorbs a modestly looser -fast-stage convergence threshold on the retained 100-case fastmode validation +fast-stage convergence threshold on the retained 100-scene fastmode validation boundary. Aggressive thresholds were rejected: `50` and above preserved convergence but moved at least one corrected state by `0.00618` AOD and `3.16 hPa` against the retained fullmode reference. -Retained fastmode-only 100-case probes against the tracked fullmode reference: +Retained fastmode-only 100-scene probes against the tracked fullmode reference: ```text threshold 15: median 0.384 s, max delta 4.18164e-4 AOD / 0.545859 hPa @@ -642,7 +642,7 @@ threshold 20: median 0.356 s, max delta 4.18164e-4 AOD / 0.545859 hPa threshold 30: median 0.349 s, max delta 4.18164e-4 AOD / 0.545859 hPa ``` -Decision: do not carry `30.0` as the case-owned fastmode OE default. It kept +Decision: do not carry `30.0` as the scene-owned fastmode OE default. It kept the retained fastmode/reference max-delta gate unchanged in this probe, but `40.0` failed the retained validation gate below. Keeping the public default at `1.0` leaves the diagnosis work on the established accuracy contract instead of @@ -651,7 +651,7 @@ shipping a near-boundary speed retune as part of the batch API change. ## 2026-05-28 Prefetch Boundary Recheck After Retune Rechecked the user-facing prefetch-session shape after the threshold retune: -one empty caller-owned `SessionCache()`, fixed scene/case/measurement, +one empty caller-owned `SessionCache()`, fixed scene/measurement, `ZDISAMAR_WORKER_LIMIT=10`, omitted controls, one warm call, varying only the state-vector initial/prior values, and no access to `result.final_evaluation` inside the timed loop. @@ -811,7 +811,7 @@ Sampling and convergence probes: state_vector_convergence_threshold=40: scene-008 diagnosis improved only 30.300 s -> 29.795 s - retained 100-case validation failed: + retained 100-scene validation failed: AOD max_abs_delta=9.155e-04 > 5.500e-04 pressure max_abs_delta=1.444 hPa > 0.700 hPa diff --git a/scaffolding/reports/performance/retrieval/fastmode-sub100-search.md b/scaffolding/reports/performance/retrieval/fastmode-sub100-search.md index 1a43fb016..253cc2ffe 100644 --- a/scaffolding/reports/performance/retrieval/fastmode-sub100-search.md +++ b/scaffolding/reports/performance/retrieval/fastmode-sub100-search.md @@ -1,6 +1,6 @@ # Fastmode Sub-100 ms Search -Goal: find fastmode control settings that bring O2 A optimal-estimation +Goal: find fastmode control settings that bring optimal-estimation retrieval below 0.1 s on a 10-worker CPU cap without increasing the retained fastmode accuracy loss against fullmode. @@ -58,7 +58,7 @@ current fastmode error envelope. Append each run with: - candidate name and changed knobs; -- case subset; +- scene subset; - median/min/max retrieval time; - max AOD and pressure deltas vs fullmode; - convergence count; @@ -67,7 +67,7 @@ Append each run with: ## 2026-05-27 Results -Exploratory subset: retained stress cases `1, 2, 3, 8, 20, 48, 74, 86, 93` +Exploratory subset: retained stress scenes `1, 2, 3, 8, 20, 48, 74, 86, 93` with `ZDISAMAR_WORKER_LIMIT=10`. Rejected paths: @@ -77,7 +77,7 @@ Rejected paths: - `max_iterations = 1`, no correction, 8-38 fast wavelengths: median `0.170-0.176 s`, still above 0.1 s and with `~2e-2` AOD / `>11 hPa` pressure deltas. -- `max_iterations = 2`: median `0.453 s`; full 100-case follow-up with 24 fast +- `max_iterations = 2`: median `0.453 s`; full 100-scene follow-up with 24 fast wavelengths and 4 correction wavelengths had median `0.411 s`, but only `23/100` fast stages converged. - Lower stream counts, spherical/source/renormalization switches, matrix @@ -87,7 +87,7 @@ Rejected paths: Accepted partial win: - `fastmode.oe.final_correction.wavelength_count = 4` -- retained 100-case sweep command: +- retained 100-scene sweep command: `ZDISAMAR_WORKER_LIMIT=10 uv run validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py` - convergence: `100/100` fastmode and `100/100` fast-stage convergence - fastmode median/mean retrieval: `0.500 s` / `0.483 s` @@ -112,12 +112,12 @@ Method: - treat the fast-stage and correction wavelength sets as the Jacobian-row sparsity knobs; - compute weighted two-column Jacobian rows on the full measurement grid for - retained stress cases; + retained stress scenes; - greedily select rows by a small Fisher-information objective; - validate each candidate by running real fastmode retrievals against fullmode references, not by trusting the information metric. -Stress subset: retained cases `1, 2, 3, 8, 20, 48, 74, 86, 93`. +Stress subset: retained scenes `1, 2, 3, 8, 20, 48, 74, 86, 93`. Previous 38-row default on the same stress run: @@ -140,7 +140,7 @@ Best stress candidate: Interpretation so far: the 38-row fast-stage grid is not the minimum useful Jacobian-row set. A sensitivity-selected 12-row set preserved the retained -stress-case envelope and reduced the stress median by about `19%`. This is a +stress-scene envelope and reduced the stress median by about `19%`. This is a real lead. Other stress observations: @@ -150,14 +150,14 @@ Other stress observations: - two-row final correction was not enough (`~8.8e-03` AOD / `~9.6 hPa` on the default fast-stage run); - three-row and greedy four-row final-correction sets could stay accurate on - stress cases, but were slower than the existing four-row correction in this - noisy run, likely because the selected endpoints widened expensive O2 A + stress scenes, but were slower than the existing four-row correction in this + noisy run, likely because the selected endpoints widened expensive O2A structure rather than just reducing sample count; -- evenly spaced 12-row fast-stage sampling was accurate on stress cases but +- evenly spaced 12-row fast-stage sampling was accurate on stress scenes but slower than the greedy 12-row set, so row placement matters more than count alone. -Retained 100-case follow-up after promoting the 12-row set as the default: +Retained 100-scene follow-up after promoting the 12-row set as the default: - command: `ZDISAMAR_WORKER_LIMIT=10 uv run validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py` @@ -177,6 +177,6 @@ Retained 100-case follow-up after promoting the 12-row set as the default: `0.815 s`. Conclusion: the fast-stage Jacobian does not need the old 38 rows for the -retained two-state O2 A retrieval. A 12-row sensitivity-selected set is a +retained two-state retrieval. A 12-row sensitivity-selected set is a validated default improvement: it is faster and stays inside the existing retrieval accuracy envelope. diff --git a/scaffolding/reports/performance/retrieval/inter-iteration-hillclimb.md b/scaffolding/reports/performance/retrieval/inter-iteration-hillclimb.md index 596c9a69f..2f5fe1498 100644 --- a/scaffolding/reports/performance/retrieval/inter-iteration-hillclimb.md +++ b/scaffolding/reports/performance/retrieval/inter-iteration-hillclimb.md @@ -1,13 +1,13 @@ -# O2 A OE Inter-Iteration Hillclimb +# OE Inter-Iteration Hillclimb This note records the May 2026 audit of repeated work in the session-backed -O2 A optimal-estimation path. +optimal-estimation path. ## Evidence The retained benchmark artifact is `validation/outputs/optimal_estimation/zdisamar_o2a_slow_rtm_jacobian_benchmark.json`. -At that audit the slow retained case reported: +At that audit the slow retained scene reported: - retrieval loop wall time: 3.142969 s - retrieval iteration RTM plus Jacobian: 3.142008 s @@ -23,7 +23,7 @@ session caches: - cached forward-miss/profile spectroscopy count: 3736 The lazy final-state evaluation also hits those caches once when requested. This -means the slow case is not repeatedly rebuilding the wavelength sampling plan, +means the slow scene is not repeatedly rebuilding the wavelength sampling plan, forward-miss list, or profile spectroscopy caches between OE iterations. The traced native Jacobian sweep is diff --git a/scaffolding/reports/performance/retrieval/measurement-provenance.md b/scaffolding/reports/performance/retrieval/measurement-provenance.md index 3546e8a3d..906808b63 100644 --- a/scaffolding/reports/performance/retrieval/measurement-provenance.md +++ b/scaffolding/reports/performance/retrieval/measurement-provenance.md @@ -6,7 +6,7 @@ The current paired retrieval evidence is the tracked plot manifest: validation/outputs/optimal_estimation/paired_oe_plot_manifest.json ``` -The generated parquet and per-case DISAMAR directories live under: +The generated parquet and per-scene DISAMAR directories live under: ```text out/validation/optimal_estimation/paired_disamar_zdisamar/ @@ -26,7 +26,7 @@ validation/outputs/optimal_estimation/zdisamar_o2a_slow_rtm_jacobian_benchmark.j ``` The paired scene ranges follow the geometry, aerosol, meteorology, and surface -parameter ranges from the AMT 2019 O2 A aerosol-height retrieval study, +parameter ranges from the AMT 2019 O2A aerosol-height retrieval study, `https://doi.org/10.5194/amt-12-6619-2019`, narrowed here to sensible cloud-free aerosol-only validation ranges. diff --git a/scaffolding/reports/performance/retrieval/multistart-calculation-telemetry.md b/scaffolding/reports/performance/retrieval/multistart-calculation-telemetry.md index 65ab798d9..84d643f14 100644 --- a/scaffolding/reports/performance/retrieval/multistart-calculation-telemetry.md +++ b/scaffolding/reports/performance/retrieval/multistart-calculation-telemetry.md @@ -1,6 +1,6 @@ # Multistart Calculation Telemetry Probe -Question: for repeated fastmode OE starts on one fixed O2 A scene, which +Question: for repeated fastmode OE starts on one fixed scene, which calculations are invariant enough to justify session-level reuse? ## Probe @@ -20,10 +20,10 @@ diagnose_retrieval -> runO2AFastmodeBatch ``` -The fast stage used the case-owned fastmode settings used by `diagnose()`: +The fast stage used the scene-owned fastmode settings used by `diagnose()`: 12 fast-stage wavelengths, sparse fast RTM settings, and then one sparse -full-physics correction on 4 wavelengths. The run used one prepared fast case, -one prepared correction case, and `batch_workers=2`, matching +full-physics correction on 4 wavelengths. The run used one prepared fast scene, +one prepared correction scene, and `batch_workers=2`, matching `diagnosis_batch_worker_count_for_limit(5, 10)`. Instrumentation added row context to the validation-only calculation telemetry: @@ -58,7 +58,7 @@ script overhead. I rechecked the current worktree before adding more native optimization machinery. Same-boundary settings: ReleaseFast package sync, -`ZDISAMAR_WORKER_LIMIT=10`, unchanged fastmode controls, fixed case and +`ZDISAMAR_WORKER_LIMIT=10`, unchanged fastmode controls, fixed scene and measurement, one reused `rtm.SessionCache()`, no `final_evaluation` access inside timed loops, two native diagnosis workers, and 25 starts. @@ -309,7 +309,7 @@ correction-stage LABOS/Fourier active-mask cache. ## Rejected Fused Worker-Session Probe I tested an internal native fastmode batch shape where each worker owned both -the fast-stage prepared case and the correction prepared case, then processed a +the fast-stage prepared scene and the correction prepared scene, then processed a start through fast stage and correction before claiming another start. The intent was to keep the worker's session resources hot across both stages and remove the separate fast-worker phase followed by a separate correction-worker diff --git a/scaffolding/reports/performance/retrieval/optimisation-notes/01-forward-session-reuse.md b/scaffolding/reports/performance/retrieval/optimisation-notes/01-forward-session-reuse.md index 5cece78df..7a5d4b706 100644 --- a/scaffolding/reports/performance/retrieval/optimisation-notes/01-forward-session-reuse.md +++ b/scaffolding/reports/performance/retrieval/optimisation-notes/01-forward-session-reuse.md @@ -1,4 +1,4 @@ -# 01. Reuse The O2 A RTM Session Cache +# 01. Reuse The RTM Session Cache Historical slow-case evidence from the session-reuse optimization: @@ -8,7 +8,7 @@ session reused elapsed time 4.098390 s same retrieved state true ``` -In short: reuse one O2 A RTM session cache across retrieval iterations for the +In short: reuse one RTM session cache across retrieval iterations for the same scene. Current retained slow-case evidence after later Jacobian and lazy-final-state @@ -26,7 +26,7 @@ Source links: - No direct Fortran analogue is used here. The optimisation is zdisamar's in-process session reuse around repeated state-vector evaluations. - zdisamar - `python/zdisamar/inverse_method/optimal_estimation/o2a.py`: routes every retrieval state through either a fresh RTM evaluation or the reusable session cache. - - `python/zdisamar/rtm/session_cache.py`: keeps reusable RTM storage alive and reloads only the changed case state. + - `python/zdisamar/rtm/session_cache.py`: keeps reusable RTM storage alive and reloads only the changed scene state. - [Paired sweep use](https://github.com/bout3fiddy/zdisamar/blob/aa3bdc776e605229b18b54a7999632fb276546e2/validation/optimal_estimation/paired_disamar_zdisamar_sweep.py#L200-L224): uses the session path in the current validation lane. The retrieval loop evaluates nearby state-vector points for the same scene. For diff --git a/scaffolding/reports/performance/retrieval/optimisation-notes/README.md b/scaffolding/reports/performance/retrieval/optimisation-notes/README.md index 35831f9a9..bd51e4aca 100644 --- a/scaffolding/reports/performance/retrieval/optimisation-notes/README.md +++ b/scaffolding/reports/performance/retrieval/optimisation-notes/README.md @@ -1,6 +1,6 @@ # Retrieval Optimisation Notes -These notes keep the retrieval-specific mechanism details behind the short O2 A +These notes keep the retrieval-specific mechanism details behind the short retrieval performance summary. They use source links and Python-shaped examples, not copied implementation excerpts. @@ -13,7 +13,7 @@ zdisamar: aa3bdc776e605229b18b54a7999632fb276546e2 Notes: -- [01. Reuse the O2 A forward session](01-forward-session-reuse.md) +- [01. Reuse the forward session](01-forward-session-reuse.md) - [02. Keep Jacobians state-vector sized](02-state-vector-jacobians.md) - [03. Isolate paired validation lanes](03-paired-validation-lanes.md) - [04. Defer final-state evaluation](04-lazy-final-evaluation.md) diff --git a/scripts/demo/README.md b/scripts/demo/README.md index 9892d4608..d29c16185 100644 --- a/scripts/demo/README.md +++ b/scripts/demo/README.md @@ -1,6 +1,6 @@ # Demo Notebooks -This directory contains executable, explanatory notebooks for Python-facing O2 A +This directory contains executable, explanatory notebooks for Python-facing tooling. They are demos, not CI harnesses and not tracked validation evidence. The notebooks should explain the wrapper flow, display plots inline, and keep demo state inside the notebook instead of depending on repository-local paths. @@ -22,11 +22,11 @@ zig build - `o2a_plot_bundle.ipynb`: demonstrates the supported `.plot` accessor surface on `Spectrum`, atmospheric budget, O2-O2 CIA diagnostics, and the instrument response table. Each chart has its own notebook cell, including SNR and the - sun-normalized radiance noise envelope from the retained O2 A baseline + sun-normalized radiance noise envelope from the retained baseline measurement-noise model. -- `optimal_estimation_demo.ipynb`: demonstrates a two-state O2 A +- `optimal_estimation_demo.ipynb`: demonstrates a two-state optimal-estimation flow using aerosol optical depth and aerosol layer mid-pressure, with convergence, measurement-fit, residual, and Jacobian plots in - separate cells. The notebook also points to the case-owned fastmode switch and + separate cells. The notebook also points to the scene-owned fastmode switch and the main fastmode control knobs for trying the fastmode OE lane and its sparse wavelength defaults. diff --git a/scripts/demo/o2a_plot_bundle.ipynb b/scripts/demo/o2a_plot_bundle.ipynb index 64559d639..63b5c0b06 100644 --- a/scripts/demo/o2a_plot_bundle.ipynb +++ b/scripts/demo/o2a_plot_bundle.ipynb @@ -4,21 +4,13 @@ "cell_type": "markdown", "id": "3dbbdba0", "metadata": {}, - "source": [ - "# O2 A Plot Accessor Demo\n", - "\n", - "This notebook is a catalog of the supported plotting surface: `.plot` accessors on zdisamar domain objects. Each plot is displayed in its own cell.\n" - ] + "source": "# Plot Accessor Demo\n\nThis notebook is a catalog of the supported plotting surface: `.plot` accessors on zdisamar domain objects. Each plot is displayed in its own cell." }, { "cell_type": "markdown", "id": "73a31fad", "metadata": {}, - "source": [ - "## Setup\n", - "\n", - "The setup creates one O2 A case, runs the RTM diagnostics, and stores one SVG chart per public accessor method.\n" - ] + "source": "## Setup\n\nThe setup creates one scene, runs the RTM diagnostics, and stores one SVG chart per public accessor method." }, { "cell_type": "code", diff --git a/scripts/demo/optimal_estimation_demo.ipynb b/scripts/demo/optimal_estimation_demo.ipynb index edfcae3d8..a5bdc2af7 100644 --- a/scripts/demo/optimal_estimation_demo.ipynb +++ b/scripts/demo/optimal_estimation_demo.ipynb @@ -4,11 +4,7 @@ "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": {}, - "source": [ - "# O2 A optimal estimation\n", - "\n", - "This notebook runs the same two-state aerosol retrieval in full mode and fastmode. The measurement is simulated from a truth case, then optimal estimation retrieves aerosol optical depth and aerosol layer mid-pressure." - ] + "source": "# Optimal estimation\n\nThis notebook runs the same two-state aerosol retrieval in full mode and fastmode. The measurement is simulated from a truth scene, then optimal estimation retrieves aerosol optical depth and aerosol layer mid-pressure." }, { "cell_type": "code", @@ -20,8 +16,8 @@ "import statistics as stats\n", "from time import perf_counter\n", "\n", + "from zdisamar import optimal_estimation as oe\n", "from zdisamar import rtm\n", - "from zdisamar.inverse_method import optimal_estimation as oe\n", "from zdisamar.rtm import SessionCache\n", "from zdisamar.wavelength_bands import o2a" ] @@ -48,11 +44,7 @@ "cell_type": "markdown", "id": "8dd0d8092fe74a7c96281538738b07e2", "metadata": {}, - "source": [ - "## Simulated truth and measurement\n", - "\n", - "The truth case generates the synthetic reflectance measurement. Measurement noise is expressed as signal-to-noise ratio, either one scalar for the whole spectrum or one value per wavelength." - ] + "source": "## Simulated truth and measurement\n\nThe truth scene generates the synthetic reflectance measurement. Measurement noise is expressed as signal-to-noise ratio, either one scalar for the whole spectrum or one value per wavelength." }, { "cell_type": "code", @@ -138,11 +130,7 @@ "cell_type": "markdown", "id": "7cdc8c89c7104fffa095e18ddfef8986", "metadata": {}, - "source": [ - "## Fastmode\n", - "\n", - "Fastmode is enabled on the same case object. The retrieval call stays the same; the case-owned fastmode settings select the tuned RTM shortcuts, sparse fast-stage wavelengths, and sparse full-physics correction." - ] + "source": "## Fastmode\n\nFastmode is enabled on the same scene object. The retrieval call stays the same; the scene-owned fastmode settings select the tuned RTM shortcuts, sparse fast-stage wavelengths, and sparse full-physics correction." }, { "cell_type": "code", @@ -178,11 +166,7 @@ "cell_type": "markdown", "id": "504fb2a444614c0babb325280ed9130a", "metadata": {}, - "source": [ - "## Wall-clock timing\n", - "\n", - "These calls include Python call overhead, case loading, native preparation, retrieval iterations, and the fastmode correction step." - ] + "source": "## Wall-clock timing\n\nThese calls include Python call overhead, scene loading, native preparation, retrieval iterations, and the fastmode correction step." }, { "cell_type": "code", @@ -220,11 +204,7 @@ "cell_type": "markdown", "id": "b43b363d81ae4b689946ece5c682cd59", "metadata": {}, - "source": [ - "## Tune fastmode knobs\n", - "\n", - "The defaults are tuned for the retained O2 A validation sweep. For research, the same fields can be adjusted before running `oe.retrieve`." - ] + "source": "## Tune fastmode knobs\n\nThe defaults are tuned for the retained validation sweep. For research, the same fields can be adjusted before running `oe.retrieve`." }, { "cell_type": "code", @@ -252,11 +232,7 @@ "cell_type": "markdown", "id": "c3933fab20d04ec698c2621248eb3be0", "metadata": {}, - "source": [ - "## Aerosol profile forward simulations\n", - "\n", - "The forward model also accepts explicit aerosol profile layers for simulations that are not single-layer scalar cases." - ] + "source": "## Aerosol profile forward simulations\n\nThe forward model also accepts explicit aerosol profile layers for simulations that are not single-layer scalar scenes." }, { "cell_type": "code", diff --git a/src/AGENTS.md b/src/AGENTS.md index 446b7303c..579dd47e5 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -1,7 +1,7 @@ # Source Tree - `src/input/` owns typed atmosphere, geometry, surface, spectroscopy, instrument, and reference-data input structures, plus JSON, defaults, and validation. -- `src/setup/` builds the named physics tables consumed by the hot path: atmosphere layers, line, CIA, aerosol, instrument, solar, phase, and O2 run tables. +- `src/setup/` builds the named physics tables consumed by the hot path: atmosphere layers, line, CIA, aerosol, instrument, solar, phase, and run tables. - `src/optics/` owns per-sample optical properties: layer depths, Rayleigh, CIA absorption, curved sun path, and source levels. - `src/rtm/` owns radiative transfer: controls, gauss angles, attenuation, phase basis, scattering orders, layer reflect/transmit, reflectance, and the `solve` entry point. - `src/spectrum/` owns spectral assembly: sampling table, line physics, solar lookup, radiance wavelengths/results, instrument averaging, and the `spectrum_run` driver. @@ -10,7 +10,7 @@ - `src/cache/` owns retained, named memory owners (session, profile-line, radiance, solar-irradiance, spectrum, transport-worker, worker-pool). These are the borrowed write targets the hot path threads through. - `src/common/` is shared support code only: errors, hashing, math, units, memory, worker partition. - `src/assets/` owns embedded asset readers. -- `src/validation/` owns validation-only code that consumes the typed O2 A baseline. +- `src/validation/` owns validation-only code that consumes the typed baseline. - `src/instrumentation/` owns narrow no-op-by-default facades for trace, telemetry, and sensitivity hooks. Retained sinks, capture scripts, and reports live under `scaffolding/`. ## Rules diff --git a/src/README.md b/src/README.md index fac0e2fed..36df74b92 100644 --- a/src/README.md +++ b/src/README.md @@ -1,4 +1,4 @@ -# `src/` — O2 A forward model and retrieval +# `src/` — forward model and retrieval `zdisamar` computes the top-of-atmosphere reflectance spectrum across the oxygen A band (about 755–775 nm) and fits it for aerosol parameters, specifically the @@ -14,9 +14,9 @@ The public functions are in [`root.zig`](root.zig) and run in this order: ``` Scene -> prepare -> warmSessionMemory -> runForwardWithSessionMemory - Prepared SessionMemory SpectrumRunResult - (scene + tables) (reused memory) (radiance, reflectance, - irradiance, Jacobian) + | | | + v v v + Prepared SessionMemory SpectrumRunResult ``` - A caller-provided or parsed JSON scene gives a `Scene` — atmosphere, geometry, @@ -24,8 +24,9 @@ Scene -> prepare -> warmSessionMemory -> runForwardWithSessionMemory - `prepare` builds the physics tables that stay fixed across runs of the same scene. - `warmSessionMemory` sets up the memory a run reuses. -- `runForwardWithSessionMemory` runs one spectrum and returns the arrays. `runForward` - does the same with throwaway memory, for a single run. +- `runForwardWithSessionMemory` runs one spectrum and returns the arrays: radiance, + reflectance, irradiance, and the Jacobian. `runForward` does the same with + throwaway memory, for a single run. A retrieval runs this forward pass many times — see below. @@ -47,30 +48,39 @@ Inside one forward pass, data flows one way: - `spectrum/` — combine samples into radiance, average through the instrument slit. - `output/` — diagnostics and the written-out spectrum. -A retrieval wraps the whole pass and repeats it: +`input/` and `setup/` run once. Then `spectrum/` loops the wavelengths, calling +`optics/` and `rtm/` for each one, and averages the per-wavelength results into the +spectrum: ``` -+------------------------------------------------------+ -| retrieval/ Rodgers optimal estimation | -| | -| apply state vector -> run forward model + Jacobian | -| -> accumulate normal system -> update the state | -| -> repeat until converged | -+------------------------------------------------------+ - | - | one forward pass per step - v - the input -> output pass above +input -> setup (once per scene) + | + v +spectrum/ + | + | for each wavelength: + | optics/ -> layer optical depths + | rtm/ -> reflectance at that wavelength + | + v +gather -> slit average -> product spectrum + | + v +output ``` -The rest of the directories support the flow rather than sit in it: +`rtm/` works one wavelength at a time and has no notion of the band; `spectrum/` is +the loop that turns those single wavelengths into a spectrum. + +A retrieval wraps the whole pass and repeats it. The rest of the directories support +the flow: - `cache/` — the reused memory (see below). - `common/` — shared helpers: errors, hashing, math, units, memory, workers. - `assets/` — readers for the packaged reference data: HITRAN O2 line lists, O2–O2 CIA tables, solar spectra, and atmosphere profiles. - `api/` — the C entry points the Python package calls. -- `validation/` — O2 A band metrics. +- `validation/` — band metrics. - `instrumentation/` — tracing and telemetry hooks, off by default. ## Reused memory diff --git a/src/api/c.zig b/src/api/c.zig index 2d2f2228f..62e97c56c 100644 --- a/src/api/c.zig +++ b/src/api/c.zig @@ -13,9 +13,9 @@ pub const links_libc = builtin.link_libc; // python/zdisamar/bindings/signatures.py binds the exported `zds_*` symbols with ctypes. | // | // boundary | -// Context owns prepared setup tables, reusable O2 session memory, returned spectrum handles, and error | +// Context owns prepared setup tables, reusable session memory, returned spectrum handles, and error | // text. Compute receives only the public root inputs: Prepared, SessionMemory, and SolveConfig. | -// JSON parsing, diagnostic tables, retrieval, and fastmode return typed failures until their O2 A/O2 A | +// JSON parsing, diagnostic tables, retrieval, and fastmode return typed failures until they land | // ------------------------------------------------------------------------------------------------------------| pub const ZdsStatus = enum(c_int) { @@ -74,40 +74,43 @@ pub const ZdsDiagnosticReport = extern struct { }; // ------------------------------------------------------------------------------------------------------------| -// ZdsOptimalEstimationStateSpec ------------------------------------------------------------------------------| -// One C-facing retrieval-state control row. Optional pressure-profile pointers borrow caller buffers. | +// ZdsOptimalEstimationScalarSpec -----------------------------------------------------------------------------| +// One C-facing scalar control field for a named fixed-state retrieval lane. | // | // layout(64-bit) | -// size: 80 B (0.078 KiB), align: 8 B | +// size: 48 B (0.047 KiB), align: 8 B | // | // memory | -// [ 0.. 0] state_id : u8 | -// [ 1.. 1] has_lower : u8 | -// [ 2.. 2] has_upper : u8 | -// [ 3.. 3] padding : 1 B | -// [ 4.. 7] interval_index_1based : u32 | -// [ 8..15] initial : f64 | -// [16..23] prior : f64 | -// [24..31] variance : f64 | -// [32..39] lower : f64 | -// [40..47] upper : f64 | -// [48..55] thickness_hpa : f64 | -// [56..63] pressure_profile_count : usize | -// [64..71] pressure_profile_altitude_km : ?[*]const f64 | -// [72..79] pressure_profile_pressure_hpa : ?[*]const f64 | -// | -// referenced storage | -// pressure-profile arrays are borrowed only while the request converts into native state specs. | -pub const ZdsOptimalEstimationStateSpec = extern struct { - state_id: u8 = 0, +// [ 0.. 0] has_lower : u8 | +// [ 1.. 1] has_upper : u8 | +// [ 2.. 7] padding : 6 B | +// [ 8..15] initial : f64 | +// [16..23] prior : f64 | +// [24..31] variance : f64 | +// [32..39] lower : f64 | +// [40..47] upper : f64 | +pub const ZdsOptimalEstimationScalarSpec = extern struct { has_lower: u8 = 0, has_upper: u8 = 0, - interval_index_1based: u32 = 0, initial: f64 = 0.0, prior: f64 = 0.0, variance: f64 = 0.0, lower: f64 = 0.0, upper: f64 = 0.0, +}; +// ------------------------------------------------------------------------------------------------------------| + +// ZdsOptimalEstimationPressureSpec ---------------------------------------------------------------------------| +// C-facing pressure retrieval lane: scalar controls plus required layer-placement metadata. | +// | +// layout(64-bit) | +// size: 88 B (0.086 KiB), align: 8 B | +// | +// referenced storage | +// pressure-profile arrays are borrowed only while the request converts into the native fixed state. | +pub const ZdsOptimalEstimationPressureSpec = extern struct { + scalar: ZdsOptimalEstimationScalarSpec = .{}, + interval_index_1based: u32 = 0, thickness_hpa: f64 = 0.0, pressure_profile_count: usize = 0, pressure_profile_altitude_km: ?[*]const f64 = null, @@ -133,29 +136,29 @@ pub const ZdsOptimalEstimationControls = extern struct { // ------------------------------------------------------------------------------------------------------------| // ZdsOptimalEstimationRequest --------------------------------------------------------------------------------| -// Single-run retrieval request with borrowed measurement arrays and state rows. | +// Single-run retrieval request with borrowed measurement arrays and named fixed-state fields. | // | // layout(64-bit) | -// size: 72 B (0.070 KiB), align: 8 B | +// size: 192 B (0.188 KiB), align: 8 B | // | // memory | -// [ 0.. 7] sample_count : usize | -// [ 8..15] wavelength_nm : ?[*]const f64 | -// [16..23] reflectance : ?[*]const f64 | -// [24..31] variance : ?[*]const f64 | -// [32..39] state_count : usize | -// [40..47] states : ?[*]const ZdsOptimalEstimationStateSpec | -// [48..71] controls : ZdsOptimalEstimationControls | +// [ 0.. 7] sample_count : usize | +// [ 8.. 15] wavelength_nm : ?[*]const f64 | +// [ 16.. 23] reflectance : ?[*]const f64 | +// [ 24.. 31] variance : ?[*]const f64 | +// [ 32.. 79] aerosol_optical_depth : ZdsOptimalEstimationScalarSpec | +// [ 80..167] aerosol_layer_pressure : ZdsOptimalEstimationPressureSpec | +// [168..191] controls : ZdsOptimalEstimationControls | // | // referenced storage | -// measurement arrays and state specs borrow caller buffers for the duration of the call. | +// measurement arrays and pressure-profile rows borrow caller buffers for the duration of the call. | pub const ZdsOptimalEstimationRequest = extern struct { sample_count: usize = 0, wavelength_nm: ?[*]const f64 = null, reflectance: ?[*]const f64 = null, variance: ?[*]const f64 = null, - state_count: usize = 0, - states: ?[*]const ZdsOptimalEstimationStateSpec = null, + aerosol_optical_depth: ZdsOptimalEstimationScalarSpec = .{}, + aerosol_layer_pressure: ZdsOptimalEstimationPressureSpec = .{}, controls: ZdsOptimalEstimationControls = .{}, }; // ------------------------------------------------------------------------------------------------------------| @@ -206,33 +209,33 @@ pub const ZdsOptimalEstimationResult = extern struct { // ------------------------------------------------------------------------------------------------------------| // ZdsOptimalEstimationBatchRequest ---------------------------------------------------------------------------| -// Multi-run retrieval request sharing one measurement grid and one state template across run-specific priors. | +// Multi-run retrieval request sharing one measurement grid and one fixed state template across run priors. | // | // layout(64-bit) | -// size: 104 B (0.102 KiB), align: 8 B | +// size: 224 B (0.219 KiB), align: 8 B | // | // memory | -// [ 0.. 7] sample_count : usize | -// [ 8.. 15] wavelength_nm : ?[*]const f64 | -// [ 16.. 23] reflectance : ?[*]const f64 | -// [ 24.. 31] variance : ?[*]const f64 | -// [ 32.. 39] state_count : usize | -// [ 40.. 47] state_template : ?[*]const ZdsOptimalEstimationStateSpec | -// [ 48.. 55] run_count : usize | -// [ 56.. 63] initial : ?[*]const f64 | -// [ 64.. 71] prior : ?[*]const f64 | -// [ 72.. 95] controls : ZdsOptimalEstimationControls | -// [ 96..103] batch_worker_count : usize | +// [ 0.. 7] sample_count : usize | +// [ 8.. 15] wavelength_nm : ?[*]const f64 | +// [ 16.. 23] reflectance : ?[*]const f64 | +// [ 24.. 31] variance : ?[*]const f64 | +// [ 32.. 79] aerosol_optical_depth : ZdsOptimalEstimationScalarSpec | +// [ 80..167] aerosol_layer_pressure : ZdsOptimalEstimationPressureSpec | +// [168..175] run_count : usize | +// [176..183] initial : ?[*]const f64 | +// [184..191] prior : ?[*]const f64 | +// [192..215] controls : ZdsOptimalEstimationControls | +// [216..223] batch_worker_count : usize | // | // referenced storage | -// measurement arrays, template state rows, and run initial/prior arrays borrow caller buffers. | +// measurement arrays, pressure-profile rows, and run initial/prior arrays borrow caller buffers. | pub const ZdsOptimalEstimationBatchRequest = extern struct { sample_count: usize = 0, wavelength_nm: ?[*]const f64 = null, reflectance: ?[*]const f64 = null, variance: ?[*]const f64 = null, - state_count: usize = 0, - state_template: ?[*]const ZdsOptimalEstimationStateSpec = null, + aerosol_optical_depth: ZdsOptimalEstimationScalarSpec = .{}, + aerosol_layer_pressure: ZdsOptimalEstimationPressureSpec = .{}, run_count: usize = 0, initial: ?[*]const f64 = null, prior: ?[*]const f64 = null, @@ -544,7 +547,7 @@ pub export fn zds_context_destroy(ctx: ?*Context) void { pub export fn zds_prepare_o2a_json(ctx: ?*Context, json_ptr: ?[*]const u8, json_len: usize) c_int { // zds_prepare_o2a_json -----------------------------------------------------------------------------------| - // Parse Python's native O2 A JSON shape and prepare the resulting typed case. | + // Parse Python's native JSON shape and prepare the resulting typed scene. | // --------------------------------------------------------------------------------------------------------| const resolved = ctx orelse return @intFromEnum(ZdsStatus.failure); @@ -580,7 +583,7 @@ pub export fn zds_prepare_o2a_json(ctx: ?*Context, json_ptr: ?[*]const u8, json_ export fn zds_warm_o2a_session(ctx: ?*Context) c_int { // zds_warm_o2a_session -----------------------------------------------------------------------------------| - // Build retained session rows for the prepared O2 A case. | + // Build retained session rows for the prepared scene. | // --------------------------------------------------------------------------------------------------------| const resolved = ctx orelse return @intFromEnum(ZdsStatus.failure); @@ -590,8 +593,8 @@ export fn zds_warm_o2a_session(ctx: ?*Context) c_int { }); var solve_config = zdisamar.solveConfig(prepared.scene); - solve_config.derivative_state_mask = 0; solve_config.derivative_mode = .none; + solve_config.wants_jacobian = false; zdisamar.warmSessionMemory( allocator, @@ -608,11 +611,9 @@ export fn zds_warm_o2a_session(ctx: ?*Context) c_int { pub export fn zds_warm_o2a_optimal_estimation( ctx: ?*Context, - state_ids: ?[*]const u8, - requested_state_count: usize, ) c_int { // zds_warm_o2a_optimal_estimation ------------------------------------------------------------------------| - // Warm the session cache for the semi-analytical Jacobian state mask needed by OE. | + // Warm the session cache for the fixed two-lane semi-analytical Jacobian needed by OE. | // --------------------------------------------------------------------------------------------------------| const resolved = ctx orelse return @intFromEnum(ZdsStatus.failure); @@ -621,14 +622,9 @@ pub export fn zds_warm_o2a_optimal_estimation( return @intFromEnum(ZdsStatus.failure); }); - const selection = jacobianSelection(state_ids, requested_state_count, true) catch |err| { - resolved.setError(@errorName(err)); - return @intFromEnum(ZdsStatus.failure); - }; - var solve_config = zdisamar.solveConfig(prepared.scene); - solve_config.derivative_state_mask = selection.mask; solve_config.derivative_mode = .semi_analytical; + solve_config.wants_jacobian = true; zdisamar.warmSessionMemory( allocator, @@ -645,28 +641,16 @@ pub export fn zds_warm_o2a_optimal_estimation( pub export fn zds_run_spectrum(ctx: ?*Context, out: ?*ZdsSpectrum) c_int { // zds_run_spectrum ---------------------------------------------------------------------------------------| - // Run the prepared case without returning Jacobian columns. | + // Run the prepared scene without returning Jacobian columns. | // --------------------------------------------------------------------------------------------------------| - return runSpectrum(ctx, out, null, 0, false); + return runSpectrum(ctx, out, false); } export fn zds_run_spectrum_jacobian(ctx: ?*Context, out: ?*ZdsSpectrum) c_int { // zds_run_spectrum_jacobian ------------------------------------------------------------------------------| - // Run the prepared case with all fixed Jacobian columns in public order. | - // --------------------------------------------------------------------------------------------------------| - return runSpectrum(ctx, out, null, 0, true); -} - -export fn zds_run_spectrum_jacobian_for_states( - ctx: ?*Context, - out: ?*ZdsSpectrum, - state_ids: ?[*]const u8, - state_count: usize, -) c_int { - // zds_run_spectrum_jacobian_for_states -------------------------------------------------------------------| - // Run the prepared case with a caller-selected compact Jacobian column order. | + // Run the prepared scene with all fixed Jacobian columns in public order. | // --------------------------------------------------------------------------------------------------------| - return runSpectrum(ctx, out, state_ids, state_count, true); + return runSpectrum(ctx, out, true); } export fn zds_spectrum_report(ctx: ?*Context, spectrum: ?*const ZdsSpectrum, out: ?*ZdsDiagnosticReport) c_int { @@ -771,27 +755,26 @@ const OptimalEstimationMeasurementSlices = struct { }; // ------------------------------------------------------------------------------------------------------------| -// OptimalEstimationStateSpecs --------------------------------------------------------------------------------| -// Request-scoped native state specs plus pressure-profile spline scratch. | +// OptimalEstimationFixedState --------------------------------------------------------------------------------| +// Request-scoped fixed native state plus pressure-profile spline scratch. | // | // layout(64-bit) | -// size: 312 B (0.305 KiB), align: 8 B | +// size: 200 B (0.195 KiB), align: 8 B | // | // memory | -// [ 0.. 95] profiles : [2]PressureAltitudeProfile | -// [ 96..303] state_specs: [2]StateSpec | -// [304..311] state_count: usize | +// [ 0..95] profiles: [2]PressureAltitudeProfile | +// [96..199] state : RetrievalState | // | // referenced storage | // Pressure-profile second-derivative arrays are owned until the C call returns and deinit frees them. | -const OptimalEstimationStateSpecs = struct { +// retrievalState refreshes borrowed profile pointers from this owner before native solver calls. | +const OptimalEstimationFixedState = struct { profiles: [zdisamar.optimal_estimation.max_state_count]zdisamar.RetrievalPressureAltitudeProfile = [_]zdisamar.RetrievalPressureAltitudeProfile{.{}} ** zdisamar.optimal_estimation.max_state_count, - state_specs: [zdisamar.optimal_estimation.max_state_count]zdisamar.RetrievalStateSpec = undefined, - state_count: usize = 0, + state: zdisamar.RetrievalState = undefined, - fn deinit(self: *OptimalEstimationStateSpecs) void { - // OptimalEstimationStateSpecs.deinit -----------------------------------------------------------------| + fn deinit(self: *OptimalEstimationFixedState) void { + // OptimalEstimationFixedState.deinit -----------------------------------------------------------------| // Release request-scoped pressure-profile spline scratch. | // ----------------------------------------------------------------------------------------------------| for (&self.profiles) |*profile| { @@ -802,11 +785,13 @@ const OptimalEstimationStateSpecs = struct { } } - fn slice(self: *const OptimalEstimationStateSpecs) []const zdisamar.RetrievalStateSpec { - // OptimalEstimationStateSpecs.slice ------------------------------------------------------------------| - // Return the active leading state-spec rows in caller order. | + fn retrievalState(self: *const OptimalEstimationFixedState) zdisamar.RetrievalState { + // OptimalEstimationFixedState.retrievalState ---------------------------------------------------------| + // Return the fixed native state parsed from the named C request fields. | // ----------------------------------------------------------------------------------------------------| - return self.state_specs[0..self.state_count]; + var state = self.state; + state.aerosol_layer_mid_pressure.placement.pressure_altitude_profile = &self.profiles[1]; + return state; } }; // ------------------------------------------------------------------------------------------------------------| @@ -820,17 +805,17 @@ const OptimalEstimationStateSpecs = struct { // memory | // [ 0.. 47] measurement: OptimalEstimationMeasurementSlices | // [ 48.. 71] controls : Controls | -// [ 72..383] state_specs: OptimalEstimationStateSpecs | +// [ 72..383] fixed_state: OptimalEstimationFixedState | const OptimalEstimationRequestView = struct { measurement: OptimalEstimationMeasurementSlices, controls: zdisamar.optimal_estimation.Controls, - state_specs: OptimalEstimationStateSpecs, + fixed_state: OptimalEstimationFixedState, fn deinit(self: *OptimalEstimationRequestView) void { // OptimalEstimationRequestView.deinit ----------------------------------------------------------------| - // Release request-scoped state-spec side data. | + // Release request-scoped fixed-state side data. | // ----------------------------------------------------------------------------------------------------| - self.state_specs.deinit(); + self.fixed_state.deinit(); } }; // ------------------------------------------------------------------------------------------------------------| @@ -844,7 +829,7 @@ const OptimalEstimationRequestView = struct { // memory | // [ 0.. 47] measurement : OptimalEstimationMeasurementSlices | // [ 48.. 71] controls : Controls | -// [ 72..383] state_template : OptimalEstimationStateSpecs | +// [ 72..383] state_template : OptimalEstimationFixedState | // [384..399] initial : []const f64 | // [400..415] prior : []const f64 | // [416..423] run_count : usize | @@ -854,7 +839,7 @@ const OptimalEstimationRequestView = struct { const OptimalEstimationBatchRequestView = struct { measurement: OptimalEstimationMeasurementSlices, controls: zdisamar.optimal_estimation.Controls, - state_template: OptimalEstimationStateSpecs, + state_template: OptimalEstimationFixedState, initial: []const f64, prior: []const f64, run_count: usize, @@ -911,7 +896,7 @@ pub export fn zds_run_o2a_optimal_estimation( request_view.measurement.wavelength_nm, request_view.measurement.reflectance, request_view.measurement.variance, - request_view.state_specs.slice(), + request_view.fixed_state.retrievalState(), request_view.controls, ) catch |err| { resolved.setError(@errorName(err)); @@ -935,7 +920,7 @@ pub export fn zds_run_o2a_optimal_estimation_correction( out: ?*ZdsOptimalEstimationResult, ) c_int { // zds_run_o2a_optimal_estimation_correction --------------------------------------------------------------| - // Run one prepared-case full-physics correction step and return a Context-owned result handle. | + // Run one prepared-scene full-physics correction step and return a Context-owned result handle. | // --------------------------------------------------------------------------------------------------------| const resolved = ctx orelse return @intFromEnum(ZdsStatus.failure); @@ -969,7 +954,7 @@ pub export fn zds_run_o2a_optimal_estimation_correction( request_view.measurement.wavelength_nm, request_view.measurement.reflectance, request_view.measurement.variance, - request_view.state_specs.slice(), + request_view.fixed_state.retrievalState(), request_view.controls, ) catch |err| { resolved.setError(@errorName(err)); @@ -1027,7 +1012,7 @@ pub export fn zds_run_o2a_optimal_estimation_batch( request_view.measurement.wavelength_nm, request_view.measurement.reflectance, request_view.measurement.variance, - request_view.state_template.slice(), + request_view.state_template.retrievalState(), request_view.initial, request_view.prior, request_view.controls, @@ -1097,9 +1082,7 @@ pub export fn zds_run_o2a_fastmode_optimal_estimation_batch( }; defer correction_view.deinit(); - if (fast_view.state_count != correction_view.state_count or - fast_view.run_count != correction_view.run_count) - { + if (fast_view.run_count != correction_view.run_count) { resolved.setError("fastmode request shapes do not match"); return @intFromEnum(ZdsStatus.failure); } @@ -1117,7 +1100,7 @@ pub export fn zds_run_o2a_fastmode_optimal_estimation_batch( fast_view.measurement.wavelength_nm, fast_view.measurement.reflectance, fast_view.measurement.variance, - fast_view.state_template.slice(), + fast_view.state_template.retrievalState(), fast_view.initial, fast_view.prior, fast_view.controls, @@ -1125,7 +1108,7 @@ pub export fn zds_run_o2a_fastmode_optimal_estimation_batch( correction_view.measurement.wavelength_nm, correction_view.measurement.reflectance, correction_view.measurement.variance, - correction_view.state_template.slice(), + correction_view.state_template.retrievalState(), correction_view.prior, correction_view.controls, ) catch |err| { @@ -1268,19 +1251,19 @@ fn optimalEstimationRequestView( // --------------------------------------------------------------------------------------------------------| const resolved_request = request orelse { resolved.setError("null optimal-estimation request"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; const measurement = optimalEstimationMeasurementSlices(resolved, resolved_request) orelse { return error.InvalidMeasurement; }; const controls = optimalEstimationControls(resolved, resolved_request) orelse { - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; - var state_specs = try optimalEstimationStateSpecs(resolved, resolved_request); - errdefer state_specs.deinit(); + var fixed_state = try optimalEstimationFixedState(resolved, resolved_request); + errdefer fixed_state.deinit(); - zdisamar.optimal_estimation.validateStateSpecs(state_specs.slice()) catch |err| { + zdisamar.optimal_estimation.validateRetrievalState(fixed_state.retrievalState()) catch |err| { resolved.setError(@errorName(err)); return err; }; @@ -1288,7 +1271,7 @@ fn optimalEstimationRequestView( return .{ .measurement = measurement, .controls = controls, - .state_specs = state_specs, + .fixed_state = fixed_state, }; } @@ -1304,57 +1287,48 @@ fn optimalEstimationBatchRequestView( // --------------------------------------------------------------------------------------------------------| const resolved_request = request orelse { resolved.setError("null optimal-estimation batch request"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; const measurement = optimalEstimationMeasurementSlices(resolved, resolved_request) orelse { return error.InvalidMeasurement; }; const controls = optimalEstimationControls(resolved, resolved_request) orelse { - return error.InvalidStateSpec; - }; - - const state_template_ptr = resolved_request.state_template orelse { - resolved.setError("null state template"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; const run_count = resolved_request.run_count; - const state_count = resolved_request.state_count; + const state_count = zdisamar.optimal_estimation.max_state_count; if (run_count == 0) { resolved.setError("empty optimal-estimation batch"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } const batch_worker_count = resolved_request.batch_worker_count; if (batch_worker_count != 1) { resolved.setError("invalid optimal-estimation batch_worker_count"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } const total_state_count = std.math.mul(usize, run_count, state_count) catch { resolved.setError("optimal-estimation batch is too large"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; const initial_ptr = resolved_request.initial orelse { resolved.setError("null batch initial states"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; const prior_ptr = resolved_request.prior orelse { resolved.setError("null batch prior states"); - return error.InvalidStateSpec; + return error.InvalidRetrievalState; }; - var state_template = try optimalEstimationStateSpecsFromRaw( - resolved, - state_count, - state_template_ptr, - ); + var state_template = try optimalEstimationFixedStateFromRaw(resolved, resolved_request); errdefer state_template.deinit(); - zdisamar.optimal_estimation.validateStateSpecs(state_template.slice()) catch |err| { + zdisamar.optimal_estimation.validateRetrievalState(state_template.retrievalState()) catch |err| { resolved.setError(@errorName(err)); return err; }; @@ -1427,94 +1401,81 @@ fn optimalEstimationControls( }; } -fn optimalEstimationStateSpecs( +fn optimalEstimationFixedState( resolved: *Context, request: *const ZdsOptimalEstimationRequest, -) !OptimalEstimationStateSpecs { - // optimalEstimationStateSpecs ----------------------------------------------------------------------------| - // Convert the single-run state row pointer into native fixed-state rows. | +) !OptimalEstimationFixedState { + // optimalEstimationFixedState ----------------------------------------------------------------------------| + // Convert the single-run fixed state fields into native fixed-state rows. | // --------------------------------------------------------------------------------------------------------| - const state_specs_ptr = request.states orelse { - resolved.setError("null state specs"); - return error.InvalidStateSpec; - }; - return optimalEstimationStateSpecsFromRaw(resolved, request.state_count, state_specs_ptr); + return optimalEstimationFixedStateFromRaw(resolved, request); } -fn optimalEstimationStateSpecsFromRaw( +fn optimalEstimationFixedStateFromRaw( resolved: *Context, - state_count: usize, - state_specs_ptr: [*]const ZdsOptimalEstimationStateSpec, -) !OptimalEstimationStateSpecs { - // optimalEstimationStateSpecsFromRaw ---------------------------------------------------------------------| - // Convert raw C rows into native StateSpec rows, owning only pressure-profile spline curvature. | + request: anytype, +) !OptimalEstimationFixedState { + // optimalEstimationFixedStateFromRaw ---------------------------------------------------------------------| + // Convert named C fixed-state fields into native retrieval state plus pressure-profile curvature. | // | - // guard | - // Public state ids are exactly 0 aerosol optical depth and 1 aerosol-layer pressure; id 2 was the | - // removed surface-albedo lane and is rejected before any native state-space math sees it. | // --------------------------------------------------------------------------------------------------------| - if (state_count == 0 or state_count > zdisamar.optimal_estimation.max_state_count) { - resolved.setError("invalid state count"); - return error.InvalidStateCount; - } - - var parsed: OptimalEstimationStateSpecs = .{ .state_count = state_count }; + var parsed: OptimalEstimationFixedState = .{}; errdefer parsed.deinit(); - const raw_states = state_specs_ptr[0..state_count]; - for (raw_states, 0..) |raw, index| { - if (raw.state_id >= zdisamar.jacobian_state_count) { - resolved.setError("UnsupportedState"); - return error.UnsupportedState; - } - - const state = std.meta.intToEnum(zdisamar.JacobianState, raw.state_id) catch unreachable; - const has_pressure_profile_side_data = - raw.pressure_profile_count != 0 or - raw.pressure_profile_altitude_km != null or - raw.pressure_profile_pressure_hpa != null; - - if (state == .aerosol_layer_mid_pressure_hpa) { - parsed.profiles[index] = try optimalEstimationPressureProfile(resolved, raw); - } else if (has_pressure_profile_side_data) { - resolved.setError("invalid pressure profile state"); - return error.InvalidStateSpec; - } - - if (raw.has_lower != 0 and !std.math.isFinite(raw.lower)) { - resolved.setError("invalid optimal-estimation lower bound"); - return error.InvalidStateSpec; - } + parsed.profiles[1] = try optimalEstimationPressureProfile(resolved, request.aerosol_layer_pressure); + parsed.state = .{ + .aerosol_optical_depth = try optimalEstimationScalarSpec( + resolved, + request.aerosol_optical_depth, + ), + .aerosol_layer_mid_pressure = .{ + .scalar = try optimalEstimationScalarSpec( + resolved, + request.aerosol_layer_pressure.scalar, + ), + .placement = .{ + .thickness_hpa = request.aerosol_layer_pressure.thickness_hpa, + .interval_index_1based = request.aerosol_layer_pressure.interval_index_1based, + .pressure_altitude_profile = &parsed.profiles[1], + }, + }, + }; + try zdisamar.optimal_estimation.validateRetrievalState(parsed.retrievalState()); + return parsed; +} - if (raw.has_upper != 0 and !std.math.isFinite(raw.upper)) { - resolved.setError("invalid optimal-estimation upper bound"); - return error.InvalidStateSpec; - } +fn optimalEstimationScalarSpec( + resolved: *Context, + raw: ZdsOptimalEstimationScalarSpec, +) !zdisamar.RetrievalStateScalar { + // optimalEstimationScalarSpec ----------------------------------------------------------------------------| + // Convert one named C scalar field into the native fixed-lane scalar. | + // --------------------------------------------------------------------------------------------------------| + if (raw.has_lower != 0 and !std.math.isFinite(raw.lower)) { + resolved.setError("invalid optimal-estimation lower bound"); + return error.InvalidRetrievalState; + } - const lower_bound = - if (raw.has_lower != 0) raw.lower else zdisamar.optimal_estimation.no_lower_bound; - const upper_bound = - if (raw.has_upper != 0) raw.upper else zdisamar.optimal_estimation.no_upper_bound; - - parsed.state_specs[index] = .{ - .state = state, - .initial = raw.initial, - .prior = raw.prior, - .variance = raw.variance, - .lower_bound = lower_bound, - .upper_bound = upper_bound, - .thickness_hpa = raw.thickness_hpa, - .interval_index_1based = raw.interval_index_1based, - .pressure_altitude_profile = parsed.profiles[index], - }; + if (raw.has_upper != 0 and !std.math.isFinite(raw.upper)) { + resolved.setError("invalid optimal-estimation upper bound"); + return error.InvalidRetrievalState; } - return parsed; + const lower_bound = if (raw.has_lower != 0) raw.lower else zdisamar.optimal_estimation.no_lower_bound; + const upper_bound = if (raw.has_upper != 0) raw.upper else zdisamar.optimal_estimation.no_upper_bound; + + return .{ + .initial = raw.initial, + .prior = raw.prior, + .variance = raw.variance, + .lower_bound = lower_bound, + .upper_bound = upper_bound, + }; } fn optimalEstimationPressureProfile( resolved: *Context, - raw: ZdsOptimalEstimationStateSpec, + raw: ZdsOptimalEstimationPressureSpec, ) !zdisamar.RetrievalPressureAltitudeProfile { // optimalEstimationPressureProfile -----------------------------------------------------------------------| // Build the request-scoped pressure-profile spline for aerosol-layer pressure retrieval rows. | @@ -1626,7 +1587,7 @@ fn runWavelengthDiagnostic( }; output.* = .{}; - var result = buildWavelengthDiagnostic( + const result = buildWavelengthDiagnostic( diagnostic, prepared, wavelengths_ptr[0..wavelength_count], @@ -1635,10 +1596,8 @@ fn runWavelengthDiagnostic( resolved.setError(@errorName(err)); return @intFromEnum(ZdsStatus.failure); }; - errdefer result.deinit(allocator); - marshalWavelengthDiagnostic(diagnostic, output, &result); - result.rows = &.{}; + publishWavelengthDiagnostic(diagnostic, output, result); resolved.setError(""); return @intFromEnum(ZdsStatus.ok); } @@ -1672,12 +1631,12 @@ fn buildWavelengthDiagnostic( }; } -fn marshalWavelengthDiagnostic( +fn publishWavelengthDiagnostic( comptime diagnostic: WavelengthDiagnostic, output: *DiagnosticOutput(diagnostic), - result: *const DiagnosticResult(diagnostic), + result: DiagnosticResult(diagnostic), ) void { - // marshalWavelengthDiagnostic --------------------------------------------------------------------------- | + // publishWavelengthDiagnostic --------------------------------------------------------------------------- | // Move native row ownership to the stable C-facing output struct without copying rows. | // --------------------------------------------------------------------------------------------------------| switch (diagnostic) { @@ -1713,8 +1672,6 @@ fn diagnosticRowsPointer(comptime Row: type, rows: []const Row) ?[*]const Row { fn runSpectrum( ctx: ?*Context, out: ?*ZdsSpectrum, - state_ids: ?[*]const u8, - requested_state_count: usize, wants_jacobian: bool, ) c_int { // runSpectrum --------------------------------------------------------------------------------------------| @@ -1732,14 +1689,9 @@ fn runSpectrum( return @intFromEnum(ZdsStatus.failure); }); - const selection = jacobianSelection(state_ids, requested_state_count, wants_jacobian) catch |err| { - resolved.setError(@errorName(err)); - return @intFromEnum(ZdsStatus.failure); - }; - var solve_config = zdisamar.solveConfig(prepared.scene); - solve_config.derivative_state_mask = selection.mask; solve_config.derivative_mode = if (wants_jacobian) .semi_analytical else .none; + solve_config.wants_jacobian = wants_jacobian; const result = allocator.create(CResult) catch |err| { resolved.setError(@errorName(err)); @@ -1758,17 +1710,15 @@ fn runSpectrum( }; errdefer result.native.deinit(allocator); - if (selection.state_count != 0) { - const selected_ids = selection.ids[0..selection.state_count]; - result.compact_jacobian = compactRadianceJacobian( + if (wants_jacobian) { + result.compact_jacobian = radianceJacobianRows( result.native.spectrum, - selected_ids, solarMu0(prepared.scene), ) catch |err| { resolved.setError(@errorName(err)); return @intFromEnum(ZdsStatus.failure); }; - result.state_count = selection.state_count; + result.state_count = zdisamar.jacobian_state_count; } errdefer allocator.free(result.compact_jacobian); @@ -1791,71 +1741,27 @@ fn runSpectrum( return @intFromEnum(ZdsStatus.ok); } -const JacobianSelection = struct { - ids: [zdisamar.jacobian_state_count]u8 = .{0} ** zdisamar.jacobian_state_count, - state_count: usize = 0, - mask: u8 = 0, -}; - -fn jacobianSelection( - state_ids: ?[*]const u8, - requested_state_count: usize, - wants_jacobian: bool, -) !JacobianSelection { - // jacobianSelection --------------------------------------------------------------------------------------| - // Validate Python state ids and build the fixed root SolveConfig mask plus compact output order. | - // --------------------------------------------------------------------------------------------------------| - if (!wants_jacobian) return .{}; - if (requested_state_count > zdisamar.jacobian_state_count) return error.UnsupportedJacobianState; - - var selection = JacobianSelection{}; - if (requested_state_count == 0) { - for (0..zdisamar.jacobian_state_count) |index| { - selection.ids[index] = @intCast(index); - selection.mask |= @as(u8, 1) << @intCast(index); - } - selection.state_count = zdisamar.jacobian_state_count; - return selection; - } - - const ids = state_ids orelse return error.UnsupportedJacobianState; - - for (ids[0..requested_state_count]) |id| { - if (id >= zdisamar.jacobian_state_count) return error.UnsupportedJacobianState; - - const bit = @as(u8, 1) << @intCast(id); - if ((selection.mask & bit) != 0) return error.UnsupportedJacobianState; - - selection.ids[selection.state_count] = id; - selection.state_count += 1; - selection.mask |= bit; - } - - return selection; -} - -fn compactRadianceJacobian( +fn radianceJacobianRows( spectrum: zdisamar.Spectrum, - state_ids: []const u8, solar_mu0: f64, ) ![]f64 { - // compactRadianceJacobian --------------------------------------------------------------------------------| - // Convert native reflectance Jacobian rows into Python's compact radiance-Jacobian ABI table. | + // radianceJacobianRows -----------------------------------------------------------------------------------| + // Convert native reflectance Jacobian rows into Python's fixed two-column radiance-Jacobian ABI table. | // | // boundary | // Python `Spectrum.reflectance_jacobian()` treats the C `jacobian` pointer as dL/dx and divides by | // mu0 * irradiance / pi. Internal root output already stores dR/dx after reflectance assembly, so the | // C boundary inverts that scale exactly once before exposing the fixed ABI buffer. | // --------------------------------------------------------------------------------------------------------| - const state_count = state_ids.len; + const state_count = zdisamar.jacobian_state_count; if (spectrum.jacobian.len != spectrum.irradiance.len) return error.ShapeMismatch; const compact = try allocator.alloc(f64, spectrum.jacobian.len * state_count); for (spectrum.jacobian, spectrum.irradiance, 0..) |row, irradiance, sample_index| { const reflectance_to_radiance_scale = solar_mu0 * irradiance / std.math.pi; - for (state_ids, 0..) |state_id, compact_index| { - compact[sample_index * state_count + compact_index] = - row[state_id] * reflectance_to_radiance_scale; + for (0..state_count) |state_index| { + compact[sample_index * state_count + state_index] = + row[state_index] * reflectance_to_radiance_scale; } } return compact; diff --git a/src/assets/readers.zig b/src/assets/readers.zig index a7fbea16f..f7d6b5ba7 100644 --- a/src/assets/readers.zig +++ b/src/assets/readers.zig @@ -6,7 +6,7 @@ const units = @import("../common/units.zig"); const Allocator = std.mem.Allocator; // VendorBranchMetadata ---------------------------------------------------------------------------------------| -// Optional O2 A branch metadata uses partition weak lines from LISA strong-line sidecars. | +// Optional branch metadata uses partition weak lines from LISA strong-line sidecars. | // | // layout(64-bit) | // size: 7 B (0.007 KiB), align: 1 B | @@ -24,7 +24,7 @@ const VendorBranchMetadata = struct { fn present(self: VendorBranchMetadata) bool { // VendorBranchMetadata.present -----------------------------------------------------------------------| - // True when all O2 A branch fields were recovered from the HITRAN row. | + // True when all branch fields were recovered from the HITRAN row. | // ----------------------------------------------------------------------------------------------------| return self.branch_ic1 != null and self.branch_ic2 != null and self.rotational_nf != null; } @@ -32,7 +32,7 @@ const VendorBranchMetadata = struct { // ------------------------------------------------------------------------------------------------------------| // readers.zig ------------------------------------------------------------------------------------------------| -// Asset readers for O2 A setup tables. | +// Asset readers for setup tables. | // | // file boundary | // Runtime file I/O and text parsing live here. Setup builders call these functions before table assembly. | @@ -40,7 +40,7 @@ const VendorBranchMetadata = struct { // | // supported formats | // profile_csv : altitude, pressure, temperature, air-number-density CSV rows | -// hitran_par_o2a : fixed-width HITRAN line rows for the O2 A line list | +// hitran_par_o2a : fixed-width HITRAN line rows for the O2A line list | // lisa_sdf : LISA/DISAMAR O2 strong-line sidecar rows | // lisa_rmf : LISA/DISAMAR O2 relaxation-matrix sidecar rows | // bira_cia : BIRA O2-O2 polynomial table with scale/header rows | @@ -415,7 +415,7 @@ pub fn readCiaTable(allocator: Allocator, path: []const u8) !CiaAsset { // readCiaTable -------------------------------------------------------------------------------------------| // Parse the BIRA header scale and every numeric polynomial coefficient row into an owned table. | // | - // parseBiraCiaPolynomial: the vendor count is a lower-bound check, while appended O2 A rows after the | + // parseBiraCiaPolynomial: the vendor count is a lower-bound check, while appended rows after the | // nominal count are retained because the optical route samples them at 758-776 nm. | // --------------------------------------------------------------------------------------------------------| const bytes = try readSmallFile(allocator, path, 4 << 20); @@ -558,7 +558,7 @@ fn parseOptionalFixedInt(comptime T: type, value: []const u8) !?T { fn inlineVendorBranchMetadata(line: []const u8) !VendorBranchMetadata { // inlineVendorBranchMetadata -----------------------------------------------------------------------------| - // Recover optional O2 A branch metadata from fixed HITRAN vendor columns when present. | + // Recover optional branch metadata from fixed HITRAN vendor columns when present. | // --------------------------------------------------------------------------------------------------------| if (line.len < 85) return .{}; @@ -575,7 +575,7 @@ fn inlineVendorBranchMetadata(line: []const u8) !VendorBranchMetadata { fn fallbackVendorBranchMetadata(line: []const u8, center_wavenumber_cm1: f64) !?VendorBranchMetadata { // fallbackVendorBranchMetadata ---------------------------------------------------------------------------| - // Reconstruct O2 A P-branch metadata from trailing branch tokens when fixed fields are absent. | + // Reconstruct P-branch metadata from trailing branch tokens when fixed fields are absent. | // --------------------------------------------------------------------------------------------------------| if (center_wavenumber_cm1 < 12800.0 or center_wavenumber_cm1 > 13250.0) return null; diff --git a/src/cache/README.md b/src/cache/README.md new file mode 100644 index 000000000..ba1bbeeb7 --- /dev/null +++ b/src/cache/README.md @@ -0,0 +1,184 @@ +# `cache/` — the memory a retrieval reuses + +A retrieval runs the forward model many times, and between runs only the aerosol +state changes. Everything that does not depend on that state, the wavelength grid, +the gas spectroscopy, the solar flux, the radiative-transfer scratch, is built +once and reused. This directory holds that memory. The parent `src/README.md` +lists what a session carries; this file covers where each buffer is allocated, +how long it lives, and when it is touched. + +The subsystem reaches a steady state where it allocates nothing. A buffer grows to +the size it needs on the first run, or rebuilds when its inputs change, then is +overwritten in place on every later run and every retrieval iteration. All of it +lives on the one allocator the C API `Context` passes down. No module keeps a +private allocator, with one scoped exception covered below. + +``` + +------------------------------------------------------------+ + | session allocator (one GeneralPurposeAllocator) | + +------------------------------+-----------------------------+ + | + v + +------------------------------------------------------------+ + | SessionMemory one per session | + +------------------------------------------------------------+ + | spectrum sampling rows, slit offsets and weights | + | radiance wavelength list, radiance, Jacobian | + | profile_lines O2 line cross-sections (the largest) | + | solar_irradiance wavelength to solar-flux memo map | + | transport_workers per-worker rtm/ scratch, one per thread | + | worker_pool helper threads for the dense loop | + +------------------------------------------------------------+ + freed once, in reverse order, at SessionMemory.deinit +``` + +## The owner and reuse (`session_memory.zig`) + +`SessionMemory` is the top-level owner, a 416-byte value that holds the six +children inline and allocates nothing itself. `init` constructs +them empty and passes the allocator only to the solar map; `deinit` frees them in +reverse dependency order, the per-worker buffers first and the spectrum rows last. +Keeping the children as fields between requests removes the per-request rebuild a +retrieval would otherwise pay. + +Three kinds of memory sit underneath it: + +1. Retained workspace, grown once and reused in place. This holds nearly the whole + footprint. +2. One scoped arena, alive only while the line cross-sections build. +3. Stack scratch on the per-wavelength read path, which touches no heap. + +`ensureSliceCapacity` (`common/memory.zig`) decides on each run whether to keep a +buffer or replace it: + +``` + ensureSliceCapacity(slice, n): + + n <= slice.len -> keep the buffer, hand it back (no alloc) + n > slice.len -> allocate n, free the old buffer, (one alloc) + swap the pointer +``` + +Callers refill the whole prefix each run, so the discarded contents cost nothing, +and a buffer settles at the largest size any run has asked for. Every buffer is +freed together at `SessionMemory.deinit`. + +A retrieval stays cheap because most inputs hold still while the aerosol state +moves. Each retained table carries a `hashing.ReuseStamp`, and `prepareSessionRows` +in `root.zig` hashes the inputs a table depends on and compares: a match reuses the +table without allocating, a miss rebuilds it. + +## Spectrum sampling table (`spectrum_memory.zig`) + +`SpectrumMemory` holds the high-resolution sampling rows and two side arrays, the +offsets and weights that average the fine grid down to each product wavelength. A +miss replaces it: `rebuildTable` frees the previous arrays, adopts the freshly +built row and side-array slices without copying, and records the new stamp. A hit +returns borrowed slices of the retained arrays through `table()`. The two side +arrays are the large part of this table. + +## Radiance work (`radiance_memory.zig`) + +`RadianceMemory` holds two parts with different reuse rules. The wavelength list, +the rows, sample indices, and dense wavelengths the radiative transfer runs over, +is adopted by `rebuildWavelengthList`, which frees the old generation and is +gated by `wavelength_stamp`. The dense radiance column and the optional Jacobian +column grow through `ensureResultCapacity`, and the Jacobian column is freed when +a run wants no derivatives. A second stamp, `result_stamp`, records whether the +dense values are still valid and resets whenever the list changes or a column +grows, so stale values are never read against fresh storage. + +## Line cross-sections (`profile_line_memory.zig`) + +`profile_line_memory.zig` holds the largest table and the only arena in the +subsystem. `ProfileLineValues` keeps the O2 line cross-section data: a dense +support-profile sigma column, and, when diagnostics ask for it, a +wavelength-by-layer-node grid. Building it sums Voigt and line-mixing terms over +every line at every wavelength, the most expensive step in setup. + +The build runs only on a stamp miss. `profileLineReuseStamp` hashes the scene id, +the exact wavelengths, the parsed line assets, and the fixed-node spectroscopy +profile, and excludes the dynamic layer grid. This is possible because the line +values sit at fixed spectroscopy nodes and are resampled per pressure state +downstream, so the cache survives the pressure changes a retrieval makes each +iteration. + +The build's scratch runs through one `std.heap.ArenaAllocator`. The layer grid, +the line tables, the cutoff grid, the collected line lists, and every prepared +weak- and strong-line state allocate into it, and the whole arena drops in one +`deinit` when the build returns. The two buffers that outlive the build, the sigma +column and the diagnostic grid, allocate on the session allocator so they reach the +cache. + +The per-wavelength reader, `fillSupportLineSigmaAtWavelengthIndex`, reads the +retained column and uses fixed stack arrays for the spline. It allocates nothing. + +## Solar irradiance (`solar_irradiance_memory.zig`) + +`SolarIrradianceMemory` maps the exact f64 bits of a wavelength to its solar flux. +It grows its capacity once with `ensureTotalCapacity`, then the per-wavelength +irradiance loop fills it with `putAssumeCapacity`, which never allocates, and reads +it with `get`. Between product simulations `reset` empties the entries with +`clearRetainingCapacity` and keeps the capacity, so the next run refills the same +storage. The only free is at teardown. + +## Per-worker radiative-transfer scratch (`transport_worker_memory.zig`) + +Each `TransportWorkerMemory` owns about 27 reused slices: the attenuation tables, +the layer reflection and transmission rows, the scattering-order fields, +the phase rows, the optics rows, and the Fourier basis cache. `ensureCapacity` and +`ensureOpticsCapacity` grow them to their largest needed size once per run, and the +radiative transfer refills and rereads them per wavelength. `solveWorkArrays` hands +that code borrowed prefixes and never allocates. + +Two caches inside survive across wavelengths and rebuild only when the sun and view +geometry change: the Fourier basis, about 14.2 KiB per Fourier term, and the cached +Gauss geometry. When a backing slice is reallocated or the geometry key changes, +the parallel validity flags reset, so the next read rebuilds. This is possible +because the geometry rarely moves across the dense grid, so the basis is built a +few times over a whole run rather than once per wavelength. + +The collection keeps one private `TransportWorkerMemory` per worker thread. +`ensureWorkerCount` grows the outer array by moving the existing workers in by +value, so their buffers ride along, then frees only the old outer array. Each +worker keeps its own buffers and none is shared, so the dense wavelength loop runs +on all workers at once without contention. + +## Worker pool (`forward_worker_pool.zig`) + +`ForwardWorkerPool` owns the `std.Thread.Pool` the dense wavelength loop runs on, +or borrows a shared one. `poolForWorkerCount` returns the owned pool when the +requested worker count matches, and rebuilds it, tearing the old one down first, +when the count changes. Its one allocation is the thread-handle slice inside the +pool. It stores no physics state. + +## Footprint + +For the packaged reference scene a session holds roughly 22 MB, most of the +program's heap, allocated once and reused for its lifetime. A few tables account +for nearly all of it: + +``` + largest first: + + slit offsets and weights ~4.5 MB each + line cross-section column several MB + per-worker scratch ~1.4 MiB each (one per worker thread) + dense radiance + Jacobian ~93 KB + Fourier basis ~14.2 KiB per term +``` + +## Where to start + +- `session_memory.zig` — the owner and the teardown order, the shortest file and + the map of the directory. +- `common/memory.zig` — `ensureSliceCapacity`, the reuse primitive everything here + is built on. +- `profile_line_memory.zig` — the line cross-section cache, the one arena, and the + reuse stamp; read `profileLineReuseStamp` and the build function. +- `transport_worker_memory.zig` — the per-worker scratch and the geometry and + Fourier-basis caches. +- `prepareSessionRows` in `root.zig` — where the stamps are checked and the + children are reused or rebuilt each run. +- the parent `src/README.md` for how these tables feed `optics/`, `rtm/`, and the + retrieval loop. diff --git a/src/cache/forward_worker_pool.zig b/src/cache/forward_worker_pool.zig index ea5d3b631..84f5a2ea1 100644 --- a/src/cache/forward_worker_pool.zig +++ b/src/cache/forward_worker_pool.zig @@ -4,18 +4,18 @@ const builtin = @import("builtin"); const Allocator = std.mem.Allocator; // forward_worker_pool.zig ------------------------------------------------------------------------------------| -// Session-retained helper-thread owner for O2 A forward prefetch phases. | +// Session-retained helper-thread owner for forward prefetch phases. | // | // Helper threads stay on product/session storage, reuse unchanged helper counts, and return null on pool | // init failure so callers use the raw-spawn path. | // | // ownership boundary | -// This owner stores threads and an optional borrowed shared pool only. It stores no case, wavelength, | +// This owner stores threads and an optional borrowed shared pool only. It stores no scene, wavelength, | // layer, optical, RTM, or output state, so worker scheduling cannot become a hidden physics input. | // ------------------------------------------------------------------------------------------------------------| // ForwardWorkerPool ------------------------------------------------------------------------------------------| -// Owned-or-borrowed thread-pool selector retained by one O2 session. | +// Owned-or-borrowed thread-pool selector retained by one session. | // | // layout(64-bit) | // Debug build: size 136 B (0.133 KiB), align: 8 B | diff --git a/src/cache/profile_line_memory.zig b/src/cache/profile_line_memory.zig index 79553e719..5b639d51f 100644 --- a/src/cache/profile_line_memory.zig +++ b/src/cache/profile_line_memory.zig @@ -30,7 +30,7 @@ const min_hitran_temperature_k = line_physics.min_hitran_temperature_k; // Retained weak-line cross-section values for exact setup wavelengths and layer profile nodes. | // | // setup boundary | -// O2 A builds a preparation-time line-value grid from the parsed HITRAN rows and the computed layer grid. | +// builds a preparation-time line-value grid from the parsed HITRAN rows and the computed layer grid. | // Rows are indexed as wavelength-major, then layer-node minor, so later optics code can read a contiguous | // profile column for each exact wavelength without rebuilding weak-line Voigt terms. | // ------------------------------------------------------------------------------------------------------------| @@ -186,7 +186,7 @@ pub fn buildProfileLineValues( scene: scene_input.Scene, ) !ProfileLineValues { // buildProfileLineValues ---------------------------------------------------------------------------------| - // Build line values over the case's evenly spaced setup wavelengths. | + // Build line values over the scene's evenly spaced setup wavelengths. | // --------------------------------------------------------------------------------------------------------| const wavelength_count = scene.spectral_grid.sample_count; const wavelengths_nm = try allocator.alloc(f64, wavelength_count); @@ -245,13 +245,15 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( // row contract | // Output rows stay wavelength-major and preserve the input wavelength order exactly. Dense spectrum | // prefetch can therefore use its `RadianceWavelengthList` index as the ProfileLineValues index. | - // `build_layer_values` keeps the O2 A diagnostic layer rows for evidence paths. The public spectrum | + // `build_layer_values` keeps the diagnostic layer rows for evidence paths. The public spectrum | // route uses only support-profile total sigma, so root skips preparing unused layer rows. | // --------------------------------------------------------------------------------------------------------| - var layers = try atmosphere_layers.build(allocator, scene); - defer layers.deinit(allocator); - var lines = try line_tables.build(allocator, scene); - defer lines.deinit(allocator); + var build_arena = std.heap.ArenaAllocator.init(allocator); + defer build_arena.deinit(); + const scratch_allocator = build_arena.allocator(); + + const layers = try atmosphere_layers.build(scratch_allocator, scene); + const lines = try line_tables.build(scratch_allocator, scene); const wavelength_count = wavelengths_nm.len; const profile_node_count = if (build_layer_values) layers.layer_pressures_hpa.len else 0; @@ -261,16 +263,14 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( const support_profile_total_sigma = try allocator.alloc(f64, wavelength_count * support_profile_node_count); errdefer allocator.free(support_profile_total_sigma); - const cutoff_grid = try prepareCutoffGrid(allocator, cutoff_grid_wavelengths_nm); - defer cutoff_grid.deinit(allocator); + const cutoff_grid = try prepareCutoffGrid(scratch_allocator, cutoff_grid_wavelengths_nm); const runtime = RuntimeControls{ .cutoff_cm1 = lines.cutoff_sim_cm1, .line_mixing_factor = lines.line_mixing_factor, .cutoff_grid_wavelengths_nm = cutoff_grid.wavelengths_nm, .cutoff_grid_wavenumbers_cm1 = cutoff_grid.wavenumbers_cm1, }; - const total_lines = try collectRuntimeLines(allocator, lines.rows, lines.isotopes_sim); - defer allocator.free(total_lines); + const total_lines = try collectRuntimeLines(scratch_allocator, lines.rows, lines.isotopes_sim); const strong_sidecars = chooseStrongLineSidecars(lines); @@ -282,13 +282,13 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( if (build_layer_values) { const line_strength_threshold = thresholdStrength(lines.rows, lines.threshold_line_sim); active_lines = try collectActiveLines( - allocator, + scratch_allocator, lines.rows, lines.isotopes_sim, line_strength_threshold, ); weak_states = try prepareLayerWeakLineStates( - allocator, + scratch_allocator, active_lines, layers.layer_temperatures_k, layers.layer_pressures_hpa, @@ -296,7 +296,7 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( cost_timing_active, ); total_weak_states = try prepareLayerWeakLineStates( - allocator, + scratch_allocator, total_lines, layers.layer_temperatures_k, layers.layer_pressures_hpa, @@ -304,7 +304,7 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( cost_timing_active, ); strong_states = try prepareLayerStrongLineStates( - allocator, + scratch_allocator, strong_sidecars.lines, strong_sidecars.relaxation_matrix, layers.layer_temperatures_k, @@ -312,14 +312,8 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( cost_timing_active, ); } - defer if (build_layer_values) { - allocator.free(strong_states); - deinitWeakLineStates(allocator, total_weak_states); - deinitWeakLineStates(allocator, weak_states); - allocator.free(active_lines); - }; - // The canonical temperature derivative is a centered finite difference at T +/- 0.5 K. The public O2 A + // The canonical temperature derivative is a centered finite difference at T +/- 0.5 K. The public // Jacobian states are surface/aerosol controls, so root spectrum runs do not read d_sigma/dT and skip // these two full weak-line state families. Explicit profile-line canonical builders still request the rows // directly, and the reuse stamp records the choice so mismatched session caches cannot mix. @@ -327,16 +321,15 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( var lower_weak_states: []WeakLinePreparedState = &.{}; if (build_layer_values and include_temperature_derivatives) { upper_weak_states = try prepareLayerWeakLineStates( - allocator, + scratch_allocator, active_lines, layers.layer_temperatures_k, layers.layer_pressures_hpa, 0.5, cost_timing_active, ); - errdefer deinitWeakLineStates(allocator, upper_weak_states); lower_weak_states = try prepareLayerWeakLineStates( - allocator, + scratch_allocator, active_lines, layers.layer_temperatures_k, layers.layer_pressures_hpa, @@ -344,25 +337,19 @@ pub fn buildProfileLineValuesForWavelengthsWithCutoffGrid( cost_timing_active, ); } - defer if (build_layer_values and include_temperature_derivatives) { - deinitWeakLineStates(allocator, lower_weak_states); - deinitWeakLineStates(allocator, upper_weak_states); - }; const support_total_weak_states = try prepareProfileWeakLineStates( - allocator, + scratch_allocator, total_lines, layers.spectroscopy_profile.rows, cost_timing_active, ); - defer deinitWeakLineStates(allocator, support_total_weak_states); const support_strong_states = try prepareProfileStrongLineStates( - allocator, + scratch_allocator, strong_sidecars.lines, strong_sidecars.relaxation_matrix, layers.spectroscopy_profile.rows, cost_timing_active, ); - defer allocator.free(support_strong_states); try buildProfileLineValuesByWavelength( pool, @@ -524,7 +511,7 @@ fn buildProfileLineValuesByWavelength( // instrumentation: trace zone: profile spectroscopy cache build ----------------------------------------- | // captures: profile-line cache build wall time and exact-wavelength count | - // why: shows setup cost that O2 A compares before forward prefetch starts. | + // why: shows setup cost that compares before forward prefetch starts. | const zone = Trace.staticZone(@src(), "profile_spectroscopy_cache.build"); zone.value(@intCast(wavelengths_nm.len)); defer zone.end(); @@ -969,17 +956,6 @@ const StrongLineSidecars = struct { const CutoffGrid = struct { wavelengths_nm: []f64 = &.{}, wavenumbers_cm1: []f64 = &.{}, - - fn deinit(self: *const CutoffGrid, allocator: Allocator) void { - // CutoffGrid.deinit --------------------------------------------------------------------------------- | - // Release the paired weak-line cutoff grid arrays. | - // | - // ownership | - // Both slices are owned by CutoffGrid and are never borrowed after the parent setup object exits. | - // ----------------------------------------------------------------------------------------------------| - allocator.free(self.wavenumbers_cm1); - allocator.free(self.wavelengths_nm); - } }; // ------------------------------------------------------------------------------------------------------------| @@ -1370,14 +1346,6 @@ fn fillProfileLineStateRows(worker: *ProfileLineStateWorker, start: usize, end: } } -fn deinitWeakLineStates(allocator: Allocator, states: []WeakLinePreparedState) void { - // deinitWeakLineStates -----------------------------------------------------------------------------------| - // Release a prepared weak-line state row set after setup evaluation completes. | - // --------------------------------------------------------------------------------------------------------| - for (states) |*state| state.deinit(allocator); - allocator.free(states); -} - fn emptyRelaxationMatrix() readers.RelaxationMatrixAsset { // emptyRelaxationMatrix ----------------------------------------------------------------------------------| // Provide an unused, well-formed relaxation-matrix view for weak-only profile-line state workers. | @@ -1391,7 +1359,7 @@ fn activeLine( line_strength_threshold: f64, ) bool { // activeLine ---------------------------------------------------------------------------------------------| - // Apply the O2 A line-gas isotope and weak-line threshold controls to one parsed HITRAN row. | + // Apply the line-gas isotope and weak-line threshold controls to one parsed HITRAN row. | // --------------------------------------------------------------------------------------------------------| if (line.gas_index != 7) return false; diff --git a/src/cache/radiance_memory.zig b/src/cache/radiance_memory.zig index 9683d0b01..a4262cb9a 100644 --- a/src/cache/radiance_memory.zig +++ b/src/cache/radiance_memory.zig @@ -60,27 +60,28 @@ pub const RadianceMemory = struct { self.* = .{}; } - pub fn takeWavelengthList( + pub fn rebuildWavelengthList( self: *RadianceMemory, allocator: Allocator, - list: *radiance_wavelengths.OwnedRadianceWavelengthList, + new_rows: []radiance_wavelengths.RadianceSampleIndexRef, + new_sample_indices: []u32, + new_wavelengths: []radiance_wavelengths.RadianceWavelength, stamp: hashing.ReuseStamp, ) void { - // RadianceMemory.takeWavelengthList ----------------------------------------------------------------- | - // Move an owned exact wavelength list into reusable radiance memory and invalidate dense results. | + // RadianceMemory.rebuildWavelengthList -------------------------------------------------------------- | + // Replace the exact wavelength route and invalidate dense results that indexed the old route. | // ----------------------------------------------------------------------------------------------------| allocator.free(self.wavelength_rows); allocator.free(self.sample_indices); allocator.free(self.wavelengths); - self.wavelength_rows = list.rows; - self.sample_indices = list.sample_indices; - self.wavelengths = list.wavelengths; - self.active.wavelength_row_count = list.rows.len; - self.active.sample_index_count = list.sample_indices.len; - self.active.wavelength_count = list.wavelengths.len; + self.wavelength_rows = new_rows; + self.sample_indices = new_sample_indices; + self.wavelengths = new_wavelengths; + self.active.wavelength_row_count = new_rows.len; + self.active.sample_index_count = new_sample_indices.len; + self.active.wavelength_count = new_wavelengths.len; self.wavelength_stamp = stamp; self.result_stamp = .{}; - list.* = .{}; } pub fn hasWavelengthList( diff --git a/src/cache/session_memory.zig b/src/cache/session_memory.zig index 2b5fd8a7e..70b7eb94b 100644 --- a/src/cache/session_memory.zig +++ b/src/cache/session_memory.zig @@ -11,7 +11,7 @@ const transport_worker_memory = @import("transport_worker_memory.zig"); const Allocator = std.mem.Allocator; // session_memory.zig ------------------------------------------------------------------------------------- | -// Named memory owners retained by one O2 forward session. | +// Named memory owners retained by one forward session. | // | // ownership boundary | // The session memory owns computed rows and reusable work arrays only. It stores no scene, request, RTM | @@ -19,7 +19,7 @@ const Allocator = std.mem.Allocator; // ------------------------------------------------------------------------------------------------------------| // SessionMemory ------------------------------------------------------------------------------------------- | -// Top-level allocation owner for reusable O2 A setup, spectrum, solar, workers, and transport work. | +// Top-level allocation owner for reusable setup, spectrum, solar, workers, and transport work. | // | // layout(64-bit) | // Debug build: size 448 B (0.438 KiB), align 8 | diff --git a/src/cache/spectrum_memory.zig b/src/cache/spectrum_memory.zig index 48aaab243..349f7fe5c 100644 --- a/src/cache/spectrum_memory.zig +++ b/src/cache/spectrum_memory.zig @@ -73,23 +73,24 @@ pub const SpectrumMemory = struct { _ = try memory.ensureSliceCapacity(f64, allocator, &self.kernel_weights, side_sample_count); } - pub fn takeTable( + pub fn rebuildTable( self: *SpectrumMemory, allocator: Allocator, - owned_table: *sampling_table.OwnedSpectrumSamplingTable, + new_rows: []sampling_table.SpectrumSamplingRow, + new_kernel_offsets_nm: []f64, + new_kernel_weights: []f64, stamp: hashing.ReuseStamp, ) void { - // SpectrumMemory.takeTable ---------------------------------------------------------------------------| - // Move an owned sampling table into retained session memory without copying the row or side arrays. | + // SpectrumMemory.rebuildTable ------------------------------------------------------------------------| + // Replace retained table storage with freshly built row and side-array slices. | // ----------------------------------------------------------------------------------------------------| allocator.free(self.rows); allocator.free(self.kernel_offsets_nm); allocator.free(self.kernel_weights); - self.rows = owned_table.rows; - self.kernel_offsets_nm = owned_table.kernel_offsets_nm; - self.kernel_weights = owned_table.kernel_weights; + self.rows = new_rows; + self.kernel_offsets_nm = new_kernel_offsets_nm; + self.kernel_weights = new_kernel_weights; self.table_stamp = stamp; - owned_table.* = .{}; } pub fn hasTable(self: SpectrumMemory, stamp: hashing.ReuseStamp) bool { diff --git a/src/cache/transport_worker_memory.zig b/src/cache/transport_worker_memory.zig index d1fd667b8..91d342d3c 100644 --- a/src/cache/transport_worker_memory.zig +++ b/src/cache/transport_worker_memory.zig @@ -22,7 +22,7 @@ pub const Error = error{ // transport_worker_memory.zig ------------------------------------------------------------------------------ | // Reusable optics and LABOS transport buffers for one forward worker. | // | -// named O2 A memory block. The borrowed row types come from the new `rtm/*` modules. | +// named memory block. The borrowed row types come from the new `rtm/*` modules. | // | // ownership boundary | // This owner stores arrays, geometry reuse state, and validity flags only. Optical-property values, phase | diff --git a/src/common/errors.zig b/src/common/errors.zig index b1c2975af..de2d74e4f 100644 --- a/src/common/errors.zig +++ b/src/common/errors.zig @@ -1,5 +1,5 @@ // errors.zig -------------------------------------------------------------------------------------------------| -// Shared setup/input errors for the O2 A table layer. | +// Shared setup/input errors for the table layer. | // ------------------------------------------------------------------------------------------------------------| pub const Error = error{ diff --git a/src/common/math/gauss_legendre.zig b/src/common/math/gauss_legendre.zig index 7981aeecc..3d669fb27 100644 --- a/src/common/math/gauss_legendre.zig +++ b/src/common/math/gauss_legendre.zig @@ -24,7 +24,7 @@ const fixed_rules = buildFixedRules(); // route choice | // Small ordinary orders use rule() when the caller wants a fixed table. Larger ordinary orders use Newton | // root solves. DISAMAR canonical paths build a tridiagonal system and diagonalize it with | -// gausq2DisamarImpl because last-bit node differences are visible in steep O2 A high-resolution support | +// gausq2DisamarImpl because last-bit node differences are visible in steep high-resolution support | // samples. Callers that reuse layer shapes should retain canonical rows by order and rescale them only. | // | // hot path | diff --git a/src/common/worker_partition.zig b/src/common/worker_partition.zig index 79442e3e4..d8e7d8799 100644 --- a/src/common/worker_partition.zig +++ b/src/common/worker_partition.zig @@ -2,7 +2,7 @@ const std = @import("std"); const builtin = @import("builtin"); // worker_partition.zig ---------------------------------------------------------------------------------------| -// Shared worker partitioning and spawn orchestration for the explicit O2 A route. | +// Shared worker partitioning and spawn orchestration for the explicit route. | // | // | // boundary | diff --git a/src/input/README.md b/src/input/README.md index f0570a623..5b67d44bc 100644 --- a/src/input/README.md +++ b/src/input/README.md @@ -7,7 +7,7 @@ and checks the scene before setup or compute sees it. ``` +----------------+ +----------------------+ -| Python O2 A | JSON | parseSceneJson | +| Python | JSON | parseSceneJson | | factory/scene +-------->| buildScene+normalize | +----------------+ +----------+-----------+ | @@ -49,11 +49,11 @@ Two things hold for every `Scene`: A `Scene` enters the model through caller-provided typed data or through `parseSceneJson`; both end at the same validator. The packaged DISAMAR-family -O2 A reference case lives in +reference scene lives in `python/zdisamar/input/wavelength_band/o2a_default.py` as `default_o2a_scene()`. Python assembles the dataclass scene, resolves reference-data paths before preparation, and passes the native JSON shape over -`zds_prepare_o2a_json`. Zig receives the packaged reference case as an ordinary +`zds_prepare_o2a_json`. Zig receives the packaged reference scene as an ordinary caller-provided scene; there is no built-in Zig default scene or zero-JSON default preparation path. @@ -62,7 +62,7 @@ Python's `Scene.to_native_json_bytes()` emits and produces a typed `Scene`. A few things to know here: - The JSON shape (`NativeSceneJson`) carries Python bookkeeping and some fields - that have no effect on this forward-only O2 A route. `buildScene` maps the + that have no effect on this forward-only route. `buildScene` maps the fields the model uses across to `Scene`, and rejects unsupported route changes (a different scattering mode, a Jacobian request, finite altitude bounds) instead of accepting them silently. diff --git a/src/input/hitran_partition_tables.zig b/src/input/hitran_partition_tables.zig index fa0703f07..6c8028c4a 100644 --- a/src/input/hitran_partition_tables.zig +++ b/src/input/hitran_partition_tables.zig @@ -5,7 +5,7 @@ const CostTiming = @import("../instrumentation/cost_timing.zig"); // hitran_partition_tables.zig ------------------------------------------------------------------------------- | // Embedded HITRAN TIPS partition-sum tables used by line-strength temperature scaling. | // | -// temperature grids and partition sums for isotopologues used by the O2 A line route. The q_* names follow | +// temperature grids and partition sums for isotopologues used by the line route. The q_* names follow | // HITRAN isotopologue codes: H2O uses 161/181/171/162/182/172, CO2 uses 626/636/628/627/638/637, and O2 | // uses 66/68/67. Codes are the same compact gas-isotope identifiers produced by the spectroscopy | // `deriveIsotopologueCode` helper. | @@ -705,26 +705,26 @@ pub fn partitionSampleMatchesEndpointSecant(isotopologue_code: i32, temperature_ // prepared sample directly; this rebuilds the endpoint-secant curvature once for equality checks. | // --------------------------------------------------------------------------------------------------------| const table = preparedPartitionTable(isotopologue_code) orelse return null; + const rebuilt = rebuildEndpointSecantSample(table, temperature_k); + const prepared = interpolatePartitionTable(table, temperature_k); + return rebuilt == prepared; +} +fn rebuildEndpointSecantSample(table: PreparedPartitionView, temperature_k: f64) f64 { + // rebuildEndpointSecantSample ----------------------------------------------------------------------------| + // Recompute the endpoint-secant oracle sample with the same clamped temperature route as production. | + // --------------------------------------------------------------------------------------------------------| const safe_temperature = std.math.clamp( temperature_k, temperature_grid[0], temperature_grid[temperature_grid.len - 1], ); + if (safe_temperature <= temperature_grid[0]) return table.values[0]; + if (safe_temperature >= temperature_grid[temperature_grid.len - 1]) return table.values[table.values.len - 1]; - const rebuilt = choose_rebuilt_sample: { - if (safe_temperature <= temperature_grid[0]) break :choose_rebuilt_sample table.values[0]; - if (safe_temperature >= temperature_grid[temperature_grid.len - 1]) { - break :choose_rebuilt_sample table.values[table.values.len - 1]; - } - - break :choose_rebuilt_sample spline.sampleEndpointSecant( - temperature_grid[0..], - table.values, - safe_temperature, - ) catch unreachable; - }; - - const prepared = interpolatePartitionTable(table, temperature_k); - return rebuilt == prepared; + return spline.sampleEndpointSecant( + temperature_grid[0..], + table.values, + safe_temperature, + ) catch unreachable; } diff --git a/src/input/json.zig b/src/input/json.zig index e3755058f..c0ca83663 100644 --- a/src/input/json.zig +++ b/src/input/json.zig @@ -10,12 +10,12 @@ pub const supported_stream_count: usize = 20; pub const route_fourier_term_limit: usize = 20; // json.zig ---------------------------------------------------------------------------------------------------| -// Python-native O2 A JSON bridge. | +// Python-native JSON bridge. | // | // boundary | // Python emits Scene.to_native_json_bytes() with resolved asset paths and a few Python bookkeeping | // fields. This parser turns that API shape into Scene, validates controls that are intentionally inert | -// for this O2 A forward-only route, and rejects unsupported route changes before compute sees the scene. | +// for this forward-only route, and rejects unsupported route changes before compute sees the scene. | // | // JSON input normalization | // Python's json encoder emits bare NaN for optional altitude placeholders. Zig's JSON scanner is strict, | @@ -51,7 +51,7 @@ pub const ParsedSceneJson = struct { pub fn parseSceneJson(allocator: Allocator, raw_json: []const u8) !ParsedSceneJson { // parseSceneJson -----------------------------------------------------------------------------------------| - // Parse Python's native O2 A JSON shape and return a typed scene borrowing parser-owned rows. | + // Parse Python's native JSON shape and return a typed scene borrowing parser-owned rows. | // --------------------------------------------------------------------------------------------------------| const normalized = try normalizePythonJson(allocator, raw_json); defer normalized.deinit(allocator); @@ -184,7 +184,7 @@ fn validateAssets(inputs: InputsJson) !void { fn validateObservation(observation: ObservationJson, solar_reference_asset_id: []const u8) !void { // validateObservation ------------------------------------------------------------------------------------| - // Accept the current O2 A instrument route and pass explicit sparse product axes to Scene validation. | + // Accept the current instrument route and pass explicit sparse product axes to Scene validation. | // --------------------------------------------------------------------------------------------------------| if (!std.mem.eql(u8, observation.regime, "nadir")) return errors.Error.UnsupportedJsonInput; if (!std.mem.eql(u8, observation.sampling, "native")) return errors.Error.UnsupportedJsonInput; @@ -200,7 +200,7 @@ fn validateRtm( performance_thresholds: transport_controls.PerformanceThresholds, ) !void { // validateRtm --------------------------------------------------------------------------------------------| - // Keep O2 A on the supported integrated-source route while consuming scene-owned LABOS threshold controls.| + // Keep on the supported integrated-source route while consuming scene-owned LABOS threshold controls.| // --------------------------------------------------------------------------------------------------------| if (!std.mem.eql(u8, rtm.scattering, "multiple")) return errors.Error.UnsupportedJsonInput; if (rtm.n_streams != supported_stream_count) return errors.Error.UnsupportedJsonInput; @@ -217,7 +217,7 @@ fn validateRtm( fn validateAerosol(aerosol: AerosolJson) !void { // validateAerosol ----------------------------------------------------------------------------------------| - // Accept scalar placement and explicit multi-layer profile payloads on the proven O2 A route. | + // Accept scalar placement and explicit multi-layer profile payloads on the proven route. | // --------------------------------------------------------------------------------------------------------| if (!std.mem.eql(u8, aerosol.placement.semantics, "explicit_interval_bounds")) { return errors.Error.UnsupportedJsonInput; @@ -290,7 +290,7 @@ fn aerosolProfileLayerFromJson(row: AerosolProfileLayerJson) scene_input.Aerosol fn validateInertAltitudePlaceholder(value: ?f64) !void { // validateInertAltitudePlaceholder -----------------------------------------------------------------------| - // Native O2 A uses pressure bounds; finite altitude bounds would be silently inert on this route. | + // Native uses pressure bounds; finite altitude bounds would be silently inert on this route. | // --------------------------------------------------------------------------------------------------------| if (value) |resolved| { if (!std.math.isNan(resolved)) return errors.Error.UnsupportedJsonInput; @@ -299,7 +299,7 @@ fn validateInertAltitudePlaceholder(value: ?f64) !void { fn validateInertPressureVariance(value: f64) !void { // validateInertPressureVariance --------------------------------------------------------------------------| - // Native O2 A consumes deterministic pressure bounds; non-zero pressure variances would be inert here. | + // Native consumes deterministic pressure bounds; non-zero pressure variances would be inert here. | // --------------------------------------------------------------------------------------------------------| if (!std.math.isFinite(value) or value != 0.0) return errors.Error.UnsupportedJsonInput; } diff --git a/src/input/scene.zig b/src/input/scene.zig index b16d2d40f..a291b0384 100644 --- a/src/input/scene.zig +++ b/src/input/scene.zig @@ -2,7 +2,7 @@ const std = @import("std"); const transport_controls = @import("../rtm/controls.zig"); // scene.zig ---------------------------------------------------------------------------------------------- | -// Typed O2 A product-scene controls consumed by setup tables and forward runs. | +// Typed product-scene controls consumed by setup tables and forward runs. | // | // runtime boundary | // These rows are setup inputs only. Asset paths and text formats are resolved by src/assets; setup tables | @@ -271,7 +271,7 @@ pub const RtmControls = struct { // ------------------------------------------------------------------------------------------------------------| // Scene ------------------------------------------------------------------------------------------------------| -// Borrowed O2 A setup row. | +// Borrowed setup row. | // | // layout(64-bit) | // size: 704 B (0.688 KiB), align: 8 B | diff --git a/src/input/validate.zig b/src/input/validate.zig index 10e0c15c5..2dbcb09cf 100644 --- a/src/input/validate.zig +++ b/src/input/validate.zig @@ -8,7 +8,7 @@ const profile_spectral_tolerance: f64 = 1.0e-12; pub fn sceneControls(scene: scene_input.Scene) !void { // sceneControls ------------------------------------------------------------------------------------------| - // Validate every control O2 A consumes so unsupported or malformed setup input cannot pass inertly. | + // Validate every control consumed so unsupported or malformed setup input cannot pass inertly. | // --------------------------------------------------------------------------------------------------------| if (scene.spectral_grid.sample_count < 2) return errors.Error.InvalidControl; diff --git a/src/instrumentation/trace.zig b/src/instrumentation/trace.zig index 78693cf0b..195e7f135 100644 --- a/src/instrumentation/trace.zig +++ b/src/instrumentation/trace.zig @@ -14,7 +14,7 @@ const SourceLocation = std.builtin.SourceLocation; // matrix builds, doubling decisions, q-series work, and fixed-kernel product gates. | // rtm/scattering_orders.zig marks initial source setup, order transport, local source passes, retained | // order accumulation, convergence exits, and paired Gaussian dot products. | -// cache/profile_line_memory.zig and setup refresh code may add retained setup zones as O2 A/O2 A land. | +// cache/profile_line_memory.zig and setup refresh code may add retained setup zones as land. | // | // primary paths | // enabled is true only when build_options.enable_ztracy is present and true. build.zig supplies either the | diff --git a/src/internal.zig b/src/internal.zig index 4c25365b4..1a09f6581 100644 --- a/src/internal.zig +++ b/src/internal.zig @@ -1,5 +1,5 @@ // internal.zig -----------------------------------------------------------------------------------------------| -// Test access router for O2 A setup, asset, cache, common, input, and instrumentation modules. | +// Test access router for setup, asset, cache, common, input, and instrumentation modules. | // | // Product callers use src/root.zig. Tests use this file to reach small table builders without widening API. | // public package surface before later modules add public forward-model entry points. | @@ -119,7 +119,7 @@ pub const cache = struct { // ------------------------------------------------------------------------------------------------------------| // optics -----------------------------------------------------------------------------------------------------| -// Namespace-only test import wrapper for O2 A optical-depth fill modules. | +// Namespace-only test import wrapper for optical-depth fill modules. | // | // layout(64-bit) | // size: 0 B (0.000 KiB), align: 1 B | @@ -157,7 +157,7 @@ pub const output = struct { // ------------------------------------------------------------------------------------------------------------| // retrieval --------------------------------------------------------------------------------------------------| -// Namespace-only test import wrapper for O2 A retrieval state-space and result owners. | +// Namespace-only test import wrapper for retrieval state-space and result owners. | // | // layout(64-bit) | // size: 0 B (0.000 KiB), align: 1 B | @@ -173,7 +173,7 @@ pub const retrieval = struct { // ------------------------------------------------------------------------------------------------------------| // rtm --------------------------------------------------------------------------------------------------------| -// Namespace-only test import wrapper for O2 A RTM state modules. | +// Namespace-only test import wrapper for RTM state modules. | // | // layout(64-bit) | // size: 0 B (0.000 KiB), align: 1 B | @@ -198,7 +198,7 @@ pub const rtm = struct { // ------------------------------------------------------------------------------------------------------------| // spectrum ---------------------------------------------------------------------------------------------------| -// Namespace-only test import wrapper for O2 A spectrum sampling and exact wavelength modules. | +// Namespace-only test import wrapper for spectrum sampling and exact wavelength modules. | // | // layout(64-bit) | // size: 0 B (0.000 KiB), align: 1 B | @@ -218,7 +218,7 @@ pub const spectrum = struct { // ------------------------------------------------------------------------------------------------------------| // validation -------------------------------------------------------------------------------------------------| -// Namespace-only test import wrapper for validation-only O2 A metric helpers. | +// Namespace-only test import wrapper for validation-only metric helpers. | // | // layout(64-bit) | // size: 0 B (0.000 KiB), align: 1 B | diff --git a/src/optics/layer_depths.zig b/src/optics/layer_depths.zig index f5abe03b8..c0ea46989 100644 --- a/src/optics/layer_depths.zig +++ b/src/optics/layer_depths.zig @@ -127,7 +127,7 @@ pub const LayerOptics = struct { // [ 520..1031] log_complex_vmr_fraction : [64]f64 | // [1032..1543] second : [64]f64 | // | -// vertical setup profile. The O2 A diagnostic cross-section rows prove this cache is active for the O2 A | +// vertical setup profile. The O2A diagnostic cross-section rows prove this cache is active for the | // reference scene. | pub const CollisionPairProfile = struct { node_count: usize = 0, @@ -563,7 +563,6 @@ pub fn reduceLayerOpticsFromSupportRows( pub fn fillLayerAerosolJacobians( aerosol: aerosol_tables.AerosolLayerTable, - state_mask: jacobian_states.StateMask, layers: []LayerOptics, ) void { // fillLayerAerosolJacobians ----------------------------------------------------------------------------- | @@ -585,7 +584,6 @@ pub fn fillLayerAerosolJacobians( layer.aerosol_phase_weight_jacobian = jacobian_states.zero(); } - if (!jacobian_states.includes(state_mask, .aerosol_optical_depth)) return; if (aerosol.profile.len != 0) return; if (aerosol.optical_depth <= 0.0) return; diff --git a/src/optics/rayleigh.zig b/src/optics/rayleigh.zig index e4e2dff61..74a70020f 100644 --- a/src/optics/rayleigh.zig +++ b/src/optics/rayleigh.zig @@ -3,7 +3,7 @@ const std = @import("std"); // rayleigh.zig -----------------------------------------------------------------------------------------------| // Dry-air Rayleigh scattering helpers for gas-scattering optical-depth and phase-coefficient paths. | // | -// O2 A optical-depth checks read Rayleigh scattering directly from atmospheric-budget rows. | +// optical-depth checks read Rayleigh scattering directly from atmospheric-budget rows. | // | // math | // sigma_um_inv = 1000 / max(wavelength_nm, 1) | diff --git a/src/output/atmospheric_budget.zig b/src/output/atmospheric_budget.zig index 08fa9fa5d..fae322cbf 100644 --- a/src/output/atmospheric_budget.zig +++ b/src/output/atmospheric_budget.zig @@ -9,7 +9,7 @@ const profile_line_memory = @import("../cache/profile_line_memory.zig"); const Allocator = std.mem.Allocator; // atmospheric_budget.zig -------------------------------------------------------------------------------------| -// Public atmospheric support-row diagnostic table for the explicit O2 A route. | +// Public atmospheric support-row diagnostic table for the explicit route. | // | // boundary | // The builder projects existing setup/profile-line/optics rows into the fixed Python C ABI row order. It | diff --git a/src/output/cia_diagnostics.zig b/src/output/cia_diagnostics.zig index 9f1e82582..14d147a8e 100644 --- a/src/output/cia_diagnostics.zig +++ b/src/output/cia_diagnostics.zig @@ -5,7 +5,7 @@ const atmospheric_budget = @import("atmospheric_budget.zig"); const Allocator = std.mem.Allocator; // cia_diagnostics.zig ----------------------------------------------------------------------------------------| -// O2-O2 collision-induced absorption diagnostic rows for the explicit O2 A route. | +// O2-O2 collision-induced absorption diagnostic rows for the explicit route. | // | // boundary | // The builder projects the already-public atmospheric-budget rows into the fixed Python C ABI row order. | diff --git a/src/output/instrument_response.zig b/src/output/instrument_response.zig index 537038c1b..8a0ced635 100644 --- a/src/output/instrument_response.zig +++ b/src/output/instrument_response.zig @@ -13,7 +13,7 @@ const allowed_channel_mask = channel_mask_radiance | channel_mask_irradiance; const integration_mode_disamar_hr_grid: u32 = 2; // instrument_response.zig ------------------------------------------------------------------------------------| -// Public instrument-response support rows for the explicit O2 A route. | +// Public instrument-response support rows for the explicit route. | // | // boundary | // The builder projects spectrum sampling kernels into the fixed Python C ABI row order. It owns no C | diff --git a/src/output/line_contributions.zig b/src/output/line_contributions.zig index 1930a54eb..e8dab5876 100644 --- a/src/output/line_contributions.zig +++ b/src/output/line_contributions.zig @@ -431,7 +431,7 @@ fn strongLineRow( fn resolveStrongAnchorFields(anchor: ?StrongAnchorMatch) StrongAnchorFields { // resolveStrongAnchorFields ------------------------------------------------------------------------------| - // Copy weak-line metadata for sidecars, or keep the complete-row O2 defaults when no anchor exists. | + // Copy weak-line metadata for sidecars, or keep the complete-row defaults when no anchor exists. | // --------------------------------------------------------------------------------------------------------| if (anchor) |owned| { return .{ @@ -570,7 +570,7 @@ fn usesVendorStrongLinePartition( strong_lines: []const readers.StrongLineAssetRow, ) bool { // usesVendorStrongLinePartition --------------------------------------------------------------------------| - // Detect the O2 A sidecar partition from retained HITRAN branch metadata. | + // Detect the sidecar partition from retained HITRAN branch metadata. | // --------------------------------------------------------------------------------------------------------| if (strong_lines.len == 0) return false; diff --git a/src/output/spectrum.zig b/src/output/spectrum.zig index b1cb8e7ee..879e7df03 100644 --- a/src/output/spectrum.zig +++ b/src/output/spectrum.zig @@ -6,7 +6,7 @@ const jacobian_states = @import("../rtm/jacobian_states.zig"); const Allocator = std.mem.Allocator; // spectrum.zig -----------------------------------------------------------------------------------------------| -// Public spectrum output owner for the explicit O2 A forward route. | +// Public spectrum output owner for the explicit forward route. | // | // ownership boundary | // Spectrum owns copied product-grid arrays returned across the root/API boundary. Session cache rows, | diff --git a/src/retrieval/algebra.zig b/src/retrieval/algebra.zig index 611a0c634..6ab7918e1 100644 --- a/src/retrieval/algebra.zig +++ b/src/retrieval/algebra.zig @@ -7,7 +7,7 @@ pub const Matrix = [max_state_count][max_state_count]f64; pub const Vector = [max_state_count]f64; // algebra.zig ------------------------------------------------------------------------------------------------| -// Fixed-size state-space algebra for native O2 A optimal estimation. | +// Fixed-size state-space algebra for native optimal estimation. | // | // route map | // root.zig builds the retrieval value layer and later solver slices on top of these fixed arrays. | @@ -15,11 +15,11 @@ pub const Vector = [max_state_count]f64; // states, so retrieval state math stays two-lane. | // | // state-space storage | -// Vector is [2]f64 and Matrix is [2][2]f64. Callers pass state_count and use only the leading rows and | -// columns, so one- and two-state retrievals share stack storage and avoid heap-backed linalg. | +// Vector is [2]f64 and Matrix is [2][2]f64. OE retrieval always uses both lanes: aerosol optical depth and | +// aerosol-layer mid-pressure. | // | // retrieval math | -// Later O2 A solver code builds the Rodgers normal system and update in these arrays: | +// Later solver code builds the Rodgers normal system and update in these arrays: | // | // G = sqrt(Sa) * Jt * Se^-1 * J * sqrt(Sa) | // b = sqrt(Sa) * Jt * Se^-1 * residual | @@ -48,36 +48,31 @@ pub fn zeroMatrix() Matrix { return .{.{0.0} ** max_state_count} ** max_state_count; } -pub fn identityMatrix(state_count: usize) Matrix { +pub fn identityMatrix() Matrix { // identityMatrix -----------------------------------------------------------------------------------------| - // Build an identity matrix over the leading state_count rows and columns. | + // Build the fixed two-state identity matrix. | // --------------------------------------------------------------------------------------------------------| - var result = zeroMatrix(); - for (0..state_count) |index| result[index][index] = 1.0; - return result; + return .{ .{ 1.0, 0.0 }, .{ 0.0, 1.0 } }; } -fn symmetrize(matrix: *Matrix, state_count: usize) void { +fn symmetrize(matrix: *Matrix) void { // symmetrize ---------------------------------------------------------------------------------------------| - // Average symmetric off-diagonal entries before symmetric eigensolve or inversion. | + // Average the single off-diagonal pair before symmetric eigensolve or inversion. | // --------------------------------------------------------------------------------------------------------| - for (0..state_count) |row| { - for (row + 1..state_count) |col| { - const value = 0.5 * (matrix[row][col] + matrix[col][row]); - matrix[row][col] = value; - matrix[col][row] = value; - } - } + const value = 0.5 * (matrix[0][1] + matrix[1][0]); + matrix[0][1] = value; + matrix[1][0] = value; } -pub fn choleskyLowerDiagonal(prior_variance: []const f64, sqrt_sa: *Vector, sqrt_inv_sa: *Vector) !void { +pub fn choleskyLowerDiagonal(prior_variance: Vector, sqrt_sa: *Vector, sqrt_inv_sa: *Vector) !void { // choleskyLowerDiagonal ----------------------------------------------------------------------------------| // Convert diagonal prior covariance values into sqrt(Sa) and sqrt(Sa)^-1 lanes. | // | // guard | // variance must be finite and positive because the retrieval update divides by sqrt(variance). | // --------------------------------------------------------------------------------------------------------| - for (prior_variance, 0..) |variance, index| { + inline for (0..max_state_count) |index| { + const variance = prior_variance[index]; if (variance <= 0.0 or !std.math.isFinite(variance)) return error.InvalidPriorCovariance; const root = @sqrt(variance); sqrt_sa[index] = root; @@ -85,7 +80,7 @@ pub fn choleskyLowerDiagonal(prior_variance: []const f64, sqrt_sa: *Vector, sqrt } } -pub fn jacobiEigenSymmetric(input: Matrix, state_count: usize) struct { +pub fn jacobiEigenSymmetric(input: Matrix) struct { values: Vector, vectors: Matrix, } { @@ -94,176 +89,111 @@ pub fn jacobiEigenSymmetric(input: Matrix, state_count: usize) struct { // | // hot path | // repeated : once per OE solver update | - // shape : at most 3x3, no heap storage, fixed 24-sweep cap | + // shape : fixed 2x2, no heap storage | // output : eigenvalues sorted descending with matching eigenvector columns | // | // math | - // Jacobi rotations zero one off-diagonal pair a[p,q] at a time while accumulating eigenvectors. | + // One Jacobi rotation zeros the only off-diagonal pair while accumulating eigenvectors. | // --------------------------------------------------------------------------------------------------------| var a = input; - symmetrize(&a, state_count); - var vectors = identityMatrix(state_count); - - for (0..24) |_| { - var changed = false; - for (0..state_count) |p| { - for (p + 1..state_count) |q| { - const apq = a[p][q]; - if (@abs(apq) <= eigen_tolerance) continue; - changed = true; - - const app = a[p][p]; - const aqq = a[q][q]; - const tau = (aqq - app) / (2.0 * apq); - const sign: f64 = if (tau < 0.0) -1.0 else 1.0; - const t = sign / (@abs(tau) + @sqrt(1.0 + tau * tau)); - const c = 1.0 / @sqrt(1.0 + t * t); - const s = t * c; - - for (0..state_count) |k| { - if (k == p or k == q) continue; - const akp = a[k][p]; - const akq = a[k][q]; - a[k][p] = c * akp - s * akq; - a[p][k] = a[k][p]; - a[k][q] = s * akp + c * akq; - a[q][k] = a[k][q]; - } - - const t_apq = t * apq; - a[p][p] = app - t_apq; - a[q][q] = aqq + t_apq; - a[p][q] = 0.0; - a[q][p] = 0.0; - - for (0..state_count) |k| { - const vkp = vectors[k][p]; - const vkq = vectors[k][q]; - vectors[k][p] = c * vkp - s * vkq; - vectors[k][q] = s * vkp + c * vkq; - } - } - } - if (!changed) break; + symmetrize(&a); + var vectors = identityMatrix(); + + const apq = a[0][1]; + if (@abs(apq) > eigen_tolerance) { + const app = a[0][0]; + const aqq = a[1][1]; + const tau = (aqq - app) / (2.0 * apq); + const sign: f64 = if (tau < 0.0) -1.0 else 1.0; + const t = sign / (@abs(tau) + @sqrt(1.0 + tau * tau)); + const c = 1.0 / @sqrt(1.0 + t * t); + const s = t * c; + + const t_apq = t * apq; + a[0][0] = app - t_apq; + a[1][1] = aqq + t_apq; + a[0][1] = 0.0; + a[1][0] = 0.0; + + vectors[0][0] = c; + vectors[0][1] = s; + vectors[1][0] = -s; + vectors[1][1] = c; } - var values = zeroVector(); - for (0..state_count) |index| values[index] = @max(a[index][index], 0.0); - sortEigenPairs(&values, &vectors, state_count); + var values = Vector{ @max(a[0][0], 0.0), @max(a[1][1], 0.0) }; + sortEigenPairs(&values, &vectors); return .{ .values = values, .vectors = vectors }; } -fn sortEigenPairs(values: *Vector, vectors: *Matrix, state_count: usize) void { +fn sortEigenPairs(values: *Vector, vectors: *Matrix) void { // sortEigenPairs -----------------------------------------------------------------------------------------| // Keep eigenvalues in descending order and swap matching eigenvector columns. | // --------------------------------------------------------------------------------------------------------| - if (state_count <= 1) return; - - for (0..state_count - 1) |i| { - var best = i; - for (i + 1..state_count) |j| { - if (values[j] > values[best]) best = j; - } - if (best == i) continue; + if (values[0] >= values[1]) return; - std.mem.swap(f64, &values[i], &values[best]); - for (0..state_count) |row| std.mem.swap(f64, &vectors[row][i], &vectors[row][best]); - } + std.mem.swap(f64, &values[0], &values[1]); + std.mem.swap(f64, &vectors[0][0], &vectors[0][1]); + std.mem.swap(f64, &vectors[1][0], &vectors[1][1]); } -pub fn matrixVector(matrix: Matrix, vector: Vector, state_count: usize) Vector { +pub fn matrixVector(matrix: Matrix, vector: Vector) Vector { // matrixVector -------------------------------------------------------------------------------------------| - // Multiply matrix rows by one state vector over the active leading state_count lanes. | + // Multiply the fixed two-state matrix rows by one state vector. | // --------------------------------------------------------------------------------------------------------| - var result = zeroVector(); - for (0..state_count) |row| { - var sum: f64 = 0.0; - for (0..state_count) |col| sum += matrix[row][col] * vector[col]; - result[row] = sum; - } - return result; + return .{ + matrix[0][0] * vector[0] + matrix[0][1] * vector[1], + matrix[1][0] * vector[0] + matrix[1][1] * vector[1], + }; } -pub fn transposeMatrixVector(matrix: Matrix, vector: Vector, state_count: usize) Vector { +pub fn transposeMatrixVector(matrix: Matrix, vector: Vector) Vector { // transposeMatrixVector ----------------------------------------------------------------------------------| - // Multiply matrix columns by one state vector over the active leading state_count lanes. | + // Multiply the fixed two-state matrix columns by one state vector. | // --------------------------------------------------------------------------------------------------------| - var result = zeroVector(); - for (0..state_count) |col| { - var sum: f64 = 0.0; - for (0..state_count) |row| sum += matrix[row][col] * vector[row]; - result[col] = sum; - } - return result; + return .{ + matrix[0][0] * vector[0] + matrix[1][0] * vector[1], + matrix[0][1] * vector[0] + matrix[1][1] * vector[1], + }; } -pub fn invertSymmetric(input: Matrix, state_count: usize) !Matrix { +pub fn invertSymmetric(input: Matrix) !Matrix { // invertSymmetric ----------------------------------------------------------------------------------------| - // Invert a small state-space matrix with Gauss-Jordan elimination and row pivoting. | + // Invert the fixed symmetric 2x2 state-space matrix. | // | // hot path | - // repeated : final posterior covariance and pressure-profile spline helper solves | - // shape : at most 3x3, no heap storage, identity matrix carried beside the working matrix | + // repeated : final posterior covariance | + // shape : fixed 2x2, no heap storage | // | // guard | - // pivot_abs <= 1.0e-24 rejects singular or badly scaled systems before row normalization. | + // det_abs <= 1.0e-24 rejects singular or badly scaled systems before normalization. | // --------------------------------------------------------------------------------------------------------| var a = input; - var inverse = identityMatrix(state_count); - - for (0..state_count) |pivot_index| { - var pivot_row = pivot_index; - var pivot_abs = @abs(a[pivot_index][pivot_index]); - - for (pivot_index + 1..state_count) |row| { - const candidate = @abs(a[row][pivot_index]); - if (candidate > pivot_abs) { - pivot_abs = candidate; - pivot_row = row; - } - } - - if (pivot_abs <= 1.0e-24 or !std.math.isFinite(pivot_abs)) return error.SingularMatrix; - - if (pivot_row != pivot_index) { - for (0..state_count) |col| { - std.mem.swap(f64, &a[pivot_index][col], &a[pivot_row][col]); - std.mem.swap(f64, &inverse[pivot_index][col], &inverse[pivot_row][col]); - } - } + symmetrize(&a); - const pivot = a[pivot_index][pivot_index]; - for (0..state_count) |col| { - a[pivot_index][col] /= pivot; - inverse[pivot_index][col] /= pivot; - } + const det = a[0][0] * a[1][1] - a[0][1] * a[0][1]; + const det_abs = @abs(det); + if (det_abs <= 1.0e-24 or !std.math.isFinite(det_abs)) return error.SingularMatrix; - for (0..state_count) |row| { - if (row == pivot_index) continue; - - const factor = a[row][pivot_index]; - if (factor == 0.0) continue; - - for (0..state_count) |col| { - a[row][col] -= factor * a[pivot_index][col]; - inverse[row][col] -= factor * inverse[pivot_index][col]; - } - } - } - return inverse; + const inv_det = 1.0 / det; + return .{ + .{ a[1][1] * inv_det, -a[0][1] * inv_det }, + .{ -a[0][1] * inv_det, a[0][0] * inv_det }, + }; } -pub fn multiply(a: Matrix, b: Matrix, state_count: usize) Matrix { +pub fn multiply(a: Matrix, b: Matrix) Matrix { // multiply -----------------------------------------------------------------------------------------------| - // Multiply two small state matrices over the active leading state_count rows and columns. | + // Multiply two fixed two-state matrices. | // --------------------------------------------------------------------------------------------------------| - var result = zeroMatrix(); - for (0..state_count) |row| { - for (0..state_count) |col| { - var sum: f64 = 0.0; - for (0..state_count) |k| sum += a[row][k] * b[k][col]; - result[row][col] = sum; - } - } - return result; + return .{ + .{ + a[0][0] * b[0][0] + a[0][1] * b[1][0], + a[0][0] * b[0][1] + a[0][1] * b[1][1], + }, + .{ + a[1][0] * b[0][0] + a[1][1] * b[1][0], + a[1][0] * b[0][1] + a[1][1] * b[1][1], + }, + }; } diff --git a/src/retrieval/root.zig b/src/retrieval/root.zig index 25fa6be79..b32ca1ab9 100644 --- a/src/retrieval/root.zig +++ b/src/retrieval/root.zig @@ -9,17 +9,25 @@ const Vector = algebra.Vector; pub const max_state_count = algebra.max_state_count; pub const max_iteration_count: usize = 1000; +const small_pressure_profile_spline_nodes: usize = 64; +const spline_work_vector_count: usize = 4; +const pressure_profile_spline_stack_bytes: usize = + small_pressure_profile_spline_nodes * spline_work_vector_count * @sizeOf(f64); pub const StateVector = Vector; pub const StateMatrix = Matrix; pub const no_lower_bound = -std.math.inf(f64); pub const no_upper_bound = std.math.inf(f64); +comptime { + std.debug.assert(max_state_count == jacobian_states.state_count); +} + // root.zig ---------------------------------------------------------------------------------------------------| -// Retrieval value layer, fixed state-space scratch, and retained result owners for O2 A O2 A release. | +// Retrieval value layer, fixed state-space scratch, and retained result owners for release. | // | // route map | -// api/c.zig converts Python/C request rows into StateSpec, MeasuredReflectanceRows, and result owners. | -// Future O2 A solver slices mutate these retrieval values and call root.zig forward functions with updated | +// api/c.zig converts Python/C request rows into RetrievalState, MeasuredReflectanceRows, and result owners. | +// Future solver slices mutate these retrieval values and call root.zig forward functions with updated | // aerosol/scalar state, while session memory stays warm. | // | // primary paths | @@ -35,8 +43,7 @@ pub const no_upper_bound = std.math.inf(f64); pub const Error = error{ EmptyMeasurement, InvalidMeasurement, - InvalidStateCount, - InvalidStateSpec, + InvalidRetrievalState, InvalidPressureProfile, WavelengthGridMismatch, MissingJacobian, @@ -46,40 +53,58 @@ pub const Error = error{ InvalidPriorCovariance, }; -// StateSpec --------------------------------------------------------------------------------------------------| -// Scalar retrieval variable description passed from Python and C into native OE. | -// | -// layout(64-bit) | -// size: 104 B (0.102 KiB), align: 8 B | -// | -// memory | -// [ 0.. 7] initial : f64 | -// [ 8.. 15] prior : f64 | -// [ 16.. 23] variance : f64 | -// [ 24.. 31] lower_bound : f64 | -// [ 32.. 39] upper_bound : f64 | -// [ 40.. 47] thickness_hpa : f64 | -// [ 48.. 95] pressure_altitude_profile: PressureAltitudeProfile | -// [ 96.. 99] interval_index_1based : u32 | -// [100..100] state : jacobian_states.State | -// [101..103] trailing padding : 3 B | -// | -// encoded fields | -// bounds use +/-inf for absence. An empty pressure profile means no pressure-state metadata. | -// | -// unused bits: 24 padding + 0 bool-storage slack = 24 bits | -// cache span: 2 cache lines at 64 B per line | -// footprint: per instance = 104 B (0.102 KiB); total = per instance * state count | -pub const StateSpec = struct { - state: jacobian_states.State, +// StateScalar ------------------------------------------------------------------------------------------------| +// Scalar OE lane knobs shared by both fixed retrieval states. | +// ------------------------------------------------------------------------------------------------------------| +pub const StateScalar = struct { initial: f64, prior: f64, variance: f64, lower_bound: f64 = no_lower_bound, upper_bound: f64 = no_upper_bound, - thickness_hpa: f64 = 0.0, - interval_index_1based: u32 = 0, - pressure_altitude_profile: PressureAltitudeProfile = .{}, +}; +// ------------------------------------------------------------------------------------------------------------| + +// PressureLayerPlacement -------------------------------------------------------------------------------------| +// Pressure-lane-only side data for moving the single retrieved aerosol layer. The profile is borrowed from | +// request-scoped storage that owns and frees the spline curvature. | +// ------------------------------------------------------------------------------------------------------------| +pub const PressureLayerPlacement = struct { + thickness_hpa: f64, + interval_index_1based: u32, + pressure_altitude_profile: *const PressureAltitudeProfile, + + pub fn hasPressureAltitudeProfile(self: PressureLayerPlacement) bool { + // PressureLayerPlacement.hasPressureAltitudeProfile --------------------------------------------------| + // Check whether the borrowed pressure-altitude profile has the rows needed for pressure conversion. | + // ----------------------------------------------------------------------------------------------------| + return self.pressure_altitude_profile.*.hasSamples(); + } + + pub fn altitudeDerivativeAtPressure(self: PressureLayerPlacement, pressure_hpa: f64) !f64 { + // PressureLayerPlacement.altitudeDerivativeAtPressure ------------------------------------------------| + // Convert a retrieved pressure step through the borrowed pressure-altitude profile. | + // ----------------------------------------------------------------------------------------------------| + return self.pressure_altitude_profile.*.altitudeDerivativeAtPressure(pressure_hpa); + } +}; +// ------------------------------------------------------------------------------------------------------------| + +// PressureState ----------------------------------------------------------------------------------------------| +// Fixed pressure retrieval lane: scalar OE knobs plus required placement metadata. | +// ------------------------------------------------------------------------------------------------------------| +pub const PressureState = struct { + scalar: StateScalar, + placement: PressureLayerPlacement, +}; +// ------------------------------------------------------------------------------------------------------------| + +// RetrievalState ---------------------------------------------------------------------------------------------| +// Complete fixed two-lane OE state, in public Jacobian order. | +// ------------------------------------------------------------------------------------------------------------| +pub const RetrievalState = struct { + aerosol_optical_depth: StateScalar, + aerosol_layer_mid_pressure: PressureState, }; // ------------------------------------------------------------------------------------------------------------| @@ -348,21 +373,20 @@ pub const Result = struct { history_state_vector_convergence: []f64 = &.{}, history_snr_normal: []u8 = &.{}, - pub fn init(allocator: Allocator, state_count: usize, max_iterations: usize) !Result { + pub fn init(allocator: Allocator, max_iterations: usize) !Result { // Result.init ----------------------------------------------------------------------------------------| // Allocate C/Python-borrowable result arrays for one retrieval run. | // ----------------------------------------------------------------------------------------------------| - if (state_count > max_state_count) return error.InvalidStateCount; - if (max_iterations > max_iteration_count) return error.InvalidStateSpec; + if (max_iterations > max_iteration_count) return error.InvalidRetrievalState; - var result: Result = .{ .state_count = @intCast(state_count) }; + var result: Result = .{ .state_count = @intCast(max_state_count) }; errdefer result.deinit(allocator); - result.state_ids = try allocator.alloc(jacobian_states.State, state_count); - result.state = try allocator.alloc(f64, state_count); - result.initial_state = try allocator.alloc(f64, state_count); - result.posterior_covariance = try allocator.alloc(f64, state_count * state_count); - result.averaging_kernel = try allocator.alloc(f64, state_count * state_count); - result.history_state = try allocator.alloc(f64, max_iterations * state_count); + result.state_ids = try allocator.alloc(jacobian_states.State, max_state_count); + result.state = try allocator.alloc(f64, max_state_count); + result.initial_state = try allocator.alloc(f64, max_state_count); + result.posterior_covariance = try allocator.alloc(f64, max_state_count * max_state_count); + result.averaging_kernel = try allocator.alloc(f64, max_state_count * max_state_count); + result.history_state = try allocator.alloc(f64, max_iterations * max_state_count); result.history_chi2 = try allocator.alloc(f64, max_iterations); result.history_chi2_reflectance = try allocator.alloc(f64, max_iterations); result.history_chi2_state_vector = try allocator.alloc(f64, max_iterations); @@ -423,25 +447,24 @@ pub const BatchResult = struct { state: []f64 = &.{}, history_state: []f64 = &.{}, - pub fn init(allocator: Allocator, run_count: usize, state_count: usize, history_capacity: usize) !BatchResult { + pub fn init(allocator: Allocator, run_count: usize, history_capacity: usize) !BatchResult { // BatchResult.init -----------------------------------------------------------------------------------| // Allocate run-major SoA arrays for a full-physics retrieval batch. | // ----------------------------------------------------------------------------------------------------| - if (run_count == 0) return error.InvalidStateSpec; - if (state_count == 0 or state_count > max_state_count) return error.InvalidStateCount; - if (history_capacity == 0 or history_capacity > max_iteration_count) return error.InvalidStateSpec; + if (run_count == 0) return error.InvalidRetrievalState; + if (history_capacity == 0 or history_capacity > max_iteration_count) return error.InvalidRetrievalState; var result: BatchResult = .{ .run_count = run_count, - .state_count = state_count, + .state_count = max_state_count, .history_capacity = history_capacity, }; errdefer result.deinit(allocator); result.iteration_count = try allocator.alloc(usize, run_count); result.converged = try allocator.alloc(u8, run_count); result.status = try allocator.alloc(u8, run_count); - result.state = try allocator.alloc(f64, run_count * state_count); - result.history_state = try allocator.alloc(f64, run_count * history_capacity * state_count); + result.state = try allocator.alloc(f64, run_count * max_state_count); + result.history_state = try allocator.alloc(f64, run_count * history_capacity * max_state_count); const batch = result.output(); for (0..batch.run_count) |run_index| resetBatchRun(batch, run_index, .pending); @@ -567,31 +590,28 @@ pub const FastmodeBatchResult = struct { pub fn init( allocator: Allocator, run_count: usize, - state_count: usize, history_capacity: usize, ) !FastmodeBatchResult { // FastmodeBatchResult.init ---------------------------------------------------------------------------| // Allocate final, fast-stage, and exact-correction result arrays for a fastmode batch. | // ----------------------------------------------------------------------------------------------------| - if (run_count == 0) return error.InvalidStateSpec; - - if (state_count == 0 or state_count > max_state_count) return error.InvalidStateCount; + if (run_count == 0) return error.InvalidRetrievalState; if (history_capacity == 0 or history_capacity > max_iteration_count * 2) { - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } var result: FastmodeBatchResult = .{ .run_count = run_count, - .state_count = state_count, + .state_count = max_state_count, .history_capacity = history_capacity, }; errdefer result.deinit(allocator); result.iteration_count = try allocator.alloc(usize, run_count); result.converged = try allocator.alloc(u8, run_count); result.status = try allocator.alloc(u8, run_count); - result.state = try allocator.alloc(f64, run_count * state_count); - result.history_state = try allocator.alloc(f64, run_count * history_capacity * state_count); + result.state = try allocator.alloc(f64, run_count * max_state_count); + result.history_state = try allocator.alloc(f64, run_count * history_capacity * max_state_count); result.fast_stage_iteration_count = try allocator.alloc(usize, run_count); result.fast_stage_converged = try allocator.alloc(u8, run_count); result.full_correction_iteration_count = try allocator.alloc(usize, run_count); @@ -644,30 +664,27 @@ pub const Controls = struct { // Dense state-space vectors derived once before an OE run or correction step. | // | // layout(64-bit) | -// size: 88 B (0.086 KiB), align: 8 B | +// size: 80 B (0.078 KiB), align: 8 B | // | // memory | -// [ 0..15] state : Vector | -// [16..31] prior : Vector | -// [32..47] variance : Vector | -// [48..63] lower : Vector | -// [64..79] upper : Vector | -// [80..80] derivative_state_mask: jacobian_states.StateMask | -// [81..87] trailing padding : 7 B | +// [ 0..15] state : Vector | +// [16..31] prior : Vector | +// [32..47] variance: Vector | +// [48..63] lower : Vector | +// [64..79] upper : Vector | // | // ownership | // No heap references. Copied by value so correction paths reuse prepared scalar state without aliases. | // | -// unused bits: 56 padding + 0 bool-storage slack = 56 bits | +// unused bits: 0 padding + 0 bool-storage slack = 0 bits | // cache span: 2 cache lines at 64 B per line | -// footprint: per instance = 88 B (0.086 KiB); one stack value per active retrieval or correction | +// footprint: per instance = 80 B (0.078 KiB); one stack value per active retrieval or correction | pub const StateSpace = struct { state: Vector, prior: Vector, variance: Vector, lower: Vector, upper: Vector, - derivative_state_mask: jacobian_states.StateMask, }; // ------------------------------------------------------------------------------------------------------------| @@ -718,15 +735,12 @@ pub const Accumulation = struct { // size: 32 B (0.031 KiB), align: 8 B | // | // memory | -// [ 0.. 1] source_index: [max_state_count]u8 | -// [ 2..15] padding : 14 B | -// [16..31] state_scale : Vector | +// [0..15] state_scale : Vector | // | -// unused bits: 112 padding + 0 bool-storage slack = 112 bits | +// unused bits: 0 padding + 0 bool-storage slack = 0 bits | // cache span: 1 cache line at 64 B per line | -// footprint: per instance = 32 B (0.031 KiB); one stack value per OE iteration | +// footprint: per instance = 16 B (0.016 KiB); one stack value per OE iteration | const JacobianProjection = struct { - source_index: [max_state_count]u8 = [_]u8{0} ** max_state_count, state_scale: Vector = algebra.zeroVector(), }; // ------------------------------------------------------------------------------------------------------------| @@ -753,13 +767,11 @@ pub const Step = struct { }; // ------------------------------------------------------------------------------------------------------------| -pub fn initializeStateSpace(state_specs: []const StateSpec, result: ?*Result) Error!StateSpace { - // initializeStateSpace ---------------------------------------------------------------------------------- | - // Convert validated state-spec rows into fixed two-lane vectors used by the Rodgers update. | - // | - // removed from the state vector. | +pub fn initializeStateSpace(retrieval_state: RetrievalState, result: ?*Result) Error!StateSpace { + // initializeStateSpace -----------------------------------------------------------------------------------| + // Convert the fixed two-lane state into vectors used by the Rodgers update. | // --------------------------------------------------------------------------------------------------------| - if (state_specs.len == 0 or state_specs.len > max_state_count) return error.InvalidStateCount; + try validateRetrievalState(retrieval_state); var state_space: StateSpace = .{ .state = algebra.zeroVector(), @@ -767,40 +779,44 @@ pub fn initializeStateSpace(state_specs: []const StateSpec, result: ?*Result) Er .variance = algebra.zeroVector(), .lower = algebra.zeroVector(), .upper = algebra.zeroVector(), - .derivative_state_mask = 0, }; - for (state_specs, 0..) |spec, index| { - try validateStateSpec(spec); - state_space.state[index] = spec.initial; - state_space.prior[index] = spec.prior; - state_space.variance[index] = spec.variance; - state_space.lower[index] = spec.lower_bound; - state_space.upper[index] = spec.upper_bound; - state_space.derivative_state_mask |= jacobian_states.stateMask(spec.state); - if (result) |full_result| { - full_result.state_ids[index] = spec.state; - full_result.initial_state[index] = spec.initial; - } + const aod = retrieval_state.aerosol_optical_depth; + const pressure = retrieval_state.aerosol_layer_mid_pressure.scalar; + state_space.state[0] = aod.initial; + state_space.prior[0] = aod.prior; + state_space.variance[0] = aod.variance; + state_space.lower[0] = aod.lower_bound; + state_space.upper[0] = aod.upper_bound; + state_space.state[1] = pressure.initial; + state_space.prior[1] = pressure.prior; + state_space.variance[1] = pressure.variance; + state_space.lower[1] = pressure.lower_bound; + state_space.upper[1] = pressure.upper_bound; + + if (result) |full_result| { + full_result.state_ids[0] = .aerosol_optical_depth; + full_result.state_ids[1] = .aerosol_layer_mid_pressure_hpa; + full_result.initial_state[0] = aod.initial; + full_result.initial_state[1] = pressure.initial; } return state_space; } pub fn preparePriorScales( state_space: StateSpace, - state_count: usize, scratch: *RetrievalIterationScratch, ) Error!void { // preparePriorScales ------------------------------------------------------------------------------------ | // Fill sqrt(Sa) and sqrt(Sa)^-1 scratch lanes from the diagonal prior covariance. | // --------------------------------------------------------------------------------------------------------| - try algebra.choleskyLowerDiagonal(state_space.variance[0..state_count], &scratch.sqrt_sa, &scratch.sqrt_inv_sa); + try algebra.choleskyLowerDiagonal(state_space.variance, &scratch.sqrt_sa, &scratch.sqrt_inv_sa); } pub fn accumulateNormalSystem( measurement: MeasuredReflectanceRows, evaluation: ReflectanceEvaluationRows, - state_specs: []const StateSpec, + retrieval_state: RetrievalState, previous: Vector, prior: Vector, sqrt_sa: Vector, @@ -830,24 +846,23 @@ pub fn accumulateNormalSystem( { return error.WavelengthGridMismatch; } - if (state_specs.len == 0 or state_specs.len > max_state_count) return error.InvalidStateCount; + try validateRetrievalState(retrieval_state); scratch.b = algebra.zeroVector(); scratch.g = algebra.zeroMatrix(); scratch.jt_invse_j = algebra.zeroMatrix(); - var projection: JacobianProjection = .{}; - for (state_specs, 0..) |spec, index| { - scratch.dx_white[index] = (previous[index] - prior[index]) / sqrt_sa[index]; - projection.source_index[index] = @intCast(jacobian_states.stateIndex(spec.state)); - projection.state_scale[index] = if (spec.state == .aerosol_layer_mid_pressure_hpa) - try spec.pressure_altitude_profile.altitudeDerivativeAtPressure(previous[index]) - else - 1.0; - } + scratch.dx_white[0] = (previous[0] - prior[0]) / sqrt_sa[0]; + scratch.dx_white[1] = (previous[1] - prior[1]) / sqrt_sa[1]; + + const projection = JacobianProjection{ + .state_scale = .{ + 1.0, + try retrieval_state.aerosol_layer_mid_pressure.placement.altitudeDerivativeAtPressure(previous[1]), + }, + }; var chi2_reflectance: f64 = 0.0; - var column_values = algebra.zeroVector(); for (measurement.wavelength_nm, 0..) |wavelength_nm, sample_index| { if (evaluation.wavelength_nm[sample_index] != wavelength_nm) return error.WavelengthGridMismatch; @@ -855,21 +870,25 @@ pub fn accumulateNormalSystem( const inv_variance = measurement.inv_variance[sample_index]; chi2_reflectance += residual * residual * inv_variance; - for (0..state_specs.len) |state_index| { - column_values[state_index] = - evaluation.jacobian[sample_index][projection.source_index[state_index]] * - projection.state_scale[state_index]; - } - - for (0..state_specs.len) |row| { - const weighted_row = column_values[row] * inv_variance; - scratch.b[row] += sqrt_sa[row] * weighted_row * residual; - for (0..state_specs.len) |col| { - const normal = weighted_row * column_values[col]; - scratch.jt_invse_j[row][col] += normal; - scratch.g[row][col] += sqrt_sa[row] * normal * sqrt_sa[col]; - } - } + const column0 = evaluation.jacobian[sample_index][0] * projection.state_scale[0]; + const column1 = evaluation.jacobian[sample_index][1] * projection.state_scale[1]; + const weighted0 = column0 * inv_variance; + const weighted1 = column1 * inv_variance; + const normal00 = weighted0 * column0; + const normal01 = weighted0 * column1; + const normal10 = weighted1 * column0; + const normal11 = weighted1 * column1; + + scratch.b[0] += sqrt_sa[0] * weighted0 * residual; + scratch.b[1] += sqrt_sa[1] * weighted1 * residual; + scratch.jt_invse_j[0][0] += normal00; + scratch.jt_invse_j[0][1] += normal01; + scratch.jt_invse_j[1][0] += normal10; + scratch.jt_invse_j[1][1] += normal11; + scratch.g[0][0] += sqrt_sa[0] * normal00 * sqrt_sa[0]; + scratch.g[0][1] += sqrt_sa[0] * normal01 * sqrt_sa[1]; + scratch.g[1][0] += sqrt_sa[1] * normal10 * sqrt_sa[0]; + scratch.g[1][1] += sqrt_sa[1] * normal11 * sqrt_sa[1]; } return .{ .chi2_reflectance = chi2_reflectance, @@ -878,7 +897,6 @@ pub fn accumulateNormalSystem( } pub fn solveStep( - state_count: usize, g: Matrix, b: Vector, prior: Vector, @@ -898,13 +916,13 @@ pub fn solveStep( // state = prior + sqrt(Sa) * eigenvectors * dx_new | // | // --------------------------------------------------------------------------------------------------------| - const eig = algebra.jacobiEigenSymmetric(g, state_count); + const eig = algebra.jacobiEigenSymmetric(g); scratch.eigenvectors = eig.vectors; - scratch.dx_trans = algebra.transposeMatrixVector(eig.vectors, scratch.dx_white, state_count); - scratch.rhs_trans = algebra.transposeMatrixVector(eig.vectors, b, state_count); + scratch.dx_trans = algebra.transposeMatrixVector(eig.vectors, scratch.dx_white); + scratch.rhs_trans = algebra.transposeMatrixVector(eig.vectors, b); var max_dx_trans: f64 = 0.0; - for (0..state_count) |index| max_dx_trans = @max(max_dx_trans, @abs(scratch.dx_trans[index])); + inline for (0..max_state_count) |index| max_dx_trans = @max(max_dx_trans, @abs(scratch.dx_trans[index])); const max_change = @max(max_change_transformed_state, max_dx_trans); var lambda_scale: f64 = 1.0; var snr_normal = true; @@ -912,11 +930,10 @@ pub fn solveStep( eig.values, scratch.rhs_trans, scratch.dx_trans, - state_count, lambda_scale, &scratch.dx_trans_new, ); - var change = transformedChange(scratch.dx_trans_new, scratch.dx_trans, state_count); + var change = transformedChange(scratch.dx_trans_new, scratch.dx_trans); if (change > 1.01 * max_change) { snr_normal = false; var factor_total: f64 = 1.0; @@ -927,11 +944,10 @@ pub fn solveStep( eig.values, scratch.rhs_trans, scratch.dx_trans, - state_count, scale2, &scratch.dx_trans_new, ); - change = transformedChange(scratch.dx_trans_new, scratch.dx_trans, state_count); + change = transformedChange(scratch.dx_trans_new, scratch.dx_trans); if (change < max_change) { lambda_scale = scale2; break; @@ -939,18 +955,18 @@ pub fn solveStep( } } - const rotated = algebra.matrixVector(eig.vectors, scratch.dx_trans_new, state_count); + const rotated = algebra.matrixVector(eig.vectors, scratch.dx_trans_new); var state = algebra.zeroVector(); - for (0..state_count) |index| { + inline for (0..max_state_count) |index| { scratch.dx_physical[index] = sqrt_sa[index] * rotated[index]; state[index] = prior[index] + scratch.dx_physical[index]; } - var posterior_white = algebra.identityMatrix(state_count); - for (0..state_count) |row| { - for (0..state_count) |col| { + var posterior_white = algebra.identityMatrix(); + inline for (0..max_state_count) |row| { + inline for (0..max_state_count) |col| { var value: f64 = 0.0; - for (0..state_count) |k| { + inline for (0..max_state_count) |k| { value += eig.vectors[row][k] * (lambda_scale * eig.values[k]) * eig.vectors[col][k]; } posterior_white[row][col] += value; @@ -958,8 +974,8 @@ pub fn solveStep( } var posterior_precision = algebra.zeroMatrix(); - for (0..state_count) |row| { - for (0..state_count) |col| { + inline for (0..max_state_count) |row| { + inline for (0..max_state_count) |col| { posterior_precision[row][col] = sqrt_inv_sa[row] * posterior_white[row][col] * sqrt_inv_sa[col]; } } @@ -970,65 +986,54 @@ pub fn solveStep( }; } -pub fn quadraticForm(matrix: Matrix, vector: Vector, state_count: usize) f64 { +pub fn quadraticForm(matrix: Matrix, vector: Vector) f64 { // quadraticForm ----------------------------------------------------------------------------------------- | - // Return vector^T * matrix * vector over the active leading state lanes. | + // Return vector^T * matrix * vector over the fixed two state lanes. | // --------------------------------------------------------------------------------------------------------| - var value: f64 = 0.0; - for (0..state_count) |row| { - var row_value: f64 = 0.0; - for (0..state_count) |col| row_value += matrix[row][col] * vector[col]; - value += vector[row] * row_value; - } - return value; + const row0 = matrix[0][0] * vector[0] + matrix[0][1] * vector[1]; + const row1 = matrix[1][0] * vector[0] + matrix[1][1] * vector[1]; + return vector[0] * row0 + vector[1] * row1; } fn computeTransformedUpdate( eigenvalues: Vector, rhs_trans: Vector, dx_trans: Vector, - state_count: usize, lambda_scale: f64, out: *Vector, ) void { // computeTransformedUpdate ------------------------------------------------------------------------------ | // Build the damped transformed-state update for the current eigenvalue scale. | // --------------------------------------------------------------------------------------------------------| - for (0..state_count) |index| { + inline for (0..max_state_count) |index| { const lambda = lambda_scale * eigenvalues[index]; out[index] = (lambda_scale * rhs_trans[index] + lambda * dx_trans[index]) / (lambda + 1.0); } } -fn transformedChange(next: Vector, previous: Vector, state_count: usize) f64 { +fn transformedChange(next: Vector, previous: Vector) f64 { // transformedChange ------------------------------------------------------------------------------------- | // Return the largest absolute transformed-state lane change. | // --------------------------------------------------------------------------------------------------------| var change: f64 = 0.0; - for (0..state_count) |index| change = @max(change, @abs(next[index] - previous[index])); + inline for (0..max_state_count) |index| change = @max(change, @abs(next[index] - previous[index])); return change; } -pub fn validateStateSpecs(state_specs: []const StateSpec) Error!void { - // validateStateSpecs -------------------------------------------------------------------------------------| - // Validate one C/Python OE state vector before the solver mutates the O2 A case. | - // | - // guard | - // fixed state count : 1..2 | - // covariance lanes : finite and positive | - // pressure-state lanes : carry interval placement, thickness, and pressure-to-altitude profile rows | - // non-pressure lanes : carry no pressure-placement side data | - // | +pub fn validateRetrievalState(retrieval_state: RetrievalState) Error!void { + // validateRetrievalState ---------------------------------------------------------------------------------| + // Validate the fixed two-lane OE state before the solver mutates the scene. | // --------------------------------------------------------------------------------------------------------| - if (state_specs.len == 0 or state_specs.len > max_state_count) return error.InvalidStateCount; - - var seen = [_]bool{false} ** jacobian_states.state_count; - for (state_specs) |spec| { - const state_index = @intFromEnum(spec.state); - if (seen[state_index]) return error.InvalidStateSpec; - - seen[state_index] = true; - try validateStateSpec(spec); + try validateStateScalar(retrieval_state.aerosol_optical_depth); + try validateStateScalar(retrieval_state.aerosol_layer_mid_pressure.scalar); + + const placement = retrieval_state.aerosol_layer_mid_pressure.placement; + if (!std.math.isFinite(placement.thickness_hpa) or + placement.thickness_hpa <= 0.0 or + placement.interval_index_1based == 0 or + !placement.hasPressureAltitudeProfile()) + { + return error.InvalidRetrievalState; } } @@ -1065,46 +1070,26 @@ pub fn freePressureProfile(allocator: Allocator, profile: PressureAltitudeProfil allocator.free(profile.second); } -fn validateStateSpec(spec: StateSpec) Error!void { - // validateStateSpec --------------------------------------------------------------------------------------| - // Validate scalar values and side-data shape for one retrieval state. | +fn validateStateScalar(scalar: StateScalar) Error!void { + // validateStateScalar ----------------------------------------------------------------------------------- | + // Validate scalar values shared by both fixed retrieval lanes. | // --------------------------------------------------------------------------------------------------------| - if (!std.math.isFinite(spec.initial) or - !std.math.isFinite(spec.prior) or - !std.math.isFinite(spec.variance) or - spec.variance <= 0.0) + if (!std.math.isFinite(scalar.initial) or + !std.math.isFinite(scalar.prior) or + !std.math.isFinite(scalar.variance) or + scalar.variance <= 0.0) { - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } - if (std.math.isNan(spec.lower_bound) or std.math.isNan(spec.upper_bound)) return error.InvalidStateSpec; - if (spec.lower_bound != no_lower_bound and !std.math.isFinite(spec.lower_bound)) { - return error.InvalidStateSpec; + if (std.math.isNan(scalar.lower_bound) or std.math.isNan(scalar.upper_bound)) return error.InvalidRetrievalState; + if (scalar.lower_bound != no_lower_bound and !std.math.isFinite(scalar.lower_bound)) { + return error.InvalidRetrievalState; } - if (spec.upper_bound != no_upper_bound and !std.math.isFinite(spec.upper_bound)) { - return error.InvalidStateSpec; - } - if (spec.lower_bound > spec.upper_bound) return error.InvalidStateSpec; - - switch (spec.state) { - .aerosol_layer_mid_pressure_hpa => { - if (!std.math.isFinite(spec.thickness_hpa) or - spec.thickness_hpa <= 0.0 or - spec.interval_index_1based == 0 or - !spec.pressure_altitude_profile.hasSamples()) - { - return error.InvalidStateSpec; - } - }, - .aerosol_optical_depth => { - if (spec.thickness_hpa != 0.0 or - spec.interval_index_1based != 0 or - spec.pressure_altitude_profile.hasSamples()) - { - return error.InvalidStateSpec; - } - }, + if (scalar.upper_bound != no_upper_bound and !std.math.isFinite(scalar.upper_bound)) { + return error.InvalidRetrievalState; } + if (scalar.lower_bound > scalar.upper_bound) return error.InvalidRetrievalState; } fn initializeFastmodeBatchResult(result: *FastmodeBatchResult) void { @@ -1210,36 +1195,7 @@ fn endpointSplineSecondDerivatives( second[1] = 0.0; return; } - if (count > max_state_count) { - return endpointSplineSecondDerivativesDynamic(allocator, x, pressure_hpa, second); - } - - var matrix = algebra.zeroMatrix(); - var rhs = algebra.zeroVector(); - const width0 = x[1] - x[0]; - matrix[0][0] = 2.0 * width0; - matrix[0][1] = width0; - rhs[0] = 0.0; - - for (1..count - 1) |index| { - const width_left = x[index] - x[index - 1]; - const width_right = x[index + 1] - x[index]; - const slope_left = (@log(pressure_hpa[index]) - @log(pressure_hpa[index - 1])) / width_left; - const slope_right = (@log(pressure_hpa[index + 1]) - @log(pressure_hpa[index])) / width_right; - matrix[index][index - 1] = width_left; - matrix[index][index] = 2.0 * (width_left + width_right); - matrix[index][index + 1] = width_right; - rhs[index] = 6.0 * (slope_right - slope_left); - } - - const last = count - 1; - const width_last = x[last] - x[last - 1]; - matrix[last][last - 1] = width_last; - matrix[last][last] = 2.0 * width_last; - - const inverse = try algebra.invertSymmetric(matrix, count); - const solved = algebra.matrixVector(inverse, rhs, count); - for (0..count) |index| second[index] = solved[index]; + return endpointSplineSecondDerivativesDynamic(allocator, x, pressure_hpa, second); } fn endpointSplineSecondDerivativesDynamic( @@ -1252,14 +1208,16 @@ fn endpointSplineSecondDerivativesDynamic( // Solve the same log-pressure spline equations for profiles too large for fixed two-state storage. | // --------------------------------------------------------------------------------------------------------| const count = x.len; - var lower = try allocator.alloc(f64, count); - defer allocator.free(lower); - var diag = try allocator.alloc(f64, count); - defer allocator.free(diag); - var upper = try allocator.alloc(f64, count); - defer allocator.free(upper); - var rhs = try allocator.alloc(f64, count); - defer allocator.free(rhs); + var stack_allocator = std.heap.stackFallback(pressure_profile_spline_stack_bytes, allocator); + const scratch_allocator = stack_allocator.get(); + var lower = try scratch_allocator.alloc(f64, count); + defer scratch_allocator.free(lower); + var diag = try scratch_allocator.alloc(f64, count); + defer scratch_allocator.free(diag); + var upper = try scratch_allocator.alloc(f64, count); + defer scratch_allocator.free(upper); + var rhs = try scratch_allocator.alloc(f64, count); + defer scratch_allocator.free(rhs); @memset(lower, 0.0); @memset(diag, 0.0); diff --git a/src/root.zig b/src/root.zig index 40443a986..4c2977232 100644 --- a/src/root.zig +++ b/src/root.zig @@ -35,7 +35,7 @@ const Allocator = std.mem.Allocator; pub const geometry_direction_cosine_floor: f64 = 0.05; // root.zig ---------------------------------------------------------------------------------------------------| -// Public explicit row surface for the O2 A forward model. | +// Public explicit row surface for the forward model. | // | // public flow | // Scene -> prepare -> warmSessionMemory -> runForwardWithSessionMemory | @@ -62,7 +62,9 @@ pub const Spectrum = output_spectrum.Spectrum; pub const SpectrumRunResult = output_spectrum.SpectrumRunResult; pub const SpectrumRunSummary = output_spectrum.SpectrumRunSummary; pub const optimal_estimation = retrieval; -pub const RetrievalStateSpec = retrieval.StateSpec; +pub const RetrievalState = retrieval.RetrievalState; +pub const RetrievalStateScalar = retrieval.StateScalar; +pub const RetrievalPressureLayerPlacement = retrieval.PressureLayerPlacement; pub const RetrievalPressureAltitudeProfile = retrieval.PressureAltitudeProfile; pub const RetrievalResult = retrieval.Result; pub const RetrievalBatchResult = retrieval.BatchResult; @@ -79,24 +81,24 @@ pub const buildRunTables = setup_tables.buildRunTables; pub const buildProfileLineValues = profile_lines.buildProfileLineValues; // Prepared ---------------------------------------------------------------------------------------------------| -// Public owner for parsed or caller-provided O2 A controls and setup tables. | +// Public owner for parsed or caller-provided controls and setup tables. | // | // layout(64-bit) | // size: 2728 B (2.664 KiB), align: 8 | // | // memory | -// [ 0.. 703] case : Scene | +// [ 0.. 703] scene : Scene | // [ 704..2727] tables: RunTables | // | // referenced storage | -// case borrows control strings/slices. tables owns loaded physical setup arrays and scalar tables. | +// scene borrows control strings/slices. tables owns loaded physical setup arrays and scalar tables. | pub const Prepared = struct { scene: Scene, tables: RunTables, pub fn deinit(self: *Prepared, allocator: Allocator) void { // Prepared.deinit ------------------------------------------------------------------------------------| - // Release setup tables; case strings and slices are borrowed from caller-owned storage. | + // Release setup tables; scene strings and slices are borrowed from caller-owned storage. | // ----------------------------------------------------------------------------------------------------| self.tables.deinit(allocator); self.* = undefined; @@ -106,14 +108,14 @@ pub const Prepared = struct { pub fn initSessionMemory(allocator: Allocator) SessionMemory { // initSessionMemory --------------------------------------------------------------------------------------| - // Create an empty reusable O2 A session cache. | + // Create an empty reusable session cache. | // --------------------------------------------------------------------------------------------------------| return SessionMemory.init(allocator); } pub fn prepare(allocator: Allocator, scene: Scene) !Prepared { // prepare ------------------------------------------------------------------------------------------------| - // Build the O2 A setup tables retained across forward runs. | + // Build the setup tables retained across forward runs. | // --------------------------------------------------------------------------------------------------------| return .{ .scene = scene, @@ -128,7 +130,7 @@ pub fn warmSessionMemory( solve_config: SolveConfig, ) !void { // warmSessionMemory --------------------------------------------------------------------------------------| - // Materialize reusable spectrum, radiance, profile-line, solar, and transport memory for one case. | + // Materialize reusable spectrum, radiance, profile-line, solar, and transport memory for one scene. | // --------------------------------------------------------------------------------------------------------| const prepared_solve_config = try controls.prepareSolveConfig(solve_config); _ = try prepareSessionRows(allocator, session, prepared, prepared_solve_config); @@ -141,7 +143,7 @@ pub fn runForwardWithSessionMemory( solve_config: SolveConfig, ) !SpectrumRunResult { // runForwardWithSessionMemory ----------------------------------------------------------------------------| - // Run the O2 A product-grid spectrum through caller-retained session memory and return owned arrays. | + // Run the product-grid spectrum through caller-retained session memory and return owned arrays. | // | // Root-level orchestration follows the integrated transport route in | // `tests/unit/spectrum/spectrum_run_test.zig` | @@ -179,7 +181,7 @@ pub fn runForwardWithSessionMemory( // runForward sampling policy -----------------------------------------------------------------------------| // Canonical expected values owned by this repository. | - // product row using integrated radiance and integrated irradiance sampling. O2 A | + // product row using integrated radiance and integrated irradiance sampling. | // `evidence/python-reference-case-native.json` exposes no Python-native key for calibration arrays, slit | // kernels, or per-channel integration overrides. Keep this policy fixed here; | // user-configurable controls enter through Scene JSON and `solveConfig`. | @@ -228,7 +230,7 @@ pub fn runForwardWithSessionMemory( pub fn runForward(allocator: Allocator, prepared: *const Prepared, solve_config: SolveConfig) !SpectrumRunResult { // runForward ---------------------------------------------------------------------------------------------| - // Run one O2 A spectrum with a short-lived session memory owner. | + // Run one spectrum with a short-lived session memory owner. | // --------------------------------------------------------------------------------------------------------| var session = initSessionMemory(allocator); defer session.deinit(allocator); @@ -242,15 +244,15 @@ pub fn runOptimalEstimation( measurement_wavelength_nm: []const f64, measurement_reflectance: []const f64, measurement_variance: []const f64, - state_specs: []const retrieval.StateSpec, + retrieval_state: retrieval.RetrievalState, retrieval_controls: retrieval.Controls, ) !retrieval.Result { // runOptimalEstimation -----------------------------------------------------------------------------------| - // Run one full-physics O2 A optimal-estimation solve through the explicit forward path. | + // Run one full-physics optimal-estimation solve through the explicit forward path. | // | // steps | // 1. copy measurement rows into dense retrieval SoA storage | - // 2. initialize the fixed two-state Rodgers vectors from StateSpec rows | + // 2. initialize the fixed two-state Rodgers vectors from RetrievalState | // 3. per iteration: apply scalar retrieval state to an Scene, refresh layer/aerosol rows, run | // spectrum plus reflectance Jacobians, stream the normal system, and solve the transformed update | // 4. write history, posterior covariance, and averaging kernel into the retained Result owner | @@ -277,16 +279,19 @@ pub fn runOptimalEstimation( ); defer measurement.deinit(allocator); - var result = try retrieval.Result.init(allocator, state_specs.len, retrieval_controls.max_iterations); + try retrieval.validateRetrievalState(retrieval_state); + + const state_count = retrieval.max_state_count; + var result = try retrieval.Result.init(allocator, retrieval_controls.max_iterations); errdefer result.deinit(allocator); - const state_space = try retrieval.initializeStateSpace(state_specs, &result); - const state_count = state_specs.len; - const has_pressure_state = retrievalStateActive(state_specs, .aerosol_layer_mid_pressure_hpa); - if (has_pressure_state and prepared.scene.atmosphere.intervals.len == 0) return error.InvalidStateSpec; + const state_space = try retrieval.initializeStateSpace(retrieval_state, &result); + if (prepared.scene.atmosphere.intervals.len == 0) return error.InvalidRetrievalState; - const mutable_interval_count = if (has_pressure_state) prepared.scene.atmosphere.intervals.len else 0; - const mutable_intervals = try allocator.alloc(scene_input.VerticalInterval, mutable_interval_count); + const mutable_intervals = try allocator.alloc( + scene_input.VerticalInterval, + prepared.scene.atmosphere.intervals.len, + ); defer allocator.free(mutable_intervals); var retrieval_layers = try atmosphere_layers.buildFromPreparedProfiles( @@ -300,7 +305,7 @@ pub fn runOptimalEstimation( var state = state_space.state; var scratch: retrieval.RetrievalIterationScratch = .{}; - try retrieval.preparePriorScales(state_space, state_count, &scratch); + try retrieval.preparePriorScales(state_space, &scratch); var final_posterior_precision: retrieval.StateMatrix = .{.{0.0} ** retrieval.max_state_count} ** retrieval.max_state_count; @@ -313,9 +318,8 @@ pub fn runOptimalEstimation( allocator, session, prepared, - state_specs, + retrieval_state, previous, - state_space.derivative_state_mask, mutable_intervals, &retrieval_layers, ); @@ -328,7 +332,7 @@ pub fn runOptimalEstimation( .reflectance = forward.spectrum.reflectance, .jacobian = forward.spectrum.jacobian, }, - state_specs, + retrieval_state, previous, state_space.prior, scratch.sqrt_sa, @@ -336,7 +340,6 @@ pub fn runOptimalEstimation( ); const step = try retrieval.solveStep( - state_count, scratch.g, scratch.b, state_space.prior, @@ -357,7 +360,7 @@ pub fn runOptimalEstimation( for (0..state_count) |index| { chi2_state += dx_iter[index] * dx_iter[index] / state_space.variance[index]; } - const state_conv = retrieval.quadraticForm(step.posterior_precision, dx_iter, state_count) / + const state_conv = retrieval.quadraticForm(step.posterior_precision, dx_iter) / @as(f64, @floatFromInt(state_count)); converged = state_conv < retrieval_controls.state_vector_convergence_threshold and step.snr_normal; @@ -375,8 +378,8 @@ pub fn runOptimalEstimation( if (converged) break; } - const posterior_covariance = try retrieval_algebra.invertSymmetric(final_posterior_precision, state_count); - const averaging_kernel = retrieval_algebra.multiply(posterior_covariance, scratch.jt_invse_j, state_count); + const posterior_covariance = try retrieval_algebra.invertSymmetric(final_posterior_precision); + const averaging_kernel = retrieval_algebra.multiply(posterior_covariance, scratch.jt_invse_j); result.iteration_count = @intCast(iteration_count); result.converged = converged; for (0..state_count) |row| { @@ -396,11 +399,11 @@ pub fn runOptimalEstimationCorrection( measurement_wavelength_nm: []const f64, measurement_reflectance: []const f64, measurement_variance: []const f64, - state_specs: []const retrieval.StateSpec, + retrieval_state: retrieval.RetrievalState, retrieval_controls: retrieval.Controls, ) !retrieval.Result { // runOptimalEstimationCorrection --------------------------------------------------------------------- | - // Apply one full-physics O2 A correction step to a prepared case already written at the fast-stage state. | + // Apply one full-physics correction step to a prepared scene already written at the fast-stage state. | // | // contract | // The correction path returns exactly one Rodgers update. Controls still validate solver thresholds, | @@ -422,16 +425,19 @@ pub fn runOptimalEstimationCorrection( ); defer measurement.deinit(allocator); - var result = try retrieval.Result.init(allocator, state_specs.len, 1); + try retrieval.validateRetrievalState(retrieval_state); + + const state_count = retrieval.max_state_count; + var result = try retrieval.Result.init(allocator, 1); errdefer result.deinit(allocator); - const state_space = try retrieval.initializeStateSpace(state_specs, &result); - const state_count = state_specs.len; - const has_pressure_state = retrievalStateActive(state_specs, .aerosol_layer_mid_pressure_hpa); - if (has_pressure_state and prepared.scene.atmosphere.intervals.len == 0) return error.InvalidStateSpec; + const state_space = try retrieval.initializeStateSpace(retrieval_state, &result); + if (prepared.scene.atmosphere.intervals.len == 0) return error.InvalidRetrievalState; - const mutable_interval_count = if (has_pressure_state) prepared.scene.atmosphere.intervals.len else 0; - const mutable_intervals = try allocator.alloc(scene_input.VerticalInterval, mutable_interval_count); + const mutable_intervals = try allocator.alloc( + scene_input.VerticalInterval, + prepared.scene.atmosphere.intervals.len, + ); defer allocator.free(mutable_intervals); var retrieval_layers = try atmosphere_layers.buildFromPreparedProfiles( @@ -444,15 +450,14 @@ pub fn runOptimalEstimationCorrection( defer retrieval_layers.deinit(allocator); var scratch: retrieval.RetrievalIterationScratch = .{}; - try retrieval.preparePriorScales(state_space, state_count, &scratch); + try retrieval.preparePriorScales(state_space, &scratch); var forward = try evaluateRetrievalState( allocator, session, prepared, - state_specs, + retrieval_state, state_space.state, - state_space.derivative_state_mask, mutable_intervals, &retrieval_layers, ); @@ -465,7 +470,7 @@ pub fn runOptimalEstimationCorrection( .reflectance = forward.spectrum.reflectance, .jacobian = forward.spectrum.jacobian, }, - state_specs, + retrieval_state, state_space.state, state_space.prior, scratch.sqrt_sa, @@ -473,7 +478,6 @@ pub fn runOptimalEstimationCorrection( ); const step = try retrieval.solveStep( - state_count, scratch.g, scratch.b, state_space.prior, @@ -495,7 +499,7 @@ pub fn runOptimalEstimationCorrection( for (0..state_count) |index| { chi2_state += dx_iter[index] * dx_iter[index] / state_space.variance[index]; } - const state_conv = retrieval.quadraticForm(step.posterior_precision, dx_iter, state_count) / + const state_conv = retrieval.quadraticForm(step.posterior_precision, dx_iter) / @as(f64, @floatFromInt(state_count)); const converged = state_conv < retrieval_controls.state_vector_convergence_threshold and step.snr_normal; @@ -506,8 +510,8 @@ pub fn runOptimalEstimationCorrection( result.history_state_vector_convergence[0] = state_conv; result.history_snr_normal[0] = if (step.snr_normal) 1 else 0; - const posterior_covariance = try retrieval_algebra.invertSymmetric(step.posterior_precision, state_count); - const averaging_kernel = retrieval_algebra.multiply(posterior_covariance, accumulation.jt_invse_j, state_count); + const posterior_covariance = try retrieval_algebra.invertSymmetric(step.posterior_precision); + const averaging_kernel = retrieval_algebra.multiply(posterior_covariance, accumulation.jt_invse_j); result.iteration_count = 1; result.converged = converged; for (0..state_count) |row| { @@ -527,7 +531,7 @@ pub fn runOptimalEstimationBatch( measurement_wavelength_nm: []const f64, measurement_reflectance: []const f64, measurement_variance: []const f64, - state_template: []const retrieval.StateSpec, + state_template: retrieval.RetrievalState, initial_states: []const f64, prior_states: []const f64, retrieval_controls: retrieval.Controls, @@ -536,40 +540,39 @@ pub fn runOptimalEstimationBatch( // Run a correctness-first full-physics OE batch over caller-provided start/prior rows. | // | // boundary | - // This is the single-worker O2 A batch slice. Each start reuses the same public measurement rows and | - // prepared case, then copies compact result rows into the run-major BatchResult owner. | + // This is the single-worker batch slice. Each start reuses the same public measurement rows and | + // prepared scene, then copies compact result rows into the run-major BatchResult owner. | // | // failure model | // OutOfMemory aborts the whole batch. Numerical or control failures mark only that start as failed, | // --------------------------------------------------------------------------------------------------------| try validateRetrievalControls(retrieval_controls); - if (state_template.len == 0 or state_template.len > retrieval.max_state_count) return error.InvalidStateCount; + try retrieval.validateRetrievalState(state_template); + + const state_count = retrieval.max_state_count; - if (initial_states.len != prior_states.len) return error.InvalidStateSpec; + if (initial_states.len != prior_states.len) return error.InvalidRetrievalState; - if (initial_states.len % state_template.len != 0) return error.InvalidStateSpec; + if (initial_states.len % state_count != 0) return error.InvalidRetrievalState; - const run_count = initial_states.len / state_template.len; - if (run_count == 0) return error.InvalidStateSpec; + const run_count = initial_states.len / state_count; + if (run_count == 0) return error.InvalidRetrievalState; var result = try retrieval.BatchResult.init( allocator, run_count, - state_template.len, retrieval_controls.max_iterations, ); errdefer result.deinit(allocator); for (0..run_count) |run_index| { - const state_offset = run_index * state_template.len; - var run_specs_buffer: [retrieval.max_state_count]retrieval.StateSpec = undefined; - for (state_template, 0..) |template, state_index| { - var spec = template; - spec.initial = initial_states[state_offset + state_index]; - spec.prior = prior_states[state_offset + state_index]; - run_specs_buffer[state_index] = spec; - } + const state_offset = run_index * state_count; + var run_state = state_template; + run_state.aerosol_optical_depth.initial = initial_states[state_offset]; + run_state.aerosol_optical_depth.prior = prior_states[state_offset]; + run_state.aerosol_layer_mid_pressure.scalar.initial = initial_states[state_offset + 1]; + run_state.aerosol_layer_mid_pressure.scalar.prior = prior_states[state_offset + 1]; var run = runOptimalEstimation( allocator, @@ -578,7 +581,7 @@ pub fn runOptimalEstimationBatch( measurement_wavelength_nm, measurement_reflectance, measurement_variance, - run_specs_buffer[0..state_template.len], + run_state, retrieval_controls, ) catch |err| switch (err) { error.OutOfMemory => return err, @@ -592,12 +595,12 @@ pub fn runOptimalEstimationBatch( result.iteration_count[run_index] = run.iteration_count; result.converged[run_index] = if (run.converged) 1 else 0; result.status[run_index] = @intFromEnum(retrieval.BatchRunStatus.ok); - for (0..state_template.len) |state_index| { + for (0..state_count) |state_index| { result.state[state_offset + state_index] = run.state[state_index]; } - const history_offset = run_index * result.history_capacity * state_template.len; - const history_len = @as(usize, run.iteration_count) * state_template.len; + const history_offset = run_index * result.history_capacity * state_count; + const history_len = @as(usize, run.iteration_count) * state_count; @memcpy( result.history_state[history_offset .. history_offset + history_len], run.history_state[0..history_len], @@ -615,7 +618,7 @@ pub fn runFastmodeOptimalEstimationBatch( fast_measurement_wavelength_nm: []const f64, fast_measurement_reflectance: []const f64, fast_measurement_variance: []const f64, - fast_state_template: []const retrieval.StateSpec, + fast_state_template: retrieval.RetrievalState, initial_states: []const f64, prior_states: []const f64, fast_controls: retrieval.Controls, @@ -623,7 +626,7 @@ pub fn runFastmodeOptimalEstimationBatch( correction_measurement_wavelength_nm: []const f64, correction_measurement_reflectance: []const f64, correction_measurement_variance: []const f64, - correction_state_template: []const retrieval.StateSpec, + correction_state_template: retrieval.RetrievalState, correction_prior_states: []const f64, correction_controls: retrieval.Controls, ) !retrieval.FastmodeBatchResult { @@ -631,20 +634,17 @@ pub fn runFastmodeOptimalEstimationBatch( // Run fast-stage starts, then run a full-physics correction batch seeded from the fast-stage states. | // | // boundary | - // Python owns the fast and correction case construction. Zig receives two prepared O2 A scenes, two | + // Python owns the fast and correction scene construction. Zig receives two prepared scenes, two | // measurement grids, and config-driven controls; no wavelength counts are hardcoded here. | // | // converged flags keep fast-stage convergence when the correction stage returns ok. | // --------------------------------------------------------------------------------------------------------| - if (fast_state_template.len != correction_state_template.len) return error.InvalidStateSpec; + try retrieval.validateRetrievalState(fast_state_template); + try retrieval.validateRetrievalState(correction_state_template); - for (fast_state_template, correction_state_template) |fast_spec, correction_spec| { - if (fast_spec.state != correction_spec.state) return error.InvalidStateSpec; - } + if (initial_states.len != prior_states.len) return error.InvalidRetrievalState; - if (initial_states.len != prior_states.len) return error.InvalidStateSpec; - - if (correction_prior_states.len != prior_states.len) return error.InvalidStateSpec; + if (correction_prior_states.len != prior_states.len) return error.InvalidRetrievalState; var fast = try runOptimalEstimationBatch( allocator, @@ -678,7 +678,6 @@ pub fn runFastmodeOptimalEstimationBatch( var result = try retrieval.FastmodeBatchResult.init( allocator, fast.run_count, - fast.state_count, total_history_capacity, ); errdefer result.deinit(allocator); @@ -768,7 +767,7 @@ pub fn buildAtmosphericBudget( wavelengths_nm: []const f64, ) !AtmosphericBudget { // buildAtmosphericBudget ---------------------------------------------------------------------------------| - // Build public atmospheric support-row diagnostic rows for the prepared O2 A case. | + // Build public atmospheric support-row diagnostic rows for the prepared scene. | // --------------------------------------------------------------------------------------------------------| return atmospheric_budget.build(allocator, prepared.scene, &prepared.tables, wavelengths_nm); } @@ -818,13 +817,12 @@ pub fn buildInstrumentResponse( pub fn solveConfig(scene: Scene) SolveConfig { // solveConfig --------------------------------------------------------------------------------------------| - // Build the exercised O2 A transport controls used by integrated transport evidence. | + // Build the exercised transport controls used by integrated transport evidence. | // --------------------------------------------------------------------------------------------------------| const performance_thresholds = performanceThresholdsWithFourierLimit(scene.rtm); return .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth) | - jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, .controls = .{ .scattering = .multiple, .n_streams = @intCast(scene.rtm.stream_count), @@ -856,14 +854,14 @@ fn validateRetrievalControls(retrieval_controls: retrieval.Controls) !void { if (retrieval_controls.max_iterations == 0 or retrieval_controls.max_iterations > retrieval.max_iteration_count) { - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } if (retrieval_controls.state_vector_convergence_threshold <= 0.0 or retrieval_controls.max_change_transformed_state <= 0.0 or !std.math.isFinite(retrieval_controls.state_vector_convergence_threshold) or !std.math.isFinite(retrieval_controls.max_change_transformed_state)) { - return error.InvalidStateSpec; + return error.InvalidRetrievalState; } } @@ -871,9 +869,8 @@ fn evaluateRetrievalState( allocator: Allocator, session: *SessionMemory, prepared: *const Prepared, - state_specs: []const retrieval.StateSpec, + retrieval_state: retrieval.RetrievalState, state: retrieval.StateVector, - derivative_state_mask: jacobian_states.StateMask, mutable_intervals: []scene_input.VerticalInterval, retrieval_layers: *atmosphere_layers.LayerGrid, ) !SpectrumRunResult { @@ -882,7 +879,7 @@ fn evaluateRetrievalState( // | // boundary | // The retrieval driver owns one retained LayerGrid. This function refills its computed rows for the | - // current state while borrowing line, CIA, phase, instrument, and solar tables from the prepared case. | + // current state while borrowing line, CIA, phase, instrument, and solar tables from the prepared scene. | // | // refresh | // aerosol optical depth : rebuild inline AerosolLayerTable | @@ -897,7 +894,7 @@ fn evaluateRetrievalState( @memcpy(mutable_intervals, prepared.scene.atmosphere.intervals); scene.atmosphere.intervals = mutable_intervals; } - try writeRetrievalStateToScene(&scene, mutable_intervals, state_specs, state); + try writeRetrievalStateToScene(&scene, mutable_intervals, retrieval_state, state); try input_validate.sceneControls(scene); try atmosphere_layers.refillFromPreparedProfiles(retrieval_layers, scene, prepared.tables.quadrature); @@ -911,7 +908,7 @@ fn evaluateRetrievalState( var solve_config = solveConfig(scene); solve_config.derivative_mode = .semi_analytical; - solve_config.derivative_state_mask = derivative_state_mask; + solve_config.wants_jacobian = true; return runForwardWithSessionMemory(allocator, session, &evaluation_prepared, solve_config); } @@ -919,67 +916,53 @@ fn evaluateRetrievalState( fn writeRetrievalStateToScene( scene: *Scene, mutable_intervals: []scene_input.VerticalInterval, - state_specs: []const retrieval.StateSpec, + retrieval_state: retrieval.RetrievalState, state: retrieval.StateVector, ) !void { // writeRetrievalStateToScene -----------------------------------------------------------------------------| - // Write active OE scalar values into the case copy used for this iteration. | + // Write active OE scalar values into the scene copy used for this iteration. | // | // pressure placement | // target interval takes the new aerosol top/bottom pressures, the adjacent interval boundaries move | // with it, and no other interval is changed. | // --------------------------------------------------------------------------------------------------------| - for (state_specs, 0..) |spec, index| { - const value = state[index]; - switch (spec.state) { - .aerosol_optical_depth => scene.aerosol.optical_depth = value, - .aerosol_layer_mid_pressure_hpa => { - if (mutable_intervals.len == 0) return error.InvalidStateSpec; - - const fit_interval_index = @as(usize, spec.interval_index_1based); - const half_thickness = 0.5 * spec.thickness_hpa; - const top_pressure = value - half_thickness; - const bottom_pressure = value + half_thickness; - - scene.aerosol.interval_index_1based = fit_interval_index; - scene.aerosol.top_pressure_hpa = top_pressure; - scene.aerosol.bottom_pressure_hpa = bottom_pressure; - - var updated = false; - for (mutable_intervals) |*interval| { - const interval_index = interval.index_1based; - - if (interval_index == fit_interval_index) { - interval.top_pressure_hpa = top_pressure; - interval.bottom_pressure_hpa = bottom_pressure; - updated = true; - continue; - } - - if (interval_index + 1 == fit_interval_index) { - interval.bottom_pressure_hpa = top_pressure; - continue; - } - - if (interval_index == fit_interval_index + 1) { - interval.top_pressure_hpa = bottom_pressure; - } - } - - if (!updated) return error.InvalidStateSpec; - }, + scene.aerosol.optical_depth = state[0]; + + if (mutable_intervals.len == 0) return error.InvalidRetrievalState; + + const placement = retrieval_state.aerosol_layer_mid_pressure.placement; + const value = state[1]; + const fit_interval_index = @as(usize, placement.interval_index_1based); + const half_thickness = 0.5 * placement.thickness_hpa; + const top_pressure = value - half_thickness; + const bottom_pressure = value + half_thickness; + + scene.aerosol.interval_index_1based = fit_interval_index; + scene.aerosol.top_pressure_hpa = top_pressure; + scene.aerosol.bottom_pressure_hpa = bottom_pressure; + + var updated = false; + for (mutable_intervals) |*interval| { + const interval_index = interval.index_1based; + + if (interval_index == fit_interval_index) { + interval.top_pressure_hpa = top_pressure; + interval.bottom_pressure_hpa = bottom_pressure; + updated = true; + continue; } - } -} -fn retrievalStateActive(state_specs: []const retrieval.StateSpec, state: jacobian_states.State) bool { - // retrievalStateActive ---------------------------------------------------------------------------------- | - // Test whether one fixed Jacobian state appears in the current OE state vector. | - // --------------------------------------------------------------------------------------------------------| - for (state_specs) |spec| { - if (spec.state == state) return true; + if (interval_index + 1 == fit_interval_index) { + interval.bottom_pressure_hpa = top_pressure; + continue; + } + + if (interval_index == fit_interval_index + 1) { + interval.top_pressure_hpa = bottom_pressure; + } } - return false; + + if (!updated) return error.InvalidRetrievalState; } const PreparedSessionRows = struct { @@ -1000,44 +983,64 @@ fn prepareSessionRows( // Rebuild shape rows and retain expensive profile-line values for one already-validated solve route. | // | // hot path | - // Root prepares SolveConfig once before this boundary. Wavelength workers receive the same sanitized | - // mask and validated stream/threshold controls, matching the prepared LABOS execution route. | + // Root prepares SolveConfig once before this boundary. Wavelength workers receive the same validated | + // stream and threshold controls, matching the prepared LABOS execution route. | // --------------------------------------------------------------------------------------------------------| const sampling_stamp = samplingTableReuseStamp(prepared); - const table = resolve_sampling_table: { - if (session.spectrum.hasTable(sampling_stamp)) { - break :resolve_sampling_table try session.spectrum.table( - session.spectrum.rows.len, - session.spectrum.kernel_offsets_nm.len, - ); - } - - var owned_sampling = try sampling_table.buildSpectrumSamplingTable( + if (!session.spectrum.hasTable(sampling_stamp)) { + const built_sampling = try sampling_table.buildSpectrumSamplingTable( allocator, prepared.scene, prepared.tables.instrument, prepared.tables.lines, ); - defer owned_sampling.deinit(allocator); - const row_count = owned_sampling.rows.len; - const side_sample_count = owned_sampling.kernel_offsets_nm.len; - session.spectrum.takeTable(allocator, &owned_sampling, sampling_stamp); - break :resolve_sampling_table try session.spectrum.table(row_count, side_sample_count); - }; + session.spectrum.rebuildTable( + allocator, + built_sampling.rows, + built_sampling.kernel_offsets_nm, + built_sampling.kernel_weights, + sampling_stamp, + ); + } + const table = try session.spectrum.table( + session.spectrum.rows.len, + session.spectrum.kernel_offsets_nm.len, + ); - var owned_wavelengths = try radiance_wavelengths.buildRadianceWavelengthList(allocator, table); - defer owned_wavelengths.deinit(allocator); - const wavelength_list_stamp = radianceWavelengthListReuseStamp(owned_wavelengths.view()); - const dense_count = owned_wavelengths.wavelengths.len; - const exact_wavelengths = try allocator.alloc(f64, dense_count); + var built_wavelengths = try radiance_wavelengths.buildRadianceWavelengthList(allocator, table); + const wavelength_list_stamp = radianceWavelengthListReuseStamp(built_wavelengths.view()); + const dense_count = built_wavelengths.wavelengths.len; + const exact_wavelengths = allocator.alloc(f64, dense_count) catch |err| { + built_wavelengths.deinit(allocator); + return err; + }; defer allocator.free(exact_wavelengths); - for (owned_wavelengths.wavelengths, exact_wavelengths) |row, *wavelength_nm| { + const wavelength_list_matches = session.radiance.hasWavelengthList( + wavelength_list_stamp, + built_wavelengths.rows.len, + built_wavelengths.sample_indices.len, + built_wavelengths.wavelengths.len, + ); + if (wavelength_list_matches) { + built_wavelengths.deinit(allocator); + } else { + session.radiance.rebuildWavelengthList( + allocator, + built_wavelengths.rows, + built_wavelengths.sample_indices, + built_wavelengths.wavelengths, + wavelength_list_stamp, + ); + } + + const active_wavelengths = session.radiance.wavelengthList().wavelengths; + for (active_wavelengths, exact_wavelengths) |row, *wavelength_nm| { wavelength_nm.* = row.wavelength_nm; } const worker_count = spectrum_run.preferredRadianceWorkerCount(dense_count); const worker_pool = session.worker_pool.poolForWorkerCount(allocator, worker_count); - // Public O2 A spectrum runs read support-profile sigma rows through optics interpolation. They do not read + // Public spectrum runs read support-profile sigma rows through optics interpolation. They do not read // diagnostic layer-node rows or profile-line d_sigma/dT rows for the current surface/aerosol Jacobian set. // Both mode bits remain in the reuse stamp so future diagnostic or temperature-profile paths split caches. const build_layer_values = false; @@ -1051,15 +1054,6 @@ fn prepareSessionRows( needs_temperature_derivatives, ); - const wavelength_list_matches = session.radiance.hasWavelengthList( - wavelength_list_stamp, - owned_wavelengths.rows.len, - owned_wavelengths.sample_indices.len, - owned_wavelengths.wavelengths.len, - ); - if (!wavelength_list_matches) { - session.radiance.takeWavelengthList(allocator, &owned_wavelengths, wavelength_list_stamp); - } const wants_dense_jacobian = radiance_results.wantsJacobian(prepared_solve_config); try session.radiance.ensureResultCapacity(allocator, dense_count, wants_dense_jacobian); @@ -1219,10 +1213,10 @@ fn radianceResultReuseStamp( }) |value| hashing.updateValue(&hasher, value); for ([_]usize{ @intFromEnum(solve_config.derivative_mode), - solve_config.derivative_state_mask, @intFromEnum(solve_config.controls.scattering), solve_config.controls.n_streams, }) |value| hashing.updateValue(&hasher, value); + hashing.updateBool(&hasher, solve_config.wants_jacobian); hashing.updateBool(&hasher, solve_config.controls.use_spherical_correction); hashing.updateBool(&hasher, solve_config.controls.integrate_source_function); hashing.updateBool(&hasher, solve_config.controls.renorm_phase_function); diff --git a/src/rtm/README.md b/src/rtm/README.md new file mode 100644 index 000000000..dbbc6ab00 --- /dev/null +++ b/src/rtm/README.md @@ -0,0 +1,291 @@ +# `rtm/` — radiative transfer for one wavelength + +This is the fourth stage of the forward pass and the core of the model. For one +wavelength, `optics/` gives it a column of layers, each with an optical depth and a +single-scatter albedo. Alongside the layers it gets the viewing and solar geometry, +the solar source, and the curved sun-path samples. `rtm/` turns all of that into the +top-of-atmosphere reflectance at this wavelength, the one number the spectrum stage +wants. The module also returns the derivatives of that reflectance with respect to +the aerosol state. `spectrum/` drives this stage, calling `solveReflectance` once for +each wavelength in the band, so the work here happens one wavelength at a time and +never sees the band as a whole. + +Sunlight enters the top of the atmosphere along the solar direction, scattered by +air molecules and aerosol and absorbed by oxygen; the scattered light travels on +and scatters again; some of it leaves the top along the viewing direction. The +reflectance is the ratio of that outgoing radiance to the incoming solar flux. There +is no closed form for it once light scatters more than once, so the stage builds the +answer numerically with the LABOS scheme that zdisamar reproduces from the DISAMAR +reference model. Four nested physical ideas turn the problem into arithmetic: + +1. Azimuth in Fourier space. The dependence on the sun-to-view azimuth is expanded + as a cosine series, so the two-dimensional angular problem becomes a set of + independent one-dimensional problems, one per Fourier term `m`. +2. Discrete streams. Each one-dimensional problem is integrated over a fixed set of + Gauss-Legendre angles (the streams), plus the exact solar and viewing + directions. +3. Layer reflection and transmission. Every layer is reduced to two operators, what + it reflects and what it transmits, and thick layers are built up from thin ones + by doubling. +4. Orders of scattering. The radiation field is accumulated one scattering event at + a time: the direct beam scatters once, that field scatters again, and so on + until the contributions die out. + +``` + per wavelength: layer optics + geometry + source + curved path + | + v + +--------------------------+ + | solveReflectance | scattering == none -> direct surface + +--------------------------+ albedo * exp(-tau/mu0 - tau/muv) + | multiple scattering + v + geometry (gauss_angles) + direct beam (attenuation) + | + v + +----------- loop over retained Fourier terms m -------------+ + | | + | phase_basis -> Z+ / Z- stream-coupling matrices | + | layer_reflect_transmit (+ matrix_12x10) -> R, T rows | + | scattering_orders -> up/down fields, order by order | + | reflectance -> rho_m (top or integrated source) | + | | + +-----------------------+------------------------------------+ + | weighted Fourier sum, tail stop + v + reflectance + aerosol Jacobian vector -> spectrum/ +``` + +## The radiative transfer and its routes (`solve.zig`, `controls.zig`) + +`solveReflectance` is the entry point and the dispatcher. `controls.zig` holds the +parameters it reads, validated once by `prepareSolveConfig` so that the +per-wavelength work never has to validate them again. This is possible because the +parameters are fixed for the whole run: the per-wavelength path only reads them, and +a retrieval changes the aerosol state between iterations, not these controls. The +validated parameters fall into five groups: + +1. stream count, the number of Gauss-Legendre directions the integration uses. +2. scattering mode: off, single scattering, or multiple scattering. +3. the source-integration and spherical-correction flags, which pick the source + route and whether the solar path is treated as curved. +4. the convergence and truncation thresholds that decide when a sum has gone far + enough. +5. which Jacobians are wanted, given as a derivative mode and a state mask. + +There are two routes. With scattering off (`scattering == .none`) the answer is the +direct surface term alone. The solar beam attenuates down the column, reflects off a +Lambertian surface (one that reflects equally in all directions), and attenuates +back up: `albedo * exp(-tau/mu0) * exp(-tau/muv)`, where `mu0` and `muv` are the +cosines of the solar and viewing angles. The other route, `.multiple`, is the +layer-resolved radiative transfer in `solveLayerResolvedScattering`, which runs the +Fourier loop above. + +Within that route the source is handled one of two ways, chosen by +`integrate_source_function`: + +1. the top route reads the upward field at the top level. +2. the integrated-source route sums the scattering source over the source-level + quadrature optics built in `optics/`. This is the default, and it is the only + route that can produce the aerosol-layer-pressure Jacobian. + +The reference scene runs multiple scattering with the integrated source and the +pseudo-spherical correction on. Its viewing geometry is nadir, so the near-normal +gate collapses the Fourier loop to the single `m = 0` term. + +## Directions and the direct beam (`gauss_angles.zig`, `attenuation.zig`) + +The radiation field is tracked along a fixed set of directions, the streams from the +list above. `gauss_angles.zig` chooses them for one pair of sun and view angles. +They are a handful of angles spread over the hemisphere, the points a numerical +integration over direction evaluates at, plus the two directions the measurement +cares about: where the sun is and where the instrument looks. For each direction it +keeps a cosine and an integration weight, and for each pair of directions a factor +the scattering step reuses, since one scattering event sends light from every +direction into every other. None of this depends on wavelength, only on the two +angles and how many streams were asked for, so it is built once for a sun/view pair +and reused. When the pair changes, the phase-function rows of the next section are +rebuilt against the new directions. + +`attenuation.zig` computes how much of the direct solar beam survives down to each +level: Beer-Lambert extinction along the slanted solar path, `exp(-tau / mu0)`. On a +flat atmosphere the path stretches by the plain `1/mu0`. With the pseudo-spherical +correction it follows the curved sun-path samples from `optics/`, +`tau * r / sqrt(r^2 - r_level^2 sin^2(theta))`, which matters when the sun is low and +the rays bend noticeably through a round atmosphere. The two source routes want this +in different shapes: the integrated-source route keeps a compact +adjacent-layer-plus-top-to-level form, and the top route a full level-to-level +table. A separate tangent fill supplies the beam's derivative for the aerosol +Jacobian. + +## The phase function in Fourier space (`phase_basis.zig`) + +A scattering event redirects light by an angle whose probability is the phase +function. Here it is the sum of two pieces: Rayleigh scattering by air, a single +`l = 2` Legendre term, and aerosol scattering, a Henyey-Greenstein lobe that +`setup/` has already expanded into a Legendre series. `phase_basis.zig` decomposes +that mixture into the azimuth Fourier terms the loop runs over. + +It works in the associated Legendre polynomials `P_l^m`, built by recurrence for +each Fourier index `m` and each stream and weighted by the stream's integration +weight. From them it forms the stream-coupling matrices `Z+` and `Z-`, the +probability that light in one stream scatters into another, for the forward and +backward halves of the hemisphere. + +## Per-layer reflection and transmission (`layer_reflect_transmit.zig`, `matrix_12x10.zig`, `rows.zig`) + +For each layer and each Fourier term, `layer_reflect_transmit.zig` builds the two +operators that say what the layer does to radiation: a reflection matrix `R` and a +transmission matrix `T`, stream to stream. For a thin layer these come straight from +one scattering event: the phase kernel scaled by the single-scatter albedo, with the +unscattered direct part removed. A thick layer cannot be treated as a single +scatter, so it is doubled. It is split into halves thin enough to scatter once, then +combined with the adding equations, which account for light bouncing back and forth +between the halves before it leaves. Whether and how many times to double comes from +the layer's effective scattering depth. + +The adding step is matrix arithmetic on fixed-size matrices. `matrix_12x10.zig` +provides the products, the diagonal scalings, and the matrix inverse `(I - R*R)^-1` +that sums the endless reflections back and forth between the two halves. The matrices +are fixed at 12 directions (10 Gauss angles plus the sun and view), so the compiler +sees constant loop bounds and the inner loops unroll, and a cheap check on each +matrix skips a multiply when both factors are too weak to matter. `rows.zig` defines +the shared row types: `LayerRT` (the `R`/`T` pair), the direction vectors, and the +up/down field rows, none of which own heap memory. + +Before the Fourier loop, the stage measures for each layer the highest Legendre term +that still carries weight and how much scattering strength its remaining terms hold, +so the loop can skip layer/term pairs that contribute nothing. + +## Orders of scattering (`scattering_orders.zig`) + +With every layer reduced to `R` and `T`, `scattering_orders.zig` assembles the +radiation field by counting scattering events. The first order is the direct beam +scattering once in each layer, reflected up and transmitted down. Each later order +takes the previous order's up and down fields and scatters them again, with `R` and +`T` mixing the directions, and the result is carried between levels through the +direct-beam attenuation. The orders accumulate into the up/down fields (the `ud` +arrays) until the newest order falls below the convergence threshold or the order cap +is hit. The cap scales with the column's scattering optical depth, since optically +thicker columns need more orders to settle. + +Layers with no scattering signal are flagged and skipped, and the 12-stream +recurrence is a specialized inlined path. For Jacobians the same recurrence runs in +tangent-linear form, where every `R*U` becomes `dR*U + R*dU`, so the derivative field +propagates alongside the base field. + +## Reflectance and the Fourier sum (`reflectance.zig`) + +`reflectance.zig` turns the converged field into the reflectance coefficient `rho_m` +for the current Fourier term, following whichever of the two routes above is active. +The top route reads the upward field directly; the integrated-source route weights +each source level by its phase coupling and direction cosines before summing. The +coefficients are then combined into the azimuth series: term `m` carries weight `1` +for `m = 0` and `2 * cos(m * dphi)` otherwise, and a tail-break gate stops the loop +once the added terms are negligible. The reported reflectance is clamped to `[0, 2]`, +since it can exceed 1 under strong forward scattering. + +The same module computes the aerosol Jacobian weightings, the sensitivity of +reflectance to the aerosol optical depth and to the aerosol layer's mid-pressure. +Those derivatives are integrated over the aerosol layer the same way the source is, +and accumulated per Fourier term beside the reflectance. + +## Jacobians (`jacobian_states.zig`) + +A retrieval needs not just the reflectance but its derivative with respect to the +state it is fitting. `jacobian_states.zig` fixes that state vocabulary for the whole +model, `aerosol_optical_depth` and `aerosol_layer_mid_pressure_hpa`, as a two-element +vector with a mask marking which lanes are active. The same vector is used at every +stage: `optics/` writes per-layer derivatives in this order, `rtm/` propagates it +through one `solveReflectance` call via the tangent paths above, and `spectrum/` +convolves the active columns with the same instrument weights as the reflectance. The +result of one call is a reflectance plus this fixed-order Jacobian vector. + +## Performance + +`rtm/` runs at every dense wavelength of every retrieval iteration. Two properties +keep that affordable; the gates that skip work are covered in the next section. + +- It allocates no memory. Every large buffer (the RT layer rows, the + scattering-order fields, the phase-basis and attenuation tables) is borrowed from + `TransportWorkArrays`, sized once per worker thread in `cache/` and overwritten + each wavelength. +- The stream geometry and the Fourier phase basis are cached on the geometry key + (stream count and the two direction cosines). Across the dense grid the geometry + rarely changes, so these are built once and reused for thousands of wavelengths. + +### Where the time goes + +The cost-timing labels split `execute` into measured stages. The work below is +listed from most to least expensive, each with what it computes, how its cost +scales, and the gate that keeps it from running away. Items 1 and 2 are the two +halves of the layer build (`rt_layer_build`) and together they dominate; the rest +are comparatively cheap. + +1. The phase matrix (`rt_layer_phase_matrix`) is the first half of the layer build, + run for every layer at every retained Fourier term. Every entry of the 12-by-12 + matrix of stream pairs is a sum over the Legendre terms of the phase function, so + a forward-peaked aerosol that needs around 150 terms costs around 150 + multiply-adds in each of the 144 entries. The per-layer Legendre limit caps that + sum at the highest term still carrying weight, and a layer/term pair whose + scattering strength is below threshold is skipped before the matrix is touched. + +2. Doubling (`rt_layer_doubling`) is the second half, and it runs only for a layer + thick enough to scatter many times. Such a layer is split into halves and + rebuilt; the number of splits grows with the logarithm of the layer's effective + scattering depth, and each split costs a 10-by-10 matrix inverse (the + `(I - R*R)^-1`) plus a few 12-by-12 products. A layer enters this loop only when + its effective scattering depth clears `threshold_doubl`, so thin layers keep their + single-scatter rows and do no doubling work at all. Inside a split, a check on the + reflection matrix skips the inverse and the product updates when the reflection is + too weak to change the answer, which the `fixed_qseries_skipped` and + `fixed_rd_skipped` counters record. This is what makes an optically thick, hazy + layer the worst case for the stage. + +3. The scattering-order recurrence (`orders_total`, from `solveOrdersWithActive`) is + the next cost. Each order is one more bounce of light through the column: it + multiplies every layer's `R` and `T` against the previous order's fields to build + new local sources, then sweeps those sources up and down the level grid. The work + is roughly the order count times the number of levels. The order count is capped + from the column's scattering optical depth (`tau_scatter + 15`), so a thick + wavelength is allowed more bounces than a thin one. In practice the loop stops + well before the cap: as soon as the newest order's strongest upward value falls + below the convergence threshold (`threshold_conv_first` for the first order, then + `threshold_conv_mult`), it ends. A clear, thin atmosphere settles in one or two + orders; a thick, hazy one needs many. Layers with no scattering signal are flagged + in `rt_active` and skipped in both the source build and the sweeps. + +4. The direct-beam fill (`attenuation_fill`) runs once per wavelength, before the + Fourier loop. On a plane-parallel atmosphere it is one exponential per stream per + level. The reference scene turns the pseudo-spherical correction on, so the + fill instead integrates the curved sun-path samples for each stream and level; + that costs more than the flat case, but it is still a single pass, small next to + the layer build and the order recurrence. + +5. The Legendre basis (`plm_basis`) is a recurrence over streams and terms, which + looks like per-wavelength work but is not. It depends only on the geometry, so it + is cached and rebuilt only when the stream count or either direction cosine + changes. Across the dense grid the geometry rarely moves, so the basis is built a + handful of times over the whole pass rather than once per wavelength. + +6. The reflectance integral (`reflectance_integral`) combines the converged field + into the coefficient `rho_m` and folds it into the azimuth sum. It is a few dot + products per level, small next to the order recurrence that produced the field, + and the tail-break gate can end the azimuth sum before the configured maximum. + +One factor sits above the whole list: the Fourier loop reruns the per-layer build, +the order recurrence, the basis, and the reflectance integral once per retained term +`m` (the direct-beam fill is done once, before the loop). Whenever either direction +is within `1e-5` of normal, the near-normal gate collapses that loop to the single +`m = 0` term. The nadir reference scene takes this path, so for it the whole list +above runs once per wavelength instead of once per Fourier term. + +## Where to start + +1. `solve.zig`: `solveReflectance` and `solveLayerResolvedScattering`, the whole + stage in call order and the clearest map of the Fourier loop. +2. `controls.zig`: the routes and thresholds that decide which path runs. +3. `scattering_orders.zig` and `layer_reflect_transmit.zig`: the two pieces of the + actual radiative transfer, building layer operators and accumulating orders. +4. `src/README.md`, the parent, for how `optics/` feeds these inputs and how the + reflectance flows on into `spectrum/` and the retrieval. diff --git a/src/rtm/controls.zig b/src/rtm/controls.zig index d797c5db3..f0e8f44d0 100644 --- a/src/rtm/controls.zig +++ b/src/rtm/controls.zig @@ -3,7 +3,7 @@ const std = @import("std"); const jacobian_states = @import("jacobian_states.zig"); // controls.zig ---------------------------------------------------------------------------------------------- | -// Validated RTM control rows for the O2 A LABOS transport implementation. | +// Validated RTM control rows for the LABOS transport implementation. | // | // The Jacobian mask field uses the state order in `rtm/jacobian_states.zig`. | // ------------------------------------------------------------------------------------------------------------| @@ -31,7 +31,7 @@ pub const ScatteringMode = enum(u2) { // Prepared derivative route selector. | // | // none : reflectance only | -// semi_analytical : same-solve Jacobian propagation for the active state mask | +// semi_analytical : same-solve Jacobian propagation for the fixed two-state aerosol vector | // ------------------------------------------------------------------------------------------------------------| pub const DerivativeMode = enum(u1) { none = 0, @@ -215,7 +215,7 @@ pub const TransportControls = struct { // memory | // [ 0..71] controls : TransportControls | // [72..72] derivative_mode : DerivativeMode | -// [73..73] derivative_state_mask : StateMask | +// [73..73] wants_jacobian : bool | // [74..79] padding : 6 B | // | // unused bits: 48 padding + 7 enum-storage slack = 55 bits | @@ -223,7 +223,7 @@ pub const TransportControls = struct { // footprint: per instance = 80 B (0.078 KiB); total = per instance * live instance count | pub const SolveConfig = struct { derivative_mode: DerivativeMode = .none, - derivative_state_mask: jacobian_states.StateMask = jacobian_states.all_states_mask, + wants_jacobian: bool = true, controls: TransportControls = .{}, }; // ------------------------------------------------------------------------------------------------------------| @@ -236,7 +236,7 @@ pub fn prepareSolveConfig(config: SolveConfig) PrepareError!SolveConfig { return .{ .derivative_mode = config.derivative_mode, - .derivative_state_mask = jacobian_states.sanitizedMask(config.derivative_state_mask), + .wants_jacobian = config.wants_jacobian, .controls = config.controls, }; } diff --git a/src/rtm/gauss_angles.zig b/src/rtm/gauss_angles.zig index b8f9fe126..a739200bb 100644 --- a/src/rtm/gauss_angles.zig +++ b/src/rtm/gauss_angles.zig @@ -15,7 +15,7 @@ pub const Error = error{ // gauss_angles.zig ------------------------------------------------------------------------------------------ | // LABOS stream direction geometry for one solar/viewing angle pair. | // | -// The Gauss nodes come from the O2 A DISAMAR division-point helper. | +// The Gauss nodes come from the DISAMAR division-point helper. | // | // numerical guards | // direction_pair_floor = 1.0e-12 keeps plus/same-stream denominators finite. | diff --git a/src/rtm/jacobian_states.zig b/src/rtm/jacobian_states.zig index e0b2517bc..45b240b09 100644 --- a/src/rtm/jacobian_states.zig +++ b/src/rtm/jacobian_states.zig @@ -7,7 +7,7 @@ // spectrum/ convolves active columns with the same instrument weights as reflectance | // | // memory | -// Vector is [2]f64. StateMask is u8 and uses the low two bits. This file owns no heap storage and has no | +// Vector is [2]f64. This file owns no heap storage and has no | // hidden mutable state. Surface albedo remains a forward scalar; it is intentionally not a retrieval lane. | // ------------------------------------------------------------------------------------------------------------| @@ -19,8 +19,6 @@ pub const State = enum(u8) { }; pub const Vector = [state_count]f64; -pub const StateMask = u8; -pub const all_states_mask: StateMask = (1 << state_count) - 1; // StateNames -------------------------------------------------------------------------------------------------| // Namespace for public Jacobian state labels. | @@ -50,39 +48,6 @@ pub fn stateIndex(state: State) usize { return @intFromEnum(state); } -pub fn stateMask(state: State) StateMask { - // stateMask --------------------------------------------------------------------------------------------- | - // Build the single-bit mask for one fixed Jacobian state. | - // --------------------------------------------------------------------------------------------------------| - return @as(StateMask, 1) << @intCast(stateIndex(state)); -} - -pub fn includes(mask: StateMask, state: State) bool { - // includes ---------------------------------------------------------------------------------------------- | - // Test whether a state bit is set after the caller chooses an active Jacobian mask. | - // --------------------------------------------------------------------------------------------------------| - return (mask & stateMask(state)) != 0; -} - -pub fn sanitizedMask(mask: StateMask) StateMask { - // sanitizedMask ----------------------------------------------------------------------------------------- | - // Keep only the low bits owned by the fixed two-state Jacobian vector. | - // --------------------------------------------------------------------------------------------------------| - return mask & all_states_mask; -} - -pub fn activeStateCount(mask: StateMask) usize { - // activeStateCount -------------------------------------------------------------------------------------- | - // Count requested Jacobian columns after dropping unknown future mask bits. | - // --------------------------------------------------------------------------------------------------------| - const active_mask = sanitizedMask(mask); - var count: usize = 0; - for (0..state_count) |index| { - if ((active_mask & (@as(StateMask, 1) << @intCast(index))) != 0) count += 1; - } - return count; -} - pub fn get(vector: Vector, state: State) f64 { // get --------------------------------------------------------------------------------------------------- | // Read one derivative lane by fixed state. | @@ -97,19 +62,17 @@ pub fn set(vector: *Vector, state: State, value: f64) void { vector[stateIndex(state)] = value; } -pub fn addScaledMasked(accumulator: *Vector, vector: Vector, factor: f64, mask: StateMask) void { - // addScaledMasked ----------------------------------------------------------------------------------------| - // Add requested derivative lanes into a fixed-size accumulator. | +pub fn addScaled(accumulator: *Vector, vector: Vector, factor: f64) void { + // addScaled ----------------------------------------------------------------------------------------------| + // Add both derivative lanes into a fixed-size accumulator. | // | // hot path | - // Spectrum assembly and LABOS propagation use this for active Jacobian columns. | + // Spectrum assembly and LABOS propagation use this for the fixed Jacobian columns. | // | // math | - // accumulator_i += factor * vector_i for active lanes i | + // accumulator_i += factor * vector_i | // --------------------------------------------------------------------------------------------------------| - const active_mask = sanitizedMask(mask); for (0..state_count) |index| { - if ((active_mask & (@as(StateMask, 1) << @intCast(index))) == 0) continue; accumulator[index] += factor * vector[index]; } } @@ -118,7 +81,7 @@ pub fn scale(vector: Vector, factor: f64) Vector { // scale ------------------------------------------------------------------------------------------------- | // Multiply every fixed derivative lane by one scalar. | // | - // fixed-vector scale over the retained two O2 A retrieval lanes. | + // fixed-vector scale over the retained two retrieval lanes. | // | // math | // result_i = factor * vector_i | @@ -127,16 +90,3 @@ pub fn scale(vector: Vector, factor: f64) Vector { for (&result) |*value| value.* *= factor; return result; } - -pub fn scaleMasked(vector: Vector, factor: f64, mask: StateMask) Vector { - // scaleMasked ------------------------------------------------------------------------------------------- | - // Scale requested derivative lanes and leave inactive lanes zero. | - // --------------------------------------------------------------------------------------------------------| - const active_mask = sanitizedMask(mask); - var result = zero(); - for (0..state_count) |index| { - if ((active_mask & (@as(StateMask, 1) << @intCast(index))) == 0) continue; - result[index] = vector[index] * factor; - } - return result; -} diff --git a/src/rtm/layer_reflect_transmit.zig b/src/rtm/layer_reflect_transmit.zig index 2fda90cce..5733b0ffa 100644 --- a/src/rtm/layer_reflect_transmit.zig +++ b/src/rtm/layer_reflect_transmit.zig @@ -46,12 +46,12 @@ const phase_odd_reciprocal = build_phase_odd_reciprocal: { // | // memory | // The functions write caller-owned Mat rows and allocate no storage. Fixed stream_count=12 keeps the | -// constant-bound loop shape used by the O2 A LABOS route. | +// constant-bound loop shape used by the LABOS route. | // | // hot path | // solve.zig enters fillLayerReflectTransmitRowsWithBasis for every retained Fourier term. Each active | // layer builds a Z+/Z- phase kernel, fills single-scatter R/T, and may run layer doubling. The 12x10 route | -// keeps the O2 A loop order and cost-timing buckets so trace output remains comparable to LABOS. | +// keeps the loop order and cost-timing buckets so trace output remains comparable to LABOS. | // | // instrumentation | // Trace counters count layer visits, skip reasons, phase-row work, doubled layers, q-series decisions, and | @@ -340,7 +340,7 @@ pub fn fillLayerReflectTransmitRowsWithBasis( // instrumentation: trace counter: phase renormalization ----------------------------------------- | // captures: zero-Fourier phase-renormalization branch | - // why: verify when O2 A uses the default normalized phase function path. | + // why: verify when the run uses the default normalized phase function path. | Trace.plotU("phase_renormalizations", 1); @@ -852,7 +852,7 @@ pub fn classifyLayerDoubling( // layers, this branch keeps the single-scatter layer and avoids repeated matrix squaring. The cost is | // that very small intra-layer multiple-scattering feedback is not added. | // | - // threshold_doubl is 0.1 by generic default and 1.0e-6 in the O2 A rtm_config. Lower values double more | + // threshold_doubl is 0.1 by generic default and 1.0e-6 in the rtm_config. Lower values double more | // layers. | if (scattering != .multiple or !(effective_scattering_depth > threshold_doubl)) { return .{ @@ -1163,7 +1163,7 @@ fn doubleLayer12x10( stage_cost: ?CostTiming.Active, ) void { // doubleLayer12x10 ---------------------------------------------------------------------------------------| - // Fixed 12x10 layer doubling for the O2 A LABOS route. | + // Fixed 12x10 layer doubling for the LABOS route. | // | // | // memory | @@ -1609,7 +1609,7 @@ fn fillSingleScatterReflection12( inline fn squareAttenuation12(attenuation: *rows.Vec) void { // squareAttenuation12 ----------------------------------------------------------------------------------- | - // Fixed 12-direction attenuation square for the LABOS O2 A rtm_config. | + // Fixed 12-direction attenuation square for the LABOS rtm_config. | // --------------------------------------------------------------------------------------------------------| inline for (0..rows.max_stream_count) |direction_index| { diff --git a/src/rtm/matrix_12x10.zig b/src/rtm/matrix_12x10.zig index 53d7b234e..73270d4ba 100644 --- a/src/rtm/matrix_12x10.zig +++ b/src/rtm/matrix_12x10.zig @@ -25,7 +25,7 @@ const lu_diagonal_floor: f64 = 1.0e-30; // | // hot path | // layer_reflect_transmit.zig calls these kernels inside every retained layer-doubling step. The fixed | -// n=12, n_gauss=10 path is the O2 A LABOS route; generic n exists for unsupported or test geometries that | +// n=12, n_gauss=10 path is the LABOS route; generic n exists for unsupported or test geometries that | // still use the same algebra. Caller-owned `Into` variants keep temporary Mat values under the layer code | // owner so the hot path does not allocate or copy through heap storage. | // | @@ -95,7 +95,7 @@ pub fn smul(n: usize, n_gauss: usize, threshold_mul: f64, a: *const Mat, b: *con // tradeoff: fixed small-multiply trace gate | // Return zero when abs(trace(A_gg) * trace(B_gg)) <= threshold_mul. | // ----------------------------------------------------------------------------------------------------| - // LABOS layers pass threshold_mul = 1.0e-12 by generic default and 1.0e-8 in O2 A. This skips a | + // LABOS layers pass threshold_mul = 1.0e-12 by generic default and 1.0e-8 in O2A. This skips a | // full 12x10 product when the Gaussian trace estimate says the product is too small to matter. | if (@abs(tra * trb) <= threshold_mul) return Mat.zero(n); // end tradeoff: fixed small-multiply trace gate ------------------------------------------------------| @@ -2377,7 +2377,7 @@ fn smulAddSemul3_12KnownTracesInto( // tradeoff: fixed fused-product trace gate | // Skip A*C when abs(trace(A_gg) * trace(C_gg)) <= threshold_mul. | // --------------------------------------------------------------------------------------------------------| - // LABOS layers pass threshold_mul = 1.0e-12 by generic default and 1.0e-8 in O2 A. The skipped path | + // LABOS layers pass threshold_mul = 1.0e-12 by generic default and 1.0e-8 in O2A. The skipped path | // keeps the base C + A*diag(e) term and drops only the small Gaussian product contribution. | if (@abs(tra * trc) <= threshold_mul) { // Product skipped ------------------------------------------------------------------------------------| diff --git a/src/rtm/reflectance.zig b/src/rtm/reflectance.zig index 66c8969d9..9da9ffb1c 100644 --- a/src/rtm/reflectance.zig +++ b/src/rtm/reflectance.zig @@ -205,7 +205,7 @@ pub fn integratedSourceCoefficient( // integratedSourceCoefficient --------------------------------------------------------------------------- | // Integrated-source LABOS reflectance for one Fourier term. | // | - // O2 A `SourceLevel` rows. This route uses the RTM-quadrature branch: level weight, scattering carrier, | + // `SourceLevel` rows. This route uses the RTM-quadrature branch: level weight, scattering carrier, | // and source phase mixture are already explicit on each source level. | // | // math | diff --git a/src/rtm/scattering_orders.zig b/src/rtm/scattering_orders.zig index f03c497d4..f7e404509 100644 --- a/src/rtm/scattering_orders.zig +++ b/src/rtm/scattering_orders.zig @@ -409,7 +409,7 @@ fn solveOrdersInternal( // tradeoff: first-order convergence stop | // Return after the first transported order when max outgoing upward light is below threshold_conv_first. | // --------------------------------------------------------------------------------------------------------| - // threshold_conv_first is 1.0e-6 by generic default and 1.5e-7 in O2 A. Lower values keep more | + // threshold_conv_first is 1.0e-6 by generic default and 1.5e-7 in O2A. Lower values keep more | // scattering-order work. Higher values stop earlier and drop weak multiple-scattering feedback. | // instrumentation: perturbation: initial stop ------------------------------------------------------------| @@ -515,7 +515,7 @@ fn solveOrdersInternal( // tradeoff: multiple-order convergence stop | // Stop adding scattering orders when the order field is below threshold_conv_mult or the cap is hit. | // ----------------------------------------------------------------------------------------------------| - // threshold_conv_mult is 1.0e-4 by generic default and 1.5e-9 in O2 A. num_orders_max is the hard | + // threshold_conv_mult is 1.0e-4 by generic default and 1.5e-9 in O2A. num_orders_max is the hard | // cap; when it is zero, the resolved cap is roughly max(scattering optical depth, 0) + 15. | // instrumentation: perturbation: multiple stop -------------------------------------------------------| @@ -1206,7 +1206,7 @@ fn transportToOtherLevels12( ud_orde: []rows.UDLocal, ) void { // transportToOtherLevels12 ------------------------------------------------------------------------------ | - // Fixed 12-direction transport for the O2 A LABOS stream shape. | + // Fixed 12-direction transport for the LABOS stream shape. | // --------------------------------------------------------------------------------------------------------| ud_orde[start_level].U = ud_local[start_level].U; for (start_level + 1..end_level + 1) |level| { diff --git a/src/rtm/solve.zig b/src/rtm/solve.zig index 5626cb3e0..e5ae44e17 100644 --- a/src/rtm/solve.zig +++ b/src/rtm/solve.zig @@ -26,7 +26,7 @@ pub const Error = controls.PrepareError || attenuation.Error || gauss_angles.Err }; // solve.zig ------------------------------------------------------------------------------------------------- | -// Transport-solve dispatch for explicit O2 A optical rows. | +// Transport-solve dispatch for explicit optical rows. | // | // `directSurfaceOnly`, geometry setup, attenuation fill, RT layer build, scattering-order propagation, | // Fourier weighting, tail stop, public clamp, and aerosol tangents. | @@ -181,7 +181,7 @@ pub fn solveReflectance( // | // prepared controls | // Callers pass the result of `controls.prepareSolveConfig`. Wavelength workers do not | - // re-validate stream counts, thresholds, or derivative masks inside each LABOS solve. | + // re-validate stream counts, thresholds, or Jacobian selection inside each LABOS solve. | // | // direct math | // direct path = exp(-tau / max(mu0, 0.05)) * exp(-tau / max(muv, 0.05)) | @@ -191,19 +191,21 @@ pub fn solveReflectance( // Pressure derivatives require the integrated-source RTM quadrature weighting route. | // --------------------------------------------------------------------------------------------------------| if (config.controls.scattering == .none) { - if (config.controls.use_spherical_correction) return error.UnsupportedRadiativeTransferControls; + if (config.controls.use_spherical_correction) { + return error.UnsupportedRadiativeTransferControls; + } + if (config.derivative_mode != .none and config.wants_jacobian) { + return error.UnsupportedDerivativeMode; + } + return directSurfaceOnly( angles, surface_albedo, totalLayerOpticalDepth(layers), - config.derivative_mode, - config.derivative_state_mask, ); } - const wants_aerosol_layer_pressure = - config.derivative_mode != .none and - jacobian_states.includes(config.derivative_state_mask, .aerosol_layer_mid_pressure_hpa); + const wants_jacobian = config.derivative_mode != .none and config.wants_jacobian; const use_integrated_source = config.controls.integrate_source_function and layers.len > 1 and @@ -211,7 +213,7 @@ pub fn solveReflectance( if (config.controls.integrate_source_function and !use_integrated_source) { return error.UnsupportedRadiativeTransferControls; } - if (!use_integrated_source and wants_aerosol_layer_pressure) { + if (!use_integrated_source and wants_jacobian) { return error.UnsupportedDerivativeMode; } @@ -239,8 +241,6 @@ pub fn directSurfaceOnly( angles: ViewAngles, surface_albedo: f64, total_optical_depth: f64, - derivative_mode: controls.DerivativeMode, - derivative_state_mask: jacobian_states.StateMask, ) ReflectanceResult { // directSurfaceOnly ------------------------------------------------------------------------------------ | // Scalar direct-surface path from LABOS execute.zig. | @@ -256,8 +256,6 @@ pub fn directSurfaceOnly( const mu0 = @max(angles.solar_mu, direct_direction_cosine_floor); const muv = @max(angles.view_mu, direct_direction_cosine_floor); const direct = math.exp(-total_optical_depth / mu0) * math.exp(-total_optical_depth / muv); - _ = derivative_mode; - _ = derivative_state_mask; const raw_reflectance = surface_albedo * direct; return .{ @@ -418,15 +416,8 @@ fn solveLayerResolvedScattering( const rt_layers = work.rt_layers[0..level_count]; const order_count: usize = @intCast(config.controls.resolvedNumOrdersMax(totalScatteringOpticalDepth(layers))); - const wants_aerosol_optical_depth = - config.derivative_mode != .none and - jacobian_states.includes(config.derivative_state_mask, .aerosol_optical_depth); - const wants_aerosol_layer_pressure = - config.derivative_mode != .none and - jacobian_states.includes(config.derivative_state_mask, .aerosol_layer_mid_pressure_hpa); - const wants_any_jacobian = - config.derivative_mode != .none and - jacobian_states.activeStateCount(config.derivative_state_mask) != 0; + const wants_jacobian = config.derivative_mode != .none and config.wants_jacobian; + std.debug.assert(use_integrated_source or !wants_jacobian); const layer_phase_row_cache = work.phase_row_cache[0..level_count]; const layer_phase_row_valid = work.phase_row_valid[0..level_count]; @@ -476,7 +467,7 @@ fn solveLayerResolvedScattering( const orders_start = CostTiming.start(stage_cost); const orders_view = choose_orders_view: { if (use_integrated_source) { - if (wants_any_jacobian) { + if (wants_jacobian) { break :choose_orders_view scattering_orders.solveOrdersWithActiveLocalSum( &work.orders, 0, @@ -549,134 +540,23 @@ fn solveLayerResolvedScattering( const evaluate_aerosol_tangent = config.controls.performance_thresholds.shouldEvaluateAerosolTangent(fourier_index); - if (use_integrated_source and evaluate_aerosol_tangent and - (wants_aerosol_optical_depth or wants_aerosol_layer_pressure)) - { - const tangent = choose_integrated_aerosol_tangent: { - if (wants_aerosol_optical_depth and wants_aerosol_layer_pressure) { - break :choose_integrated_aerosol_tangent reflectance_helpers.integratedAerosolDerivativeWeighting( - layers, - level_sources, - orders_view.ud, - orders_view.ud_sum_local, - layer_count, - fourier_index, - config.controls.use_spherical_correction, - geometry, - plm_basis, - phase, - ); - } - - if (wants_aerosol_optical_depth) { - break :choose_integrated_aerosol_tangent reflectance_helpers.AerosolDerivativeWeighting{ - .aerosol_optical_depth = reflectance_helpers.integratedAerosolOpticalDepthWeighting( - layers, - level_sources, - orders_view.ud, - orders_view.ud_sum_local, - layer_count, - fourier_index, - config.controls.use_spherical_correction, - geometry, - plm_basis, - phase, - ), - }; - } - - break :choose_integrated_aerosol_tangent reflectance_helpers.AerosolDerivativeWeighting{ - .aerosol_layer_mid_pressure_hpa = reflectance_helpers.integratedAerosolLayerPressureShiftWeighting( - layers, - level_sources, - orders_view.ud, - orders_view.ud_sum_local, - layer_count, - fourier_index, - config.controls.use_spherical_correction, - geometry, - plm_basis, - phase, - ), - }; - }; - - if (wants_aerosol_optical_depth) { - - // instrumentation: perturbation: integrated aerosol-depth tangent --------------------------- | - // captures: Fourier-weighted RTM-quadrature aerosol optical-depth contribution | - // why: test sensitivity of the integrated-source AOD Jacobian without changing base physics. | - - aerosol_optical_depth_tangent += Perturbation.scalar( - .aerosol_aod_tangent, - .{ - .fourier_index = @intCast(fourier_index), - .state_index = @intCast(jacobian_states.stateIndex(.aerosol_optical_depth)), - }, - contribution.weight * tangent.aerosol_optical_depth, - ); - - // end instrumentation: perturbation: integrated aerosol-depth tangent ----------------------- | - - } - - if (wants_aerosol_layer_pressure) { - - // instrumentation: perturbation: integrated aerosol-pressure tangent ------------------------ | - // captures: Fourier-weighted RTM-quadrature aerosol pressure-shift contribution | - // why: test sensitivity of the pressure Jacobian inherited by retrieval work. | - - aerosol_layer_pressure_tangent += Perturbation.scalar( - .aerosol_pressure_tangent, - .{ - .fourier_index = @intCast(fourier_index), - .state_index = @intCast(jacobian_states.stateIndex(.aerosol_layer_mid_pressure_hpa)), - }, - contribution.weight * tangent.aerosol_layer_mid_pressure_hpa, - ); - - // end instrumentation: perturbation: integrated aerosol-pressure tangent -------------------- | - - } - } else if (!use_integrated_source and wants_aerosol_optical_depth and evaluate_aerosol_tangent) { - const tangent_attenuation = try attenuation.fillDynamicTangent( - work.dynamic_attenuation_tangent_data, - layers, - .aerosol_optical_depth, - geometry, - ); - const rt_layers_tangent = work.rt_layers_tangent[0..level_count]; - layer_reflect_transmit.fillLayerReflectTransmitTangentRowsWithBasis( - rt_layers_tangent, + if (use_integrated_source and evaluate_aerosol_tangent and wants_jacobian) { + const tangent = reflectance_helpers.integratedAerosolDerivativeWeighting( layers, - .aerosol_optical_depth, + level_sources, + orders_view.ud, + orders_view.ud_sum_local, + layer_count, fourier_index, + config.controls.use_spherical_correction, geometry, - config.controls, - phase, - rayleigh_phase_coefficient2, plm_basis, + phase, ); - const tangent_orders = scattering_orders.solveOrdersTangent( - &work.orders, - 0, - layer_count, - geometry, - dynamic_attenuation.?, - tangent_attenuation, - rt_layers, - rt_layers_tangent, - config.controls, - order_count, - ); - - const tangent_rho_m = - reflectance_helpers.topReflectanceCoefficient(tangent_orders.ud, layer_count, geometry); - - // instrumentation: perturbation: non-integrated aerosol-depth tangent --------------------------- | - // captures: Fourier-weighted tangent-order aerosol optical-depth contribution | - // why: compare non-integrated tangent-order sensitivity against retained AOD gates. | + // instrumentation: perturbation: integrated aerosol-depth tangent ------------------------------- | + // captures: Fourier-weighted RTM-quadrature aerosol optical-depth contribution | + // why: test sensitivity of the integrated-source AOD Jacobian without changing base physics. | aerosol_optical_depth_tangent += Perturbation.scalar( .aerosol_aod_tangent, @@ -684,20 +564,33 @@ fn solveLayerResolvedScattering( .fourier_index = @intCast(fourier_index), .state_index = @intCast(jacobian_states.stateIndex(.aerosol_optical_depth)), }, - contribution.weight * tangent_rho_m, + contribution.weight * tangent.aerosol_optical_depth, ); - // end instrumentation: perturbation: non-integrated aerosol-depth tangent ----------------------- | + // end instrumentation: perturbation: integrated aerosol-depth tangent --------------------------- | + + // instrumentation: perturbation: integrated aerosol-pressure tangent ---------------------------- | + // captures: Fourier-weighted RTM-quadrature aerosol pressure-shift contribution | + // why: test sensitivity of the pressure Jacobian inherited by retrieval work. | + + aerosol_layer_pressure_tangent += Perturbation.scalar( + .aerosol_pressure_tangent, + .{ + .fourier_index = @intCast(fourier_index), + .state_index = @intCast(jacobian_states.stateIndex(.aerosol_layer_mid_pressure_hpa)), + }, + contribution.weight * tangent.aerosol_layer_mid_pressure_hpa, + ); + + // end instrumentation: perturbation: integrated aerosol-pressure tangent ------------------------ | } if (contribution.tail_break) break; } var jacobian = jacobian_states.zero(); - if (wants_aerosol_optical_depth) { + if (wants_jacobian) { jacobian_states.set(&jacobian, .aerosol_optical_depth, aerosol_optical_depth_tangent); - } - if (wants_aerosol_layer_pressure) { jacobian_states.set(&jacobian, .aerosol_layer_mid_pressure_hpa, aerosol_layer_pressure_tangent); } return .{ @@ -759,7 +652,7 @@ fn resolvedFourierMax( // | // tradeoff: near-normal scalar Fourier route | // When either direction cosine is within 1.0e-5 of normal, LABOS evaluates only m=0. | - // The O2 A reference case exercises this route because the viewing angle is nadir. | + // The reference scene exercises this route because the viewing angle is nadir. | // --------------------------------------------------------------------------------------------------------| if (layer_count == 0) return 0; if ((1.0 - view_mu) < near_normal_mu_delta or diff --git a/src/setup/README.md b/src/setup/README.md index a12b72299..8626c4013 100644 --- a/src/setup/README.md +++ b/src/setup/README.md @@ -65,7 +65,7 @@ and no reallocation. ## Gas absorption (`line_tables.zig`, `cia_table.zig`) -These hold the spectroscopy of the O2 A band. `line_tables.zig` loads the HITRAN +These hold the spectroscopy of the O2A band. `line_tables.zig` loads the HITRAN line parameters together with the LISA line-mixing relaxation matrix and the strong-line sidecar, and carries the runtime line-filter controls (isotopes, strength threshold, wavenumber cutoff, mixing factor). It does not compute diff --git a/src/setup/atmosphere_layers.zig b/src/setup/atmosphere_layers.zig index d902a05c5..4b4d5157f 100644 --- a/src/setup/atmosphere_layers.zig +++ b/src/setup/atmosphere_layers.zig @@ -311,17 +311,17 @@ pub fn buildWithQuadrature( .{ .rows = raw_profile_rows }, scene.atmosphere.surface_pressure_hpa, ); - var dense_profile_owned = true; - errdefer if (dense_profile_owned) allocator.free(dense_profile_rows); const profile = ProfileView{ .rows = dense_profile_rows }; - const spectroscopy_profile_rows = try buildSpectroscopyProfileRows( + const spectroscopy_profile_rows = buildSpectroscopyProfileRows( allocator, .{ .rows = raw_profile_rows }, profile, - ); + ) catch |err| { + allocator.free(dense_profile_rows); + return err; + }; - dense_profile_owned = false; return buildWithOwnedProfiles(allocator, scene, dense_profile_rows, spectroscopy_profile_rows, quadrature); } @@ -342,21 +342,18 @@ pub fn buildFromPreparedProfiles( // ownership | // The returned LayerGrid owns duplicated profile rows so ordinary LayerGrid.deinit remains correct. | // --------------------------------------------------------------------------------------------------------| - const owned_source_profile_rows = try allocator.dupe(readers.AtmosphereProfileRow, source_profile_rows); - var source_profile_owned = true; - errdefer if (source_profile_owned) allocator.free(owned_source_profile_rows); - const owned_spectroscopy_profile_rows = - try allocator.dupe(readers.AtmosphereProfileRow, spectroscopy_profile_rows); - var spectroscopy_profile_owned = true; - errdefer if (spectroscopy_profile_owned) allocator.free(owned_spectroscopy_profile_rows); - - source_profile_owned = false; - spectroscopy_profile_owned = false; + const duplicated_source_profile_rows = try allocator.dupe(readers.AtmosphereProfileRow, source_profile_rows); + const duplicated_spectroscopy_profile_rows = + allocator.dupe(readers.AtmosphereProfileRow, spectroscopy_profile_rows) catch |err| { + allocator.free(duplicated_source_profile_rows); + return err; + }; + return buildWithOwnedProfiles( allocator, scene, - owned_source_profile_rows, - owned_spectroscopy_profile_rows, + duplicated_source_profile_rows, + duplicated_spectroscopy_profile_rows, quadrature, ); } @@ -391,24 +388,25 @@ fn buildWithOwnedProfiles( quadrature: LayerQuadrature, ) !LayerGrid { // buildWithOwnedProfiles ---------------------------------------------------------------------------------| - // Compute layer/support placement from caller-owned profile rows that transfer into the returned grid. | + // Compute layer/support placement from profile rows now owned by the returned grid. | // --------------------------------------------------------------------------------------------------------| - var dense_profile_owned = true; - errdefer if (dense_profile_owned) allocator.free(dense_profile_rows); - var spectroscopy_profile_owned = true; - errdefer if (spectroscopy_profile_owned) allocator.free(spectroscopy_profile_rows); - - const shape = try layerGridShape(scene, quadrature); + const shape = layerGridShape(scene, quadrature) catch |err| { + allocator.free(dense_profile_rows); + allocator.free(spectroscopy_profile_rows); + return err; + }; - var grid = try allocate( + var grid = allocate( allocator, dense_profile_rows, spectroscopy_profile_rows, shape.layer_count, shape.support_count, - ); - dense_profile_owned = false; - spectroscopy_profile_owned = false; + ) catch |err| { + allocator.free(dense_profile_rows); + allocator.free(spectroscopy_profile_rows); + return err; + }; errdefer grid.deinit(allocator); try fillLayerGrid(&grid, scene, quadrature, shape.support_order); @@ -671,7 +669,7 @@ fn densifyVendorPressureGrid( surface_pressure_hpa: f64, ) ![]readers.AtmosphereProfileRow { // densifyVendorPressureGrid ------------------------------------------------------------------------------| - // Build the O2 A setup profile used before layer/support placement. | + // Build the setup profile used before layer/support placement. | // | // ClimatologyProfile.densifyVendorPressureGrid before state_build vertical setup. | // | @@ -681,6 +679,10 @@ fn densifyVendorPressureGrid( // --------------------------------------------------------------------------------------------------------| if (profile.rows.len < 2) return allocator.dupe(readers.AtmosphereProfileRow, profile.rows); + var hydrostatic_arena = std.heap.ArenaAllocator.init(allocator); + defer hydrostatic_arena.deinit(); + const scratch_allocator = hydrostatic_arena.allocator(); + const scale_height_guess_km = 8.0; var dense_row_count: usize = 1; for (profile.rows[0 .. profile.rows.len - 1], profile.rows[1..]) |lower, upper| { @@ -691,14 +693,10 @@ fn densifyVendorPressureGrid( dense_row_count += additional_levels + 1; } - const dense_pressures_hpa = try allocator.alloc(f64, dense_row_count); - defer allocator.free(dense_pressures_hpa); - const dense_temperatures_k = try allocator.alloc(f64, dense_row_count); - defer allocator.free(dense_temperatures_k); - const dense_altitudes_km = try allocator.alloc(f64, dense_row_count); - defer allocator.free(dense_altitudes_km); - const dense_altitudes_gp_km = try allocator.alloc(f64, (dense_row_count - 1) * 2); - defer allocator.free(dense_altitudes_gp_km); + const dense_pressures_hpa = try scratch_allocator.alloc(f64, dense_row_count); + const dense_temperatures_k = try scratch_allocator.alloc(f64, dense_row_count); + const dense_altitudes_km = try scratch_allocator.alloc(f64, dense_row_count); + const dense_altitudes_gp_km = try scratch_allocator.alloc(f64, (dense_row_count - 1) * 2); var dense_index: usize = 0; dense_pressures_hpa[dense_index] = profile.rows[0].pressure_hpa; @@ -731,8 +729,7 @@ fn densifyVendorPressureGrid( const universal_gas_constant = 8.3144621; const mean_molecular_weight_air = 28.964e-3; const safe_surface_pressure_hpa = @max(surface_pressure_hpa, 1.0e-9); - const dense_log_pressures = try allocator.alloc(f64, dense_row_count); - defer allocator.free(dense_log_pressures); + const dense_log_pressures = try scratch_allocator.alloc(f64, dense_row_count); for (dense_pressures_hpa, 0..) |pressure_hpa, index| { dense_log_pressures[index] = @log(@max(pressure_hpa, 1.0e-9)); @@ -750,8 +747,7 @@ fn densifyVendorPressureGrid( } } - const previous_altitudes_km = try allocator.dupe(f64, dense_altitudes_km); - defer allocator.free(previous_altitudes_km); + const previous_altitudes_km = try scratch_allocator.dupe(f64, dense_altitudes_km); var iteration: usize = 0; while (iteration < 6) : (iteration += 1) { @@ -931,7 +927,7 @@ fn fillSupportRow( // fillSupportRow -----------------------------------------------------------------------------------------| // Fill one boundary or active support row from the DISAMAR profile thermodynamics route. | // | - // reference O2 A route, so support pressure and temperature come from profile spline sampling at the | + // reference route, so support pressure and temperature come from profile spline sampling at the | // support altitude. The geometric-mean pressure fallback is for non-canonical interval grids. | // --------------------------------------------------------------------------------------------------------| const pressure_hpa = profile.interpolatePressureLogSpline(altitude_km); diff --git a/src/setup/run_tables.zig b/src/setup/run_tables.zig index 6a4e5802a..d98e70569 100644 --- a/src/setup/run_tables.zig +++ b/src/setup/run_tables.zig @@ -61,7 +61,7 @@ pub const RunTables = struct { pub fn buildRunTables(allocator: Allocator, scene: scene_input.Scene) !RunTables { // buildRunTables -----------------------------------------------------------------------------------------| - // Validate the O2 A scene and build every physical setup table. | + // Validate the scene and build every physical setup table. | // --------------------------------------------------------------------------------------------------------| try validate.sceneControls(scene); diff --git a/src/spectrum/README.md b/src/spectrum/README.md new file mode 100644 index 000000000..4354835a3 --- /dev/null +++ b/src/spectrum/README.md @@ -0,0 +1,196 @@ +# `spectrum/` — from fine-grid radiance to the instrument spectrum + +This is the fifth stage of the forward pass, and the one that turns the radiative +transfer into the spectrum an instrument would record. The instrument reports +radiance at a fixed set of product wavelengths, each one an average over a band set +by its slit. The model works at a much finer resolution, because oxygen absorption +swings sharply across the A band and a coarse grid would miss the line shapes. +`spectrum/` picks the fine wavelengths to evaluate, runs the per-wavelength radiance +there through `optics/` and `rtm/`, and averages the result back down through the +slit. `rtm/` computes the radiance one wavelength at a time, and `spectrum/` is the +loop around it that turns those single wavelengths into a band. + +Oxygen absorption varies fast near a strong line and slowly between lines, so the +model places its sample points densely around the strong lines and sparsely +elsewhere. It then evaluates far fewer wavelengths than a uniform grid would, and a +slit average over those points still reproduces what the instrument records. + +``` + +----------------------+ + | sampling_table | choose the fine wavelengths, dense near O2 lines + +----------------------+ + | + v + +----------------------+ + | radiance_wavelengths | reduce to the unique wavelengths, run each once + +----------------------+ + | + v + +----------------------+ + | spectrum_run | radiance at each wavelength via optics/ and rtm/, parallel + +----------------------+ + | + v + +----------------------+ + | radiance_results | gather the fine grid onto the product wavelengths + +----------------------+ + | + v + +----------------------+ + | instrument_average | slit average, calibration, reflectance + +----------------------+ + | + v + product spectrum -> output/, retrieval/ +``` + +## Choosing where to evaluate (`sampling_table.zig`, `radiance_wavelengths.zig`) + +`sampling_table.zig` builds the fine wavelength grid. It starts from the product +wavelengths and the instrument slit width, widens the span to catch the line wings, +and then splits it into intervals cut at the strong O2 line positions. Each interval +carries a set of Gauss-Legendre quadrature points: wide intervals far from any line +get few, narrow intervals near a line center get many. For each product wavelength it +stores the offsets and weights that say which fine samples average into that product +wavelength and how much each one counts. The table depends on the spectral grid, the +instrument, and the line positions, none of which change during a retrieval, so it is +built once and reused. + +`radiance_wavelengths.zig` collects the unique fine wavelengths those weights refer +to. Two product wavelengths near each other share many of the same fine samples, and +the model should evaluate each one only once. The wavelength's exact f64 bit pattern +is the key, with no rounding or tolerance, so a sample shared between two product +wavelengths collapses to one entry. The result, `RadianceWavelengthList`, is the list +of wavelengths the per-wavelength loop runs over, plus the index map from each product +wavelength back to its samples in that list. + +## Evaluating one wavelength (`spectrum_run.zig`) + +`spectrum_run.zig` is the orchestrator, and `radianceAtWavelength` is its core. For +one fine wavelength it reads the cached O2 line cross-section, fills the support and +layer optical depths through `optics/` (gas absorption, Rayleigh, the O2-O2 continuum, +aerosol), builds the source levels and curved sun-path samples when the route asks for +them, then calls `rtm/`'s `solveReflectance` and scales the returned reflectance into a +radiance. + +`runForwardSpectrum` runs the whole stage: it fills the dense radiance grid, gathers +it onto the product wavelengths, then hands off to the slit average. The dense grid is +the expensive part, so it is split across worker threads, each writing its own slice +of the output; Performance covers how that stays race-free. + +## O2 line spectroscopy (`line_physics.zig`) + +`line_physics.zig` is the oxygen A-band line-by-line spectroscopy: given a +temperature, a pressure, and a wavelength, it returns the O2 absorption cross-section +per molecule. `cache/profile_line_memory.zig` sums it over the lines at every fine +wavelength and caches the result; the per-wavelength loop reads that cache rather than +calling this file, and `output/` calls it for per-line diagnostics. That is why it +sits in the folder but not in the dataflow above. + +Each oxygen line absorbs in a profile around its center wavelength, and its width +comes from two effects. Thermal motion of the molecules spreads it into a Gaussian +(Doppler broadening); collisions with other molecules spread it into a Lorentzian +(pressure broadening). The true profile is the convolution of the two, the Voigt +profile, evaluated here through a complex-probability-function approximation. On top +of the shape, three things scale a line: + +1. Strength with temperature. A line's strength is set by how many molecules sit in + its lower energy state, which follows the Boltzmann population and the partition + function, so the reference strength at 296 K is rescaled to the layer temperature. +2. Pressure shift. The center moves slightly with pressure. +3. Cutoff. Past a set distance in wavenumber from the center, a line is dropped. + +In the A band the strong lines sit close enough that they do not absorb +independently: a collision can move a molecule between two lines, so their shapes +couple. Summing isolated Voigt profiles would be wrong there. zdisamar instead +uses a line-mixing treatment for the strong lines, built from the LISA relaxation +matrix for up to 128 of them, and sums the remaining weaker lines as independent Voigt +profiles. The cross-section it returns is what `optics/` multiplies by the O2 number +density and the path length to get the gas absorption optical depth of a layer. + +## Solar irradiance (`solar_lookup.zig`) + +Reflectance is radiance divided by the incoming solar flux, so the model needs the +solar irradiance at each wavelength. `solar_lookup.zig` reads it from the solar spline +table that `setup/` prepared, keyed on the exact wavelength bits through the cached +`SolarIrradianceMemory`, and integrates it with the same offsets and weights as the +radiance so the two share a sampling. A positive floor on the source value +keeps the later division from blowing up. + +## Gathering and the instrument slit (`radiance_results.zig`, `instrument_average.zig`) + +`radiance_results.zig` holds the dense radiance grid and gathers it. The `rtm/` stage +returns a reflectance, which this module scales into a radiance with +`L = reflectance * mu0 * E0 / pi`, because radiance is the physical quantity the slit +averages. Gathering then applies each product wavelength's offsets and weights exactly +once, summing the fine radiances (and the aerosol Jacobian lanes alongside them) into +one product radiance. + +`instrument_average.zig` builds the product rows. When the sampling already folded the +slit into those weights, the product radiance is the convolved value directly; +otherwise it applies the slit weighting here. It then calibrates each channel +(gain, offset, wavelength shift, stray light) and forms reflectance from the +calibrated radiance and the averaged solar irradiance, `rho = pi * L / (mu0 * E0)`, +with a floor on the denominator. The aerosol Jacobian columns ride through the same +convolution and calibration, and because the fitted states perturb the radiance but +not the solar flux or geometry, the reflectance derivative is just the radiance +derivative over the same denominator. + +## Performance + +The spectrum stage runs the per-wavelength loop that dominates the whole forward +model, so the cost lives in how many wavelengths it evaluates and how much each one +costs. Two properties hold the rest down. + +- It allocates no memory. The dense grid, the product rows, and each worker's optics + scratch are caller-owned, sized once in `cache/` and reused across every wavelength + and every retrieval iteration. +- The fine grid is concurrent and race-free by construction. Each worker is handed + its own `TransportWorkerMemory` by index, so no worker touches another's scratch, + and the output is partitioned into disjoint wavelength ranges (static ranges, or + disjoint chunks pulled from a shared queue). The only shared mutable state is the + chunk queue's atomic claim and a first-worker-wins error slot; the per-wavelength + work touches neither. Shared inputs (the tables, the cached line sigma, the solar + memory) are read-only while the workers run. + +### Where the time goes + +Listed from most to least expensive, each with what it costs and the gate that keeps +it down. + +1. The dense radiance loop (`radianceAtWavelength`, once per unique fine wavelength) + is the bulk of forward time. Each call fills the optical depths and runs `rtm/`'s + `solveReflectance`, whose own cost is the subject of the `rtm/` README. The number + of calls is set by the sampling table, and the adaptive grid is the gate: by + sampling densely only where absorption varies fast, it keeps that count far below a + uniform fine grid. +2. The O2 line summation (`line_physics.zig`, through `cache/profile_line_memory.zig`) + is the most expensive single build. For every fine wavelength and every profile + node it sums the contribution of every nearby line, each one a Voigt evaluation, + and the strong-line line-mixing prep builds a relaxation matrix once per node, + which grows with the square of the strong-line count. It is done once per scene and + reused across retrieval iterations, since the gas spectroscopy does not depend on + the aerosol state. Within the build, work is skipped four ways: a binary-search + window narrows to the lines near each wavelength, the weak and strong lines are + split, the wavenumber cutoff drops distant lines, and the temperature- and + pressure-dependent terms are prepared once per node and reused across all its + wavelengths. +3. Building the sampling table (`sampling_table.zig`) is a one-time per-scene cost. It + places the adaptive quadrature points, parallelized across product rows, and is + cached and rebuilt only when the spectral grid or the instrument changes. +4. The gather and the slit average (`radiance_results.zig`, `instrument_average.zig`) + are cheap. Each product wavelength is a handful of weighted sums over its samples, + and the slit convolution is short, so neither is a bottleneck next to the + per-wavelength loop. + +## Where to start + +1. `spectrum_run.zig`: `runForwardSpectrum` for the whole stage in order, and + `radianceAtWavelength` for the per-wavelength work that drives `optics/` and + `rtm/`. +2. `sampling_table.zig`: how the fine grid is placed around the O2 lines. +3. `line_physics.zig`: the line-by-line O2 spectroscopy, read with + `cache/profile_line_memory.zig`, which is where it is summed and cached. +4. `instrument_average.zig`: the slit average, calibration, and reflectance. +5. the parent `src/README.md` for how this stage sits between `optics/`, `rtm/`, and + the retrieval. diff --git a/src/spectrum/instrument_average.zig b/src/spectrum/instrument_average.zig index db937e6e4..647c2e0af 100644 --- a/src/spectrum/instrument_average.zig +++ b/src/spectrum/instrument_average.zig @@ -203,14 +203,10 @@ pub fn postprocessRadianceResults( } applyRadianceCalibration(calibration, out_rows); - const active_state_count = jacobian_states.activeStateCount(solve_config.derivative_state_mask); - const wants_jacobian = solve_config.derivative_mode != .none and active_state_count != 0; + const wants_jacobian = solve_config.derivative_mode != .none and solve_config.wants_jacobian; if (!wants_jacobian) return; for (0..jacobian_states.state_count) |state_index| { - const state: jacobian_states.State = @enumFromInt(state_index); - if (!jacobian_states.includes(solve_config.derivative_state_mask, state)) continue; - for (raw_rows, out_rows, 0..) |raw_row, *out_row, index| { out_row.jacobian[state_index] = choose_jacobian: { if (uses_integrated_sampling) break :choose_jacobian raw_row.jacobian[state_index]; @@ -247,8 +243,7 @@ pub fn assembleReflectanceResults( if (radiance.len != irradiance.len or radiance.len != out_reflectance.len) return error.ShapeMismatch; if (!std.math.isFinite(solar_cosine)) return error.InvalidSolarCosine; - const active_state_count = jacobian_states.activeStateCount(solve_config.derivative_state_mask); - const wants_jacobian = solve_config.derivative_mode != .none and active_state_count != 0; + const wants_jacobian = solve_config.derivative_mode != .none and solve_config.wants_jacobian; if (wants_jacobian and out_jacobian.len != radiance.len) return error.ShapeMismatch; if (!wants_jacobian and out_jacobian.len != 0 and out_jacobian.len != radiance.len) { return error.ShapeMismatch; @@ -263,11 +258,7 @@ pub fn assembleReflectanceResults( var row_jacobian = jacobian_states.zero(); if (wants_jacobian) { - row_jacobian = jacobian_states.scaleMasked( - radiance_row.jacobian, - scale, - solve_config.derivative_state_mask, - ); + row_jacobian = jacobian_states.scale(radiance_row.jacobian, scale); out_jacobian[index] = row_jacobian; for (0..jacobian_states.state_count) |state_index| { summary.jacobian_sum[state_index] += row_jacobian[state_index]; diff --git a/src/spectrum/line_physics.zig b/src/spectrum/line_physics.zig index 6fc15bdf2..94e11a4e7 100644 --- a/src/spectrum/line_physics.zig +++ b/src/spectrum/line_physics.zig @@ -431,7 +431,7 @@ fn usesVendorStrongLinePartition( strong_lines: []const readers.StrongLineAssetRow, ) bool { // usesVendorStrongLinePartition ------------------------------------------------------------------------- | - // Detect the O2 A sidecar partition from retained HITRAN branch metadata. | + // Detect the sidecar partition from retained HITRAN branch metadata. | // --------------------------------------------------------------------------------------------------------| if (strong_lines.len == 0) return false; diff --git a/src/spectrum/radiance_results.zig b/src/spectrum/radiance_results.zig index 214245992..0a03dffe6 100644 --- a/src/spectrum/radiance_results.zig +++ b/src/spectrum/radiance_results.zig @@ -80,8 +80,7 @@ pub fn wantsJacobian(solve_config: controls.SolveConfig) bool { // wantsJacobian ----------------------------------------------------------------------------------------- | // Decide whether dense radiance storage and nominal gathers need active Jacobian lanes. | // --------------------------------------------------------------------------------------------------------| - return solve_config.derivative_mode != .none and - jacobian_states.activeStateCount(solve_config.derivative_state_mask) != 0; + return solve_config.derivative_mode != .none and solve_config.wants_jacobian; } pub fn scaleReflectanceToRadiance( @@ -108,7 +107,7 @@ pub fn scaleReflectanceToRadiance( const scale = solar_cosine * solar_irradiance / std.math.pi; const radiance_jacobian = if (wantsJacobian(solve_config)) - jacobian_states.scaleMasked(reflectance.jacobian, scale, solve_config.derivative_state_mask) + jacobian_states.scale(reflectance.jacobian, scale) else jacobian_states.zero(); @@ -181,11 +180,10 @@ pub fn integratePrefetchedRadianceAtNominal( radiance_sum += weight * results.radiance[result_index]; if (wants_jacobian) { - jacobian_states.addScaledMasked( + jacobian_states.addScaled( &jacobian_sum, results.jacobian[result_index], weight, - solve_config.derivative_state_mask, ); } } diff --git a/src/spectrum/sampling_table.zig b/src/spectrum/sampling_table.zig index 6831718b6..c34aacb35 100644 --- a/src/spectrum/sampling_table.zig +++ b/src/spectrum/sampling_table.zig @@ -226,7 +226,7 @@ pub const SpectrumSamplingTable = struct { // ------------------------------------------------------------------------------------------------------------| // OwnedSpectrumSamplingTable -------------------------------------------------------------------------------- | -// Owned sampling rows and side-array kernels returned by the O2 A sampling builder. | +// Owned sampling rows and side-array kernels returned by the sampling builder. | // | // layout(64-bit) | // size: 48 B (0.047 KiB), align: 8 B | @@ -562,7 +562,7 @@ pub fn buildSpectrumSamplingTable( lines: line_tables.LineTable, ) !OwnedSpectrumSamplingTable { // buildSpectrumSamplingTable -------------------------------------------------------------------------- | - // Build the O2 A spectrum sampling plan from explicit setup tables. | + // Build the spectrum sampling plan from explicit setup tables. | // | // This module owns the sampling, integration-kernel, and adaptive-interval rows that feed spectrum | // transport. | @@ -627,7 +627,7 @@ fn buildSpectrumSamplingTableWithNominals( errdefer allocator.free(rows); var kernel_storage_builder = KernelStorageBuilder.init(row_count * 2); - defer kernel_storage_builder.deinit(allocator); + errdefer kernel_storage_builder.deinit(allocator); try fillSamplingRows( allocator, @@ -969,7 +969,7 @@ fn buildAdaptiveIntervalPlan( // buildAdaptiveIntervalPlan ----------------------------------------------------------------------------- | // Partition the expanded spectral support around strong O2 line centers and assign Gauss orders. | // | - // Builds the adaptive interval plan for the DISAMAR reference case. | + // Builds the adaptive interval plan for the DISAMAR reference scene. | // --------------------------------------------------------------------------------------------------------| const support_window = adaptiveKernelSupportWindow(grid, instrument, grid.start_nm); const fwhm_nm = @max(instrument.line_fwhm_nm, safe_fwhm_floor_nm); diff --git a/src/spectrum/spectrum_run.zig b/src/spectrum/spectrum_run.zig index 678c669cd..e17eb4869 100644 --- a/src/spectrum/spectrum_run.zig +++ b/src/spectrum/spectrum_run.zig @@ -52,7 +52,7 @@ pub const radiance_prefetch_pooled_chunk_size: usize = 8; const RadiancePrefetchErrorState = worker_partition.FirstWorkerErrorState(anyerror); // SamplingPolicy ----------------------------------------------------------------------------------------- | -// Product-channel sampling and calibration policy for the exercised O2 A spectrum route. | +// Product-channel sampling and calibration policy for the exercised spectrum route. | // | // layout(64-bit) | // size: 104 B (0.102 KiB), align: 8 B | @@ -128,7 +128,7 @@ pub const RadianceWorkRows = struct { // ------------------------------------------------------------------------------------------------------------| // RadiancePrefetchWorker ------------------------------------------------------------------------------------ | -// Site-local worker row for dense O2 A radiance prefetch. | +// Site-local worker row for dense radiance prefetch. | // | // layout(64-bit) | // The row is intentionally not size-pinned because Telemetry.Context is sink-selected: zero-size in normal | @@ -216,7 +216,7 @@ pub fn radianceAtWavelength( // instrumentation: trace zone: one exact-wavelength forward sample -------------------------------------- | // captures: setup-optics, optional source/curved rows, transport, and radiance scaling for one wavelength | - // why: separates solve-count effects from per-solve cost when O2 A compares same-sitting reruns. | + // why: separates solve-count effects from per-solve cost when comparing same-sitting reruns. | var sample_zone = Trace.staticZone(@src(), "spectrum.radiance_at_wavelength"); defer sample_zone.end(); Trace.plotU("forward_samples", 1); @@ -249,7 +249,9 @@ pub fn radianceAtWavelength( stage_cost, ); try layer_depths.reduceLayerOpticsFromSupportRows(layer_grid, work_rows.support, work_rows.layers, stage_cost); - layer_depths.fillLayerAerosolJacobians(aerosol, prepared_solve_config.derivative_state_mask, work_rows.layers); + if (prepared_solve_config.derivative_mode != .none and prepared_solve_config.wants_jacobian) { + layer_depths.fillLayerAerosolJacobians(aerosol, work_rows.layers); + } } const source_rows = source_rows: { @@ -288,14 +290,13 @@ pub fn radianceAtWavelength( angles, surface_albedo, totalOpticalDepth(work_rows.layers), - prepared_solve_config.derivative_mode, - prepared_solve_config.derivative_state_mask, ); } const needs_order_local_sum = prepared_solve_config.controls.integrate_source_function and - prepared_solve_config.derivative_mode != .none; + prepared_solve_config.derivative_mode != .none and + prepared_solve_config.wants_jacobian; var work = try worker_memory.solveWorkArrays( layer_count, prepared_solve_config.controls.n_streams, @@ -377,13 +378,13 @@ fn prefetchRadianceRows( if (wavelengths.wavelengths.len == 0) return; // instrumentation: trace counter: forward worker count ---------------------------------------------------| - // captures: selected O2 A dense-prefetch worker count | + // captures: selected dense-prefetch worker count | // why: ties dense-prefetch wall time to the concurrency shape selected by root/session code. | Trace.plotU("forward_worker_count", @intCast(worker_count)); // end instrumentation: trace counter: forward worker count -----------------------------------------------| // instrumentation: trace counter: high-resolution radiance rows ------------------------------------------| - // captures: unique exact radiance wavelengths evaluated by this O2 A prefetch pass | + // captures: unique exact radiance wavelengths evaluated by this prefetch pass | // why: distinguishes fewer transport solves from cheaper work per solve. | Trace.plotU("high_resolution_misses", @intCast(wavelengths.wavelengths.len)); // end instrumentation: trace counter: high-resolution radiance rows --------------------------------------| @@ -462,7 +463,7 @@ fn transportWorkerShapeMatches( fn radiancePrefetchWorkerMain(worker: *RadiancePrefetchWorker) void { // radiancePrefetchWorkerMain -----------------------------------------------------------------------------| - // Worker loop for dense O2 A radiance rows. Each worker drains either a static range or the pooled queue, | + // Worker loop for dense radiance rows. Each worker drains either a static range or the pooled queue, | // writes disjoint dense rows, and stores the first error for the joined caller to return. | // --------------------------------------------------------------------------------------------------------| var thread_name_buffer: [64]u8 = undefined; @@ -488,13 +489,13 @@ fn radiancePrefetchWorkerMain(worker: *RadiancePrefetchWorker) void { while (nextRadiancePrefetchChunk(worker)) |chunk| { - // instrumentation: trace zone: O2 A radiance prefetch chunk ------------------------------------------| - // captures: one O2 A prefetch chunk size and wall time | + // instrumentation: trace zone: radiance prefetch chunk ------------------------------------------| + // captures: one prefetch chunk size and wall time | // why: preserves the chunk boundary while keeping per-wavelength optics visible separately. | const chunk_zone = Trace.deepStaticZone(@src(), "forward_prefetch.chunk"); chunk_zone.value(@intCast(chunk.len())); defer chunk_zone.end(); - // end instrumentation: trace zone: O2 A radiance prefetch chunk --------------------------------------| + // end instrumentation: trace zone: radiance prefetch chunk --------------------------------------| for (chunk.start..chunk.end) |index| { const wavelength = worker.wavelengths.wavelengths[index]; @@ -580,7 +581,7 @@ pub fn runForwardSpectrum( solar_memory: *solar_irradiance_memory.SolarIrradianceMemory, ) !instrument_average.ReflectanceAssemblySummary { // runForwardSpectrum -------------------------------------------------------------------------------------| - // Run the explicit O2 A spectrum path from exact radiance wavelengths to product reflectance. | + // Run the explicit spectrum path from exact radiance wavelengths to product reflectance. | // | // `grid_calculation/spectral_forward.zig` dense prefetch and `grid_calculation/simulate.zig` nominal | // gather, channel postprocess, and reflectance assembly. The individual formulas stay in the stage | @@ -611,13 +612,13 @@ pub fn runForwardSpectrum( product_rows.reflectance.len == row_count; if (!product_shapes_match) return error.ShapeMismatch; - // instrumentation: trace zone: O2 A spectrum run ---------------------------------------------------------| + // instrumentation: trace zone: spectrum run ---------------------------------------------------------| // captures: dense prefetch, product gather, channel postprocess, and reflectance assembly wall time | - // why: gives O2 A a same-boundary phase around the full explicit spectrum path. | + // why: gives a same-boundary phase around the full explicit spectrum path. | const run_zone = Trace.staticZone(@src(), "spectrum.o2a_run"); run_zone.value(@intCast(row_count)); defer run_zone.end(); - // end instrumentation: trace zone: O2 A spectrum run -----------------------------------------------------| + // end instrumentation: trace zone: spectrum run -----------------------------------------------------| if (prefetch_dense_radiance) { try prefetchRadianceRows( diff --git a/src/validation/band_metrics.zig b/src/validation/band_metrics.zig index bb3c63c14..446854973 100644 --- a/src/validation/band_metrics.zig +++ b/src/validation/band_metrics.zig @@ -5,22 +5,22 @@ pub const Error = error{ }; // band_metrics.zig ------------------------------------------------------------------------------------- | -// Validation-only O2 A spectrum comparison metrics. | +// Validation-only spectrum comparison metrics. | // | -// Replaces the removed broad runtime-case metrics helper with explicit validation vectors. | -// Runtime-case wrappers stay out of O2 A because they depend on broad pre-refactor owner shapes. | +// Replaces the removed broad runtime-scene metrics helper with explicit validation vectors. | +// Runtime-scene wrappers stay out because they depend on broad pre-refactor owner shapes. | // | // boundary | // Callers pass explicit wavelength/reflectance vectors and reference rows. This module performs no file | // I/O, imports no RTM/product modules, and stores no retained state. | // | -// O2 A windows | +// O2A windows | // blue wing 755.0..758.5 nm; trough 760.2..761.1 nm; rebound 761.8..762.4 nm; mid band 763.8..765.5 nm; | // red wing 769.5..771.0 nm. | // ----------------------------------------------------------------------------------------------------------- | // ReferenceSample ------------------------------------------------------------------------------------------ | -// One retained O2 A reference reflectance sample. | +// One retained O2A reference reflectance sample. | // | // layout(64-bit) | // size: 16 B (0.016 KiB), align: 8 B | @@ -50,7 +50,7 @@ pub const RangeExtremum = struct { // ----------------------------------------------------------------------------------------------------------- | // ComparisonMetrics ---------------------------------------------------------------------------------------- | -// Scalar residual, correlation, and O2 A morphology metrics for one generated/reference comparison. | +// Scalar residual, correlation, and morphology metrics for one generated/reference comparison. | // | // layout(64-bit) | // size: 120 B (0.117 KiB), align: 8 B | @@ -460,7 +460,7 @@ pub fn computeComparisonMetrics( zero_tolerance_abs: f64, ) Error!ComparisonMetrics { // computeComparisonMetrics ----------------------------------------------------------------------------- | - // Compare generated reflectance to reference rows using the residual and O2 A morphology metrics. | + // Compare generated reflectance to reference rows using the residual and morphology metrics. | // ------------------------------------------------------------------------------------------------------- | if (wavelengths_nm.len != reflectance.len) return error.ShapeMismatch; diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 69cdaf52d..37c97fe02 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -22,10 +22,11 @@ - Allocation-failure and `std.heap.DebugAllocator` coverage belong in the fast suites when they validate lifecycle or cleanup behavior. - Kernel and construction experiments live under `scaffolding/experiments/`; `tests/` contains asserting coverage. - `tests/validation/o2a_vendor_reflectance_assessment_test.zig` is an opt-in DISAMAR-reference assessment lane, not a default correctness gate. -- Run it with `zig build test-validation-o2a-vendor` when you need to compare zdisamar's O2 A forward reflectance against the stored DISAMAR reference for `Config_O2_with_CIA.in`. +- Run it with `zig build test-validation-o2a-vendor` when you need to compare zdisamar's forward reflectance against the stored DISAMAR reference for `Config_O2_with_CIA.in`. - The lane emits JSON on every run and only fails when the tracked metrics regress beyond the stored baseline tolerances. Treat the emitted JSON as the real output: compare `current` against `baseline`, look at `trend`, and decide whether `improved`, `flat`, or `regressed` matches the files that changed. -- A `flat` result is acceptable when the change did not touch forward physics or O2 A reference assets. A `regressed` result is a concern when work touched `src/rtm/`, `src/optics/`, `src/spectrum/`, `src/setup/`, `src/input/case.zig`, `data/reference_data/cross_sections`, or the O2 A reference CSV. A zero-difference pass is exceptional and should be called out explicitly. +- A `flat` result is acceptable when the change did not touch forward physics or reference assets. A `regressed` result is a concern when work touched `src/rtm/`, `src/optics/`, `src/spectrum/`, `src/setup/`, `src/input/case.zig`, `data/reference_data/cross_sections`, or the reference CSV. A zero-difference pass is exceptional and should be called out explicitly. - New config/control surfaces should get three kinds of coverage when practical: one valid-path test, one invalid or unsupported-input test, and one DISAMAR-reference test against the legacy or alternate path when both are meant to agree. - Shard and focused-lane tests must assert the intended case set or semantic outcome directly, not merely that some cases ran. - When a change introduces derived hints, prepared metadata, or delayed config application, add at least one test that omits redundant legacy fields so stale fallback values cannot mask ordering bugs. - Wavelength-dependent controls need at least two-point coverage when they affect prepared optics or morphology, so reference-only sampling mistakes fail deterministically. +- Use narrow bands for testing control flow and narrow-band physics; full band only for full-band-specific tests. diff --git a/tests/python/rtm_scene.py b/tests/python/rtm_scene.py new file mode 100644 index 000000000..96faf1f68 --- /dev/null +++ b/tests/python/rtm_scene.py @@ -0,0 +1,29 @@ +"""Shared minimal RTM scene for fast tests. + +The expensive part of every native RTM call is the one-time "prepare" step: building the O2 +A-band line sampling table across the scene's spectral band. That cost tracks the BAND WIDTH +(number of absorption lines spanned), not the sample count, so a 701-sample and an 11-sample +full-band scene both pay ~1.0s, while a ~1.5nm window pays ~0.25-0.3s. The forward itself is +~5ms once prepared. Tests that check a contract, route parity, structure, or relative +behavior run on the narrow window; tests that pin canonical full-band physics keep the full +scene and live elsewhere. +""" + + +def narrow(scene=None): + """Narrow a scene to a representative O2 A-band window (759.5-761nm, 16 samples). + + Pass an existing scene to narrow it in place (e.g. one carrying an aerosol profile), or + omit it to build and narrow the default reference scene. The window sits on the strong + R-branch with deep line cores, so a forward over it is a real radiative-transfer + computation, just over a band that prepares ~4x faster than the full 755-776nm band. + """ + + if scene is None: + from zdisamar.wavelength_bands import o2a + + scene = o2a.reference_scene() + scene.spectral_grid.start_nm = 759.5 + scene.spectral_grid.end_nm = 761.0 + scene.spectral_grid.sample_count = 16 + return scene diff --git a/tests/python/test_aerosol_profile_spectrum.py b/tests/python/test_aerosol_profile_spectrum.py index b2cf25481..035fbaaf0 100644 --- a/tests/python/test_aerosol_profile_spectrum.py +++ b/tests/python/test_aerosol_profile_spectrum.py @@ -5,12 +5,16 @@ from unittest.mock import patch import pytest -from zdisamar import rtm +from rtm_scene import narrow +from zdisamar import optimal_estimation, rtm from zdisamar.bindings.handles import RtmHandle -from zdisamar.inverse_method import optimal_estimation pytestmark = [pytest.mark.integration, pytest.mark.native] +# These tests assert serialization parity, "aerosol changes the spectrum", cache reuse, and +# retrieval rejection -- all band-independent. They run on a narrow representative window so +# every forward prepares ~4x faster. The canonical full-band physics lives in the Zig goldens. + def one_layer_profile() -> rtm.AerosolProfileLayer: return rtm.AerosolProfileLayer( @@ -47,11 +51,11 @@ def multi_layer_profile() -> tuple[rtm.AerosolProfileLayer, ...]: def profile_scene(profile: tuple[rtm.AerosolProfileLayer, ...]): scene = rtm.reference_scene() scene.set_aerosol_profile(profile) - return scene + return narrow(scene) def test_one_layer_aerosol_profile_serializes_as_legacy_layer_view() -> None: - scene = rtm.reference_scene() + scene = narrow(rtm.reference_scene()) assert len(scene.aerosol.profile) == 1 assert b'"profile"' not in scene.to_json_bytes() @@ -86,7 +90,7 @@ def test_one_layer_aerosol_profile_serializes_as_legacy_layer_view() -> None: def test_invalid_aerosol_profile_placements_are_rejected() -> None: - scene = rtm.reference_scene() + scene = narrow(rtm.reference_scene()) partially_off_grid_scene = deepcopy(scene) partially_off_grid_scene.set_aerosol_profile( ( @@ -125,7 +129,7 @@ def test_invalid_aerosol_profile_placements_are_rejected() -> None: def test_multi_layer_aerosol_profile_changes_spectrum_and_reuses_session_cache() -> None: - scene = rtm.reference_scene() + scene = narrow(rtm.reference_scene()) profiled_scene = profile_scene(multi_layer_profile()) assert b'"profile"' in profiled_scene.to_json_bytes() @@ -168,7 +172,12 @@ def test_multi_layer_aerosol_profile_is_forward_only_for_retrieval() -> None: initial=0.3, prior=0.3, prior_uncertainty=math.sqrt(0.8), - ) + ), + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + ), ] ), ) @@ -191,7 +200,7 @@ def expect_spectrum_rejected(scene) -> None: def expect_profile_jacobian_rejected(cache: rtm.SessionCache) -> None: try: - cache.spectrum(jacobian=True, jacobian_state_names=("aerosol_optical_depth",)) + cache.spectrum(jacobian=True) except ValueError as error: assert "profile Jacobians" in str(error) else: @@ -218,7 +227,12 @@ def expect_profile_oe_correction_rejected(scene, spectrum) -> None: initial=0.3, prior=0.3, prior_uncertainty=math.sqrt(0.8), - ) + ), + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + ), ] ), controls=optimal_estimation.RetrievalControls(max_iterations=1), diff --git a/tests/python/test_import_boundaries.py b/tests/python/test_import_boundaries.py index 9dd414b5f..fc6f5bda2 100644 --- a/tests/python/test_import_boundaries.py +++ b/tests/python/test_import_boundaries.py @@ -20,14 +20,18 @@ def test_import_laziness() -> None: assert "numpy" not in sys.modules assert "pandas" not in sys.modules assert "altair" not in sys.modules -assert zd.__all__ == ["reference_data", "rtm", "wavelength_bands"] +assert zd.__all__ == ["reference_data", "rtm", "optimal_estimation", "wavelength_bands"] assert not hasattr(zd, "prepare") assert not hasattr(zd, "forward") assert not hasattr(zd, "Scene") import zdisamar.api as api -assert api.__all__ == ["reference_data", "rtm", "wavelength_bands"] +assert api.__all__ == ["reference_data", "rtm", "optimal_estimation", "wavelength_bands"] + +from zdisamar import optimal_estimation + +assert optimal_estimation.Measurement.__name__ == "Measurement" """ subprocess.run([sys.executable, "-c", script], check=True, env=env) @@ -95,10 +99,10 @@ def test_plot_jacobian_uses_rtm_conversion() -> None: from zdisamar.plot.jacobian import jacobian_frame class Spectrum: - jacobian_state_names = ("aerosol_optical_depth",) + jacobian_state_names = ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa") wavelength_nm = np.array([755.0, 760.0], dtype=np.float64) irradiance = np.array([10.0, 12.0], dtype=np.float64) - radiance_jacobian = np.array([[0.2], [0.3]], dtype=np.float64) + radiance_jacobian = np.array([[0.2, 0.02], [0.3, 0.03]], dtype=np.float64) def reflectance_jacobian(self, state: str): diff --git a/tests/python/test_o2a_measurement_noise.py b/tests/python/test_o2a_measurement_noise.py index e90c32622..f063fefb0 100644 --- a/tests/python/test_o2a_measurement_noise.py +++ b/tests/python/test_o2a_measurement_noise.py @@ -1,4 +1,4 @@ -"""Check retained O2 A baseline measurement-noise scaling.""" +"""Check retained baseline measurement-noise scaling.""" import math import sys diff --git a/tests/python/test_o2a_prepare_external_cwd.py b/tests/python/test_o2a_prepare_external_cwd.py index a334d056d..c71c1524a 100644 --- a/tests/python/test_o2a_prepare_external_cwd.py +++ b/tests/python/test_o2a_prepare_external_cwd.py @@ -1,7 +1,7 @@ import numpy as np import pytest +from rtm_scene import narrow from zdisamar import rtm -from zdisamar.wavelength_bands import o2a pytestmark = pytest.mark.integration @@ -18,11 +18,13 @@ def spectrum_arrays(spectrum) -> dict[str, np.ndarray]: def test_reference_data_resolves_from_external_cwd(tmp_path, monkeypatch) -> None: monkeypatch.chdir(tmp_path) - scene = o2a.reference_scene() + # Narrow representative band: this test verifies external-cwd asset resolution, route + # determinism, and finiteness -- all independent of band width. + scene = narrow() assert "vendor/disamar-fortran" not in scene.o2_lines.line_list_asset.path spectrum = rtm.spectrum(scene) - reference_spectrum = rtm.spectrum(o2a.reference_scene()) + reference_spectrum = rtm.spectrum(narrow()) assert len(spectrum.wavelength_nm) == int(scene.spectral_grid.sample_count) assert len(reference_spectrum.wavelength_nm) == int(scene.spectral_grid.sample_count) diff --git a/tests/python/test_o2a_setup_roundtrip.py b/tests/python/test_o2a_setup_roundtrip.py index ff4481416..41f7d3ccb 100644 --- a/tests/python/test_o2a_setup_roundtrip.py +++ b/tests/python/test_o2a_setup_roundtrip.py @@ -17,7 +17,7 @@ def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( description=( - "Verify that the typed Python O2 A setup matches the default DISAMAR parity entrypoint." + "Verify that the typed Python setup matches the default DISAMAR parity entrypoint." ) ) parser.add_argument( @@ -39,16 +39,23 @@ def spectrum_arrays(spectrum: Any) -> dict[str, np.ndarray]: } -def run_roundtrip() -> dict[str, Any]: +def run_roundtrip(scene_factory: Any = None) -> dict[str, Any]: from zdisamar import rtm from zdisamar.wavelength_bands import o2a + # scene_factory builds the scene under test. The contract checks below are all route + # parities and structural invariants (typed == reference, session == functional, + # jacobian shape/determinism), so they hold on any band; the test passes a narrow + # representative scene for speed while the CLI keeps the full default band. + if scene_factory is None: + scene_factory = o2a.reference_scene + tolerance = 1.0e-12 start_s = time.perf_counter() scene_start_s = time.perf_counter() - scene = o2a.reference_scene() + scene = scene_factory() scene_s = time.perf_counter() - scene_start_s typed_rtm_start_s = time.perf_counter() @@ -58,7 +65,7 @@ def run_roundtrip() -> dict[str, Any]: typed_arrays = spectrum_arrays(typed_spectrum) reference_rtm_start_s = time.perf_counter() - reference_spectrum = rtm.spectrum(o2a.reference_scene()) + reference_spectrum = rtm.spectrum(scene_factory()) reference_rtm_s = time.perf_counter() - reference_rtm_start_s reference_report = reference_spectrum.diagnostic_report reference_arrays = spectrum_arrays(reference_spectrum) @@ -86,37 +93,17 @@ def run_roundtrip() -> dict[str, Any]: perturbed_session_spectrum = cache.spectrum(jacobian=True) perturbed_session_rtm_s = time.perf_counter() - perturbed_session_rtm_start_s perturbed_session_arrays = spectrum_arrays(perturbed_session_spectrum) - perturbed_session_names = perturbed_session_spectrum.jacobian_state_names perturbed_session_jacobian = np.asarray( perturbed_session_spectrum.radiance_jacobian, dtype=np.float64, ).copy() - requested_jacobian_names = ( - "aerosol_optical_depth", - "aerosol_layer_mid_pressure_hpa", - ) - compact_session_spectrum = cache.spectrum( - jacobian=True, - jacobian_state_names=requested_jacobian_names, - ) - compact_session_names = compact_session_spectrum.jacobian_state_names - compact_session_jacobian = np.asarray( - compact_session_spectrum.radiance_jacobian, + repeated_session_spectrum = cache.spectrum(jacobian=True) + repeated_session_names = repeated_session_spectrum.jacobian_state_names + repeated_session_jacobian = np.asarray( + repeated_session_spectrum.radiance_jacobian, dtype=np.float64, ).copy() - empty_jacobian_state_selection_rejected = False - - try: - cache.spectrum(jacobian=True, jacobian_state_names=()) - except ValueError: - empty_jacobian_state_selection_rejected = True - - full_state_indices = {name: index for index, name in enumerate(perturbed_session_names)} - selected_full_jacobian = np.stack( - [perturbed_session_jacobian[:, full_state_indices[name]] for name in compact_session_names], - axis=1, - ) perturbed_functional_spectrum = rtm.spectrum(perturbed_scene, jacobian=True) perturbed_functional_arrays = spectrum_arrays(perturbed_functional_spectrum) @@ -215,26 +202,29 @@ def run_roundtrip() -> dict[str, Any]: rtol=tolerance, ) ), - "requested_jacobian_dimension_matches_state_vector": bool( - compact_session_names == requested_jacobian_names - and compact_session_jacobian.shape - == (perturbed_session_arrays["wavelength_nm"].size, len(requested_jacobian_names)) + "jacobian_dimension_matches_fixed_state_vector": bool( + repeated_session_names + == ( + "aerosol_optical_depth", + "aerosol_layer_mid_pressure_hpa", + ) + and repeated_session_jacobian.shape + == (perturbed_session_arrays["wavelength_nm"].size, 2) ), - "requested_jacobian_columns_match_full_native_columns": bool( + "repeated_jacobian_matches_full_native_columns": bool( np.allclose( - compact_session_jacobian, - selected_full_jacobian, + repeated_session_jacobian, + perturbed_session_jacobian, atol=tolerance, rtol=tolerance, ) ), - "empty_jacobian_state_selection_rejected": empty_jacobian_state_selection_rejected, "parity_route_used": True, "tolerance": tolerance, } return { - "question": "Does the typed Python O2 A setup match the default DISAMAR parity entrypoint?", + "question": "Does the typed Python setup match the default DISAMAR parity entrypoint?", "answer": { "matches": checks["reference_matches_typed_arrays"], "sample_count": checks["typed_sample_count"], @@ -279,12 +269,10 @@ def main() -> int: f"session_matches_typed_arrays={checks['session_matches_typed_arrays']}, " "session_reload_matches_functional_jacobian=" f"{checks['session_reload_matches_functional_jacobian']}, " - "requested_jacobian_dimension_matches_state_vector=" - f"{checks['requested_jacobian_dimension_matches_state_vector']}, " - "requested_jacobian_columns_match_full_native_columns=" - f"{checks['requested_jacobian_columns_match_full_native_columns']}, " - "empty_jacobian_state_selection_rejected=" - f"{checks['empty_jacobian_state_selection_rejected']}" + "jacobian_dimension_matches_fixed_state_vector=" + f"{checks['jacobian_dimension_matches_fixed_state_vector']}, " + "repeated_jacobian_matches_full_native_columns=" + f"{checks['repeated_jacobian_matches_full_native_columns']}" ) print(f"json: {output_path}") excluded = {"tolerance", "typed_sample_count", "reference_sample_count"} @@ -298,10 +286,12 @@ def main() -> int: def test_o2a_setup_roundtrip_contracts() -> None: - summary = run_roundtrip() + from rtm_scene import narrow + + summary = run_roundtrip(scene_factory=narrow) checks = summary["checks"] excluded = {"tolerance", "typed_sample_count", "reference_sample_count"} assert all(value for key, value in checks.items() if key not in excluded) assert summary["answer"]["matches"] is True - assert summary["answer"]["sample_count"] == 701 + assert summary["answer"]["sample_count"] == 16 diff --git a/tests/python/test_optimal_estimation_diagnosis.py b/tests/python/test_optimal_estimation_diagnosis.py index 02fc81d07..15b796bb1 100644 --- a/tests/python/test_optimal_estimation_diagnosis.py +++ b/tests/python/test_optimal_estimation_diagnosis.py @@ -8,8 +8,8 @@ def test_optimal_estimation_diagnosis_display() -> None: - from zdisamar.inverse_method.optimal_estimation.diagnosis import RetrievalDiagnosis - from zdisamar.inverse_method.optimal_estimation.retrieval import Result + from zdisamar.optimal_estimation.diagnosis import RetrievalDiagnosis + from zdisamar.optimal_estimation.retrieval import Result calls = 0 @@ -21,13 +21,13 @@ def diagnosis_factory(*, n: int | None = None): return {"n": n} result = Result( - state_names=("aerosol_optical_depth",), - state=(0.1,), + state_names=("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), + state=(0.1, 600.0), iterations=1, converged=True, history=(), - posterior_covariance=((1.0,),), - averaging_kernel=((1.0,),), + posterior_covariance=((1.0, 0.0), (0.0, 1.0)), + averaging_kernel=((1.0, 0.0), (0.0, 1.0)), diagnosis_factory=diagnosis_factory, ) diagnosis = result.diagnose(n=3) @@ -109,18 +109,21 @@ def diagnosis_factory(*, n: int | None = None): grouped_payload = basin_sweep.to_dict() assert grouped_payload["basin_count"] == 2 - one_axis_sweep = RetrievalDiagnosis( - state_names=("aerosol_optical_depth",), - start_state=((0.1,), (0.2,)), - retrieved_state=((0.12,), (0.22,)), - iterations=(3, 4), - converged=(True, True), - retrieval_paths=(((0.12,),), ((0.22,),)), - start_bounds=((0.02, 2.0),), - batch_workers=1, - ) - assert one_axis_sweep.basins() == () - assert one_axis_sweep.to_dict()["basin_count"] == 0 + try: + RetrievalDiagnosis( + state_names=("aerosol_optical_depth",), + start_state=((0.1,), (0.2,)), + retrieved_state=((0.12,), (0.22,)), + iterations=(3, 4), + converged=(True, True), + retrieval_paths=(((0.12,),), ((0.22,),)), + start_bounds=((0.02, 2.0),), + batch_workers=1, + ) + except ValueError as error: + assert "aerosol_layer_mid_pressure_hpa" in str(error) + else: + raise AssertionError("one-axis retrieval diagnosis was accepted") two_axis_sweep = RetrievalDiagnosis( state_names=( @@ -177,10 +180,10 @@ def diagnosis_factory(*, n: int | None = None): def test_optimal_estimation_diagnosis_auto_workers() -> None: from zdisamar.input.wavelength_band.o2a import Scene - from zdisamar.inverse_method.optimal_estimation.diagnosis import diagnose_retrieval - from zdisamar.inverse_method.optimal_estimation.o2a import BatchResult - from zdisamar.inverse_method.optimal_estimation.retrieval import Measurement, RetrievalControls - from zdisamar.inverse_method.optimal_estimation.state_vector import ( + from zdisamar.optimal_estimation.diagnosis import diagnose_retrieval + from zdisamar.optimal_estimation.o2a import BatchResult + from zdisamar.optimal_estimation.retrieval import Measurement, RetrievalControls + from zdisamar.optimal_estimation.state_vector import ( AerosolLayerMidPressure, AerosolOpticalDepth, StateVector, @@ -225,8 +228,8 @@ def diagnosis_batch(**kwargs): return batch - from zdisamar.inverse_method import optimal_estimation - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe + from zdisamar import optimal_estimation + from zdisamar.optimal_estimation import o2a as o2a_oe banned_batch_api = "retrieve" + "_many" assert not hasattr(optimal_estimation, banned_batch_api) @@ -235,7 +238,7 @@ def diagnosis_batch(**kwargs): with ( patch.dict(os.environ, {"ZDISAMAR_WORKER_LIMIT": "10"}), patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=diagnosis_batch, ), ): @@ -260,7 +263,7 @@ def diagnosis_batch(**kwargs): calls.clear() with patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=diagnosis_batch, ): explicit = diagnose_retrieval( @@ -296,7 +299,7 @@ def explicit_diagnosis_batch(**kwargs): ) with patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=explicit_diagnosis_batch, ): explicit_large = diagnose_retrieval( @@ -316,10 +319,10 @@ def explicit_diagnosis_batch(**kwargs): def test_optimal_estimation_diagnosis_adaptive_grid() -> None: from zdisamar.input.wavelength_band.o2a import Scene - from zdisamar.inverse_method.optimal_estimation.diagnosis import diagnose_retrieval - from zdisamar.inverse_method.optimal_estimation.o2a import BatchResult - from zdisamar.inverse_method.optimal_estimation.retrieval import Measurement, RetrievalControls - from zdisamar.inverse_method.optimal_estimation.state_vector import ( + from zdisamar.optimal_estimation.diagnosis import diagnose_retrieval + from zdisamar.optimal_estimation.o2a import BatchResult + from zdisamar.optimal_estimation.retrieval import Measurement, RetrievalControls + from zdisamar.optimal_estimation.state_vector import ( AerosolLayerMidPressure, AerosolOpticalDepth, StateVector, @@ -386,7 +389,7 @@ def diagnosis_batch(**kwargs): with ( patch.dict(os.environ, {"ZDISAMAR_WORKER_LIMIT": "10"}), patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=diagnosis_batch, ), ): @@ -410,7 +413,7 @@ def diagnosis_batch(**kwargs): calls.clear() with patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=diagnosis_batch, ): dynamic = diagnose_retrieval( @@ -432,7 +435,7 @@ def diagnosis_batch(**kwargs): assert any(0.37 <= row[0] <= 0.55 for row in dynamic_refinement_rows) calls.clear() - from zdisamar.inverse_method.optimal_estimation.diagnosis import BoundaryBisectionStrategy + from zdisamar.optimal_estimation.diagnosis import BoundaryBisectionStrategy strategy = BoundaryBisectionStrategy( refinement_batch_size=3, @@ -440,7 +443,7 @@ def diagnosis_batch(**kwargs): ) with patch( - "zdisamar.inverse_method.optimal_estimation.o2a.diagnosis_batch", + "zdisamar.optimal_estimation.o2a.diagnosis_batch", side_effect=diagnosis_batch, ): custom = diagnose_retrieval( @@ -455,7 +458,7 @@ def diagnosis_batch(**kwargs): assert len(calls[1]) == 3 assert any(0.46 <= row[0] <= 0.49 for row in custom.start_state[9:]) - from zdisamar.inverse_method.optimal_estimation.diagnosis import DiagnosisBatchSummary + from zdisamar.optimal_estimation.diagnosis import DiagnosisBatchSummary def synthetic_boundary_batch(rows: tuple[tuple[float, ...], ...]) -> DiagnosisBatchSummary: diff --git a/tests/python/test_optimal_estimation_fastmode.py b/tests/python/test_optimal_estimation_fastmode.py index 59f638bcb..d6013ec20 100644 --- a/tests/python/test_optimal_estimation_fastmode.py +++ b/tests/python/test_optimal_estimation_fastmode.py @@ -8,14 +8,14 @@ def test_fastmode_oe_runs_single_full_correction() -> None: from zdisamar.input.wavelength_band.o2a import Scene from zdisamar.input.wavelength_band.optimisation import Optimisation - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Iteration, Measurement, Result, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector from zdisamar.rtm.session_cache import SessionCache @dataclass(frozen=True) @@ -209,13 +209,13 @@ def test_fastmode_oe_uses_sparse_fast_stage_sampling() -> None: FastModeWavelengthWindow, Optimisation, ) - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Measurement, Result, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector from zdisamar.rtm.session_cache import SessionCache optimisation = Optimisation.defaults() @@ -240,7 +240,18 @@ def test_fastmode_oe_uses_sparse_fast_stage_sampling() -> None: ), instrument_response=SimpleNamespace(measured_wavelengths_nm=wavelengths), ) - state_vector = cast(StateVector, SimpleNamespace(parameters=(), names=())) + state_names = ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa") + state_vector = cast( + StateVector, + SimpleNamespace( + parameters=( + SimpleNamespace(name="aerosol_optical_depth"), + SimpleNamespace(name="aerosol_layer_mid_pressure_hpa"), + ), + names=state_names, + jacobian_names=state_names, + ), + ) calls: list[dict[str, object]] = [] loads: list[tuple[object, bool]] = [] @@ -249,15 +260,15 @@ def fake_native_retrieval(**kwargs): calls.append(kwargs) return Result( - state_names=(), - state=(), + state_names=state_names, + state=(0.2, 800.0), iterations=1, converged=True, history=(), - posterior_covariance=(), - averaging_kernel=(), + posterior_covariance=((1.0, 0.0), (0.0, 1.0)), + averaging_kernel=((1.0, 0.0), (0.0, 1.0)), measurement=kwargs["measurement"], - initial_state=(), + initial_state=(0.2, 800.0), ) class Cache: @@ -319,12 +330,12 @@ def test_fastmode_batch_uses_native_fused_correction() -> None: from zdisamar.input.wavelength_band.o2a import Scene from zdisamar.input.wavelength_band.optimisation import Optimisation - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Measurement, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector @dataclass(frozen=True) class Parameter: @@ -514,13 +525,13 @@ def test_fastmode_pressure_profile_uses_loaded_sparse_cache() -> None: from zdisamar.input.instrument import SpectralGrid from zdisamar.input.wavelength_band.o2a import Scene from zdisamar.input.wavelength_band.optimisation import Optimisation - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Measurement, Result, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector from zdisamar.rtm.session_cache import SessionCache @dataclass(frozen=True) @@ -585,7 +596,19 @@ def write_to(self, target: object, value: float) -> None: ), instrument_response=SimpleNamespace(measured_wavelengths_nm=wavelengths), ) - state_vector = StateVector((PressureParameter(),)) + state_vector = StateVector( + ( + ResolvedParameter( + name="aerosol_optical_depth", + initial=0.2, + prior=0.2, + prior_uncertainty=0.1, + lower=0.0, + upper=1.0, + ), + PressureParameter(), + ) + ) loads: list[tuple[object, bool]] = [] match_checks: list[object] = [] profile_calls: list[tuple[object, object]] = [] @@ -620,12 +643,12 @@ def fake_native_retrieval(**kwargs): return Result( state_names=kwargs["state_vector"].names, - state=(800.0,), + state=(0.2, 800.0), iterations=1, converged=True, history=(), - posterior_covariance=((1.0,),), - averaging_kernel=((1.0,),), + posterior_covariance=((1.0, 0.0), (0.0, 1.0)), + averaging_kernel=((1.0, 0.0), (0.0, 1.0)), measurement=kwargs["measurement"], initial_state=kwargs["state_vector"].initial_state(), ) @@ -658,4 +681,4 @@ def fake_native_retrieval(**kwargs): active_controls = cast(RetrievalControls, retrieval_calls[0]["controls"]) assert active_controls.max_iterations == 4 resolved_state_vector = cast(StateVector, retrieval_calls[0]["state_vector"]) - assert isinstance(resolved_state_vector.parameters[0], ResolvedParameter) + assert isinstance(resolved_state_vector.parameters[1], ResolvedParameter) diff --git a/tests/python/test_optimal_estimation_models.py b/tests/python/test_optimal_estimation_models.py index 7c839fc7c..9a3b42cdb 100644 --- a/tests/python/test_optimal_estimation_models.py +++ b/tests/python/test_optimal_estimation_models.py @@ -8,10 +8,10 @@ def test_optimal_estimation_grid_mismatch_rejected() -> None: import numpy as np - from zdisamar.inverse_method.optimal_estimation import ( + from zdisamar.optimal_estimation import ( WavelengthGridMismatchError, ) - from zdisamar.inverse_method.optimal_estimation.measurement import ( + from zdisamar.optimal_estimation.measurement import ( require_matching_wavelength_grid, ) @@ -30,19 +30,22 @@ def test_optimal_estimation_grid_mismatch_rejected() -> None: def test_optimal_estimation_result_dataclass() -> None: - from zdisamar.inverse_method.optimal_estimation.retrieval import FastCorrection, Result - from zdisamar.inverse_method.optimal_estimation.rtm_evaluation import RtmEvaluation + from zdisamar.optimal_estimation.retrieval import FastCorrection, Result + from zdisamar.optimal_estimation.rtm_evaluation import RtmEvaluation first = cast(RtmEvaluation, object()) second = cast(RtmEvaluation, object()) + state_names = ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa") + state = (0.3, 850.0) + covariance = ((1.0, 0.0), (0.0, 1.0)) result = Result( - state_names=(), - state=(), + state_names=state_names, + state=state, iterations=0, converged=True, history=(), - posterior_covariance=(), - averaging_kernel=(), + posterior_covariance=covariance, + averaging_kernel=covariance, final_evaluation=first, ) assert result.final_evaluation is first @@ -51,13 +54,13 @@ def test_optimal_estimation_result_dataclass() -> None: assert "final_evaluation" in {field.name for field in fields(result)} positional = Result( - (), - (), + state_names, + state, 0, True, (), - (), - (), + covariance, + covariance, ) assert positional.initial_state is None assert positional.fast_correction is None @@ -65,7 +68,7 @@ def test_optimal_estimation_result_dataclass() -> None: correction = FastCorrection( fast_iterations=2, fast_converged=True, - fast_state=(0.4,), + fast_state=(0.4, 850.0), full_correction=None, full_correction_converged=False, full_correction_state_vector_convergence=1.5, @@ -75,21 +78,21 @@ def test_optimal_estimation_result_dataclass() -> None: def test_final_evaluation_reuses_last_rtm_evaluation() -> None: - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import Result - from zdisamar.inverse_method.optimal_estimation.rtm_evaluation import RtmEvaluation + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import Result + from zdisamar.optimal_estimation.rtm_evaluation import RtmEvaluation sentinel = cast(RtmEvaluation, object()) calls = 0 result = Result( - state_names=("aerosol_optical_depth",), - state=(1.0,), + state_names=("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), + state=(1.0, 850.0), iterations=1, converged=True, history=(), - posterior_covariance=((1.0,),), - averaging_kernel=((1.0,),), - last_evaluated_state=(1.0,), + posterior_covariance=((1.0, 0.0), (0.0, 1.0)), + averaging_kernel=((1.0, 0.0), (0.0, 1.0)), + last_evaluated_state=(1.0, 850.0), last_evaluation=sentinel, ) @@ -107,9 +110,9 @@ def evaluate_state(_state): def test_lazy_final_evaluator_snapshots_scene() -> None: - from zdisamar.inverse_method import optimal_estimation - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.rtm_evaluation import RtmEvaluation + from zdisamar import optimal_estimation + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.rtm_evaluation import RtmEvaluation from zdisamar.wavelength_bands import o2a sentinel = cast(RtmEvaluation, object()) @@ -132,20 +135,25 @@ def fake_evaluate_state(template, _state, _state_vector): initial=0.3, prior=0.3, prior_uncertainty=math.sqrt(0.8), - ) + ), + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + ), ] ), ) scene.geometry.solar_zenith_deg = 0.0 - assert evaluator([1.0]) is sentinel + assert evaluator([1.0, 850.0]) is sentinel assert observed_solar_zenith == [original_solar_zenith] def test_state_vector_uncertainty_rejected() -> None: - from zdisamar.inverse_method import optimal_estimation + from zdisamar import optimal_estimation for uncertainty in (-1.0, 0.0, math.inf, math.nan): try: @@ -156,6 +164,11 @@ def test_state_vector_uncertainty_rejected() -> None: prior=0.3, prior_uncertainty=uncertainty, ), + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + ), ) ) except ValueError as error: @@ -164,10 +177,50 @@ def test_state_vector_uncertainty_rejected() -> None: raise AssertionError("invalid state-vector uncertainty was accepted") +def test_state_vector_requires_canonical_two_state_order() -> None: + + from zdisamar import optimal_estimation + + try: + optimal_estimation.StateVector( + ( + optimal_estimation.AerosolOpticalDepth( + initial=0.3, + prior=0.3, + prior_uncertainty=math.sqrt(0.8), + ), + ) + ) + except ValueError as error: + assert "aerosol_layer_mid_pressure_hpa" in str(error) + else: + raise AssertionError("one-state vector was accepted") + + try: + optimal_estimation.StateVector( + ( + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + ), + optimal_estimation.AerosolOpticalDepth( + initial=0.3, + prior=0.3, + prior_uncertainty=math.sqrt(0.8), + ), + ) + ) + except ValueError as error: + assert "in that order" in str(error) + else: + raise AssertionError("reordered state vector was accepted") + + def test_measurement_accepts_scientific_numeric_scalars() -> None: import numpy as np - from zdisamar.inverse_method import optimal_estimation + from zdisamar import optimal_estimation measurement = optimal_estimation.Measurement( wavelength_nm=(cast(float, np.float64(760.0)), cast(float, np.float64(760.1))), diff --git a/tests/python/test_optimal_estimation_native.py b/tests/python/test_optimal_estimation_native.py index 1fac3c6d8..17d69af1d 100644 --- a/tests/python/test_optimal_estimation_native.py +++ b/tests/python/test_optimal_estimation_native.py @@ -11,19 +11,25 @@ def test_native_oe_loads_requested_scene_into_supplied_cache() -> None: from zdisamar.input.wavelength_band.o2a import Scene - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Measurement, Result, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector from zdisamar.rtm.session_cache import SessionCache events: list[tuple[str, object, object]] = [] requested_scene = SimpleNamespace(scene_id="requested") measurement = Measurement((760.0,), (0.2,), signal_to_noise=100.0) - state_vector = SimpleNamespace(parameters=(), jacobian_names=("aerosol_optical_depth",)) + state_vector = SimpleNamespace( + parameters=( + SimpleNamespace(name="aerosol_optical_depth"), + SimpleNamespace(name="aerosol_layer_mid_pressure_hpa"), + ), + jacobian_names=("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), + ) controls = RetrievalControls(max_iterations=1) native_result = Result((), (), 0, True, (), (), ()) @@ -67,7 +73,7 @@ def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: assert events == [ ("has_loaded_case", requested_scene, None), ("load", requested_scene, False), - ("warm_oe", ("aerosol_optical_depth",), None), + ("warm_oe", ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), None), ("optimal_estimation", measurement, controls), ] @@ -75,19 +81,25 @@ def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: def test_native_oe_reuses_matching_supplied_cache() -> None: from zdisamar.input.wavelength_band.o2a import Scene - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe - from zdisamar.inverse_method.optimal_estimation.retrieval import ( + from zdisamar.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation.retrieval import ( Measurement, Result, RetrievalControls, ) - from zdisamar.inverse_method.optimal_estimation.state_vector import StateVector + from zdisamar.optimal_estimation.state_vector import StateVector from zdisamar.rtm.session_cache import SessionCache events: list[tuple[str, object, object]] = [] requested_scene = SimpleNamespace(scene_id="requested") measurement = Measurement((760.0,), (0.2,), signal_to_noise=100.0) - state_vector = SimpleNamespace(parameters=(), jacobian_names=("aerosol_optical_depth",)) + state_vector = SimpleNamespace( + parameters=( + SimpleNamespace(name="aerosol_optical_depth"), + SimpleNamespace(name="aerosol_layer_mid_pressure_hpa"), + ), + jacobian_names=("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), + ) controls = RetrievalControls(max_iterations=1) native_result = Result((), (), 0, True, (), (), ()) @@ -109,7 +121,7 @@ def has_loaded_scene(self, scene) -> bool: def load(self, scene, *, copy_scene: bool = True) -> None: - raise AssertionError("matching OE cache reloaded its prepared case") + raise AssertionError("matching OE cache reloaded its prepared scene") def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: @@ -130,15 +142,15 @@ def warm_optimal_estimation(self, state_names: tuple[str, ...]) -> None: assert result is native_result assert events == [ ("has_loaded_case", requested_scene, None), - ("warm_oe", ("aerosol_optical_depth",), None), + ("warm_oe", ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa"), None), ("optimal_estimation", measurement, controls), ] def test_native_oe_marshaling_bounds() -> None: + from zdisamar import optimal_estimation from zdisamar.bindings.handles import RtmHandle - from zdisamar.inverse_method import optimal_estimation handle = object.__new__(RtmHandle) measurement = optimal_estimation.Measurement( @@ -146,6 +158,10 @@ def test_native_oe_marshaling_bounds() -> None: reflectance=[0.2], signal_to_noise=100.0, ) + pressure_profile = SimpleNamespace( + altitude_km=(0.0, 1.0), + pressure_hpa=(900.0, 800.0), + ) state_vector = SimpleNamespace( parameters=[ SimpleNamespace( @@ -156,7 +172,18 @@ def test_native_oe_marshaling_bounds() -> None: lower=None, upper=None, interval_index_1based=0, - ) + ), + SimpleNamespace( + name="aerosol_layer_mid_pressure_hpa", + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + lower=600.0, + upper=1000.0, + interval_index_1based=2, + thickness_hpa=10.0, + pressure_altitude_profile=pressure_profile, + ), ] ) @@ -272,14 +299,17 @@ def test_native_oe_marshaling_bounds() -> None: def test_native_oe_runs_after_reference_scene_prepare() -> None: + from rtm_scene import narrow + from zdisamar import optimal_estimation from zdisamar.bindings.handles import RtmHandle - from zdisamar.inverse_method import optimal_estimation - from zdisamar.wavelength_bands import o2a + from zdisamar.optimal_estimation import o2a as o2a_oe handle = RtmHandle() try: - scene = o2a.reference_scene() + # Narrow representative band: this test asserts retrieval STRUCTURE (iteration and + # state counts), not full-band physics values, so the band does not affect coverage. + scene = narrow() measurement = optimal_estimation.simulate_measurement( scene, signal_to_noise=100.0, @@ -291,8 +321,16 @@ def test_native_oe_runs_after_reference_scene_prepare() -> None: prior=0.3, prior_uncertainty=math.sqrt(0.8), ), + optimal_estimation.AerosolLayerMidPressure( + initial=850.0, + prior=850.0, + prior_uncertainty=100.0, + lower=600.0, + upper=1000.0, + ), ) ) + state_vector = o2a_oe.resolved_state_vector_for_scene(scene, state_vector) handle.load_scene(scene) result = handle.optimal_estimation( measurement=measurement, @@ -300,13 +338,13 @@ def test_native_oe_runs_after_reference_scene_prepare() -> None: controls=optimal_estimation.RetrievalControls(max_iterations=1), ) assert result["iteration_count"] == 1 - assert result["state_count"] == 1 + assert result["state_count"] == 2 correction = handle.optimal_estimation_correction( measurement=measurement, state_vector=state_vector, controls=optimal_estimation.RetrievalControls(max_iterations=10), ) assert correction["iteration_count"] == 1 - assert correction["state_count"] == 1 + assert correction["state_count"] == 2 finally: handle.close() diff --git a/tests/python/test_optimal_estimation_plot.py b/tests/python/test_optimal_estimation_plot.py index ed629fa0c..932d8946e 100644 --- a/tests/python/test_optimal_estimation_plot.py +++ b/tests/python/test_optimal_estimation_plot.py @@ -4,8 +4,8 @@ from typing import cast import numpy as np -from zdisamar.inverse_method.optimal_estimation.retrieval import Iteration, Measurement, Result -from zdisamar.inverse_method.optimal_estimation.rtm_evaluation import RtmEvaluation +from zdisamar.optimal_estimation.retrieval import Iteration, Measurement, Result +from zdisamar.optimal_estimation.rtm_evaluation import RtmEvaluation from zdisamar.plot.fields import TOTAL_OPTICAL_DEPTH, WAVELENGTH_NM from zdisamar.plot.optimal_estimation import ( JACOBIAN_PANEL_SPACING, diff --git a/tests/python/test_reference_data_and_rtm_tables.py b/tests/python/test_reference_data_and_rtm_tables.py index 76a8dbf19..9f43f5e81 100644 --- a/tests/python/test_reference_data_and_rtm_tables.py +++ b/tests/python/test_reference_data_and_rtm_tables.py @@ -6,6 +6,7 @@ from typing import cast import pytest +from rtm_scene import narrow pytestmark = [pytest.mark.integration, pytest.mark.native] @@ -195,7 +196,12 @@ def test_reference_data_and_rtm_tables() -> None: else: raise AssertionError("negative nominal spectral sample count was accepted") - mutable_scene = copy.deepcopy(scene) + # The config assertions above need the full band (fastmode windows reference + # specific wavelengths). The forward/budget/plot calls below only check structure + # and route behavior, so they run on a narrow representative copy for speed. + forward_scene = narrow(copy.deepcopy(scene)) + + mutable_scene = copy.deepcopy(forward_scene) with rtm.SessionCache() as cache: cache.load(mutable_scene) @@ -204,7 +210,7 @@ def test_reference_data_and_rtm_tables() -> None: assert spectrum.scene is not None assert spectrum.scene.geometry.solar_zenith_deg == scene.geometry.solar_zenith_deg - budget = rtm.atmospheric_budget(scene, np.array([760.76], dtype=np.float64)) + budget = rtm.atmospheric_budget(forward_scene, np.array([760.76], dtype=np.float64)) assert budget.row_count > 0 assert len(budget.column("wavelength_nm")) == budget.row_count first_table = budget.table @@ -215,7 +221,7 @@ def test_reference_data_and_rtm_tables() -> None: assert len(rows) == budget.row_count assert "support_row_kind_label" in rows[0] - spectrum = rtm.spectrum(scene) + spectrum = rtm.spectrum(forward_scene) output = Path(tmpdir) / "reflectance" chart = spectrum.plot.reflectance(save=output) assert chart is not None diff --git a/tests/python/test_session_cache.py b/tests/python/test_session_cache.py index 27233541c..a4a8b2188 100644 --- a/tests/python/test_session_cache.py +++ b/tests/python/test_session_cache.py @@ -31,13 +31,21 @@ def close(self) -> None: try: cache.warm_optimal_estimation(("aerosol_optical_depth",)) except RuntimeError as error: - assert "no loaded wavelength-band case" in str(error) + assert "no loaded wavelength-band scene" in str(error) else: raise AssertionError("unloaded OE cache warm was accepted") cache.load(scene) - cache.warm_optimal_estimation(("aerosol_optical_depth",)) - cache.warm_optimal_estimation(("aerosol_optical_depth",)) + try: + cache.warm_optimal_estimation(("aerosol_optical_depth",)) + except ValueError as error: + assert "aerosol_layer_mid_pressure_hpa" in str(error) + else: + raise AssertionError("one-state OE cache warm was accepted") + + cache.warm_optimal_estimation( + ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa") + ) cache.warm_optimal_estimation( ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa") ) @@ -50,7 +58,6 @@ def close(self) -> None: assert calls == [ ("load", (scene, True)), - ("warm_oe", ("aerosol_optical_depth",)), ("warm_oe", ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa")), ("load", (scene, True)), ("warm_oe", ("aerosol_optical_depth", "aerosol_layer_mid_pressure_hpa")), diff --git a/tests/unit/api/c_abi_retained_test.zig b/tests/unit/api/c_abi_retained_test.zig index 16b33a6be..d10817194 100644 --- a/tests/unit/api/c_abi_retained_test.zig +++ b/tests/unit/api/c_abi_retained_test.zig @@ -4,47 +4,29 @@ const c_api = @import("c_api"); const o2a_json = @import("o2a_json.zig"); test "optimal-estimation C ABI result rows keep ctypes layout" { - try std.testing.expectEqual(@as(usize, 80), @sizeOf(c_api.ZdsOptimalEstimationStateSpec)); + try std.testing.expectEqual(@as(usize, 48), @sizeOf(c_api.ZdsOptimalEstimationScalarSpec)); + try std.testing.expectEqual(@as(usize, 88), @sizeOf(c_api.ZdsOptimalEstimationPressureSpec)); try std.testing.expectEqual(@as(usize, 24), @sizeOf(c_api.ZdsOptimalEstimationControls)); - try std.testing.expectEqual(@as(usize, 72), @sizeOf(c_api.ZdsOptimalEstimationRequest)); + try std.testing.expectEqual(@as(usize, 192), @sizeOf(c_api.ZdsOptimalEstimationRequest)); try std.testing.expectEqual(@as(usize, 120), @sizeOf(c_api.ZdsOptimalEstimationResult)); - try std.testing.expectEqual(@as(usize, 104), @sizeOf(c_api.ZdsOptimalEstimationBatchRequest)); + try std.testing.expectEqual(@as(usize, 224), @sizeOf(c_api.ZdsOptimalEstimationBatchRequest)); try std.testing.expectEqual(@as(usize, 72), @sizeOf(c_api.ZdsOptimalEstimationBatchResult)); try std.testing.expectEqual(@as(usize, 104), @sizeOf(c_api.ZdsOptimalEstimationFastmodeBatchResult)); } -test "warm optimal-estimation cache accepts the two retained Jacobian states" { +test "warm optimal-estimation cache uses the fixed two-state Jacobian route" { const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(ctx); - try prepareDefault(ctx); + try prepareNarrow(ctx); - const state_ids = [_]u8{ 0, 1 }; try std.testing.expectEqual( @intFromEnum(c_api.ZdsStatus.ok), - c_api.zds_warm_o2a_optimal_estimation(ctx, state_ids[0..].ptr, state_ids.len), + c_api.zds_warm_o2a_optimal_estimation(ctx), ); try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } -test "warm optimal-estimation cache rejects removed surface-albedo state lane" { - const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; - defer c_api.zds_context_destroy(ctx); - - try prepareDefault(ctx); - - const removed_surface_albedo_id = [_]u8{2}; - try std.testing.expectEqual( - @intFromEnum(c_api.ZdsStatus.failure), - c_api.zds_warm_o2a_optimal_estimation( - ctx, - removed_surface_albedo_id[0..].ptr, - removed_surface_albedo_id.len, - ), - ); - try std.testing.expectEqualStrings("UnsupportedJacobianState", std.mem.span(c_api.zds_last_error(ctx))); -} - test "optimal-estimation free hooks clear borrowed result rows" { const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(ctx); @@ -94,14 +76,17 @@ test "single-run optimal-estimation returns result handle for full-grid measurem defer std.testing.allocator.free(variance); @memset(variance, 1.0e-6); - var states = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - states[0].initial = 0.3; - states[0].prior = 0.3; + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; + var aod = aerosolOpticalDepthSpec(); + aod.initial = 0.3; + aod.prior = 0.3; var request = singleRequest( spectrum.wavelength_nm[0..spectrum.len], spectrum.reflectance[0..spectrum.len], variance, - &states, + aod, + aerosolPressureSpec(&altitude_km, &pressure_hpa), ); request.controls.max_iterations = 1; var result: c_api.ZdsOptimalEstimationResult = .{}; @@ -113,20 +98,30 @@ test "single-run optimal-estimation returns result handle for full-grid measurem defer c_api.zds_optimal_estimation_result_free(ctx, &result); try std.testing.expect(result.result_handle != null); - try std.testing.expectEqual(@as(usize, 1), result.state_count); + try std.testing.expectEqual(@as(usize, 2), result.state_count); try std.testing.expectEqual(@as(usize, 1), result.iteration_count); - try std.testing.expectEqual(@as(u8, 1), result.converged); + try std.testing.expectEqual(@as(u8, 0), result.converged); try std.testing.expectEqual(@as(u8, 0), result.state_ids.?[0]); - try std.testing.expectApproxEqAbs(0.3, result.state.?[0], 1.0e-12); + try std.testing.expectEqual(@as(u8, 1), result.state_ids.?[1]); + try std.testing.expectApproxEqRel(0.3218022557145076, result.state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(841.0563427719196, result.state.?[1], 1.0e-12); try std.testing.expectApproxEqAbs(0.3, result.initial_state.?[0], 0.0); - try std.testing.expectApproxEqRel(1.3229305526174265e-7, result.posterior_covariance.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.9999867706944716, result.averaging_kernel.?[0], 1.0e-12); - try std.testing.expectApproxEqAbs(0.3, result.history_state.?[0], 0.0); - try std.testing.expectApproxEqAbs(0.0, result.history_chi2.?[0], 0.0); - try std.testing.expectApproxEqAbs(0.0, result.history_chi2_reflectance.?[0], 0.0); - try std.testing.expectApproxEqAbs(0.0, result.history_chi2_state_vector.?[0], 0.0); - try std.testing.expectApproxEqAbs(0.0, result.history_state_vector_convergence.?[0], 0.0); - try std.testing.expectEqual(@as(u8, 1), result.history_snr_normal.?[0]); + try std.testing.expectApproxEqAbs(850.0, result.initial_state.?[1], 0.0); + try std.testing.expectApproxEqRel(5.0327224966474805e-5, result.posterior_covariance.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(0.005484945752473451, result.posterior_covariance.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(0.005484945752473451, result.posterior_covariance.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(98.14024630925327, result.posterior_covariance.?[3], 1.0e-12); + try std.testing.expectApproxEqRel(313.7498523160241, result.averaging_kernel.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(-0.017296055445342524, result.averaging_kernel.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(-172.96055445340608, result.averaging_kernel.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(5.864488802889277, result.averaging_kernel.?[3], 1.0e-12); + try std.testing.expectApproxEqRel(0.3218022557145076, result.history_state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(841.0563427719196, result.history_state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(17581.186238836166, result.history_chi2.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(17580.338814954608, result.history_chi2_reflectance.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(0.8474238815580228, result.history_chi2_state_vector.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(5.379307392874958, result.history_state_vector_convergence.?[0], 1.0e-12); + try std.testing.expectEqual(@as(u8, 0), result.history_snr_normal.?[0]); try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } @@ -147,12 +142,14 @@ test "correction optimal-estimation returns one-step result handle" { defer std.testing.allocator.free(variance); @memset(variance, 1.0e-6); - var states = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; var request = singleRequest( spectrum.wavelength_nm[0..spectrum.len], spectrum.reflectance[0..spectrum.len], variance, - &states, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), ); request.controls.max_iterations = 5; var result: c_api.ZdsOptimalEstimationResult = .{}; @@ -164,21 +161,30 @@ test "correction optimal-estimation returns one-step result handle" { defer c_api.zds_optimal_estimation_result_free(ctx, &result); try std.testing.expect(result.result_handle != null); - try std.testing.expectEqual(@as(usize, 1), result.state_count); + try std.testing.expectEqual(@as(usize, 2), result.state_count); try std.testing.expectEqual(@as(usize, 1), result.iteration_count); try std.testing.expectApproxEqAbs(0.1, result.initial_state.?[0], 0.0); - try std.testing.expectApproxEqRel(0.3358862191767914, result.state.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(2.3333920905514787e-7, result.posterior_covariance.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.9999766660790946, result.averaging_kernel.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.3358862191767914, result.history_state.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(242005.5066494241, result.history_chi2.?[0], 1.0e-12); + try std.testing.expectApproxEqAbs(850.0, result.initial_state.?[1], 0.0); + try std.testing.expectApproxEqRel(0.36656240097651505, result.state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(847.531818925324, result.state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(3.212187341505087e-7, result.posterior_covariance.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(0.0015179266918172764, result.posterior_covariance.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(0.0015179266918172764, result.posterior_covariance.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(72.29518736871741, result.posterior_covariance.?[3], 1.0e-12); + try std.testing.expectApproxEqRel(0.9999678781265839, result.averaging_kernel.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(-1.5179266918172786e-5, result.averaging_kernel.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(-0.15179266918767098, result.averaging_kernel.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(0.2770481263128252, result.averaging_kernel.?[3], 1.0e-12); + try std.testing.expectApproxEqRel(0.36656240097651505, result.history_state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(847.531818925324, result.history_state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(265603.46505356533, result.history_chi2.?[0], 1.0e-12); try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } test "batch optimal-estimation returns run-major result handle" { const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(ctx); - try prepareDefault(ctx); + try prepareNarrow(ctx); var spectrum: c_api.ZdsSpectrum = .{}; try std.testing.expectEqual( @@ -186,20 +192,22 @@ test "batch optimal-estimation returns run-major result handle" { c_api.zds_run_spectrum(ctx, &spectrum), ); defer c_api.zds_spectrum_free(ctx, &spectrum); - try expectDefaultProductSpectrum(spectrum); + try std.testing.expectEqual(@as(usize, 16), spectrum.len); const variance = try std.testing.allocator.alloc(f64, spectrum.len); defer std.testing.allocator.free(variance); @memset(variance, 1.0e-6); - var state_template = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - const initial = [_]f64{ 0.08, 0.09 }; - const prior = [_]f64{ 0.10, 0.10 }; + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; + const initial = [_]f64{ 0.08, 850.0, 0.09, 850.0 }; + const prior = [_]f64{ 0.10, 850.0, 0.10, 850.0 }; var request = batchRequest( spectrum.wavelength_nm[0..spectrum.len], spectrum.reflectance[0..spectrum.len], variance, - &state_template, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), &initial, &prior, ); @@ -214,16 +222,23 @@ test "batch optimal-estimation returns run-major result handle" { try std.testing.expect(result.result_handle != null); try std.testing.expectEqual(@as(usize, 2), result.run_count); - try std.testing.expectEqual(@as(usize, 1), result.state_count); + try std.testing.expectEqual(@as(usize, 2), result.state_count); try std.testing.expectEqual(@as(usize, 1), result.history_capacity); try std.testing.expectEqual(@as(usize, 1), result.iteration_count.?[0]); try std.testing.expectEqual(@as(usize, 1), result.iteration_count.?[1]); try std.testing.expectEqual(@as(u8, 1), result.status.?[0]); try std.testing.expectEqual(@as(u8, 1), result.status.?[1]); - try std.testing.expectApproxEqRel(0.3488838954119376, result.state.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.3419559672154927, result.state.?[1], 1.0e-12); + // Narrow-band goldens (regenerated for the 759.5-761nm window). Canonical full-band OE + // physics is pinned by the single-run and correction tests above; this test pins the + // run-major batch layout and that history mirrors the final state. + try std.testing.expectApproxEqRel(0.1651899607587543, result.state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(849.8922457984752, result.state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(0.16665664954904796, result.state.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(849.8774905141596, result.state.?[3], 1.0e-12); try std.testing.expectApproxEqRel(result.state.?[0], result.history_state.?[0], 0.0); try std.testing.expectApproxEqRel(result.state.?[1], result.history_state.?[1], 0.0); + try std.testing.expectApproxEqRel(result.state.?[2], result.history_state.?[2], 0.0); + try std.testing.expectApproxEqRel(result.state.?[3], result.history_state.?[3], 0.0); try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } @@ -232,8 +247,8 @@ test "fastmode optimal-estimation batch returns per-stage metadata" { defer c_api.zds_context_destroy(fast_ctx); const correction_ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(correction_ctx); - try prepareDefault(fast_ctx); - try prepareDefault(correction_ctx); + try prepareNarrow(fast_ctx); + try prepareNarrow(correction_ctx); var spectrum: c_api.ZdsSpectrum = .{}; try std.testing.expectEqual( @@ -241,21 +256,22 @@ test "fastmode optimal-estimation batch returns per-stage metadata" { c_api.zds_run_spectrum(fast_ctx, &spectrum), ); defer c_api.zds_spectrum_free(fast_ctx, &spectrum); - try expectDefaultProductSpectrum(spectrum); + try std.testing.expectEqual(@as(usize, 16), spectrum.len); const variance = try std.testing.allocator.alloc(f64, spectrum.len); defer std.testing.allocator.free(variance); @memset(variance, 1.0e-6); - var fast_state_template = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - var correction_state_template = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - const initial = [_]f64{ 0.08, 0.09 }; - const prior = [_]f64{ 0.10, 0.10 }; + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; + const initial = [_]f64{ 0.08, 850.0, 0.09, 850.0 }; + const prior = [_]f64{ 0.10, 850.0, 0.10, 850.0 }; var fast_request = batchRequest( spectrum.wavelength_nm[0..spectrum.len], spectrum.reflectance[0..spectrum.len], variance, - &fast_state_template, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), &initial, &prior, ); @@ -265,7 +281,8 @@ test "fastmode optimal-estimation batch returns per-stage metadata" { spectrum.wavelength_nm[0..spectrum.len], spectrum.reflectance[0..spectrum.len], variance, - &correction_state_template, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), &initial, &prior, ); @@ -286,7 +303,7 @@ test "fastmode optimal-estimation batch returns per-stage metadata" { try std.testing.expect(result.result_handle != null); try std.testing.expectEqual(@as(usize, 2), result.run_count); - try std.testing.expectEqual(@as(usize, 1), result.state_count); + try std.testing.expectEqual(@as(usize, 2), result.state_count); try std.testing.expectEqual(@as(usize, 2), result.history_capacity); try std.testing.expectEqual(@as(usize, 2), result.iteration_count.?[0]); try std.testing.expectEqual(@as(usize, 2), result.iteration_count.?[1]); @@ -294,46 +311,40 @@ test "fastmode optimal-estimation batch returns per-stage metadata" { try std.testing.expectEqual(@as(usize, 1), result.full_correction_iteration_count.?[0]); try std.testing.expectEqual(@as(u8, 1), result.status.?[0]); try std.testing.expectEqual(@as(u8, 1), result.status.?[1]); - try std.testing.expectApproxEqRel(0.3007235613338587, result.state.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.30054519061204255, result.state.?[1], 1.0e-12); - try std.testing.expectApproxEqRel(0.3488838954119376, result.history_state.?[0], 1.0e-12); - try std.testing.expectApproxEqRel(0.3007235613338587, result.history_state.?[1], 1.0e-12); - try std.testing.expectApproxEqRel(0.3419559672154927, result.history_state.?[2], 1.0e-12); - try std.testing.expectApproxEqRel(0.30054519061204255, result.history_state.?[3], 1.0e-12); + // Narrow-band goldens (regenerated for the 759.5-761nm window). This test pins the + // two-stage fastmode history layout; canonical full-band physics lives in the single-run + // and correction tests. Final state and the stage-1 (batch) history rows are absolute; + // the full-correction history rows must mirror the final state, asserted relationally. + try std.testing.expectApproxEqRel(0.21803451863669004, result.state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(849.603112982948, result.state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(0.21821872416099974, result.state.?[2], 1.0e-12); + try std.testing.expectApproxEqRel(849.5992085372817, result.state.?[3], 1.0e-12); + try std.testing.expectApproxEqRel(0.1651899607587543, result.history_state.?[0], 1.0e-12); + try std.testing.expectApproxEqRel(849.8922457984752, result.history_state.?[1], 1.0e-12); + try std.testing.expectApproxEqRel(0.16665664954904796, result.history_state.?[4], 1.0e-12); + try std.testing.expectApproxEqRel(849.8774905141596, result.history_state.?[5], 1.0e-12); + try std.testing.expectApproxEqRel(result.state.?[0], result.history_state.?[2], 0.0); + try std.testing.expectApproxEqRel(result.state.?[1], result.history_state.?[3], 0.0); + try std.testing.expectApproxEqRel(result.state.?[2], result.history_state.?[6], 0.0); + try std.testing.expectApproxEqRel(result.state.?[3], result.history_state.?[7], 0.0); try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(fast_ctx))); } -test "optimal-estimation request rejects removed surface-albedo state lane" { - const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; - defer c_api.zds_context_destroy(ctx); - try prepareDefault(ctx); - - const wavelength_nm = [_]f64{ 758.0, 760.0 }; - const reflectance = [_]f64{ 0.12, 0.13 }; - const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var states = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - states[0].state_id = 2; - var request = singleRequest(&wavelength_nm, &reflectance, &variance, &states); - var result: c_api.ZdsOptimalEstimationResult = .{}; - - try std.testing.expectEqual( - @intFromEnum(c_api.ZdsStatus.failure), - c_api.zds_run_o2a_optimal_estimation(ctx, &request, &result), - ); - try std.testing.expectEqual(c_api.ZdsOptimalEstimationResult{}, result); - try std.testing.expectEqualStrings("UnsupportedState", std.mem.span(c_api.zds_last_error(ctx))); -} - test "optimal-estimation pressure state requires profile rows" { const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(ctx); - try prepareDefault(ctx); + try prepareNarrow(ctx); const wavelength_nm = [_]f64{ 758.0, 760.0 }; const reflectance = [_]f64{ 0.12, 0.13 }; const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var states = [_]c_api.ZdsOptimalEstimationStateSpec{pressureStateWithoutProfile()}; - var request = singleRequest(&wavelength_nm, &reflectance, &variance, &states); + var request = singleRequest( + &wavelength_nm, + &reflectance, + &variance, + aerosolOpticalDepthSpec(), + pressureStateWithoutProfile(), + ); var result: c_api.ZdsOptimalEstimationResult = .{}; try std.testing.expectEqual( @@ -345,10 +356,18 @@ test "optimal-estimation pressure state requires profile rows" { } fn prepareDefault(ctx: *c_api.Context) !void { - const allocator = std.testing.allocator; - const rendered = try o2a_json.nativeJson(allocator); - defer allocator.free(rendered); + try prepareJson(ctx, try o2a_json.nativeJson(std.testing.allocator)); +} + +// prepareNarrow feeds the default scene over a narrow representative band. Tests that exercise +// control flow, batching/fastmode structure, or error paths use it so the one-time prepare is +// ~4x cheaper; tests pinning canonical full-band physics keep prepareDefault. +fn prepareNarrow(ctx: *c_api.Context) !void { + try prepareJson(ctx, try o2a_json.narrowJson(std.testing.allocator)); +} +fn prepareJson(ctx: *c_api.Context, rendered: []u8) !void { + defer std.testing.allocator.free(rendered); try std.testing.expectEqual( @intFromEnum(c_api.ZdsStatus.ok), c_api.zds_prepare_o2a_json(ctx, rendered.ptr, rendered.len), @@ -356,9 +375,8 @@ fn prepareDefault(ctx: *c_api.Context) !void { try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } -fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationStateSpec { +fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationScalarSpec { return .{ - .state_id = 0, .has_lower = 1, .has_upper = 1, .initial = 0.1, @@ -369,21 +387,44 @@ fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationStateSpec { }; } -fn pressureStateWithoutProfile() c_api.ZdsOptimalEstimationStateSpec { +fn pressureStateWithoutProfile() c_api.ZdsOptimalEstimationPressureSpec { return .{ - .state_id = 1, - .has_lower = 1, - .has_upper = 1, + .scalar = .{ + .has_lower = 1, + .has_upper = 1, + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower = 600.0, + .upper = 1000.0, + }, .interval_index_1based = 1, - .initial = 850.0, - .prior = 850.0, - .variance = 100.0, - .lower = 600.0, - .upper = 1000.0, .thickness_hpa = 10.0, }; } +fn aerosolPressureSpec( + altitude_km: []const f64, + pressure_hpa: []const f64, +) c_api.ZdsOptimalEstimationPressureSpec { + return .{ + .scalar = .{ + .has_lower = 1, + .has_upper = 1, + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower = 600.0, + .upper = 1000.0, + }, + .interval_index_1based = 2, + .thickness_hpa = 10.0, + .pressure_profile_count = altitude_km.len, + .pressure_profile_altitude_km = altitude_km.ptr, + .pressure_profile_pressure_hpa = pressure_hpa.ptr, + }; +} + fn expectDefaultProductSpectrum(spectrum: c_api.ZdsSpectrum) !void { try std.testing.expectEqual(@as(usize, 701), spectrum.len); try expectProductRow(spectrum, 0, 755.0, 17800959849043.055, 480585461471192.25, 0.23273015591193758); @@ -409,15 +450,16 @@ fn singleRequest( wavelength_nm: []const f64, reflectance: []const f64, variance: []const f64, - states: []const c_api.ZdsOptimalEstimationStateSpec, + aod: c_api.ZdsOptimalEstimationScalarSpec, + pressure: c_api.ZdsOptimalEstimationPressureSpec, ) c_api.ZdsOptimalEstimationRequest { return .{ .sample_count = wavelength_nm.len, .wavelength_nm = wavelength_nm.ptr, .reflectance = reflectance.ptr, .variance = variance.ptr, - .state_count = states.len, - .states = states.ptr, + .aerosol_optical_depth = aod, + .aerosol_layer_pressure = pressure, }; } @@ -425,7 +467,8 @@ fn batchRequest( wavelength_nm: []const f64, reflectance: []const f64, variance: []const f64, - state_template: []const c_api.ZdsOptimalEstimationStateSpec, + aod: c_api.ZdsOptimalEstimationScalarSpec, + pressure: c_api.ZdsOptimalEstimationPressureSpec, initial: []const f64, prior: []const f64, ) c_api.ZdsOptimalEstimationBatchRequest { @@ -434,9 +477,9 @@ fn batchRequest( .wavelength_nm = wavelength_nm.ptr, .reflectance = reflectance.ptr, .variance = variance.ptr, - .state_count = state_template.len, - .state_template = state_template.ptr, - .run_count = initial.len / state_template.len, + .aerosol_optical_depth = aod, + .aerosol_layer_pressure = pressure, + .run_count = initial.len / 2, .initial = initial.ptr, .prior = prior.ptr, .batch_worker_count = 1, diff --git a/tests/unit/api/c_abi_test.zig b/tests/unit/api/c_abi_test.zig index bc3a77ea1..ba97f6fce 100644 --- a/tests/unit/api/c_abi_test.zig +++ b/tests/unit/api/c_abi_test.zig @@ -5,11 +5,12 @@ const o2a_json = @import("o2a_json.zig"); test "optimal-estimation C ABI result rows keep ctypes layout" { try std.testing.expect(c_api.links_libc); - try std.testing.expectEqual(@as(usize, 80), @sizeOf(c_api.ZdsOptimalEstimationStateSpec)); + try std.testing.expectEqual(@as(usize, 48), @sizeOf(c_api.ZdsOptimalEstimationScalarSpec)); + try std.testing.expectEqual(@as(usize, 88), @sizeOf(c_api.ZdsOptimalEstimationPressureSpec)); try std.testing.expectEqual(@as(usize, 24), @sizeOf(c_api.ZdsOptimalEstimationControls)); - try std.testing.expectEqual(@as(usize, 72), @sizeOf(c_api.ZdsOptimalEstimationRequest)); + try std.testing.expectEqual(@as(usize, 192), @sizeOf(c_api.ZdsOptimalEstimationRequest)); try std.testing.expectEqual(@as(usize, 120), @sizeOf(c_api.ZdsOptimalEstimationResult)); - try std.testing.expectEqual(@as(usize, 104), @sizeOf(c_api.ZdsOptimalEstimationBatchRequest)); + try std.testing.expectEqual(@as(usize, 224), @sizeOf(c_api.ZdsOptimalEstimationBatchRequest)); try std.testing.expectEqual(@as(usize, 72), @sizeOf(c_api.ZdsOptimalEstimationBatchResult)); try std.testing.expectEqual(@as(usize, 104), @sizeOf(c_api.ZdsOptimalEstimationFastmodeBatchResult)); } @@ -165,27 +166,6 @@ test "optimal-estimation C ABI rejects null request and output pointers" { ); } -test "optimal-estimation request rejects removed surface-albedo state lane" { - const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; - defer c_api.zds_context_destroy(ctx); - try prepareTinyJson(ctx); - - const wavelength_nm = [_]f64{ 758.0, 760.0 }; - const reflectance = [_]f64{ 0.12, 0.13 }; - const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var states = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - states[0].state_id = 2; - var request = singleRequest(&wavelength_nm, &reflectance, &variance, &states); - var result: c_api.ZdsOptimalEstimationResult = .{}; - - try std.testing.expectEqual( - @intFromEnum(c_api.ZdsStatus.failure), - c_api.zds_run_o2a_optimal_estimation(ctx, &request, &result), - ); - try std.testing.expectEqual(c_api.ZdsOptimalEstimationResult{}, result); - try std.testing.expectEqualStrings("UnsupportedState", std.mem.span(c_api.zds_last_error(ctx))); -} - test "optimal-estimation pressure state requires profile rows" { const ctx = c_api.zds_context_create() orelse return error.OutOfMemory; defer c_api.zds_context_destroy(ctx); @@ -194,8 +174,13 @@ test "optimal-estimation pressure state requires profile rows" { const wavelength_nm = [_]f64{ 758.0, 760.0 }; const reflectance = [_]f64{ 0.12, 0.13 }; const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var states = [_]c_api.ZdsOptimalEstimationStateSpec{pressureStateWithoutProfile()}; - var request = singleRequest(&wavelength_nm, &reflectance, &variance, &states); + var request = singleRequest( + &wavelength_nm, + &reflectance, + &variance, + aerosolOpticalDepthSpec(), + pressureStateWithoutProfile(), + ); var result: c_api.ZdsOptimalEstimationResult = .{}; try std.testing.expectEqual( @@ -214,10 +199,19 @@ test "optimal-estimation batch rejects unsupported worker counts" { const wavelength_nm = [_]f64{ 758.0, 760.0 }; const reflectance = [_]f64{ 0.12, 0.13 }; const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var state_template = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - const initial = [_]f64{0.1}; - const prior = [_]f64{0.1}; - var request = batchRequest(&wavelength_nm, &reflectance, &variance, &state_template, &initial, &prior); + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; + const initial = [_]f64{ 0.1, 850.0 }; + const prior = [_]f64{ 0.1, 850.0 }; + var request = batchRequest( + &wavelength_nm, + &reflectance, + &variance, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), + &initial, + &prior, + ); request.batch_worker_count = 2; var result: c_api.ZdsOptimalEstimationBatchResult = .{}; @@ -237,78 +231,31 @@ test "optimal-estimation batch rejects state-count overflow before slicing run r defer c_api.zds_context_destroy(ctx); try prepareTinyJson(ctx); - const wavelength_nm = [_]f64{ 758.0, 760.0 }; - const reflectance = [_]f64{ 0.12, 0.13 }; - const variance = [_]f64{ 1.0e-4, 1.0e-4 }; - var state_template = [_]c_api.ZdsOptimalEstimationStateSpec{aerosolOpticalDepthSpec()}; - const initial = [_]f64{0.1}; - const prior = [_]f64{0.1}; - var request = batchRequest(&wavelength_nm, &reflectance, &variance, &state_template, &initial, &prior); - request.run_count = std.math.maxInt(usize); - request.state_count = 2; - var result: c_api.ZdsOptimalEstimationBatchResult = .{}; - - try expectFailure( - ctx, - c_api.zds_run_o2a_optimal_estimation_batch(ctx, &request, &result), - "optimal-estimation batch is too large", - ); - try std.testing.expectEqual(c_api.ZdsOptimalEstimationBatchResult{}, result); -} - -test "fastmode optimal-estimation batch rejects mismatched state ordering" { - const fast_ctx = c_api.zds_context_create() orelse return error.OutOfMemory; - defer c_api.zds_context_destroy(fast_ctx); - const correction_ctx = c_api.zds_context_create() orelse return error.OutOfMemory; - defer c_api.zds_context_destroy(correction_ctx); - try prepareTinyJson(fast_ctx); - try prepareTinyJson(correction_ctx); - const wavelength_nm = [_]f64{ 758.0, 760.0 }; const reflectance = [_]f64{ 0.12, 0.13 }; const variance = [_]f64{ 1.0e-4, 1.0e-4 }; const altitude_km = [_]f64{ 0.0, 1.0 }; const pressure_hpa = [_]f64{ 900.0, 800.0 }; - var fast_state_template = [_]c_api.ZdsOptimalEstimationStateSpec{ - aerosolOpticalDepthSpec(), - aerosolPressureSpec(&altitude_km, &pressure_hpa), - }; - var correction_state_template = [_]c_api.ZdsOptimalEstimationStateSpec{ - aerosolPressureSpec(&altitude_km, &pressure_hpa), - aerosolOpticalDepthSpec(), - }; const initial = [_]f64{ 0.1, 850.0 }; const prior = [_]f64{ 0.1, 850.0 }; - var fast_request = batchRequest( + var request = batchRequest( &wavelength_nm, &reflectance, &variance, - &fast_state_template, - &initial, - &prior, - ); - var correction_request = batchRequest( - &wavelength_nm, - &reflectance, - &variance, - &correction_state_template, + aerosolOpticalDepthSpec(), + aerosolPressureSpec(&altitude_km, &pressure_hpa), &initial, &prior, ); - var result: c_api.ZdsOptimalEstimationFastmodeBatchResult = .{}; + request.run_count = std.math.maxInt(usize); + var result: c_api.ZdsOptimalEstimationBatchResult = .{}; - try std.testing.expectEqual( - @intFromEnum(c_api.ZdsStatus.failure), - c_api.zds_run_o2a_fastmode_optimal_estimation_batch( - fast_ctx, - correction_ctx, - &fast_request, - &correction_request, - &result, - ), + try expectFailure( + ctx, + c_api.zds_run_o2a_optimal_estimation_batch(ctx, &request, &result), + "optimal-estimation batch is too large", ); - try std.testing.expectEqual(c_api.ZdsOptimalEstimationFastmodeBatchResult{}, result); - try std.testing.expectEqualStrings("InvalidStateSpec", std.mem.span(c_api.zds_last_error(fast_ctx))); + try std.testing.expectEqual(c_api.ZdsOptimalEstimationBatchResult{}, result); } fn expectFailure(ctx: *c_api.Context, status: c_int, expected_error: []const u8) !void { @@ -365,9 +312,8 @@ fn prepareTinyJson(ctx: *c_api.Context) !void { try std.testing.expectEqualStrings("", std.mem.span(c_api.zds_last_error(ctx))); } -fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationStateSpec { +fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationScalarSpec { return .{ - .state_id = 0, .has_lower = 1, .has_upper = 1, .initial = 0.1, @@ -378,17 +324,18 @@ fn aerosolOpticalDepthSpec() c_api.ZdsOptimalEstimationStateSpec { }; } -fn pressureStateWithoutProfile() c_api.ZdsOptimalEstimationStateSpec { +fn pressureStateWithoutProfile() c_api.ZdsOptimalEstimationPressureSpec { return .{ - .state_id = 1, - .has_lower = 1, - .has_upper = 1, + .scalar = .{ + .has_lower = 1, + .has_upper = 1, + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower = 600.0, + .upper = 1000.0, + }, .interval_index_1based = 1, - .initial = 850.0, - .prior = 850.0, - .variance = 100.0, - .lower = 600.0, - .upper = 1000.0, .thickness_hpa = 10.0, }; } @@ -396,17 +343,18 @@ fn pressureStateWithoutProfile() c_api.ZdsOptimalEstimationStateSpec { fn aerosolPressureSpec( altitude_km: []const f64, pressure_hpa: []const f64, -) c_api.ZdsOptimalEstimationStateSpec { +) c_api.ZdsOptimalEstimationPressureSpec { return .{ - .state_id = 1, - .has_lower = 1, - .has_upper = 1, + .scalar = .{ + .has_lower = 1, + .has_upper = 1, + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower = 600.0, + .upper = 1000.0, + }, .interval_index_1based = 2, - .initial = 850.0, - .prior = 850.0, - .variance = 100.0, - .lower = 600.0, - .upper = 1000.0, .thickness_hpa = 10.0, .pressure_profile_count = altitude_km.len, .pressure_profile_altitude_km = altitude_km.ptr, @@ -418,15 +366,16 @@ fn singleRequest( wavelength_nm: []const f64, reflectance: []const f64, variance: []const f64, - states: []const c_api.ZdsOptimalEstimationStateSpec, + aod: c_api.ZdsOptimalEstimationScalarSpec, + pressure: c_api.ZdsOptimalEstimationPressureSpec, ) c_api.ZdsOptimalEstimationRequest { return .{ .sample_count = wavelength_nm.len, .wavelength_nm = wavelength_nm.ptr, .reflectance = reflectance.ptr, .variance = variance.ptr, - .state_count = states.len, - .states = states.ptr, + .aerosol_optical_depth = aod, + .aerosol_layer_pressure = pressure, }; } @@ -434,7 +383,8 @@ fn batchRequest( wavelength_nm: []const f64, reflectance: []const f64, variance: []const f64, - state_template: []const c_api.ZdsOptimalEstimationStateSpec, + aod: c_api.ZdsOptimalEstimationScalarSpec, + pressure: c_api.ZdsOptimalEstimationPressureSpec, initial: []const f64, prior: []const f64, ) c_api.ZdsOptimalEstimationBatchRequest { @@ -443,9 +393,9 @@ fn batchRequest( .wavelength_nm = wavelength_nm.ptr, .reflectance = reflectance.ptr, .variance = variance.ptr, - .state_count = state_template.len, - .state_template = state_template.ptr, - .run_count = initial.len / state_template.len, + .aerosol_optical_depth = aod, + .aerosol_layer_pressure = pressure, + .run_count = initial.len / 2, .initial = initial.ptr, .prior = prior.ptr, .batch_worker_count = 1, diff --git a/tests/unit/api/o2a_json.zig b/tests/unit/api/o2a_json.zig index 43de44a75..d66be1435 100644 --- a/tests/unit/api/o2a_json.zig +++ b/tests/unit/api/o2a_json.zig @@ -80,7 +80,7 @@ const native_json = \\ ], \\ "layer_count": 3, \\ "metadata": { - \\ "description": "DISAMAR O2 A reference scene defined in Python.", + \\ "description": "DISAMAR O2A reference scene defined in Python.", \\ "id": "disamar_reference_o2a", \\ "storage": "disamar-reference-o2a" \\ }, @@ -167,6 +167,33 @@ const native_json = \\} ; +// The full-band spectral grid block, verbatim from native_json (unique substring). narrowJson +// swaps it for a ~1.5nm window on the O2 A-band R-branch so control-flow / structural tests +// prepare ~4x faster. native_json itself is left byte-identical so the canonical full-band +// golden tests keep their exact inputs. +const full_grid_block = + \\ "end_nm": 776.0, + \\ "sample_count": 701, + \\ "start_nm": 755.0 +; +const narrow_grid_block = + \\ "end_nm": 761.0, + \\ "sample_count": 16, + \\ "start_nm": 759.5 +; + pub fn nativeJson(allocator: std.mem.Allocator) ![]u8 { return allocator.dupe(u8, native_json); } + +pub fn narrowJson(allocator: std.mem.Allocator) ![]u8 { + // narrowJson ------------------------------------------------------------------------------------------- | + // Default scene over a narrow representative band; use for tests that check control flow, | + // batching/fastmode structure, or error paths rather than canonical full-band physics values. | + // ------------------------------------------------------------------------------------------------------ | + const size = std.mem.replacementSize(u8, native_json, full_grid_block, narrow_grid_block); + const rendered = try allocator.alloc(u8, size); + const count = std.mem.replace(u8, native_json, full_grid_block, narrow_grid_block, rendered); + std.debug.assert(count == 1); + return rendered; +} diff --git a/tests/unit/cache/profile_line_memory_test.zig b/tests/unit/cache/profile_line_memory_test.zig index 4c7a23c39..b35944a3a 100644 --- a/tests/unit/cache/profile_line_memory_test.zig +++ b/tests/unit/cache/profile_line_memory_test.zig @@ -2,6 +2,7 @@ const std = @import("std"); const builtin = @import("builtin"); const internal = @import("internal"); const o2a_scene = @import("../support/o2a_scene.zig"); +const CountingAllocator = @import("../support/counting_allocator.zig").CountingAllocator; const profile_line_test_sample_count = 7; @@ -386,6 +387,53 @@ test "ProfileLineValues parallel wavelength build matches serial rows" { ); } +test "ProfileLineValues retain only output rows on the caller allocator" { + if (builtin.mode == .Debug) return error.SkipZigTest; + + const wavelengths_nm = [_]f64{ 758.0, 760.0 }; + var expected = try internal.cache.profile_line_memory.buildProfileLineValuesForWavelengthsWithCutoffGrid( + std.testing.allocator, + o2a_scene.reference(), + wavelengths_nm[0..], + wavelengths_nm[0..], + true, + false, + null, + 1, + &.{}, + ); + defer expected.deinit(std.testing.allocator); + + var counter = CountingAllocator.init(std.testing.allocator); + const counting_allocator = counter.allocator(); + var actual = try internal.cache.profile_line_memory.buildProfileLineValuesForWavelengthsWithCutoffGrid( + counting_allocator, + o2a_scene.reference(), + wavelengths_nm[0..], + wavelengths_nm[0..], + true, + false, + null, + 1, + &.{}, + ); + errdefer actual.deinit(counting_allocator); + + try std.testing.expectEqual(@as(usize, 2), counter.live_allocations); + try std.testing.expectEqual(expected.wavelength_count, actual.wavelength_count); + try std.testing.expectEqual(expected.profile_node_count, actual.profile_node_count); + try std.testing.expectEqual(expected.support_profile_node_count, actual.support_profile_node_count); + try std.testing.expectEqualSlices(u8, std.mem.sliceAsBytes(expected.values), std.mem.sliceAsBytes(actual.values)); + try std.testing.expectEqualSlices( + u8, + std.mem.sliceAsBytes(expected.support_profile_total_sigma_cm2_per_molecule), + std.mem.sliceAsBytes(actual.support_profile_total_sigma_cm2_per_molecule), + ); + + actual.deinit(counting_allocator); + try std.testing.expectEqual(@as(usize, 0), counter.live_allocations); +} + test "ProfileLineValues fill support-row sigma from atmospheric-budget evidence" { if (builtin.mode == .Debug) return error.SkipZigTest; @@ -467,7 +515,7 @@ const SupportLineSigmaEvidence = struct { // ------------------------------------------------------------------------------------------------------------| // Canonical expected values owned by this repository. -// public-python-baseline.json .diagnostics.atmospheric_budget.rows, support row 1 at the five O2 A probes. +// public-python-baseline.json .diagnostics.atmospheric_budget.rows, support row 1 at the five probes. const support_line_sigma_evidence = [_]SupportLineSigmaEvidence{ .{ .wavelength_nm = 758.0, @@ -527,8 +575,8 @@ const ProfileLineTotalProbeEvidence = struct { }; // ------------------------------------------------------------------------------------------------------------| -// Source: canonical O2 A profile-line evidence fixtures. -// The O2 A artifacts do not expose per-profile-node total line sidecar rows, so this table is derived by running +// Source: canonical profile-line evidence fixtures. +// The artifacts do not expose per-profile-node total line sidecar rows, so this table is derived by running // nodes at the five diagnostic wavelengths used by public-python-baseline.json. const profile_line_total_probe_evidence = [_]ProfileLineTotalProbeEvidence{ .{ @@ -651,8 +699,8 @@ fn findProfileLineProbeEvidence( return null; } -// Source: canonical O2 A profile-line evidence fixtures. -// The O2 A public artifacts contain diagnostic optical depths, not per-layer line-cache rows, so this table is +// Source: canonical profile-line evidence fixtures. +// The public artifacts contain diagnostic optical depths, not per-layer line-cache rows, so this table is // wavelengths used by public-python-baseline.json. const profile_line_probe_evidence = [_]ProfileLineProbeEvidence{ .{ diff --git a/tests/unit/cache/radiance_memory_test.zig b/tests/unit/cache/radiance_memory_test.zig index 0b011e44f..642c33322 100644 --- a/tests/unit/cache/radiance_memory_test.zig +++ b/tests/unit/cache/radiance_memory_test.zig @@ -8,7 +8,7 @@ const radiance_results = internal.spectrum.radiance_results; const radiance_wavelengths = internal.spectrum.radiance_wavelengths; const sampling_table = internal.spectrum.sampling_table; -test "RadianceMemory takes exact wavelength list ownership and exposes active views" { +test "RadianceMemory rebuilds exact wavelength list from slices and exposes active views" { const rows = [_]sampling_table.SpectrumSamplingRow{ .{ .nominal_wavelength_nm = 760.0, @@ -25,18 +25,22 @@ test "RadianceMemory takes exact wavelength list ownership and exposes active vi .irradiance_integration = .disabled(), }, }; - var list = try radiance_wavelengths.buildRadianceWavelengthList( + const list = try radiance_wavelengths.buildRadianceWavelengthList( std.testing.allocator, .{ .rows = rows[0..] }, ); - errdefer list.deinit(std.testing.allocator); var memory = radiance_memory.RadianceMemory{}; defer memory.deinit(std.testing.allocator); const stamp = hashing.ReuseStamp{ .value = 0x1234 }; - memory.takeWavelengthList(std.testing.allocator, &list, stamp); + memory.rebuildWavelengthList( + std.testing.allocator, + list.rows, + list.sample_indices, + list.wavelengths, + stamp, + ); - try std.testing.expectEqual(@as(usize, 0), list.rows.len); try std.testing.expectEqual(@as(usize, 2), memory.wavelength_rows.len); try std.testing.expectEqual(@as(usize, 2), memory.sample_indices.len); try std.testing.expectEqual(@as(usize, 1), memory.wavelengths.len); diff --git a/tests/unit/cache/spectrum_memory_test.zig b/tests/unit/cache/spectrum_memory_test.zig index b6f27bac3..b1ccec575 100644 --- a/tests/unit/cache/spectrum_memory_test.zig +++ b/tests/unit/cache/spectrum_memory_test.zig @@ -47,18 +47,22 @@ test "SpectrumMemory rejects active prefixes beyond retained capacity" { try std.testing.expectError(error.ShapeMismatch, memory.table(1, 2)); } -test "SpectrumMemory publishes sampling table stamp when taking ownership" { +test "SpectrumMemory publishes sampling table stamp when rebuilding from slices" { var memory = spectrum_memory.SpectrumMemory{}; defer memory.deinit(std.testing.allocator); const allocator = std.testing.allocator; - var owned = sampling_table.OwnedSpectrumSamplingTable{ - .rows = try allocator.alloc(sampling_table.SpectrumSamplingRow, 1), - .kernel_offsets_nm = try allocator.alloc(f64, 0), - .kernel_weights = try allocator.alloc(f64, 0), + const rows = try allocator.alloc(sampling_table.SpectrumSamplingRow, 1); + const kernel_offsets_nm = allocator.alloc(f64, 0) catch |err| { + allocator.free(rows); + return err; }; - errdefer owned.deinit(allocator); - owned.rows[0] = .{ + const kernel_weights = allocator.alloc(f64, 0) catch |err| { + allocator.free(rows); + allocator.free(kernel_offsets_nm); + return err; + }; + rows[0] = .{ .nominal_wavelength_nm = 760.0, .radiance_wavelength_nm = 760.0, .irradiance_wavelength_nm = 760.0, @@ -67,8 +71,7 @@ test "SpectrumMemory publishes sampling table stamp when taking ownership" { }; const stamp = ReuseStamp{ .value = 0xabc }; - memory.takeTable(allocator, &owned, stamp); - try std.testing.expectEqual(@as(usize, 0), owned.rows.len); + memory.rebuildTable(allocator, rows, kernel_offsets_nm, kernel_weights, stamp); try std.testing.expect(memory.hasTable(stamp)); try std.testing.expect(!memory.hasTable(.{ .value = 0xdef })); } diff --git a/tests/unit/input/scene_test.zig b/tests/unit/input/scene_test.zig index dcd4ebd84..8001cbc8d 100644 --- a/tests/unit/input/scene_test.zig +++ b/tests/unit/input/scene_test.zig @@ -2,7 +2,7 @@ const std = @import("std"); const internal = @import("internal"); const o2a_scene = @import("../support/o2a_scene.zig"); -test "O2 A scene fixture consumes every setup control" { +test "scene fixture consumes every setup control" { const scene = o2a_scene.reference(); try internal.input.validate.sceneControls(scene); @@ -60,7 +60,7 @@ test "invalid controls are rejected instead of carried inertly" { ); } -test "Python native O2 scene JSON round-trips into typed controls" { +test "Python native scene JSON round-trips into typed controls" { const allocator = std.testing.allocator; const rendered = try o2a_scene.nativeJson(allocator); diff --git a/tests/unit/optics/curved_sun_path_test.zig b/tests/unit/optics/curved_sun_path_test.zig index e0aaee912..fd0b76e03 100644 --- a/tests/unit/optics/curved_sun_path_test.zig +++ b/tests/unit/optics/curved_sun_path_test.zig @@ -9,7 +9,7 @@ const setup = internal.setup.run_tables; const allocator = std.testing.allocator; // CurvedSampleEvidence ---------------------------------------------------------------------------------------| -// Test-local pseudo-spherical evidence from O2 A baseline artifacts: | +// Test-local pseudo-spherical evidence from baseline artifacts: | // Canonical expected values owned by this repository. | // internal-dump-baseline.json .probe_forward_inputs[].pseudo_spherical.samples plus | // public-python-baseline.json .diagnostics.atmospheric_budget.rows for the line-absorption support values. | diff --git a/tests/unit/optics/layer_depths_test.zig b/tests/unit/optics/layer_depths_test.zig index 291ae8189..235c2e6cb 100644 --- a/tests/unit/optics/layer_depths_test.zig +++ b/tests/unit/optics/layer_depths_test.zig @@ -9,7 +9,7 @@ const jacobian = internal.rtm.jacobian_states; const allocator = std.testing.allocator; // SupportEvidence ------------------------------------------------------------------------------------------- | -// Test-local support-row optics evidence from O2 A baseline artifact: | +// Test-local support-row optics evidence from baseline artifact: | // Canonical expected values owned by this repository. | // public-python-baseline.json .diagnostics.atmospheric_budget.rows. | // | @@ -168,7 +168,7 @@ const layer_zero_758_rows = [_]SupportEvidence{ }; // LayerAerosolEvidence ---------------------------------------------------------------------------------------| -// Test-local aerosol layer evidence from O2 A baseline artifact: | +// Test-local aerosol layer evidence from baseline artifact: | // Canonical expected values owned by this repository. | // internal-dump-baseline.json .probe_forward_inputs[0].layers. | // | @@ -430,11 +430,7 @@ test "layer optics fill aerosol optical-depth jacobian lanes from current route const layers = try allocator.alloc(layer_depths.LayerOptics, tables.layers.layer_pressures_hpa.len); defer allocator.free(layers); try layer_depths.reduceLayerOpticsFromSupportRows(tables.layers, support_rows, layers, null); - layer_depths.fillLayerAerosolJacobians( - tables.aerosol, - jacobian.stateMask(.aerosol_optical_depth), - layers, - ); + layer_depths.fillLayerAerosolJacobians(tables.aerosol, layers); for (aerosol_jacobian_layers_758) |expected| { const layer_index: usize = @intCast(expected.layer_index); diff --git a/tests/unit/optics/source_levels_test.zig b/tests/unit/optics/source_levels_test.zig index 0a2c27e30..80f216e17 100644 --- a/tests/unit/optics/source_levels_test.zig +++ b/tests/unit/optics/source_levels_test.zig @@ -10,7 +10,7 @@ const source_levels = internal.optics.source_levels; const allocator = std.testing.allocator; // SourceLevelEvidence ----------------------------------------------------------------------------------------| -// Test-local RTM source-level evidence from O2 A baseline artifact: | +// Test-local RTM source-level evidence from baseline artifact: | // Canonical expected values owned by this repository. | // internal-dump-baseline.json .probe_forward_inputs[].rtm_quadrature_levels. | // | @@ -316,11 +316,7 @@ test "source levels reproduce shared RTM quadrature evidence rows" { null, ); try layer_depths.reduceLayerOpticsFromSupportRows(tables.layers, support_rows, layer_rows, null); - layer_depths.fillLayerAerosolJacobians( - tables.aerosol, - jacobian.stateMask(.aerosol_optical_depth), - layer_rows, - ); + layer_depths.fillLayerAerosolJacobians(tables.aerosol, layer_rows); try source_levels.fillSourceLevelsAtWavelength( expected.wavelength_nm, tables.layers, diff --git a/tests/unit/output/atmospheric_budget_test.zig b/tests/unit/output/atmospheric_budget_test.zig index ba6b91c57..a98f27056 100644 --- a/tests/unit/output/atmospheric_budget_test.zig +++ b/tests/unit/output/atmospheric_budget_test.zig @@ -7,7 +7,7 @@ const atmospheric_budget = internal.output.atmospheric_budget; const allocator = std.testing.allocator; // RowEvidence ------------------------------------------------------------------------------------------------| -// Test-local public atmospheric-budget evidence from O2 A baseline artifact: | +// Test-local public atmospheric-budget evidence from baseline artifact: | // Canonical expected values owned by this repository. | // public-python-baseline.json .diagnostics.atmospheric_budget.rows. | // | @@ -191,7 +191,7 @@ const expected_budget_rows = [_]RowEvidence{ }, }; -test "atmospheric budget rows match O2 A public Python evidence at probe wavelengths" { +test "atmospheric budget rows match public Python evidence at probe wavelengths" { var prepared = try internal.public.prepare(allocator, o2a_scene.reference()); defer prepared.deinit(allocator); @@ -246,7 +246,7 @@ fn expectRowEqual( fn expectF64Bits(expected: f64, actual: f64) !void { // expectF64Bits ------------------------------------------------------------------------------------------| - // Enforce the O2 A diagnostic rule: use exact f64 bits when O2 A JSON round-trips the value. | + // Enforce the diagnostic rule: use exact f64 bits when JSON round-trips the value. | // --------------------------------------------------------------------------------------------------------| const expected_bits: u64 = @bitCast(expected); const actual_bits: u64 = @bitCast(actual); @@ -255,8 +255,8 @@ fn expectF64Bits(expected: f64, actual: f64) !void { fn expectOpticalDepthUlp(expected: f64, actual: f64) !void { // expectOpticalDepthUlp ----------------------------------------------------------------------------------| - // Optical-depth products are compared as ULPs rather than prose tolerance. O2 A JSON values are the | - // expected source; this permits only the one-ULP product-order difference already present in the O2 A | + // Optical-depth products are compared as ULPs rather than prose tolerance. JSON values are the | + // expected source; this permits only the one-ULP product-order difference already present in the | // optics rows, and still fails larger drift. | // --------------------------------------------------------------------------------------------------------| const expected_bits: u64 = @bitCast(expected); diff --git a/tests/unit/output/cia_diagnostics_test.zig b/tests/unit/output/cia_diagnostics_test.zig index fa2f8f02f..98726f45e 100644 --- a/tests/unit/output/cia_diagnostics_test.zig +++ b/tests/unit/output/cia_diagnostics_test.zig @@ -7,7 +7,7 @@ const cia_diagnostics = internal.output.cia_diagnostics; const allocator = std.testing.allocator; // RowEvidence ------------------------------------------------------------------------------------------------| -// Test-local public O2-O2 CIA evidence from O2 A baseline artifact: | +// Test-local public O2-O2 CIA evidence from baseline artifact: | // Canonical expected values owned by this repository. | // public-python-baseline.json .diagnostics.collision_induced_absorption.rows. | // | @@ -131,7 +131,7 @@ const expected_cia_rows = [_]RowEvidence{ }, }; -test "O2-O2 CIA diagnostics match O2 A public Python evidence at probe wavelengths" { +test "O2-O2 CIA diagnostics match public Python evidence at probe wavelengths" { var prepared = try internal.public.prepare(allocator, o2a_scene.reference()); defer prepared.deinit(allocator); @@ -175,7 +175,7 @@ fn expectRowEqual(expected: cia_diagnostics.CiaRow, actual: cia_diagnostics.CiaR fn expectF64Bits(expected: f64, actual: f64) !void { // expectF64Bits ------------------------------------------------------------------------------------------| - // Enforce exact f64 bits for O2 A JSON values that round-trip without local derived math. | + // Enforce exact f64 bits for JSON values that round-trip without local derived math. | // --------------------------------------------------------------------------------------------------------| const expected_bits: u64 = @bitCast(expected); const actual_bits: u64 = @bitCast(actual); diff --git a/tests/unit/output/instrument_response_test.zig b/tests/unit/output/instrument_response_test.zig index 8bea54e37..e7bf91c48 100644 --- a/tests/unit/output/instrument_response_test.zig +++ b/tests/unit/output/instrument_response_test.zig @@ -8,7 +8,7 @@ const allocator = std.testing.allocator; // RowEvidence ------------------------------------------------------------------------------------------------| // `rtm.instrument_response(o2a.reference_scene(), [758.0, 760.0, 765.0, 767.0, 776.0])`. | -// The O2 A public artifact records no instrument-response rows, so this canonical probe is the source oracle. | +// The public artifact records no instrument-response rows, so this canonical probe is the source oracle. | // | // layout(64-bit) | // size: 104 B (0.102 KiB), align: 8 B | diff --git a/tests/unit/output/line_contributions_test.zig b/tests/unit/output/line_contributions_test.zig index 65860273d..4e824df30 100644 --- a/tests/unit/output/line_contributions_test.zig +++ b/tests/unit/output/line_contributions_test.zig @@ -7,7 +7,7 @@ const line_contributions = internal.output.line_contributions; const allocator = std.testing.allocator; // RowEvidence ------------------------------------------------------------------------------------------------| -// Test-local O2 line-contribution evidence from O2 A baseline artifact: | +// Test-local O2 line-contribution evidence from baseline artifact: | // Canonical expected values owned by this repository. | // public-python-baseline.json .diagnostics.line_contributions.rows. | // | @@ -176,7 +176,7 @@ const expected_rows = [_]RowEvidence{ }, }; -test "O2 line contributions match O2 A public Python evidence at probe wavelengths" { +test "O2 line contributions match public Python evidence at probe wavelengths" { var prepared = try internal.public.prepare(allocator, o2a_scene.reference()); defer prepared.deinit(allocator); @@ -238,7 +238,7 @@ fn expectRowEqual( fn expectF64Bits(expected: f64, actual: f64) !void { // expectF64Bits ------------------------------------------------------------------------------------------| - // Enforce exact f64 bits for O2 A JSON values that round-trip without local derived-math exceptions. | + // Enforce exact f64 bits for JSON values that round-trip without local derived-math exceptions. | // --------------------------------------------------------------------------------------------------------| const expected_bits: u64 = @bitCast(expected); const actual_bits: u64 = @bitCast(actual); diff --git a/tests/unit/public_surface_test.zig b/tests/unit/public_surface_test.zig index 1be1ac8aa..9c75e13c7 100644 --- a/tests/unit/public_surface_test.zig +++ b/tests/unit/public_surface_test.zig @@ -20,7 +20,9 @@ test "public root exposes setup session and spectrum surface" { try std.testing.expect(@hasDecl(zdisamar, "Spectrum")); try std.testing.expect(@hasDecl(zdisamar, "SpectrumRunResult")); try std.testing.expect(@hasDecl(zdisamar, "optimal_estimation")); - try std.testing.expect(@hasDecl(zdisamar, "RetrievalStateSpec")); + try std.testing.expect(@hasDecl(zdisamar, "RetrievalState")); + try std.testing.expect(@hasDecl(zdisamar, "RetrievalStateScalar")); + try std.testing.expect(@hasDecl(zdisamar, "RetrievalPressureLayerPlacement")); try std.testing.expect(@hasDecl(zdisamar, "RetrievalPressureAltitudeProfile")); try std.testing.expect(@hasDecl(zdisamar, "RetrievalResult")); try std.testing.expect(@hasDecl(zdisamar, "RetrievalBatchResult")); @@ -43,14 +45,14 @@ test "public root exposes setup session and spectrum surface" { try std.testing.expect(!@hasDecl(zdisamar, "zds_context_create")); } -test "public O2 A root surface keeps route-only spectrum knobs internal" { +test "public root surface keeps route-only spectrum knobs internal" { const zdisamar = internal.public; - // Source: canonical O2 A evidence fixtures owned by this repository. + // Source: canonical evidence fixtures owned by this repository. // Canonical expected values owned by this repository. // integrated irradiance rows. `python-reference-case-native.json` exposes no calibration, slit-kernel, // radiance-integration, or irradiance-integration override keys, so the root call keeps those six - // `runForwardSpectrum` arguments as fixed route constants rather than public case fields. + // `runForwardSpectrum` arguments as fixed route constants rather than public scene fields. try std.testing.expect(!@hasDecl(zdisamar, "buildReferenceRunTables")); try std.testing.expect(!@hasDecl(zdisamar, "deinitReferenceRunTables")); try std.testing.expect(!@hasDecl(zdisamar, "deinitRunTables")); diff --git a/tests/unit/retrieval/iteration_math_test.zig b/tests/unit/retrieval/iteration_math_test.zig index afc1fb44c..5e6f34dc2 100644 --- a/tests/unit/retrieval/iteration_math_test.zig +++ b/tests/unit/retrieval/iteration_math_test.zig @@ -13,22 +13,40 @@ test "normal-system accumulation uses reflectance Jacobians directly" { ); defer measurements.deinit(std.testing.allocator); - const specs = [_]retrieval.StateSpec{ - .{ - .state = .aerosol_optical_depth, + const pressure_altitude_profile = retrieval.PressureAltitudeProfile{ + .altitude_km = &.{ 0.0, 1.0 }, + .pressure_hpa = &.{ 900.0, 800.0 }, + .second = &.{ 0.0, 0.0 }, + }; + const retrieval_state: retrieval.RetrievalState = .{ + .aerosol_optical_depth = .{ .initial = 0.3, .prior = 0.2, .variance = 4.0, .lower_bound = 0.0, .upper_bound = 1.0, }, + .aerosol_layer_mid_pressure = .{ + .scalar = .{ + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower_bound = 600.0, + .upper_bound = 1000.0, + }, + .placement = .{ + .thickness_hpa = 10.0, + .interval_index_1based = 2, + .pressure_altitude_profile = &pressure_altitude_profile, + }, + }, }; - var result = try retrieval.Result.init(std.testing.allocator, specs.len, 1); + var result = try retrieval.Result.init(std.testing.allocator, 1); defer result.deinit(std.testing.allocator); - const state_space = try retrieval.initializeStateSpace(&specs, &result); + const state_space = try retrieval.initializeStateSpace(retrieval_state, &result); var scratch: retrieval.RetrievalIterationScratch = .{}; - try retrieval.preparePriorScales(state_space, specs.len, &scratch); + try retrieval.preparePriorScales(state_space, &scratch); const jacobian_rows = [_]internal.rtm.jacobian_states.Vector{ .{ 0.5, 0.0 }, @@ -41,7 +59,7 @@ test "normal-system accumulation uses reflectance Jacobians directly" { .reflectance = &.{ 0.10, 0.20 }, .jacobian = &jacobian_rows, }, - &specs, + retrieval_state, state_space.state, state_space.prior, scratch.sqrt_sa, @@ -54,7 +72,9 @@ test "normal-system accumulation uses reflectance Jacobians directly" { try std.testing.expectApproxEqAbs(4.5, scratch.g[0][0], 1.0e-15); try std.testing.expectApproxEqAbs(1.125, accumulation.jt_invse_j[0][0], 1.0e-15); try std.testing.expectEqual(.aerosol_optical_depth, result.state_ids[0]); + try std.testing.expectEqual(.aerosol_layer_mid_pressure_hpa, result.state_ids[1]); try std.testing.expectApproxEqAbs(0.3, result.initial_state[0], 0.0); + try std.testing.expectApproxEqAbs(850.0, result.initial_state[1], 0.0); } test "solve step keeps zero-residual state at the prior" { @@ -62,17 +82,18 @@ test "solve step keeps zero-residual state at the prior" { scratch.dx_white = algebra.zeroVector(); const step = try retrieval.solveStep( - 1, - .{ .{ 2.0, 0.0 }, .{ 0.0, 0.0 } }, + .{ .{ 2.0, 0.0 }, .{ 0.0, 3.0 } }, .{ 0.0, 0.0 }, - .{ 0.3, 0.0 }, - .{ 2.0, 0.0 }, - .{ 0.5, 0.0 }, + .{ 0.3, 850.0 }, + .{ 2.0, 10.0 }, + .{ 0.5, 0.1 }, 1.0, &scratch, ); try std.testing.expectApproxEqAbs(0.3, step.state[0], 0.0); + try std.testing.expectApproxEqAbs(850.0, step.state[1], 0.0); try std.testing.expect(step.snr_normal); try std.testing.expectApproxEqAbs(0.75, step.posterior_precision[0][0], 1.0e-15); + try std.testing.expectApproxEqAbs(0.04, step.posterior_precision[1][1], 1.0e-15); } diff --git a/tests/unit/retrieval/pressure_profile_test.zig b/tests/unit/retrieval/pressure_profile_test.zig index 169227618..b1a254caa 100644 --- a/tests/unit/retrieval/pressure_profile_test.zig +++ b/tests/unit/retrieval/pressure_profile_test.zig @@ -1,5 +1,6 @@ const std = @import("std"); const internal = @import("internal"); +const CountingAllocator = @import("../support/counting_allocator.zig").CountingAllocator; const retrieval = internal.retrieval.root; @@ -66,3 +67,36 @@ test "pressure profile derivative matches the canonical expectation two-point lo 2.0e-13, ); } + +test "pressure profile dynamic spline uses stack scratch for small profiles" { + const altitude_km = [_]f64{ 0.0, 1.0, 2.0, 3.0, 4.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0, 710.0, 630.0, 560.0 }; + + const expected = try retrieval.buildPressureProfile( + std.testing.allocator, + &altitude_km, + &pressure_hpa, + ); + defer retrieval.freePressureProfile(std.testing.allocator, expected); + + var counter = CountingAllocator.init(std.testing.allocator); + const counting_allocator = counter.allocator(); + const actual = try retrieval.buildPressureProfile( + counting_allocator, + &altitude_km, + &pressure_hpa, + ); + errdefer retrieval.freePressureProfile(counting_allocator, actual); + + try std.testing.expectEqual(@as(usize, 1), counter.alloc_calls); + try std.testing.expectEqual(@as(usize, 1), counter.live_allocations); + try std.testing.expectEqualSlices(f64, expected.second, actual.second); + try std.testing.expectApproxEqAbs( + try expected.altitudeDerivativeAtPressure(700.0), + try actual.altitudeDerivativeAtPressure(700.0), + 0.0, + ); + + retrieval.freePressureProfile(counting_allocator, actual); + try std.testing.expectEqual(@as(usize, 0), counter.live_allocations); +} diff --git a/tests/unit/retrieval/retrieval_state_validation_test.zig b/tests/unit/retrieval/retrieval_state_validation_test.zig new file mode 100644 index 000000000..df25c0ed9 --- /dev/null +++ b/tests/unit/retrieval/retrieval_state_validation_test.zig @@ -0,0 +1,65 @@ +const std = @import("std"); +const internal = @import("internal"); + +const retrieval = internal.retrieval.root; + +test "fixed retrieval state rejects invalid scalar and pressure placement" { + const altitude_km = [_]f64{ 0.0, 1.0 }; + const pressure_hpa = [_]f64{ 900.0, 800.0 }; + const second = [_]f64{ 0.0, 0.0 }; + const pressure_altitude_profile = retrieval.PressureAltitudeProfile{ + .altitude_km = altitude_km[0..], + .pressure_hpa = pressure_hpa[0..], + .second = second[0..], + }; + + const invalid_aod: retrieval.RetrievalState = .{ + .aerosol_optical_depth = .{ + .initial = 0.3, + .prior = 0.2, + .variance = -4.0, + .lower_bound = 0.0, + .upper_bound = 1.0, + }, + .aerosol_layer_mid_pressure = .{ + .scalar = .{ + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower_bound = 600.0, + .upper_bound = 1000.0, + }, + .placement = .{ + .thickness_hpa = 10.0, + .interval_index_1based = 1, + .pressure_altitude_profile = &pressure_altitude_profile, + }, + }, + }; + try std.testing.expectError(error.InvalidRetrievalState, retrieval.validateRetrievalState(invalid_aod)); + + const invalid_pressure: retrieval.RetrievalState = .{ + .aerosol_optical_depth = .{ + .initial = 0.3, + .prior = 0.2, + .variance = 4.0, + .lower_bound = 0.0, + .upper_bound = 1.0, + }, + .aerosol_layer_mid_pressure = .{ + .scalar = .{ + .initial = 850.0, + .prior = 850.0, + .variance = 100.0, + .lower_bound = 600.0, + .upper_bound = 1000.0, + }, + .placement = .{ + .thickness_hpa = std.math.nan(f64), + .interval_index_1based = 1, + .pressure_altitude_profile = &pressure_altitude_profile, + }, + }, + }; + try std.testing.expectError(error.InvalidRetrievalState, retrieval.validateRetrievalState(invalid_pressure)); +} diff --git a/tests/unit/retrieval/state_layout_test.zig b/tests/unit/retrieval/state_layout_test.zig index d7534a515..eec964648 100644 --- a/tests/unit/retrieval/state_layout_test.zig +++ b/tests/unit/retrieval/state_layout_test.zig @@ -4,7 +4,10 @@ const internal = @import("internal"); const retrieval = internal.retrieval.root; test "native retrieval layouts match canonical optimal-estimation value owners" { - try std.testing.expectEqual(@as(usize, 104), @sizeOf(retrieval.StateSpec)); + try std.testing.expectEqual(@as(usize, 40), @sizeOf(retrieval.StateScalar)); + try std.testing.expectEqual(@as(usize, 24), @sizeOf(retrieval.PressureLayerPlacement)); + try std.testing.expectEqual(@as(usize, 64), @sizeOf(retrieval.PressureState)); + try std.testing.expectEqual(@as(usize, 104), @sizeOf(retrieval.RetrievalState)); try std.testing.expectEqual(@as(usize, 48), @sizeOf(retrieval.PressureAltitudeProfile)); try std.testing.expectEqual(@as(usize, 48), @sizeOf(retrieval.MeasuredReflectanceRows)); try std.testing.expectEqual(@as(usize, 256), @sizeOf(retrieval.RetrievalIterationScratch)); @@ -13,5 +16,5 @@ test "native retrieval layouts match canonical optimal-estimation value owners" try std.testing.expectEqual(@as(usize, 120), @sizeOf(retrieval.BatchOutput)); try std.testing.expectEqual(@as(usize, 168), @sizeOf(retrieval.FastmodeBatchResult)); try std.testing.expectEqual(@as(usize, 24), @sizeOf(retrieval.Controls)); - try std.testing.expectEqual(@as(usize, 88), @sizeOf(retrieval.StateSpace)); + try std.testing.expectEqual(@as(usize, 80), @sizeOf(retrieval.StateSpace)); } diff --git a/tests/unit/retrieval/state_spec_validation_test.zig b/tests/unit/retrieval/state_spec_validation_test.zig deleted file mode 100644 index 5f5bd1eee..000000000 --- a/tests/unit/retrieval/state_spec_validation_test.zig +++ /dev/null @@ -1,48 +0,0 @@ -const std = @import("std"); -const internal = @import("internal"); - -const retrieval = internal.retrieval.root; - -test "retrieval state specs reject duplicate states and non-finite pressure thickness" { - const duplicate_states = [_]retrieval.StateSpec{ - .{ - .state = .aerosol_optical_depth, - .initial = 0.3, - .prior = 0.2, - .variance = 4.0, - .lower_bound = 0.0, - .upper_bound = 1.0, - }, - .{ - .state = .aerosol_optical_depth, - .initial = 0.4, - .prior = 0.2, - .variance = 4.0, - .lower_bound = 0.0, - .upper_bound = 1.0, - }, - }; - try std.testing.expectError(error.InvalidStateSpec, retrieval.validateStateSpecs(&duplicate_states)); - - const altitude_km = [_]f64{ 0.0, 1.0 }; - const pressure_hpa = [_]f64{ 900.0, 800.0 }; - const second = [_]f64{ 0.0, 0.0 }; - const pressure_state = [_]retrieval.StateSpec{ - .{ - .state = .aerosol_layer_mid_pressure_hpa, - .initial = 850.0, - .prior = 850.0, - .variance = 100.0, - .lower_bound = 600.0, - .upper_bound = 1000.0, - .thickness_hpa = std.math.nan(f64), - .interval_index_1based = 1, - .pressure_altitude_profile = .{ - .altitude_km = altitude_km[0..], - .pressure_hpa = pressure_hpa[0..], - .second = second[0..], - }, - }, - }; - try std.testing.expectError(error.InvalidStateSpec, retrieval.validateStateSpecs(&pressure_state)); -} diff --git a/tests/unit/root.zig b/tests/unit/root.zig index 32a22dc5b..e3f57e397 100644 --- a/tests/unit/root.zig +++ b/tests/unit/root.zig @@ -47,7 +47,7 @@ test { _ = @import("retrieval/state_layout_test.zig"); _ = @import("retrieval/pressure_profile_test.zig"); _ = @import("retrieval/measured_reflectance_rows_test.zig"); - _ = @import("retrieval/state_spec_validation_test.zig"); + _ = @import("retrieval/retrieval_state_validation_test.zig"); _ = @import("retrieval/iteration_math_test.zig"); _ = @import("validation/band_metrics_test.zig"); _ = @import("instrumentation/facades_test.zig"); diff --git a/tests/unit/rtm/controls_test.zig b/tests/unit/rtm/controls_test.zig index a4a5d5dc1..05723e5e9 100644 --- a/tests/unit/rtm/controls_test.zig +++ b/tests/unit/rtm/controls_test.zig @@ -3,7 +3,6 @@ const std = @import("std"); const internal = @import("internal"); const controls = internal.rtm.controls; -const jacobian_states = internal.rtm.jacobian_states; // ControlLayoutEvidence --------------------------------------------------------------------------------------| // Compile-time layout pins for transport control rows copied from the RTM control contract. | @@ -116,15 +115,15 @@ test "performance thresholds keep order and Fourier cap helpers" { try std.testing.expect(uncapped.shouldEvaluateAerosolTangent(100)); } -test "prepare solve config validates controls and sanitizes Jacobian state mask" { +test "prepare solve config validates controls and preserves Jacobian request" { const prepared = try controls.prepareSolveConfig(.{ .derivative_mode = .semi_analytical, - .derivative_state_mask = 0xff, + .wants_jacobian = true, .controls = .{ .n_streams = 16 }, }); try std.testing.expectEqual(controls.DerivativeMode.semi_analytical, prepared.derivative_mode); - try std.testing.expectEqual(jacobian_states.all_states_mask, prepared.derivative_state_mask); + try std.testing.expect(prepared.wants_jacobian); try std.testing.expectEqual(@as(u16, 16), prepared.controls.n_streams); try std.testing.expectError( diff --git a/tests/unit/rtm/jacobian_states_test.zig b/tests/unit/rtm/jacobian_states_test.zig index 9bfeccc29..b9cfbb037 100644 --- a/tests/unit/rtm/jacobian_states_test.zig +++ b/tests/unit/rtm/jacobian_states_test.zig @@ -3,25 +3,17 @@ const internal = @import("internal"); const jacobian = internal.rtm.jacobian_states; -test "jacobian state masks preserve fixed aerosol retrieval order" { +test "jacobian states preserve fixed aerosol retrieval order" { try std.testing.expectEqual(@as(usize, 2), jacobian.state_count); try std.testing.expectEqual(@as(usize, 0), jacobian.stateIndex(.aerosol_optical_depth)); try std.testing.expectEqual(@as(usize, 1), jacobian.stateIndex(.aerosol_layer_mid_pressure_hpa)); - try std.testing.expectEqual(@as(jacobian.StateMask, 0b11), jacobian.all_states_mask); - - const active_mask = jacobian.stateMask(.aerosol_layer_mid_pressure_hpa); - try std.testing.expect(!jacobian.includes(active_mask, .aerosol_optical_depth)); - try std.testing.expect(jacobian.includes(active_mask, .aerosol_layer_mid_pressure_hpa)); - try std.testing.expectEqual(@as(usize, 1), jacobian.activeStateCount(active_mask)); } -test "jacobian masked vector helpers leave inactive lanes untouched" { +test "jacobian vector helpers operate on both fixed lanes" { var accumulator = jacobian.Vector{ 10.0, 20.0 }; const vector = jacobian.Vector{ 1.0, 2.0 }; - const mask = jacobian.stateMask(.aerosol_optical_depth); - jacobian.addScaledMasked(&accumulator, vector, 4.0, mask); - try std.testing.expectEqual(jacobian.Vector{ 14.0, 20.0 }, accumulator); + jacobian.addScaled(&accumulator, vector, 4.0); + try std.testing.expectEqual(jacobian.Vector{ 14.0, 28.0 }, accumulator); try std.testing.expectEqual(jacobian.Vector{ 2.0, 4.0 }, jacobian.scale(vector, 2.0)); - try std.testing.expectEqual(jacobian.Vector{ 2.0, 0.0 }, jacobian.scaleMasked(vector, 2.0, mask)); } diff --git a/tests/unit/rtm/solve_test.zig b/tests/unit/rtm/solve_test.zig index df9a7426c..afb837fa3 100644 --- a/tests/unit/rtm/solve_test.zig +++ b/tests/unit/rtm/solve_test.zig @@ -11,8 +11,6 @@ const rows = internal.rtm.rows; const solve = internal.rtm.solve; const source_levels = internal.optics.source_levels; -const tangent_step: f64 = 1.0e-5; - test "direct surface solve applies scalar formula without retrieval Jacobian lanes" { var storage = TestSolveWorkStorage{}; var work = storage.work(); @@ -25,8 +23,8 @@ test "direct surface solve applies scalar formula without retrieval Jacobian lan .view_mu = 0.25, }; const config = controls.SolveConfig{ - .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.all_states_mask, + .derivative_mode = .none, + .wants_jacobian = false, .controls = .{ .scattering = .none }, }; const result = try solve.solveReflectance( @@ -47,23 +45,19 @@ test "direct surface solve applies scalar formula without retrieval Jacobian lan } test "direct surface solve keeps retrieval Jacobian zero and respects public clamp" { - const masked = solve.directSurfaceOnly( + const direct = solve.directSurfaceOnly( .{ .solar_mu = 1.0, .view_mu = 1.0 }, 0.3, 0.0, - .semi_analytical, - jacobian_states.stateMask(.aerosol_optical_depth), ); const clamped = solve.directSurfaceOnly( .{ .solar_mu = 1.0, .view_mu = 1.0 }, 3.0, 0.0, - .semi_analytical, - jacobian_states.all_states_mask, ); - try std.testing.expectEqual(@as(f64, 0.3), masked.reflectance); - try std.testing.expectEqual(jacobian_states.zero(), masked.jacobian); + try std.testing.expectEqual(@as(f64, 0.3), direct.reflectance); + try std.testing.expectEqual(jacobian_states.zero(), direct.jacobian); try std.testing.expectEqual(@as(f64, 2.0), clamped.reflectance); try std.testing.expectEqual(jacobian_states.zero(), clamped.jacobian); } @@ -272,9 +266,6 @@ test "integrated scattering route propagates aerosol AOD and pressure tangents" var work = storage.work(); const layers = integratedAerosolLayers(); const levels = integratedAerosolSourceLevels(); - const mask = - jacobian_states.stateMask(.aerosol_optical_depth) | - jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa); const result = try solve.solveReflectance( .{ .solar_mu = 0.58, .view_mu = 0.64 }, @@ -286,7 +277,7 @@ test "integrated scattering route propagates aerosol AOD and pressure tangents" 0.5, .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = mask, + .wants_jacobian = true, .controls = .{ .scattering = .single, .n_streams = 8, @@ -303,10 +294,10 @@ test "integrated scattering route propagates aerosol AOD and pressure tangents" try std.testing.expectApproxEqAbs(0.00926416296449234, pressure, 1.0e-15); } -test "scattering route propagates AOD tangent and rejects pressure lane" { +test "non-integrated scattering route rejects Jacobian requests" { var storage = TestSolveWorkStorage{}; var work = storage.work(); - var layers = [_]layer_depths.LayerOptics{ + const layers = [_]layer_depths.LayerOptics{ .{ .gas_absorption_optical_depth = 0.08, .aerosol_optical_depth = 0.02, @@ -316,61 +307,6 @@ test "scattering route propagates AOD tangent and rejects pressure lane" { .single_scatter_albedo = 0.10, }, }; - jacobian_states.set(&layers[0].optical_depth_jacobian, .aerosol_optical_depth, 1.0); - jacobian_states.set(&layers[0].scattering_optical_depth_jacobian, .aerosol_optical_depth, 0.5); - jacobian_states.set(&layers[0].single_scatter_albedo_jacobian, .aerosol_optical_depth, 4.0); - - const aod = try solve.solveReflectance( - .{ .solar_mu = 0.58, .view_mu = 0.64 }, - 0.3, - &layers, - &.{}, - &.{}, - testPhase(), - 0.5, - .{ - .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth), - .controls = .{ - .scattering = .single, - .n_streams = 8, - .integrate_source_function = false, - }, - }, - &work, - ); - var plus_layers = layers; - var minus_layers = layers; - perturbAodLayer(&plus_layers[0], tangent_step); - perturbAodLayer(&minus_layers[0], -tangent_step); - const plus = try solve.solveReflectance( - .{ .solar_mu = 0.58, .view_mu = 0.64 }, - 0.3, - &plus_layers, - &.{}, - &.{}, - testPhase(), - 0.5, - .{ .controls = .{ .scattering = .single, .n_streams = 8, .integrate_source_function = false } }, - &work, - ); - const minus = try solve.solveReflectance( - .{ .solar_mu = 0.58, .view_mu = 0.64 }, - 0.3, - &minus_layers, - &.{}, - &.{}, - testPhase(), - 0.5, - .{ .controls = .{ .scattering = .single, .n_streams = 8, .integrate_source_function = false } }, - &work, - ); - const expected_aod_tangent = (plus.reflectance - minus.reflectance) / (2.0 * tangent_step); - try std.testing.expectApproxEqAbs( - expected_aod_tangent, - aod.jacobian[jacobian_states.stateIndex(.aerosol_optical_depth)], - 1.0e-8, - ); try std.testing.expectError( error.UnsupportedDerivativeMode, @@ -384,7 +320,7 @@ test "scattering route propagates AOD tangent and rejects pressure lane" { 0.5, .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, .controls = .{ .scattering = .single, .n_streams = 8, @@ -520,28 +456,6 @@ fn anisotropicPhase() phase_table.PhaseTable { }; } -fn perturbAodLayer(layer: *layer_depths.LayerOptics, signed_step: f64) void { - // perturbAodLayer --------------------------------------------------------------------------------------- | - // Apply the non-integrated AOD finite-difference variables to one layer row. | - // --------------------------------------------------------------------------------------------------------| - layer.total_optical_depth = @max( - layer.total_optical_depth + - signed_step * jacobian_states.get(layer.optical_depth_jacobian, .aerosol_optical_depth), - 0.0, - ); - layer.total_scattering_optical_depth = @max( - layer.total_scattering_optical_depth + - signed_step * jacobian_states.get(layer.scattering_optical_depth_jacobian, .aerosol_optical_depth), - 0.0, - ); - layer.single_scatter_albedo = std.math.clamp( - layer.single_scatter_albedo + - signed_step * jacobian_states.get(layer.single_scatter_albedo_jacobian, .aerosol_optical_depth), - 0.0, - 1.0, - ); -} - fn integratedAerosolLayers() [2]layer_depths.LayerOptics { // integratedAerosolLayers ------------------------------------------------------------------------------ | // Synthetic positive-scattering layers for solve-level integrated aerosol derivative tests. | diff --git a/tests/unit/session_reuse_parity_test.zig b/tests/unit/session_reuse_parity_test.zig index 1259548fa..f069352fd 100644 --- a/tests/unit/session_reuse_parity_test.zig +++ b/tests/unit/session_reuse_parity_test.zig @@ -24,7 +24,7 @@ test "runForwardWithSessionMemory reuses profile-line rows across repeated scene const solve_config = zdisamar.SolveConfig{ .derivative_mode = .none, - .derivative_state_mask = 0, + .wants_jacobian = false, .controls = .{ .scattering = .none, .n_streams = @intCast(scene.rtm.stream_count), diff --git a/tests/unit/setup/run_tables_test.zig b/tests/unit/setup/run_tables_test.zig index a0951625a..1f95c62fc 100644 --- a/tests/unit/setup/run_tables_test.zig +++ b/tests/unit/setup/run_tables_test.zig @@ -1,8 +1,9 @@ const std = @import("std"); const internal = @import("internal"); const o2a_scene = @import("../support/o2a_scene.zig"); +const CountingAllocator = @import("../support/counting_allocator.zig").CountingAllocator; -test "RunTables match O2 A baseline table evidence" { +test "RunTables match baseline table evidence" { var tables = try internal.setup.run_tables.buildRunTables( std.testing.allocator, o2a_scene.reference(), @@ -163,6 +164,26 @@ test "LayerGrid refill from prepared profiles matches fresh builds for pressure } } +test "LayerGrid build keeps hydrostatic scratch out of retained allocations" { + const allocator = std.testing.allocator; + const atmosphere_layers = internal.setup.atmosphere_layers; + const scene = o2a_scene.reference(); + + var expected = try atmosphere_layers.build(allocator, scene); + defer expected.deinit(allocator); + + var counter = CountingAllocator.init(allocator); + const counting_allocator = counter.allocator(); + var actual = try atmosphere_layers.build(counting_allocator, scene); + errdefer actual.deinit(counting_allocator); + + try std.testing.expectEqual(@as(usize, 22), counter.live_allocations); + try expectLayerGridEqual(expected, actual); + + actual.deinit(counting_allocator); + try std.testing.expectEqual(@as(usize, 0), counter.live_allocations); +} + fn findLineByWavenumber( rows: []const internal.assets.readers.LineAssetRow, center_wavenumber_cm1: f64, @@ -213,7 +234,7 @@ fn moveAerosolLayerPressure( } } - if (!updated) return error.InvalidStateSpec; + if (!updated) return error.InvalidRetrievalState; } fn expectEqualF64Slices(expected: []const f64, actual: []const f64) !void { diff --git a/tests/unit/spectrum/instrument_average_test.zig b/tests/unit/spectrum/instrument_average_test.zig index 5abdad538..cbc0db05e 100644 --- a/tests/unit/spectrum/instrument_average_test.zig +++ b/tests/unit/spectrum/instrument_average_test.zig @@ -60,7 +60,7 @@ test "postprocessSignal rejects in-place non-integrated convolution" { ); } -test "postprocessRadianceResults convolves and calibrates active row-vector Jacobian lanes" { +test "postprocessRadianceResults convolves and calibrates fixed row-vector Jacobian lanes" { const raw = [_]radiance_results.RadianceResult{ .{ .radiance = 2.0, .jacobian = .{ 1.0, 100.0 } }, .{ .radiance = 4.0, .jacobian = .{ 2.0, 200.0 } }, @@ -73,13 +73,12 @@ test "postprocessRadianceResults convolves and calibrates active row-vector Jaco .offset = 0.5, .stray_light = 0.25, }; - const mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa); var output = [_]radiance_results.RadianceResult{.{}} ** raw.len; try instrument_average.postprocessRadianceResults( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = mask, + .wants_jacobian = true, }, false, calibration, @@ -89,9 +88,11 @@ test "postprocessRadianceResults convolves and calibrates active row-vector Jaco ); var convolved_radiance: [raw.len]f64 = undefined; + var convolved_aod: [raw.len]f64 = undefined; var convolved_pressure: [raw.len]f64 = undefined; for (0..raw.len) |index| { convolved_radiance[index] = expectedConvolutionRow(raw[0..], kernel[0..], index, .radiance, 0); + convolved_aod[index] = expectedConvolutionRow(raw[0..], kernel[0..], index, .jacobian, 0); convolved_pressure[index] = expectedConvolutionRow(raw[0..], kernel[0..], index, .jacobian, 1); } @@ -101,7 +102,11 @@ test "postprocessRadianceResults convolves and calibrates active row-vector Jaco actual.radiance, 1.0e-15, ); - try std.testing.expectApproxEqAbs(0.0, actual.jacobian[0], 0.0); + try std.testing.expectApproxEqAbs( + expectedDerivativeCalibration(calibration, convolved_aod[0..], index), + actual.jacobian[0], + 1.0e-12, + ); try std.testing.expectApproxEqAbs( expectedDerivativeCalibration(calibration, convolved_pressure[0..], index), actual.jacobian[1], @@ -110,7 +115,7 @@ test "postprocessRadianceResults convolves and calibrates active row-vector Jaco } } -test "postprocessRadianceResults permits in-place integrated sampling and zeros inactive lanes" { +test "postprocessRadianceResults permits in-place integrated sampling and zeros Jacobian lanes when disabled" { var rows = [_]radiance_results.RadianceResult{ .{ .radiance = 1.0, .jacobian = .{ 2.0, 4.0 } }, .{ .radiance = 3.0, .jacobian = .{ 6.0, 12.0 } }, @@ -124,7 +129,7 @@ test "postprocessRadianceResults permits in-place integrated sampling and zeros try instrument_average.postprocessRadianceResults( .{ .derivative_mode = .none, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, true, calibration, @@ -218,8 +223,7 @@ test "assembleReflectanceResults keeps denominator floor and clamp summary" { try std.testing.expectApproxEqAbs(reflectance[1], summary.max_reflectance, 0.0); } -test "assembleReflectanceResults scales active radiance Jacobian lanes into reflectance units" { - const mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa); +test "assembleReflectanceResults scales fixed radiance Jacobian lanes into reflectance units" { const radiance = [_]radiance_results.RadianceResult{ .{ .radiance = 2.0, .jacobian = .{ 0.25, 250.0 } }, .{ .radiance = 4.0, .jacobian = .{ 0.5, 500.0 } }, @@ -231,7 +235,7 @@ test "assembleReflectanceResults scales active radiance Jacobian lanes into refl const summary = try instrument_average.assembleReflectanceResults( controls.SolveConfig{ .derivative_mode = .semi_analytical, - .derivative_state_mask = mask, + .wants_jacobian = true, }, 0.5, radiance[0..], @@ -244,11 +248,11 @@ test "assembleReflectanceResults scales active radiance Jacobian lanes into refl const scale1 = std.math.pi / (16.0 * 0.5); try std.testing.expectApproxEqAbs(2.0 * scale0, reflectance[0], 1.0e-15); try std.testing.expectApproxEqAbs(4.0 * scale1, reflectance[1], 1.0e-15); - try std.testing.expectApproxEqAbs(0.0, jacobian[0][0], 0.0); + try std.testing.expectApproxEqAbs(0.25 * scale0, jacobian[0][0], 1.0e-15); try std.testing.expectApproxEqAbs(250.0 * scale0, jacobian[0][1], 1.0e-13); - try std.testing.expectApproxEqAbs(0.0, jacobian[1][0], 0.0); + try std.testing.expectApproxEqAbs(0.5 * scale1, jacobian[1][0], 1.0e-15); try std.testing.expectApproxEqAbs(500.0 * scale1, jacobian[1][1], 1.0e-13); - try std.testing.expectApproxEqAbs(0.0, summary.jacobian_sum[0], 0.0); + try std.testing.expectApproxEqAbs(jacobian[0][0] + jacobian[1][0], summary.jacobian_sum[0], 1.0e-15); try std.testing.expectApproxEqAbs(jacobian[0][1] + jacobian[1][1], summary.jacobian_sum[1], 1.0e-13); } @@ -265,7 +269,7 @@ test "assembleReflectanceResults requires output rows for requested Jacobians" { instrument_average.assembleReflectanceResults( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth), + .wants_jacobian = true, }, 1.0, radiance[0..], @@ -291,7 +295,7 @@ test "assembleReflectanceResults zeros provided Jacobian rows when derivative mo const summary = try instrument_average.assembleReflectanceResults( .{ .derivative_mode = .none, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, 1.0, radiance[0..], diff --git a/tests/unit/spectrum/radiance_results_test.zig b/tests/unit/spectrum/radiance_results_test.zig index 5fc7387ee..3a617b4b4 100644 --- a/tests/unit/spectrum/radiance_results_test.zig +++ b/tests/unit/spectrum/radiance_results_test.zig @@ -9,8 +9,7 @@ const radiance_wavelengths = internal.spectrum.radiance_wavelengths; const sampling_table = internal.spectrum.sampling_table; const solve = internal.rtm.solve; -test "scaleReflectanceToRadiance applies solar radiance scale and active Jacobian lanes" { - const mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa); +test "scaleReflectanceToRadiance applies solar radiance scale and fixed Jacobian lanes" { const reflectance = solve.ReflectanceResult{ .reflectance = 0.42, .jacobian = .{ 0.1, 0.2 }, @@ -21,7 +20,7 @@ test "scaleReflectanceToRadiance applies solar radiance scale and active Jacobia const actual = radiance_results.scaleReflectanceToRadiance( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = mask, + .wants_jacobian = true, }, reflectance, solar_cosine, @@ -30,7 +29,7 @@ test "scaleReflectanceToRadiance applies solar radiance scale and active Jacobia const scale = solar_cosine * solar_irradiance / std.math.pi; try std.testing.expectApproxEqAbs(0.42 * scale, actual.radiance, 1.0e-15); - try std.testing.expectApproxEqAbs(0.0, actual.jacobian[0], 0.0); + try std.testing.expectApproxEqAbs(0.1 * scale, actual.jacobian[0], 1.0e-15); try std.testing.expectApproxEqAbs(0.2 * scale, actual.jacobian[1], 1.0e-15); } @@ -38,7 +37,7 @@ test "scaleReflectanceToRadiance zeros Jacobian when derivative mode is off" { const actual = radiance_results.scaleReflectanceToRadiance( .{ .derivative_mode = .none, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, .{ .reflectance = 0.25, @@ -59,7 +58,7 @@ test "integratePrefetchedRadianceAtNominal returns direct prefetched row" { const actual = try radiance_results.integratePrefetchedRadianceAtNominal( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, .{ .radiance = radiance[0..], .jacobian = jacobian[0..] }, .{ .start = 0 }, @@ -73,7 +72,7 @@ test "integratePrefetchedRadianceAtNominal returns direct prefetched row" { try std.testing.expectApproxEqAbs(0.5, actual.jacobian[1], 0.0); } -test "integratePrefetchedRadianceAtNominal weights radiance and active Jacobian lanes" { +test "integratePrefetchedRadianceAtNominal weights radiance and fixed Jacobian lanes" { var radiance = [_]f64{ 10.0, 20.0, 40.0 }; var jacobian = [_]jacobian_states.Vector{ .{ 1.0, 100.0 }, .{ 2.0, 200.0 }, .{ 4.0, 400.0 } }; const sample_indices = [_]u32{ 2, 0, 1 }; @@ -84,12 +83,10 @@ test "integratePrefetchedRadianceAtNominal weights radiance and active Jacobian .sample_count = 3, .encoding = .side_samples, }; - const mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa); - const actual = try radiance_results.integratePrefetchedRadianceAtNominal( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = mask, + .wants_jacobian = true, }, .{ .radiance = radiance[0..], .jacobian = jacobian[0..] }, .{ .start = 0 }, @@ -102,7 +99,7 @@ test "integratePrefetchedRadianceAtNominal weights radiance and active Jacobian ); try std.testing.expectApproxEqAbs(20.0, actual.radiance, 0.0); - try std.testing.expectApproxEqAbs(0.0, actual.jacobian[0], 0.0); + try std.testing.expectApproxEqAbs(2.0, actual.jacobian[0], 0.0); try std.testing.expectApproxEqAbs(200.0, actual.jacobian[1], 0.0); } @@ -119,7 +116,7 @@ test "integratePrefetchedRadianceAtNominal omits Jacobian when derivative mode i const actual = try radiance_results.integratePrefetchedRadianceAtNominal( controls.SolveConfig{ .derivative_mode = .none, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, .{ .radiance = radiance[0..] }, .{ .start = 0 }, @@ -165,7 +162,7 @@ test "integratePrefetchedRadianceAtNominal rejects missing Jacobian storage when radiance_results.integratePrefetchedRadianceAtNominal( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.all_states_mask, + .wants_jacobian = true, }, .{ .radiance = radiance[0..] }, radiance_wavelengths.RadianceSampleIndexRef{ .start = 0 }, diff --git a/tests/unit/spectrum/radiance_wavelengths_test.zig b/tests/unit/spectrum/radiance_wavelengths_test.zig index 2b7bb387b..101f9a0ac 100644 --- a/tests/unit/spectrum/radiance_wavelengths_test.zig +++ b/tests/unit/spectrum/radiance_wavelengths_test.zig @@ -62,9 +62,9 @@ test "RadianceWavelengthList deduplicates only exact f64 bit keys" { try std.testing.expectEqual(@as(u32, 2), list.sample_indices[4]); } -test "RadianceWavelengthList evidence anchors keep O2 A exact-key route shape" { +test "RadianceWavelengthList evidence anchors keep exact-key route shape" { - // Source: canonical O2 A radiance-wavelength evidence fixtures. + // Source: canonical radiance-wavelength evidence fixtures. // Canonical expected values owned by this repository. try std.testing.expectEqual(@as(usize, 3874), radiance_wavelength_evidence.count); @@ -83,7 +83,7 @@ test "RadianceWavelengthList evidence anchors keep O2 A exact-key route shape" { } // RadianceWavelengthEvidence -------------------------------------------------------------------------------- | -// Exact-key canonical radiance wavelength evidence from the O2 A internal dump. | +// Exact-key canonical radiance wavelength evidence from the internal dump. | // | // layout(64-bit) | // size: 40 B (0.039 KiB), align: 8 B | diff --git a/tests/unit/spectrum/sampling_table_test.zig b/tests/unit/spectrum/sampling_table_test.zig index ed95bd185..2db0c5bae 100644 --- a/tests/unit/spectrum/sampling_table_test.zig +++ b/tests/unit/spectrum/sampling_table_test.zig @@ -46,7 +46,7 @@ test "IntegrationKernelRef keeps direct inline and side sample contracts" { try std.testing.expectApproxEqAbs(0.2, side_ref.weight(side_storage, 0), 0.0); } -test "SpectrumSamplingTable summary matches O2 A aggregate sampling evidence shape" { +test "SpectrumSamplingTable summary matches aggregate sampling evidence shape" { const side_offsets = [_]f64{ -0.02, 0.0, 0.02, -0.01, 0.01 }; const side_weights = [_]f64{ 0.25, 0.5, 0.25, 0.5, 0.5 }; const rows = [_]sampling_table.SpectrumSamplingRow{ @@ -90,7 +90,7 @@ test "SpectrumSamplingTable summary matches O2 A aggregate sampling evidence sha try std.testing.expectEqual(@as(usize, 5), summary.side_sample_count); try std.testing.expectEqual(@as(usize, 3), summary.max_kernel_sample_count); - // Source: canonical O2 A sampling evidence fixtures. + // Source: canonical sampling evidence fixtures. // Canonical expected values owned by this repository. try std.testing.expectEqual(@as(usize, 701), sampling_evidence.row_count); try std.testing.expectEqual(@as(usize, 565776), sampling_evidence.kernel_offsets_len); @@ -98,7 +98,7 @@ test "SpectrumSamplingTable summary matches O2 A aggregate sampling evidence sha try std.testing.expectEqual(@as(usize, 3874), sampling_evidence.forward_miss_count); } -test "O2 SpectrumSamplingTable builder matches O2 A aggregate and exact-key evidence" { +test "SpectrumSamplingTable builder matches aggregate and exact-key evidence" { var tables = try internal.setup.run_tables.buildRunTables( std.testing.allocator, o2a_scene.reference(), @@ -116,7 +116,7 @@ test "O2 SpectrumSamplingTable builder matches O2 A aggregate and exact-key evid const table = owned.view(); const summary = sampling_table.summarize(table); - // Source: canonical O2 A sampling evidence fixtures. + // Source: canonical sampling evidence fixtures. // Canonical expected values owned by this repository. try std.testing.expectEqual(@as(usize, 701), summary.row_count); try std.testing.expectEqual(@as(usize, 701), summary.radiance_integrated_rows); @@ -128,7 +128,7 @@ test "O2 SpectrumSamplingTable builder matches O2 A aggregate and exact-key evid var list = try radiance_wavelengths.buildRadianceWavelengthList(std.testing.allocator, table); defer list.deinit(std.testing.allocator); - // Source: same O2 A internal dump, sampling_table.forward_misses first/last rows. + // Source: same internal dump, sampling_table.forward_misses first/last rows. try std.testing.expectEqual(@as(usize, 3874), list.wavelengths.len); try std.testing.expectEqual(@as(u64, 4649845583965292708), list.wavelengths[0].key); try std.testing.expectApproxEqAbs(754.240334835155, list.wavelengths[0].wavelength_nm, 0.0); @@ -136,7 +136,7 @@ test "O2 SpectrumSamplingTable builder matches O2 A aggregate and exact-key evid try std.testing.expectApproxEqAbs(776.8246811031544, list.wavelengths[list.wavelengths.len - 1].wavelength_nm, 0.0); } -test "O2 SpectrumSamplingTable consumes explicit sparse measured wavelengths" { +test "SpectrumSamplingTable consumes explicit sparse measured wavelengths" { const explicit_wavelengths = [_]f64{ 758.0, 758.06, 758.21, 758.27 }; var scene = o2a_scene.reference(); scene.spectral_grid = .{ @@ -168,7 +168,7 @@ test "O2 SpectrumSamplingTable consumes explicit sparse measured wavelengths" { } } -test "O2 SpectrumSamplingTable parallel fill keeps resolved samples deterministic" { +test "SpectrumSamplingTable parallel fill keeps resolved samples deterministic" { var tables = try internal.setup.run_tables.buildRunTables( std.testing.allocator, o2a_scene.reference(), @@ -282,7 +282,7 @@ fn expectResolvedKernelEqual( } // SamplingEvidence ------------------------------------------------------------------------------------------ | -// Aggregate canonical sampling-table evidence from the O2 A internal dump. | +// Aggregate canonical sampling-table evidence from the internal dump. | // | // layout(64-bit) | // size: 32 B (0.031 KiB), align: 8 B | diff --git a/tests/unit/spectrum/solar_lookup_test.zig b/tests/unit/spectrum/solar_lookup_test.zig index a0930b827..ff05b7f99 100644 --- a/tests/unit/spectrum/solar_lookup_test.zig +++ b/tests/unit/spectrum/solar_lookup_test.zig @@ -44,7 +44,7 @@ test "irradianceAtWavelength matches operational spline and clamp behavior" { ); defer tables.deinit(std.testing.allocator); - // Source: `OperationalSolarSpectrum.interpolateIrradiance` route for the O2 A reference solar asset. + // Source: `OperationalSolarSpectrum.interpolateIrradiance` route for the reference solar asset. try expectSolar(4.87640000000000000e14, try solar_lookup.irradianceAtWavelength(tables.solar, 753.0)); try expectSolar(4.86690022241158375e14, try solar_lookup.irradianceAtWavelength(tables.solar, 753.005)); try expectSolar(4.87870339978322125e14, try solar_lookup.irradianceAtWavelength(tables.solar, 760.005)); @@ -82,7 +82,7 @@ test "integrateIrradianceAtNominalAssumeCapacity weights cached irradiance sampl }, ); - // Source: `spectral_eval.zig` integrateIrradianceAtNominal route with the O2 A reference solar asset. + // Source: `spectral_eval.zig` integrateIrradianceAtNominal route with the reference solar asset. try expectSolar(4.87366765517170125e14, actual); try std.testing.expectEqual(@as(u32, 3), memory.values.count()); } @@ -159,7 +159,7 @@ test "integrateIrradianceAtNominalAssumeCapacity rejects malformed side storage" fn expectSolar(expected: f64, actual: f64) !void { // expectSolar --------------------------------------------------------------------------------------------| - // Compare large O2 A irradiance values with an absolute tolerance small enough to catch spline changes. | + // Compare large irradiance values with an absolute tolerance small enough to catch spline changes. | // --------------------------------------------------------------------------------------------------------| try std.testing.expectApproxEqAbs(expected, actual, 1.0e1); } diff --git a/tests/unit/spectrum/spectrum_run_test.zig b/tests/unit/spectrum/spectrum_run_test.zig index 9410a713d..3fe107664 100644 --- a/tests/unit/spectrum/spectrum_run_test.zig +++ b/tests/unit/spectrum/spectrum_run_test.zig @@ -80,7 +80,7 @@ test "radianceAtWavelength wires optics direct transport and radiance scaling" { }; const solve_config = controls.SolveConfig{ .derivative_mode = .none, - .derivative_state_mask = 0, + .wants_jacobian = false, .controls = .{ .scattering = .none, .integrate_source_function = false, @@ -99,14 +99,14 @@ test "radianceAtWavelength wires optics direct transport and radiance scaling" { null, ); try layer_depths.reduceLayerOpticsFromSupportRows(tables.layers, expected_support, expected_layers, null); - layer_depths.fillLayerAerosolJacobians(tables.aerosol, solve_config.derivative_state_mask, expected_layers); + if (solve_config.derivative_mode != .none and solve_config.wants_jacobian) { + layer_depths.fillLayerAerosolJacobians(tables.aerosol, expected_layers); + } const expected_reflectance = solve.directSurfaceOnly( angles, surface_albedo, testTotalOpticalDepth(expected_layers), - solve_config.derivative_mode, - solve_config.derivative_state_mask, ); const expected = radiance_results.scaleReflectanceToRadiance( solve_config, @@ -276,8 +276,7 @@ test "radianceAtWavelength matches canonical transport probes" { }; const solve_config = controls.SolveConfig{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth) | - jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, .controls = .{ .scattering = .multiple, .n_streams = @intCast(scene.rtm.stream_count), @@ -466,8 +465,7 @@ test "runForwardSpectrum matches canonical full spectrum" { }; const solve_config = controls.SolveConfig{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth) | - jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, .controls = .{ .scattering = .multiple, .n_streams = @intCast(scene.rtm.stream_count), @@ -652,7 +650,7 @@ test "runForwardSpectrum assembles direct-route product reflectance across worke const surface_albedo = 0.35; const solve_config = controls.SolveConfig{ .derivative_mode = .none, - .derivative_state_mask = 0, + .wants_jacobian = false, .controls = .{ .scattering = .none, .integrate_source_function = false }, }; @@ -706,14 +704,14 @@ test "runForwardSpectrum assembles direct-route product reflectance across worke null, ); try layer_depths.reduceLayerOpticsFromSupportRows(tables.layers, expected_support, expected_layers, null); - layer_depths.fillLayerAerosolJacobians(tables.aerosol, solve_config.derivative_state_mask, expected_layers); + if (solve_config.derivative_mode != .none and solve_config.wants_jacobian) { + layer_depths.fillLayerAerosolJacobians(tables.aerosol, expected_layers); + } const expected_transport = solve.directSurfaceOnly( angles, surface_albedo, testTotalOpticalDepth(expected_layers), - solve_config.derivative_mode, - solve_config.derivative_state_mask, ); expected_reflectance_sum += expected_transport.reflectance; @@ -799,7 +797,7 @@ test "gatherProductRows gathers radiance irradiance and active Jacobian lanes" { try spectrum_run.gatherProductRows( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, }, table, .{ @@ -823,7 +821,7 @@ test "gatherProductRows gathers radiance irradiance and active Jacobian lanes" { try std.testing.expectApproxEqAbs(20.0, out_radiance[0].radiance, 0.0); try std.testing.expectEqual(jacobian_states.Vector{ 2.0, 200.0 }, out_radiance[0].jacobian); try std.testing.expectApproxEqAbs(25.0, out_radiance[1].radiance, 0.0); - try std.testing.expectApproxEqAbs(0.0, out_radiance[1].jacobian[0], 0.0); + try std.testing.expectApproxEqAbs(2.5, out_radiance[1].jacobian[0], 0.0); try std.testing.expectApproxEqAbs(250.0, out_radiance[1].jacobian[1], 0.0); try std.testing.expectApproxEqAbs(200.0, out_irradiance[0], 0.0); try std.testing.expectApproxEqAbs(425.0, out_irradiance[1], 0.0); @@ -901,7 +899,7 @@ test "gatherProductRows resets stale solar memory and rejects malformed shapes" test "postprocessAndAssembleProductRows carries integrated radiance irradiance and Jacobians" { const solve_config: controls.SolveConfig = .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_layer_mid_pressure_hpa), + .wants_jacobian = true, }; const raw_radiance = [_]radiance_results.RadianceResult{ .{ .radiance = 10.0, .jacobian = .{ 1.0, 100.0 } }, @@ -933,15 +931,15 @@ test "postprocessAndAssembleProductRows carries integrated radiance irradiance a const row0_scale = std.math.pi / (52.0 * 0.25); const row1_scale = std.math.pi / (102.0 * 0.25); try std.testing.expectApproxEqAbs(21.0, out_radiance[0].radiance, 0.0); - try std.testing.expectEqual(jacobian_states.Vector{ 0.0, 200.0 }, out_radiance[0].jacobian); + try std.testing.expectEqual(jacobian_states.Vector{ 2.0, 200.0 }, out_radiance[0].jacobian); try std.testing.expectApproxEqAbs(41.0, out_radiance[1].radiance, 0.0); - try std.testing.expectEqual(jacobian_states.Vector{ 0.0, 400.0 }, out_radiance[1].jacobian); + try std.testing.expectEqual(jacobian_states.Vector{ 4.0, 400.0 }, out_radiance[1].jacobian); try std.testing.expectEqual([2]f64{ 52.0, 102.0 }, out_irradiance); try std.testing.expectApproxEqAbs(21.0 * row0_scale, out_reflectance[0], 1.0e-14); try std.testing.expectApproxEqAbs(41.0 * row1_scale, out_reflectance[1], 1.0e-14); - try std.testing.expectApproxEqAbs(0.0, out_jacobian[0][0], 0.0); + try std.testing.expectApproxEqAbs(2.0 * row0_scale, out_jacobian[0][0], 1.0e-14); try std.testing.expectApproxEqAbs(200.0 * row0_scale, out_jacobian[0][1], 1.0e-14); - try std.testing.expectApproxEqAbs(0.0, out_jacobian[1][0], 0.0); + try std.testing.expectApproxEqAbs(4.0 * row1_scale, out_jacobian[1][0], 1.0e-14); try std.testing.expectApproxEqAbs(400.0 * row1_scale, out_jacobian[1][1], 1.0e-14); try std.testing.expectEqual(@as(usize, 2), summary.sample_count); try std.testing.expectApproxEqAbs(out_reflectance[0] + out_reflectance[1], summary.reflectance_sum, 1.0e-14); @@ -950,7 +948,7 @@ test "postprocessAndAssembleProductRows carries integrated radiance irradiance a test "postprocessAndAssembleProductRows convolves non-integrated product rows" { const solve_config: controls.SolveConfig = .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth), + .wants_jacobian = true, }; const kernel = [_]f64{ 1.0, 1.0, 1.0 }; const raw_radiance = [_]radiance_results.RadianceResult{ @@ -983,16 +981,23 @@ test "postprocessAndAssembleProductRows convolves non-integrated product rows" { const expected_radiance = [_]f64{ 15.0, 70.0 / 3.0, 30.0 }; const expected_irradiance = [_]f64{ 150.0, 200.0, 250.0 }; - const expected_jacobian = [_]f64{ 1.5, 7.0 / 3.0, 3.0 }; - for (expected_radiance, expected_irradiance, expected_jacobian, 0..) |radiance, irradiance, jacobian, index| { + const expected_aod_jacobian = [_]f64{ 1.5, 7.0 / 3.0, 3.0 }; + const expected_pressure_jacobian = [_]f64{ 150.0, 700.0 / 3.0, 300.0 }; + for ( + expected_radiance, + expected_irradiance, + expected_aod_jacobian, + expected_pressure_jacobian, + 0.., + ) |radiance, irradiance, aod_jacobian, pressure_jacobian, index| { const scale = std.math.pi / (irradiance * 0.5); try std.testing.expectApproxEqAbs(radiance, out_radiance[index].radiance, 1.0e-14); - try std.testing.expectApproxEqAbs(jacobian, out_radiance[index].jacobian[0], 1.0e-14); - try std.testing.expectApproxEqAbs(0.0, out_radiance[index].jacobian[1], 0.0); + try std.testing.expectApproxEqAbs(aod_jacobian, out_radiance[index].jacobian[0], 1.0e-14); + try std.testing.expectApproxEqAbs(pressure_jacobian, out_radiance[index].jacobian[1], 1.0e-12); try std.testing.expectApproxEqAbs(irradiance, out_irradiance[index], 1.0e-14); try std.testing.expectApproxEqAbs(radiance * scale, out_reflectance[index], 1.0e-14); - try std.testing.expectApproxEqAbs(jacobian * scale, out_jacobian[index][0], 1.0e-14); - try std.testing.expectApproxEqAbs(0.0, out_jacobian[index][1], 0.0); + try std.testing.expectApproxEqAbs(aod_jacobian * scale, out_jacobian[index][0], 1.0e-14); + try std.testing.expectApproxEqAbs(pressure_jacobian * scale, out_jacobian[index][1], 1.0e-12); } try std.testing.expectEqual(@as(usize, 3), summary.sample_count); try std.testing.expectApproxEqAbs(15.0 + 70.0 / 3.0 + 30.0, summary.radiance_sum, 1.0e-14); @@ -1035,7 +1040,7 @@ test "postprocessAndAssembleProductRows rejects inconsistent product shapes" { spectrum_run.postprocessAndAssembleProductRows( .{ .derivative_mode = .semi_analytical, - .derivative_state_mask = jacobian_states.stateMask(.aerosol_optical_depth), + .wants_jacobian = true, }, true, true, @@ -1055,7 +1060,7 @@ test "postprocessAndAssembleProductRows rejects inconsistent product shapes" { } // TransportProbeEvidence -------------------------------------------------------------------------------------| -// One-wavelength transport result at one O2 A probe wavelength. | +// One-wavelength transport result at one probe wavelength. | // | // layout(64-bit) | // size: 64 B (0.062 KiB), align: 8 B | @@ -1163,7 +1168,7 @@ fn expectFullSpectrumApproxAbs( } // PublicPythonBaselineEvidence -------------------------------------------------------------------------------| -// Minimal typed view of the O2 A public Python baseline JSON used by the full-spectrum canonical test. | +// Minimal typed view of the public Python baseline JSON used by the full-spectrum canonical test. | // | // layout(64-bit) | // size: 312 B (0.305 KiB), align: 8 B | @@ -1176,7 +1181,7 @@ const PublicPythonBaselineEvidence = struct { // ------------------------------------------------------------------------------------------------------------| // PublicPythonSpectrumEvidence -------------------------------------------------------------------------------| -// Spectrum run variants recorded by the O2 A public Python baseline. | +// Spectrum run variants recorded by the public Python baseline. | // | // layout(64-bit) | // size: 312 B (0.305 KiB), align: 8 B | @@ -1216,7 +1221,7 @@ const FullSpectrumEvidence = struct { // ------------------------------------------------------------------------------------------------------------| // FullSpectrumReflectanceJacobianEvidence --------------------------------------------------------------------| -// Two reflectance-space Jacobian columns requested by the O2 A public baseline capture. | +// Two reflectance-space Jacobian columns requested by the public baseline capture. | // | // layout(64-bit) | // size: 32 B (0.031 KiB), align: 8 B | diff --git a/tests/unit/support/counting_allocator.zig b/tests/unit/support/counting_allocator.zig new file mode 100644 index 000000000..e9ebab3fc --- /dev/null +++ b/tests/unit/support/counting_allocator.zig @@ -0,0 +1,73 @@ +const std = @import("std"); + +const Allocator = std.mem.Allocator; +const Alignment = std.mem.Alignment; + +pub const CountingAllocator = struct { + child: Allocator, + alloc_calls: usize = 0, + resize_calls: usize = 0, + remap_calls: usize = 0, + free_calls: usize = 0, + live_allocations: usize = 0, + requested_bytes: usize = 0, + + pub fn init(child: Allocator) CountingAllocator { + return .{ .child = child }; + } + + pub fn allocator(self: *CountingAllocator) Allocator { + return .{ + .ptr = self, + .vtable = &.{ + .alloc = alloc, + .resize = resize, + .remap = remap, + .free = free, + }, + }; + } + + fn alloc(ctx: *anyopaque, len: usize, alignment: Alignment, ret_addr: usize) ?[*]u8 { + const self: *CountingAllocator = @ptrCast(@alignCast(ctx)); + const ptr = self.child.rawAlloc(len, alignment, ret_addr) orelse return null; + self.alloc_calls += 1; + self.live_allocations += 1; + self.requested_bytes += len; + return ptr; + } + + fn resize( + ctx: *anyopaque, + memory: []u8, + alignment: Alignment, + new_len: usize, + ret_addr: usize, + ) bool { + const self: *CountingAllocator = @ptrCast(@alignCast(ctx)); + const ok = self.child.rawResize(memory, alignment, new_len, ret_addr); + if (ok) self.resize_calls += 1; + return ok; + } + + fn remap( + ctx: *anyopaque, + memory: []u8, + alignment: Alignment, + new_len: usize, + ret_addr: usize, + ) ?[*]u8 { + const self: *CountingAllocator = @ptrCast(@alignCast(ctx)); + const ptr = self.child.rawRemap(memory, alignment, new_len, ret_addr) orelse return null; + self.remap_calls += 1; + if (new_len > memory.len) self.requested_bytes += new_len - memory.len; + return ptr; + } + + fn free(ctx: *anyopaque, memory: []u8, alignment: Alignment, ret_addr: usize) void { + const self: *CountingAllocator = @ptrCast(@alignCast(ctx)); + self.free_calls += 1; + self.live_allocations -= 1; + self.child.rawFree(memory, alignment, ret_addr); + } +}; diff --git a/tests/unit/support/o2a_scene.zig b/tests/unit/support/o2a_scene.zig index 97d47698d..550ba8056 100644 --- a/tests/unit/support/o2a_scene.zig +++ b/tests/unit/support/o2a_scene.zig @@ -29,7 +29,7 @@ const output_isotopes = [_]usize{ 1, 2, 3 }; pub fn reference() scene_input.Scene { // reference --------------------------------------------------------------------------------------------- | - // Test fixture matching the Python-owned O2 A default scene. Product code must receive scenes as input. | + // Test fixture matching the Python-owned default scene. Product code must receive scenes as input. | // --------------------------------------------------------------------------------------------------------| return .{ .id = "o2a_disamar_reference_python", @@ -138,7 +138,7 @@ fn nativeJsonPayload(scene: scene_input.Scene) NativeSceneJson { .metadata = .{ .id = "disamar_reference_o2a", .storage = "disamar-reference-o2a", - .description = "DISAMAR O2 A reference scene defined in Python.", + .description = "DISAMAR O2A reference scene defined in Python.", }, .plan = .{ .derivative_mode = "none" }, diff --git a/validation/AGENTS.md b/validation/AGENTS.md index 5bcf20130..b508514b1 100644 --- a/validation/AGENTS.md +++ b/validation/AGENTS.md @@ -1,18 +1,18 @@ # Validation -- Use this tree for tracked O2 A output-validation evidence that is intentionally committed: zdisamar versus DISAMAR reference outputs, zdisamar normal versus fast-mode outputs, and Python optimal-estimation retrieval outputs. -- Keep validation-specific O2 A maintenance scripts in this directory when they run reference cases, compare outputs, or rewrite the tracked validation bundle. +- Use this tree for tracked output-validation evidence that is intentionally committed: zdisamar versus DISAMAR reference outputs, zdisamar normal versus fast-mode outputs, and Python optimal-estimation retrieval outputs. +- Keep validation-specific maintenance scripts in this directory when they run reference scenes, compare outputs, or rewrite the tracked validation bundle. - Keep code-level invariants, API behavior checks, and algorithm unit checks under `tests/`, not in validation experiments. - Keep disposable traces, scratch runs, and exploratory validation output under `out/`. - Keep spectra validation under `validation/spectra/` and inverse-method validation under `validation/optimal_estimation/`. -- Keep shared O2 A validation setup under `validation/o2a/`; reserve +- Keep shared validation setup under `validation/o2a/`; reserve `validation/common/` for generic validation plumbing. - Keep validation-owned reference fixtures under `validation/reference_data/`. - Use Altair for validation plots. Do not add Matplotlib back to validation scripts. - Keep DISAMAR baseline retrieval configs on `aerosolLayerHeight 0`. `aerosolLayerHeight 1` activates a Fortran DISAMAR shortcut path, and these validation lanes are meant to compare zdisamar against the normal physical inverse problem instead. - Keep tracked benchmark outputs under `validation/outputs/`; these files are generated evidence and should only be refreshed by the validation scripts. - Invoke validation scripts directly with `uv run ...`; do not add one-off Python validation or plotting scripts to `zig build`. -- The committed O2 A spectra comparison evidence lives under `validation/outputs/spectra/`. Regenerate it with `uv run validation/spectra/validate_spectra.py`; do not hand-edit its generated contents. +- The committed O2A spectra comparison evidence lives under `validation/outputs/spectra/`. Regenerate it with `uv run validation/spectra/validate_spectra.py`; do not hand-edit its generated contents. - Validation thresholds are sacred contracts, not tuning knobs. Do not loosen a threshold to make a failing lane pass. If a threshold fails, fix the configuration alignment, reference generation, implementation, or explicitly diff --git a/validation/README.md b/validation/README.md index 2a6277cfc..d98d271aa 100644 --- a/validation/README.md +++ b/validation/README.md @@ -1,9 +1,9 @@ # Validation Assets -This directory stores tracked O2 A output-validation evidence that is +This directory stores tracked output-validation evidence that is intentionally kept in git. The scripts compare zdisamar outputs against DISAMAR reference outputs, zdisamar normal-mode outputs against fastmode outputs, and -Python optimal-estimation retrieval outputs against retained reference cases. +Python optimal-estimation retrieval outputs against retained reference scenes. Disposable validation traces, scratch runs, and exploratory analysis belong under `out/`, not under this directory. @@ -13,7 +13,7 @@ The validation tree is split by target: - `spectra/`: forward reflectance and reflectance-Jacobian parity evidence. - `optimal_estimation/`: inverse-method retrieval evidence. -- `o2a/`: shared O2 A scene, baseline, and measurement-noise helpers used by +- `o2a/`: shared scene, baseline, and measurement-noise helpers used by both spectra and inverse-method validation. - `common/`: generic validation plumbing such as paths and timers. - `reference_data/`: committed DISAMAR/reference fixtures and baselines. @@ -36,7 +36,7 @@ The validation tree is split by target: - `outputs/spectra/o2a_fast_mode_spectra.png`, `outputs/spectra/o2a_fast_mode_spectra_data.csv`, and `outputs/spectra/o2a_fast_mode_spectra_metrics.json`: retained comparison of - reference settings against the O2 A fastmode preset over several scene + reference settings against the fastmode preset over several scene geometries. - `reference_data/spectra/o2a_with_cia_disamar_reference.csv`: retained DISAMAR reference spectrum still consumed by older forward validation tests. @@ -94,9 +94,9 @@ The tracked validation plot compares reflectance-space quantities because those are the retrieval-facing parity signal. The retained threshold for every row in the bundle is `1e-13` max absolute residual. -## Baseline Case +## Baseline Scene -Validation lanes use the typed O2 A baseline exposed by `zdisamar.wavelength_bands.o2a.reference_scene()` +Validation lanes use the typed baseline exposed by `zdisamar.wavelength_bands.o2a.reference_scene()` and mirrored in `validation/o2a/case.py`. Keep Zig validation -tests and Python validation scripts aligned with that case instead of introducing +tests and Python validation scripts aligned with that scene instead of introducing a second serialized validation input. diff --git a/validation/o2a/baseline.py b/validation/o2a/baseline.py index d338ac8e1..d4f9a5e52 100644 --- a/validation/o2a/baseline.py +++ b/validation/o2a/baseline.py @@ -1,4 +1,4 @@ -"""Shared O2 A optimal-estimation retrieval baseline settings.""" +"""Shared optimal-estimation retrieval baseline settings.""" WAVELENGTH_START_NM = 758.0 WAVELENGTH_END_NM = 770.0 @@ -30,7 +30,7 @@ def configure_case(case) -> None: - """Apply the retrieval baseline to a mutable zdisamar O2 A case.""" + """Apply the retrieval baseline to a mutable zdisamar scene.""" case.spectral_grid.start_nm = WAVELENGTH_START_NM case.spectral_grid.end_nm = WAVELENGTH_END_NM diff --git a/validation/o2a/case.py b/validation/o2a/case.py index aeff788d5..1dd01ae68 100644 --- a/validation/o2a/case.py +++ b/validation/o2a/case.py @@ -1,10 +1,10 @@ -"""Canonical O2 A validation scene construction.""" +"""Canonical validation scene construction.""" from dataclasses import replace def build_o2a_case(band, *, jacobian_reference_layer: bool = False): - """Return the packaged O2 A case, with optional Jacobian-validation geometry.""" + """Return the packaged scene, with optional Jacobian-validation geometry.""" case = band.reference_scene() @@ -75,7 +75,7 @@ def build_o2a_case(band, *, jacobian_reference_layer: bool = False): def build_o2a_jacobian_case(band): - """Return the O2 A validation case used by retained Jacobian comparisons.""" + """Return the validation scene used by retained Jacobian comparisons.""" case = build_o2a_case(band, jacobian_reference_layer=True) @@ -84,7 +84,7 @@ def build_o2a_jacobian_case(band): metadata={ "id": "disamar_reference_o2a_jacobian_validation", "storage": "disamar-reference-o2a-jacobian-validation", - "description": "DISAMAR O2 A Jacobian validation case.", + "description": "DISAMAR O2A Jacobian validation case.", }, scene_id="o2a_disamar_reference_jacobian_validation", instrument_response=replace( diff --git a/validation/o2a/measurement_noise.py b/validation/o2a/measurement_noise.py index fc4f1bb7e..7d3459eb2 100644 --- a/validation/o2a/measurement_noise.py +++ b/validation/o2a/measurement_noise.py @@ -1,11 +1,10 @@ -"""Retained O2 A baseline measurement-noise helpers.""" +"""Retained baseline measurement-noise helpers.""" from dataclasses import dataclass from functools import cache import numpy as np -from zdisamar import reference_data, rtm -from zdisamar.inverse_method import optimal_estimation +from zdisamar import optimal_estimation, reference_data, rtm from validation.o2a import baseline @@ -32,7 +31,7 @@ def snr_table(self) -> tuple[list[float], list[float]]: def measurement_from_o2a_baseline_noise(case) -> optimal_estimation.Measurement: - """Build a reflectance measurement with retained O2 A baseline SNR semantics.""" + """Build a reflectance measurement with retained baseline SNR semantics.""" spectrum = rtm.spectrum(case) wavelength_nm = np.asarray(spectrum.wavelength_nm, dtype=np.float64).copy() @@ -61,7 +60,7 @@ def components_from_spectrum( irradiance, reflectance, ) -> O2ANoiseComponents: - """Return SNR components for one O2 A spectrum.""" + """Return SNR components for one O2A spectrum.""" wavelength = _one_dimensional_array("wavelength_nm", wavelength_nm) radiance_values = _positive_array("radiance", radiance, wavelength.size) @@ -114,7 +113,7 @@ def components_from_spectrum( def instrument_mapped_s5_reference_radiance( wavelength_nm, ) -> np.ndarray: - """Map the S5 reference radiance spectrum onto the O2 A instrument grid.""" + """Map the S5 reference radiance spectrum onto the instrument grid.""" target = _one_dimensional_array("wavelength_nm", wavelength_nm) reference_wavelength, reference_radiance = _load_s5_reference_radiance() diff --git a/validation/optimal_estimation/benchmark_slow_rtm_jacobian.py b/validation/optimal_estimation/benchmark_slow_rtm_jacobian.py index 96439f9e6..f038c641d 100644 --- a/validation/optimal_estimation/benchmark_slow_rtm_jacobian.py +++ b/validation/optimal_estimation/benchmark_slow_rtm_jacobian.py @@ -24,7 +24,7 @@ sync_release_fast_binding(REPO_ROOT) from zdisamar import rtm # noqa: E402 -from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe # noqa: E402 +from zdisamar.optimal_estimation import o2a as o2a_oe # noqa: E402 from zdisamar.wavelength_bands import o2a # noqa: E402 from validation.common.paths import write_json # noqa: E402 @@ -63,10 +63,7 @@ def stats(values: list[float]) -> dict[str, float]: } -def probe_rtm_calls( - probe_case: o2a.Scene, - jacobian_names: tuple[str, ...], -) -> dict[str, Any]: +def probe_rtm_calls(probe_case: o2a.Scene) -> dict[str, Any]: # Cold no-session forwards: a warm SessionCache fully memoizes the forward # result, so repeated identical calls would measure cache hits, not RTM @@ -78,11 +75,7 @@ def radiance_reflectance_only() -> None: def radiance_reflectance_and_jacobian() -> None: - spectrum = rtm.spectrum( - probe_case, - jacobian=True, - jacobian_state_names=jacobian_names, - ) + spectrum = rtm.spectrum(probe_case, jacobian=True) _ = list(spectrum.reflectance) rtm_only_s = [time_call(radiance_reflectance_only) for _ in range(PROBE_RUNS)] @@ -147,7 +140,7 @@ def main() -> int: _ = result.final_evaluation lazy_final_evaluation_s = time.perf_counter() - lazy_final_start - probe = probe_rtm_calls(probe_case, state_vector.jacobian_names) + probe = probe_rtm_calls(probe_case) finally: cache.close() @@ -155,7 +148,7 @@ def main() -> int: "validation_case": "zdisamar_o2a_slow_rtm_jacobian_latency", "source": "Slow retained scene from shared DISAMAR OE reference sweep case 71.", "regime": ( - "Baseline O2 A per-wavelength noise; two-state aerosol retrieval " + "Baseline per-wavelength noise; two-state aerosol retrieval " "(AOD, mid-pressure); retrieval_controls(); reused SessionCache for " "the retrieval; cold no-session forwards for direct_rtm_call_probe." ), diff --git a/validation/optimal_estimation/multistart_fast_mode_retrieval.py b/validation/optimal_estimation/multistart_fast_mode_retrieval.py index cac780402..c013e9e09 100644 --- a/validation/optimal_estimation/multistart_fast_mode_retrieval.py +++ b/validation/optimal_estimation/multistart_fast_mode_retrieval.py @@ -9,7 +9,7 @@ # ] # /// -"""Prototype multi-start O2 A fast-mode optimal-estimation retrieval outputs.""" +"""Prototype multi-start fast-mode optimal-estimation retrieval outputs.""" import argparse import copy @@ -34,9 +34,8 @@ os.environ.setdefault(NATIVE_WORKER_LIMIT_ENV, str(DEFAULT_NATIVE_WORKER_LIMIT)) sys.path[:0] = [str(REPO_ROOT), str(PYTHON_ROOT)] -from zdisamar import rtm # noqa: E402 -from zdisamar.inverse_method import optimal_estimation # noqa: E402 -from zdisamar.inverse_method.optimal_estimation.diagnosis import ( # noqa: E402 +from zdisamar import optimal_estimation, rtm # noqa: E402 +from zdisamar.optimal_estimation.diagnosis import ( # noqa: E402 diagnose_retrieval, diagnosis_batch_worker_count_for_limit, ) @@ -72,7 +71,7 @@ def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="Run multi-start fast-mode O2 A optimal-estimation retrievals." + description="Run multi-start fast-mode optimal-estimation retrievals." ) parser.add_argument("--scene-count", type=int, default=DEFAULT_SCENE_COUNT) parser.add_argument( @@ -1034,7 +1033,7 @@ def build_summary( "method": "case.optimisation.fastmode.enabled = True", "fast_stage_only": fast_stage_only, "note": ( - "Uses the case-owned fast-mode OE path, including sparse fast-stage " + "Uses the scene-owned fast-mode OE path, including sparse fast-stage " "sampling. The sparse full-physics final correction is disabled only " "when --fast-stage-only is passed." ), diff --git a/validation/optimal_estimation/reference_cases.py b/validation/optimal_estimation/reference_cases.py index 90dd3cf3c..225cd649f 100644 --- a/validation/optimal_estimation/reference_cases.py +++ b/validation/optimal_estimation/reference_cases.py @@ -1,4 +1,4 @@ -"""Shared O2 A optimal-estimation reference sweep cases.""" +"""Shared optimal-estimation reference sweep scenes.""" import json from typing import Any @@ -44,7 +44,7 @@ def truth_scenes(*, count: int | None = None) -> list[dict[str, float]]: if requested > available: raise ValueError( - f"requested {requested} OE reference cases but manifest only samples {available}" + f"requested {requested} OE reference scenes but manifest only samples {available}" ) return oe_setup.sampled_scenes(available, int(payload["seed"]))[:requested] diff --git a/validation/optimal_estimation/setup.py b/validation/optimal_estimation/setup.py index ce5505141..a2ce9ebe9 100644 --- a/validation/optimal_estimation/setup.py +++ b/validation/optimal_estimation/setup.py @@ -1,11 +1,11 @@ -"""Shared O2 A optimal-estimation validation setup.""" +"""Shared optimal-estimation validation setup.""" import copy import math from typing import Any import numpy as np -from zdisamar.inverse_method import optimal_estimation +from zdisamar import optimal_estimation from validation.o2a import baseline as oe_baseline diff --git a/validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py b/validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py index 9c09a4be5..1380e1ae1 100644 --- a/validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py +++ b/validation/optimal_estimation/sweep_fast_mode_optimal_estimation.py @@ -9,7 +9,7 @@ # ] # /// -"""Run retained O2 A fastmode optimal-estimation sweep outputs.""" +"""Run retained fastmode optimal-estimation sweep outputs.""" import copy import math @@ -31,7 +31,7 @@ sync_release_fast_binding(REPO_ROOT) -from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe # noqa: E402 +from zdisamar.optimal_estimation import o2a as o2a_oe # noqa: E402 from zdisamar.plot.properties import PLOT # noqa: E402 from zdisamar.wavelength_bands import o2a # noqa: E402 @@ -376,9 +376,9 @@ def build_summary(data: pd.DataFrame) -> dict[str, Any]: "method": "case.optimisation.fastmode.enabled = True", "defaults": fastmode_defaults(), "note": ( - "Reference and fastmode rows use the same deterministic zdisamar O2 A " - "optimal-estimation sweep cases, measurement vectors, priors, and initial " - "states. Fastmode enables case-owned RTM controls, sparse fast-stage OE " + "Reference and fastmode rows use the same deterministic zdisamar " + "optimal-estimation sweep scenes, measurement vectors, priors, and initial " + "states. Fastmode enables scene-owned RTM controls, sparse fast-stage OE " "wavelength sampling, and the default sparse full-physics OE correction." ), }, @@ -1020,7 +1020,7 @@ def create_plot(data: pd.DataFrame, output_path: Path) -> None: spacing=34, ).properties( title={ - "text": "O2A Optimal-Estimation Sweep: zdisamar Fullmode and Fastmode", + "text": "Optimal-Estimation Sweep: zdisamar Fullmode and Fastmode", "subtitle": ( "Each paired scene uses identical measurements, priors, and initial states; " "error bars are posterior 1-sigma for retrieved states and combined " diff --git a/validation/optimal_estimation/sweep_optimal_estimation.py b/validation/optimal_estimation/sweep_optimal_estimation.py index cc90ca957..4f09171ed 100644 --- a/validation/optimal_estimation/sweep_optimal_estimation.py +++ b/validation/optimal_estimation/sweep_optimal_estimation.py @@ -30,7 +30,7 @@ sync_release_fast_binding(REPO_ROOT) from zdisamar import rtm # noqa: E402 -from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe # noqa: E402 +from zdisamar.optimal_estimation import o2a as o2a_oe # noqa: E402 from zdisamar.wavelength_bands import o2a # noqa: E402 from validation.common.paths import stable_repo_path, write_json # noqa: E402 diff --git a/validation/spectra/validate_fast_mode_spectra.py b/validation/spectra/validate_fast_mode_spectra.py index facf79087..349d3b882 100644 --- a/validation/spectra/validate_fast_mode_spectra.py +++ b/validation/spectra/validate_fast_mode_spectra.py @@ -9,7 +9,7 @@ # ] # /// -"""Validate retained O2 A fast-mode spectra outputs.""" +"""Validate retained fast-mode spectra outputs.""" import copy import sys @@ -420,7 +420,8 @@ def write_metrics(metrics: list[dict[str, Any]], output_path: Path) -> None: "method": "case.optimisation.fastmode.enabled = True", "overrides": fast_mode_overrides(), "note": ( - "Fast mode preserves each case's physical scene and output wavelength grid, " + "Fast mode preserves each scene's physical configuration and output " + "wavelength grid, " "then applies validated radiative-transfer and adaptive-reference-grid " "performance overrides." ), diff --git a/validation/spectra/validate_spectra.py b/validation/spectra/validate_spectra.py index 6d2f082a3..47a631ee1 100644 --- a/validation/spectra/validate_spectra.py +++ b/validation/spectra/validate_spectra.py @@ -90,7 +90,7 @@ class MetricRow(TypedDict): def run_zdisamar_validation(case) -> dict[str, np.ndarray]: - spectrum = rtm.spectrum(case, jacobian=True, jacobian_state_names=STATE_NAMES) + spectrum = rtm.spectrum(case, jacobian=True) wavelength_nm = np.asarray(spectrum.wavelength_nm, dtype=np.float64).copy() reflectance = np.asarray(spectrum.reflectance, dtype=np.float64).copy() state_names = spectrum.jacobian_state_names @@ -110,7 +110,7 @@ def run_zdisamar_validation(case) -> dict[str, np.ndarray]: def mid_pressure_jacobian_scale(case) -> float: - from zdisamar.inverse_method.optimal_estimation import o2a as o2a_oe + from zdisamar.optimal_estimation import o2a as o2a_oe profile = o2a_oe.pressure_altitude_profile_from_scene(case) @@ -403,7 +403,7 @@ def write_manifest(output_path: Path) -> None: "policy": { "validation_case": "hardcoded_o2a_jacobian_reference", "note": ( - "The tracked O2 A validation plot is generated by the " + "The tracked validation plot is generated by the " "Python API and compares zdisamar reflectance plus " "reflectance Jacobians against committed DISAMAR reference " "derivatives. Thresholds apply to the interior instrument " diff --git a/verify b/verify index a7305ebce..125c69163 100755 --- a/verify +++ b/verify @@ -1,20 +1,68 @@ #!/usr/bin/env bash -set -euo pipefail +# verify — run every gate, but overlap them instead of summing their wall time. +# +# The sub-commands are unchanged: same Zig retained tests, same Python tests, +# same pre-commit hooks, same whitespace checks. Only the scheduling changed. +# +# Why: each gate already saturates most cores on its own, yet the old script ran +# them strictly one after another, so the total wall time was the SUM. The only +# true ordering constraint is that the Python tests need the freshly synced +# native .so. Everything else is independent, so we run four lanes concurrently: +# +# lane 1 sync the native package -> run the Python tests against it +# lane 2 Zig retained test suite +# lane 3 pre-commit hooks (lint / format / type / styleguide) +# lane 4 whitespace checks +# +# We buffer each lane's output and print it when every lane has finished, then +# exit non-zero if any lane failed (showing which). Nothing is skipped or +# weakened. Written for bash 3.2 (macOS default): no associative arrays. +set -uo pipefail cd "$(git rev-parse --show-toplevel)" -echo "== zig retained tests" -zig build test +log_dir=$(mktemp -d "${TMPDIR:-/tmp}/verify.XXXXXX") +trap 'rm -rf "$log_dir"' EXIT -echo "== sync python package" -zig build sync-python-package -Doptimize=ReleaseFast +# Lane 1: native Python boundary. sync must finish before pytest, so this lane +# chains them; pytest sees the .so this run just built. +lane_native_python() { + zig build sync-python-package -Doptimize=ReleaseFast && uv run pytest tests/python +} +# Lane 2: Zig retained tests (independent of the Python lane). +lane_zig_tests() { zig build test; } +# Lane 3: pre-commit hooks. +lane_precommit() { uv run prek run --all-files; } +# Lane 4: whitespace checks. +lane_whitespace() { git diff --check && git diff --cached --check; } -echo "== python tests" -uv run pytest tests/python +lanes=(zig_tests native_python precommit whitespace) -echo "== pre-commit hooks" -uv run prek run --all-files +pids=() +for name in "${lanes[@]}"; do + "lane_$name" >"$log_dir/$name.log" 2>&1 & + pids+=("$!") +done -echo "== whitespace checks" -git diff --check -git diff --cached --check +fail=0 +rcs=() +for i in "${!lanes[@]}"; do + if wait "${pids[$i]}"; then rcs+=(0); else rcs+=("$?"); fi +done + +for i in "${!lanes[@]}"; do + name=${lanes[$i]} + echo "== $name" + cat "$log_dir/$name.log" + if [ "${rcs[$i]}" -ne 0 ]; then + echo "== $name FAILED (exit ${rcs[$i]})" + fail=1 + fi + echo +done + +if [ "$fail" -ne 0 ]; then + echo "verify: FAILED" + exit 1 +fi +echo "verify: all gates passed"