Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ scratch/
specs/
docs/specs/

# Disposable validation output. Tracked O2 A evidence lives directly in validation/.
# Disposable validation output. Tracked evidence lives directly in validation/.
validation/.cache/
validation/function_diff/
validation/latest/
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Repo Notes

- `zdisamar` is an O2 A RTM lab. Treat DISAMAR as the reference family used for validation, not as the codebase architecture.
- `zdisamar` is an O2A RTM lab. Treat DISAMAR as the reference family used for validation, not as the codebase architecture.
- Keep the public flow simple: input -> RTM -> output.
- Keep routines under `src/rtm/`, `src/optics/`, `src/spectrum/`, and `src/retrieval/` free of file I/O, CLI wiring, text parsing, and hidden global state.
- Keep scientific assets under `data/reference_data/`; loaders and parsers live under `src/input/reference_data/`.
Expand Down
6 changes: 3 additions & 3 deletions DISAMAR_IMPLEMENTATION_DIFFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ zdisamar links use the current GitHub PR branch.

| Area | Fortran does | zdisamar does | Why | Impact |
| --- | --- | --- | --- | --- |
| Polarization | Can model scalar light and polarized light. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Uses the scalar O2 A path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L326-L452) | The current O2 A validation target is scalar. | Faster and smaller. This route does not produce polarized-light output. |
| Polarization | Can model scalar light and polarized light. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Uses the scalar path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L326-L452) | The current validation target is scalar. | Faster and smaller. This route does not produce polarized-light output. |
| Scattering angle basis | Stores separate values for positive and negative viewing directions. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1065-1517) | Stores one side and derives the other when needed. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L127-L305) | In the scalar case, the two sides are mathematically linked. | Less stored data, with the same scalar result. |
| Scattering matrix work | Builds full scattering matrices. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1520-1762) | Builds the full matrix when layers need it, but builds only one row when reflectance or aerosol weighting only reads one row. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L584-L689) | Some later steps only use one row. | Avoids matrix work that would be thrown away. |
| Gas and aerosol scattering | Carries a more general phase-matrix representation. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1549-1562) | Combines gas and aerosol scattering into the scalar form needed by the current layer or interface. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/phase_basis.zig#L454-L582) | The scalar model only needs the combined scattering shape at the point being computed. | Keeps the layer math smaller while still carrying gas and aerosol contributions. |
| Path attenuation storage | Stores attenuation for many direction and level pairs. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L971-1062) | Uses either full path storage or a smaller storage layout, depending on what the route will read. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/attenuation.zig#L93-L231) | Not every route reads every path. | Saves memory on the common route, while keeping full lookup available when needed. |
| Curved direct beam | Starts from a flat-layer direct beam, then corrects the top-of-atmosphere path for curvature. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1005-1025) | Uses a fast curved-path calculation for normal support grids, and a slower fallback for larger grids. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/attenuation.zig#L644-L725) | The solar beam bends through a spherical atmosphere. | Keeps the spherical correction, without letting large grids overflow fixed local storage. |
| Layer doubling | Decides whether each layer needs the more expensive doubling calculation. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L1855-1908) | Makes the same kind of decision using an explicit cutoff before choosing the cheap or expensive layer path. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/layer_reflect_transmit.zig#L1280-L1339) | Very thin weak-scattering layers often do not need the expensive path. | Faster layer builds. The cutoff must stay covered by validation because it can slightly affect results. |
| Small matrix multiply | Uses general small-matrix routines. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3528-3720) | Adds fixed-size routines for the common O2 A matrix shape. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L384-L595) | The common O2 A shape is known before runtime. | Less loop overhead and better compiler output in the hot matrix path. |
| Small matrix multiply | Uses general small-matrix routines. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3528-3720) | Adds fixed-size routines for the common matrix shape. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L384-L595) | The common shape is known before runtime. | Less loop overhead and better compiler output in the hot matrix path. |
| Multiple reflection solve | Computes the repeated-reflection correction in a general helper. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L3616-3720) | Solves the main repeated-reflection part first, then fills the viewing and solar pieces around it. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/matrix_12x10.zig#L2023-L2537) | Most repeated reflection happens through the Gaussian stream block. | Makes the optimized solve match the physical block structure. |
| Local source fields | Computes local radiation fields for source-function work. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/addingToolsModule.f90#L1667-1816) | Carries those fields only on routes that need source integration or aerosol Jacobians. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/scattering_orders.zig#L1394-L1518) | Direct reflectance does not use them. | Simpler routes avoid extra arrays and transport work. |
| Jacobians | Computes many weighting functions for gases, aerosols, clouds, absorption, scattering, and interface movement. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L2699-3525) | Implements the derivative outputs used by the current O2 A model. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/reflectance.zig#L460-L858) | Unsupported derivative outputs should not appear as if they are supported. | Public Jacobians are narrower, but less misleading. |
| Jacobians | Computes many weighting functions for gases, aerosols, clouds, absorption, scattering, and interface movement. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L2699-3525) | Implements the derivative outputs used by the current model. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/rtm/reflectance.zig#L460-L858) | Unsupported derivative outputs should not appear as if they are supported. | Public Jacobians are narrower, but less misleading. |
| Source quadrature | Builds layer-interface source weighting inside the radiative-transfer flow. [code](https://gitlab.com/KNMI-OSS/disamar/disamar/-/blob/d17c52884a875cb87b98e4c4ea7f722659e685ac/src/LabosModule.f90#L92-105) | Prepares the within-layer source samples before LABOS runs. [code](https://github.com/bout3fiddy/zdisamar/blob/codex/explicit-dataflow-refactor/src/optics/source_levels.zig#L76-L318) | Aerosol source weighting needs samples inside the layer, not only at layer edges. | LABOS gets ready-to-use scalar source data. |
2 changes: 1 addition & 1 deletion NOTICE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
zdisamar is an independent implementation of the O2 A radiative-transfer
zdisamar is an independent implementation of the O2A radiative-transfer
problem used in DISAMAR-family validation studies. The Fortran DISAMAR project
is a scientific reference for validation; this repository does not include
DISAMAR source code.
Expand Down
57 changes: 28 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ spectral response all affect the measured spectrum.

The Fortran DISAMAR code family is the scientific reference for this work.
`zdisamar` keeps the same radiative-transfer problem and reorganizes the
repeated oxygen A-band calculations so validation cases can be run, timed, and
repeated oxygen A-band calculations so validation scenes can be run, timed, and
inspected through generated spectra and timing files.

The Python wrapper is demonstrated in executable notebooks under
Expand All @@ -26,9 +26,9 @@ uv run --with jupyterlab --with ipykernel python -m jupyter lab scripts/demo

The two demos are
[`o2a_plot_bundle.ipynb`](./scripts/demo/o2a_plot_bundle.ipynb), which shows the
Python-facing O2 A output and plotting accessors, and
Python-facing output and plotting accessors, and
[`optimal_estimation_demo.ipynb`](./scripts/demo/optimal_estimation_demo.ipynb),
which shows a two-state O2 A optimal-estimation flow.
which shows a two-state optimal-estimation flow.

The published Python package has no third-party runtime dependencies: the RTM,
optimal-estimation helpers, and notebook-facing SVG plots ship with the native
Expand All @@ -53,7 +53,7 @@ clouds, so aerosol retrievals need a detailed RTM.

![Aerosol scene and oxygen A-band reflectance](./docs/assets/o2a-aerosol-spectrum-context.png)

The figure links a visible aerosol scene to the O2 A reflectance spectrum seen
The figure links a visible aerosol scene to the O2A reflectance spectrum seen
by the instrument: the aerosol contribution changes both the absolute
reflectance and the structure inside the absorption band. Aerosol optical
thickness, aerosol vertical distribution, and surface reflection all affect the
Expand Down Expand Up @@ -81,11 +81,11 @@ it reads a retrieval configuration, prepares atmospheric and surface inputs,
calculates spectra and Jacobians, and runs inverse methods such as optimal
estimation. Its strength is breadth. It supports many retrieval families,
spectral ranges, configuration options, and operational/research use cases. That
breadth also makes focused O2 A benchmarking difficult: a single aerosol-height
case still passes through general setup, broad configuration handling, and
breadth also makes focused benchmarking difficult: a single aerosol-height
scene still passes through general setup, broad configuration handling, and
general numerical routines built for a much wider set of retrieval problems.

Both implementations target the same O2 A retrieval RTM: line-by-line
Both implementations target the same retrieval RTM: line-by-line
oxygen absorption, multiple scattering, instrument-grid convolution, and
reflectance derivatives for optimal estimation. The performance improvements
come from reducing repeated setup around that calculation:
Expand All @@ -94,17 +94,17 @@ come from reducing repeated setup around that calculation:
across repeated RTM calls;
- optimal-estimation retrievals call the RTM several times while the
scene, instrument grid, spectroscopy, and many optical inputs stay the same.
Each iteration reuses that O2 A calculation state in memory;
Each iteration reuses that calculation state in memory;
- retrieval Jacobians are calculated for the active state-vector columns;
- common O2 A LABOS matrix shapes for the 20-stream case use specialized
- common LABOS matrix shapes for the 20-stream case use specialized
implementations in the repeated layer-doubling calculations;
- validation and benchmark evidence is stored under
[`validation/outputs/`](./validation/outputs/).

The benchmark cases use `nstreamsSim = 20` and `nstreamsRetr = 20`. Streams are
The benchmark scenes use `nstreamsSim = 20` and `nstreamsRetr = 20`. Streams are
the angular quadrature directions used by the multiple-scattering
radiative-transfer solver; more streams resolve the angular radiation field more
finely, but each RTM call costs more. The production DISAMAR O2 A
finely, but each RTM call costs more. The production DISAMAR
setup usually uses 16 streams, so these retained 20-stream timings are
deliberately slower than a production-tuned Fortran run.

Expand All @@ -122,12 +122,12 @@ optimal-estimation retrieval timing.
The timings in this section were recorded on the local benchmark machine: Mac
mini `Mac16,10`, Apple M4, 10 CPU cores (4 performance, 6 efficiency), 24 GB
RAM. Treat the absolute seconds as machine-specific wall-clock measurements; the
linked validation artifacts are the source for the reported case counts,
linked validation artifacts are the source for the reported scene counts,
retrieved-state deltas, and timing summaries.

### RTM

The RTM benchmark calculates one O2 A spectrum over 755-776 nm. The
The RTM benchmark calculates one O2A spectrum over 755-776 nm. The
reported spectrum has 701 instrument-grid wavelengths, but each instrument
channel is an average over sharper oxygen absorption structure at higher
spectral resolution:
Expand Down Expand Up @@ -172,8 +172,8 @@ aerosol optical depth: median +1.688e-08, mean -3.025e-07, range -3.703e-0
aerosol mid pressure [hPa]: median -0.0016, mean -0.0020, range -0.0522 to +0.0821
```

Fastmode retrieval is a zdisamar-only optimisation lane on the same O2 A case.
The normal case remains the full-physics reference. Enabling
Fastmode retrieval is a zdisamar-only optimisation lane on the same scene.
The normal scene remains the full-physics reference. Enabling
`case.optimisation.fastmode.enabled` resolves inspectable RTM, adaptive-grid,
OE, sparse fast-stage wavelength sampling, and final-correction defaults before
execution. The shipped fastmode path solves the fast stage on the sparse
Expand All @@ -192,7 +192,7 @@ correction wavelengths on the validation measurement grid. Median speedup versus
fullmode is `+1.382 s`. The maximum fastmode-minus-fullmode deltas are
`4.197e-04` aerosol optical depth and `0.551 hPa` aerosol mid pressure.
These timings are wall-clock durations around the public retrieval call. They
include session/cache creation, native case load and preparation, native OE work,
include session/cache creation, native scene load and preparation, native OE work,
and the sparse full-physics correction; they do not include scene construction,
simulated measurement construction, CSV writing, or plot rendering.
The technical note is
Expand All @@ -215,13 +215,13 @@ OE 5-case fastmode sweep retrieval total 4.516 s

That benchmark artifact is
[`benchmark/results.json`](./benchmark/results.json). It is a shorter
production-path timing check; the retained 100-case validation sweep above
production-path timing check; the retained 100-scene validation sweep above
remains the accuracy and convergence contract.

The wrapper-overhead probe used the 10-worker baseline case directly. Under that
The wrapper-overhead probe used the 10-worker baseline scene directly. Under that
boundary, the current one-shot fastmode public call is `0.230 s` median, and a
repeated-start loop with a caller-owned session cache is `0.179 s` median after
the first sparse-case load. That focused evidence is tracked in
the first sparse-scene load. That focused evidence is tracked in
[`scaffolding/reports/performance/retrieval/fastmode-session-overhead.md`](./scaffolding/reports/performance/retrieval/fastmode-session-overhead.md).

The tracked paired DISAMAR/zdisamar summary is
Expand Down Expand Up @@ -256,26 +256,26 @@ counts are in
## Python Package

Use `uv` to download the published package into an isolated environment. The
wheel includes the native RTM library and bundled O2 A reference data. The
wheel includes the native RTM library and bundled reference data. The
Python package has no third-party runtime dependencies.

```bash
uv run --with zdisamar python
```

The public flow is one case object: configure the simulated O2 A scene, pass it
The public flow is one scene object: configure the simulated scene, pass it
to the RTM, and pass it to optimal estimation when you want a retrieval.

```python
from zdisamar import rtm
from zdisamar.inverse_method import optimal_estimation as oe
from zdisamar import optimal_estimation as oe
from zdisamar.rtm import SessionCache
from zdisamar.wavelength_bands import o2a
```

### Simulated Scene

Start from the packaged O2 A reference case and set the aerosol state directly.
Start from the packaged reference scene and set the aerosol state directly.

```python
case = o2a.reference_scene()
Expand All @@ -285,7 +285,7 @@ case.aerosol_layer.mid_pressure_hpa = 760.0

### Forward Model

Run the native forward model on the case. The result exposes wavelength,
Run the native forward model on the scene. The result exposes wavelength,
radiance, irradiance, reflectance, optional Jacobians, and dependency-free SVG
plot accessors.

Expand All @@ -296,7 +296,6 @@ spectrum.plot.reflectance()
jacobian_spectrum = rtm.spectrum(
case,
jacobian=True,
jacobian_state_names=("aerosol_optical_depth",),
)
jacobian_spectrum.plot.jacobian("aerosol_optical_depth")
```
Expand Down Expand Up @@ -330,9 +329,9 @@ profile_spectrum = rtm.spectrum(profile_case)

### Fastmode Controls

Fastmode is a case-owned optimisation mode. The case keeps the physical scene
and exposes the RTM, adaptive-grid, OE, sparse fast-stage, and final-correction
controls in one place.
Fastmode is a scene-owned optimisation mode. The scene keeps the physical
configuration and exposes the RTM, adaptive-grid, OE, sparse fast-stage, and
final-correction controls in one place.

```python
fast_case = o2a.reference_scene()
Expand All @@ -359,7 +358,7 @@ fastmode.oe.final_correction.wavelength_count = 4

### Optimal Estimation

Create a simulated measurement from a truth case. Public measurement noise is
Create a simulated measurement from a truth scene. Public measurement noise is
expressed as signal-to-noise ratio; pass one scalar SNR or one value per
wavelength. State-vector prior uncertainty is the one-sigma prior spread in the
same units as the state value.
Expand Down
Loading