Skip to content

RUST_BACKEND.md

Latest commit

 

History

History
472 lines (400 loc) · 23.6 KB

RUST_BACKEND.md

File metadata and controls

472 lines (400 loc) · 23.6 KB
Raw

Rust-primary backend

SHARPpy Reimagined supports sharpmod_rs as its primary numerical and direct pressure-level GRIB point-decoding backend. Official Windows executables bundle the extension, and default auto mode selects Rust after validating its package version, backend API, and required operations. Rust works behind the Python facade; the PySide6 GUI, renderer, command-line tools, SHARPpy widget stack, model retrieval, and profile orchestration remain shared Python application layers.

The independently optimized Python backend remains fully functional and is the portable fallback. Source and Python-only installations therefore do not need Rust, Cargo, or maturin, and a missing or unloadable extension does not prevent the application from starting in default auto mode.

Backend selection

Set SHARPMOD_BACKEND before starting Python:

Value Behavior
auto Default. Use Rust when the extension loads successfully; otherwise use Python and retain the fallback reason for diagnostics.
python Always use the optimized Python implementation. The Rust extension is not required.
rust Require the Rust implementation. Missing or unloadable native code produces a clear error instead of silently falling back.

PowerShell examples:

$env:SHARPMOD_BACKEND = "auto"
$env:SHARPMOD_BACKEND = "python"
$env:SHARPMOD_BACKEND = "rust"

POSIX shell examples:

export SHARPMOD_BACKEND=auto
export SHARPMOD_BACKEND=python
export SHARPMOD_BACKEND=rust

An unset variable is equivalent to auto. An unsupported value is an error; it is not treated as auto.

Inspect the resolved state through the Python backend layer:

from sharpmod.backends import backend_info

print(backend_info())

The diagnostic reports the requested and active backends, whether sharpmod_rs is discoverable, its version when it was loaded, and the fallback reason when auto selected Python. In forced python mode the selector does not import the native module, so rust_installed can be true while rust_version remains None. Application code should import through sharpmod.backends; direct sharpmod_rs imports are reserved for extension development and benchmark verification.

An importable native module is not accepted on import success alone. Before activation, the selector requires its package version to match sharpmod exactly, its integer backend API version to equal the Python contract (currently 7), and all 18 required operations listed below to be callable. In auto mode an old or incomplete extension falls back to Python and records the compatibility failure in fallback_reason; in forced rust mode the same mismatch raises BackendUnavailableError instead of reaching an operation with stale native code.

Backend selection is process-local. Set the environment variable before the first backend operation rather than changing it while the GUI or workers are already running.

Accelerated scope

Both backends expose the same named operations:

  • wind_to_components converts meteorological wind direction and speed to unit-preserving u and v components.
  • components_to_wind converts u and v components to meteorological wind direction and unit-preserving speed.
  • interpolate_1d performs the backend's one-dimensional profile interpolation contract.
  • basic_sounding_qc performs the documented structural and basic-value checks for sounding columns.
  • pressure_sort_dedup_indices returns the ordering/selection needed to sort pressure levels and remove duplicates consistently across all aligned columns.
  • parse_sounding_rows parses the deliberately narrow, simple sounding-row representation supported by the backend contract.
  • profile_kinematics computes Bunkers storm motion plus the standard SFC-to-0.5/1/3/4/6 km shear, mean-wind, storm-relative-wind, and SRH workspace in one coarse-grained call.
  • profile_parcels computes surface-based, lowest-300-hPa most-unstable, and 100-hPa mixed-layer parcel summaries—including CAPE/CIN, LCL/LFC/EL, and CAPE through 3/6 km—in one coarse-grained call. Cached 0–6 km CAPE is used only when reported levels in that layer are no more than 500 m apart; coarser soundings retain the reference integration.
  • profile_convective_parcels computes surface, forecast, most-unstable, mixed-layer, and mean-effective parcel summaries and virtual-temperature plotting traces together with effective-inflow-layer pressure bounds.
  • lift_parcel computes one explicit user-selected parcel summary and plotting trace.
  • profile_dcape computes DCAPE, its source pressure/downrush temperature, and the descending parcel-temperature trace.
  • profile_thermodynamics prepares one owned profile snapshot and shares it across the three-parcel, traced convective, and DCAPE workspaces. Native traces cross the extension boundary as contiguous NumPy buffers plus offsets; the public Python API still exposes the existing immutable tuples.
  • profile_batch_analysis evaluates complete independent soundings in stable input order. Calls below eight profiles are serial; larger calls default to a process-wide Rayon pool bounded by the available CPUs and four workers, remain serial when already inside Rayon, and accept explicit worker/cutoff overrides. Sharing the default pool keeps simultaneous GUI callers under the same native thread ceiling.
  • decode_grib_point and decode_grib_points reuse a bounded immutable message inventory, select the nearest point once per distinct grid definition, and return each sounding as one C-contiguous NumPy matrix. The multipoint form opens each selected message once and asks ecCodes for every requested grid index together while retaining request order and duplicate grid cells. When verified surface messages are present, both forms remove below-ground isobars and prepend surface pressure/height, 2-m thermodynamics, and 10-m wind.
  • The remaining required callables enable, clear, and report the native GRIB inventory cache so tests and benchmarks can distinguish cold scans from inventory reuse.

The shared behavior includes missing values, masks, NaNs, stable ordering, boundary results, error conditions, units, and output types. Python/Rust equivalence tests use explicit floating-point tolerances. Unit conversions remain the caller's responsibility unless an operation's contract says otherwise.

The wind and interpolation operations default to no explicit sentinel. The QC, pressure ordering, and row-parser operations default to -9999; passing missing=None disables explicit-sentinel matching. Parser blanks and non-finite values are normalized to NaN in that mode. Both backends apply these rules identically.

The simple-row parser follows Python float() for Unicode decimal digits: the Nd blocks in the target interpreter's Unicode table—including Arabic-Indic, Devanagari, fullwidth, and mathematical digits—are normalized to ASCII before the native parse. CPython 3.12's Unicode 15 table additionally accepts Kawi and Nag Mundari digits; the CPython 3.11 wheel follows its Unicode 14 table and rejects those two newer blocks. Numeric characters outside the decimal-digit category, such as Roman numerals and superscripts, remain errors. This deliberately mirrors Python float() rather than Rust's broader char::is_numeric predicate.

No Qt, rendering, or forecast downloading has been ported. A Python AcceleratedConvectiveProfile keeps SHARPpy's orchestration and public parcel objects while delegating its standard parcel/effective-layer searches and DCAPE integrations to the backend. The Skew-T user-parcel action delegates its explicit ascent through the same boundary. Backend failure falls back to the pinned SHARPpy routines. Advanced parcel fields that are not part of the traced parcel summary contract (for example cap strength and exact upstream MPL bookkeeping) remain Python-side compatibility concerns. ECAPE also retains a Python MUCAPE check for its exact pinned-oracle upper-bound contract. The Rust decoder receives a cache-owned local file from the Python retrieval layer. Rayon is used only for complete-profile analysis, where the measured 8-profile and 32-profile workloads benefit. GRIB/ecCodes calls remain behind a process-wide native mutex; batch analysis never enters ecCodes.

Repository-owned application call sites delegate wind-component creation, generic profile interpolation, ERA5/WRF wind conversion, pressure sorting and deduplication, standard SharpTab kinematics, standard/full parcel workspaces, user parcel ascents, DCAPE, and forecast-model GRIB point extraction through the facade. A real Profile caches its immutable kinematics and three-parcel summary workspaces; full convective profiles consume the extended workspace once during construction. Box fast-tier analysis and built-in ensemble summaries submit complete ordered profile groups through profile_batch_analysis; load failures retain their original positions and fall back per profile without discarding successful members. The bounded metric cache fingerprints all source columns and rebuilds prepared results after either profile replacement or an in-place array edit. Nonstandard layers and any unavailable workspace continue through the existing Python routines. Basic QC and simple-row parsing remain equivalence-tested APIs without changing the SPC, UWyo, BUFKIT, WRF, or other decoder pipelines.

Direct GRIB behavior

Forecast retrieval still uses Herbie and the existing provider/subregion routes. The selected inventory is reused for transfer planning, and a complete local subset is handed to the active decoder without constructing xarray data.

The Python decoder scans message headers once per file identity, performs one ecCodes nearest-point lookup per distinct grid definition, vector-reads all uncached requested indexes from each selected field/level message, and keeps bounded inventory, nearest-point, and decoded-point LRUs. The Rust decoder memory-maps the file, locates message boundaries without copying, uses ecCodes handles that borrow each mapped message, and returns all nine columns in one NumPy-compatible matrix per point. Its adapter batches unique uncached requests and keeps a bounded exact-result cache without conflating different surface grids.

Inventory identity includes the canonical path and size plus replacement-safe filesystem metadata: volume/file index and creation/write timestamps on Windows, or device/inode plus modification/change timestamps on Unix. An atomic same-size replacement therefore invalidates both Python and native state. The inventory stores immutable offsets and metadata only--never ecCodes handles--so model-cache leases and mapped-file lifetimes remain unchanged. Python ecCodes access uses its own re-entrant lock and all native ecCodes access uses one native mutex.

For HRRR, retrieval includes PRES and HGT at the surface, TMP and DPT at 2 m, and UGRD and VGRD at 10 m. Both direct decoders and the cfgrib/Zarr compatibility routes use surface pressure to remove all subterranean isobars before inserting the real ground row. Model extraction fails closed if this verified merge is unavailable. The JSON sidecar records surface_pressure_hpa, surface_merged, and below_ground_levels_removed.

Optimized Python and Rust return matching omega values and missing masks at the published pressure levels. For GEFS, the value published only at 850 hPa stays missing at every other level. The old/new benchmark's 12 -> 1 valid omega difference records removal of the frozen legacy xarray full-column broadcast; it is not missing Rust functionality or a Python/Rust equivalence exception.

SHARPMOD_GRIB_DECODER=auto is the default. Unsupported layouts fall back to cfgrib with its persistent on-disk index; split groups are reduced to at most a 3-by-3 neighborhood before xarray merging. This preserves the existing neighbor-wind surface-vorticity calculation without materializing whole model grids. Use xarray to force that compatibility path or direct to make a direct-decoder error explicit.

Python-only installation

Existing setup continues to work unchanged:

py -3.11 -m venv .gribenv
.\.gribenv\Scripts\Activate.ps1
python scripts\install_sharppy_compat.py --extras dev,render

With no extension installed, auto uses Python. python also works normally, while an explicit rust request reports that the extension is unavailable.

Build the local Rust extension

Install Rust 1.88 or newer from the stable channel and make cargo available on PATH. Then install the optional build frontend into the same .gribenv that will import the extension:

.\.gribenv\Scripts\Activate.ps1
python -m pip install -e ".[dev,rust-build]"
sharpmod-rust-sync
sharpmod-rust-sync --check

sharpmod-rust-sync compares the installed native distribution with the checkout version. It performs a locked release rebuild only when the extension is missing or stale, installs it into the active virtual environment, and then requires forced-Rust selection in a fresh Python process. --check performs the version and selector checks without modifying the environment. Use sharpmod-rust-sync --force after changing native source even when the package version did not change.

The lower-level extension-development path remains available:

Set-Location rust\sharpmod-rs
maturin develop --release --locked
Set-Location ..\..

maturin develop installs sharpmod_rs into the active environment; it does not modify the normal setuptools build backend for sharpmod. From a POSIX shell, activate .gribenv/bin/activate and use the same sync command or crate directory commands. Windows source builds also require the MSVC C/C++ linker toolchain (normally installed through Visual Studio Build Tools).

Confirm the extension and selector:

python -c "import sharpmod_rs; print(sharpmod_rs.__version__)"
$env:SHARPMOD_BACKEND = "rust"
python -c "from sharpmod.backends import backend_info; print(backend_info())"

Tests and checks

Python-only tests must pass without building Rust:

Remove-Item Env:SHARPMOD_BACKEND -ErrorAction SilentlyContinue
python -m pytest
$env:SHARPMOD_BACKEND = "python"
python -m pytest sharpmod/tests -k "backend or rust"

Run the native checks from the crate and then run the equivalence tests with Rust required:

Set-Location rust\sharpmod-rs
cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked
maturin develop --release --locked
Set-Location ..\..

$env:SHARPMOD_BACKEND = "rust"
python -m pytest sharpmod/tests -k "backend or rust"

Rust-specific Python tests skip with an explicit reason when sharpmod_rs is not installed during a normal Python-only run. Forced-Rust selector tests must still verify the error path without depending on a globally installed extension.

Benchmarks

The cross-backend harness requires a built extension and compares the Python and Rust backend methods on small profiles, ordinary profiles, a 100,000-value vector, a 2,048-by-128 array batch, repeated calls, scalar 700 hPa lookups on 32/128 levels, six repeated fields, and real cached-selector Profile construction. It also measures 32/128-level kinematics, diagnostics-only parcels, traced convective parcels, DCAPE, shared thermodynamic preparation, and complete 8-by-32 and 32-by-128 profile batches. Shared thermodynamics are split into public-adapter, buffered-adapter, and direct-native rows; complete batches split serial and bounded four-worker rows. An equal-work sequential-versus-two-thread diagnostic remains observational:

$env:SHARPMOD_BACKEND = "rust"
python benchmarks\benchmark_backends.py

See the benchmark guide for methodology and reproducibility guidance. Benchmark output is evidence to inspect, not a pass criterion. No operation is described as faster without recorded results on the relevant workload, and common single-profile operations may remain preferable in Python when boundary-crossing or conversion overhead dominates.

The two-thread table is observational, not a pass/fail result or a promise of parallel scaling. Small array kernels retain the Python GIL while borrowing NumPy input slices; releasing it could allow another Python thread to mutate storage that Rust is reading. The profile-kinematics and profile-parcel bindings instead create one Rust-owned snapshot before releasing the GIL; their internal prepared structures borrow that snapshot instead of copying the same columns again. Complete-batch workers likewise receive only owned vectors. GRIB decoding releases the GIL and returns capsule-backed NumPy matrices, but native ecCodes calls remain serialized. Python ecCodes/cfgrib calls use a separate lock, so callers should not mix decoder implementations concurrently in one process.

Recorded local result

The following snapshot was recorded on 2026-07-15 on a 12th Gen Intel Core i5-1235U in Windows Balanced mode, with Python 3.11.14, NumPy 2.4.6, sharpmod_rs 0.3.1, and Rust 1.97.0. The harness alternated measurement order across 20 repeats after five warmups. The complete header, command, medians, minima, and maxima for every measured row are retained in the raw result.

Representative median milliseconds per call were:

Workload Operation Python Rust Local observation
100,000 values wind_to_components 4.619510 3.816955 Rust median about 17% lower
100,000 values components_to_wind 5.626450 4.531640 Rust median about 19% lower
100,000 values interpolate_1d 3.766315 4.706360 Python median about 20% lower
2,048 x 128 values wind_to_components 11.361770 7.825710 Rust median about 31% lower
2,048 x 128 values components_to_wind 16.628920 12.481090 Rust median about 25% lower

This is a noisy local observation, not a cross-platform guarantee; several small and repeated-operation ranges overlapped substantially and are not used for speed claims. Large wind-vector batches showed the clearest Rust benefit, while NumPy remained preferable for the large interpolation case. auto selects one backend for all operations rather than making a per-call performance decision, so benchmark the real workload and use python explicitly when its mix favors the Python implementation.

Recorded GRIB decoder result

On 2026-07-16, a three-sample run used a 181,267,625-byte HRRR pressure-level subset at 35.18 N, 97.44 W (40 output levels). The OS file cache was warm, while application decoder caches and cfgrib indexes were disabled for each cold sample. "Old Rust" is the historical hybrid path: cfgrib/xarray still performs the complete decode and Rust only converts the extracted wind column.

Comparison Old median Optimized median Speedup
Python 33.961649 s 6.601793 s 5.14x
Rust / historical hybrid 31.398744 s 6.614370 s 4.75x

The optimized cold decoders were effectively tied in this run (Python was 0.19% lower). With a persistent cfgrib index versus a warm direct-Python inventory at a new point, the medians were 19.569670 s and 7.254023 s (2.70x). For an exact decoded-point cache hit, 100 samples had medians of 0.165650 ms in Python and 0.110250 ms in Rust. See the dated raw result in benchmarks/results for minima, maxima, environment details, and the exact command.

The current API-7 optimization measurements, including the release baselines, Criterion confidence intervals, complete-adapter JSON, fixed-fixture GRIB cache states, batch worker evaluation, regressions, and validation commands, are in 2026-09-12-rust-backend-optimization.md.

Platform status

The crate is portable Rust stable code. The dedicated .github/workflows/rust.yml workflow is configured to build, install, smoke-test, equivalence-test, and retain CPython 3.11 and 3.12 wheel artifacts on:

  • Windows x86_64
  • Linux x86_64
  • macOS arm64
  • macOS x86_64

Those jobs prepare downloadable native-wheel CI artifacts; the wheels are not published as a separate package-index release. The official Windows release workflow builds and installs the extension before constructing and runtime-checking the one-folder and one-file PyInstaller applications. Rust is therefore the supported primary backend in those binaries. The Python-only package remains the portable fallback for source installs and platforms without a compatible extension.

Current limitations

  • Native wheels are CI/build artifacts rather than a separately published package-index release. Official Windows desktop binaries bundle the native extension directly.
  • The normal sharpmod wheel and sdist contain the Python backend layer but not the Rust crate or extension. The [rust-build] extra installs maturin only; a repository checkout is required for the documented source build.
  • The prepared wheels target CPython 3.11 and 3.12 separately and are not abi3 wheels. Each additional Python minor version requires its own native wheel matrix entry.
  • NumPy masked arrays, non-contiguous views, non-float64 data, and sentinel normalization can require Python-side validation or copies. Zero-copy is an optimization for compatible arrays, not an API guarantee.
  • The declared NumPy range spans NumPy 1.x and 2.x; native compatibility must be tested across that range before publishing wheels.
  • Small arrays can cost more to cross the Python/Rust boundary than they save in computation. The workspace APIs amortize that cost by returning all standard layers or parcels together and caching each result on a Profile.
  • Borrowed array kernels retain the GIL. Profile kinematics and parcels own copies before releasing it. Direct GRIB decoding also releases it, but Rust-decoder ecCodes access is serialized; concurrent point decodes are not currently expected to scale. Python ecCodes/cfgrib calls are protected separately rather than by a shared cross-language lock.
  • A Profile's original internal SharpTab caches still follow its normal derived-attribute lifecycle. The cross-profile metric cache separately fingerprints pressure, height, thermodynamic, and wind columns, so its prepared result is discarded if those arrays are edited in place.
  • Native parcel results intentionally exclude several advanced Parcel fields outside the traced parcel summary contract, including cap-strength bookkeeping and the pinned ECAPE upper-bound check. The Python compatibility layer derives or masks those fields and retains the pinned SHARPpy fallback.
  • Direct GRIB caches are process-local and bounded. A truly cold request still scans message headers; inventory reuse removes that scan but ecCodes must still unpack every selected field/level message. Multipoint calls amortize that unpack across all requested indexes rather than making repeated scalar passes.
  • When a downloaded inventory has no usable relative/absolute-vorticity field, model extraction keeps the xarray compatibility path so the existing neighbor-wind vorticity estimate is preserved.
  • Parsing is intentionally limited to the backend's simple row format. Existing SPC, UWyo, BUFKIT, WRF, and other decoders remain Python implementations.
  • QC is basic and structural. It does not replace the complete SHARPpy profile validation and meteorological analysis paths.
  • The PyInstaller spec includes sharpmod_rs only when the extension is discoverable and collected successfully in the environment performing the build. The official release job builds and installs it before packaging; custom local builds that omit it log a warning and produce a fully functional Python-fallback bundle.