NGWPC Closeout Delivery 8/31/2026

[cmaynard-ngwpc]

AWS Migration, Slurm-Based Job Execution, and Tuning Time Control Overhaul
Server operations moved off the legacy PW workflow onto AWS, with jobs now submitted and monitored directly against the Slurm REST API and a CLI-based binary build pipeline for distribution. Forecast verification was retired in favor of routing evaluation through the new eval-mgr service, while calibration/validation tuning periods gained a full rework of start/end/gap time-control math, multi-file tuning-parameter uploads, and better gage/domain handling. The service also picked up a health-check endpoint, CSRF/reverse-proxy hardening, and a Python 3.12 upgrade.

Note: Pull Request includes commits OWP has yet to merge from previous deliveries

Additions

• Slurm REST API integration for starting and monitoring jobs directly from the server, replacing the previous PW-based job control path
• CLI binaries built and baked into images for user download, with improved CLI logging and normalized API URLs
• Health-check endpoint, plus env-driven CSRF trusted origins and X-Forwarded-Proto trust for reverse-proxy deployments
• Multi-file tuning-parameter upload handling with distinct validation messages per file
• New validation_window_gap field and supporting time-control math for calibration/validation periods
• Hindcast data collection now passes observation data through to verification/evaluation

Removals

• Forecast verification workflow and the polymorphic VerificationRun code supporting it
• Automatic validation behavior and its supporting fields
• Obsolete fields, legacy GitLab docker env files, and the old manylinux2014-based CLI build in favor of staticx
• Inactive Alaska gages not present in the new hydrofabric

Changes

• Migrated compute execution from GitLab/PW to AWS PCS, including S3 stack and dependency updates
• Reworked tuning-tab time-control validation and defaults (validation_window renamed to validation_window_after_calibration, more accurate min/max logic)
• Upgraded to Python 3.12 on a python:3.14-slim-bookworm base image; adopted Whitenoise for static file serving and dropped the fiona dependency
• Refactored gage/domain handling and combined forecast/hindcast status filtering; corrected credential-saving logic