Validation of the Neural Hydrology Framework for Discharge Forecasting in Small to Medium-Sized Catchments for the OpenRiverSense Platform
Short Description: The OpenRiverSense (ORS) project develops an open system for monitoring, analyzing, and predicting hydrological measurements in small and medium-sized waterways. This pilot project investigates whether modern neural time-series models — built on the open-source NeuralHydrology framework, using LSTM networks — can provide reliable streamflow predictions for German catchments, including ones with no historical discharge measurements. The LSTM approach is particularly promising for data-scarce regions because a model can be trained simultaneously across many catchments and then fine-tuned for specific locations. As a data basis we use the open-source Caravan reference dataset (forcings + static catchment attributes) and its Caravan-MultiMet extension, which includes both ERA5-Land reanalysis and ECMWF HRES forecast meteorology.
Demonstrator: an interactive showcase is online at demonstrator-openriversense.aisc.hpi.de.
-
Objective 1: Model initialization and validation on known catchments — DONE. Train a regional LSTM on European basins; benchmark against published results; demonstrate operational forecasting from HRES weather forecasts. See Project status below for results.
-
Objective 2: Training on a new catchment with measurement data — open. Extend the model to a previously unused catchment. Acquire meteorological time series and catchment attributes. Test transfer learning from the regional model and combined training (Caravan + local data).
-
Objective 3: Pilot application on an ungauged basin — open (ORS is building the live forecast pipeline). Deploy the trained model on catchments without historical discharge data. Predictions based exclusively on meteorological inputs and catchment attributes, validated against ORS Open-Hardware sensor stations.
As of April 2026 Objective 1 is complete. Several end-to-end pipelines reach median test NSE 0.79–0.80 on 20 spatially held-out Bremen catchments using HRES weather forecasts at 1-day lead time — within the statistical resolution of the test set. The simplest of those pipelines, Config 25 (h256 CuDaLSTM, pre-train on ERA5-Land + fine-tune on HRES, constant LR=1e-4), can be seen as a checkpoint; it reaches median NSE 0.8 and is marginally best on secondary metrics.
Key takeaway: the methodology works; the dominant remaining work for ORS is the live data infrastructure (HRES feed, basin coverage, harmonisation), not the model.
The full chronological record (Phases 1–5, Configs 01–28, lessons learned, recommended training recipes) is in EXPERIMENT_LOG.md.
Most relevant analysis notebooks:
notebooks/01_objective1/showcase_pilotproject.ipynb— short demo: European basin map + hydrograph at an unseen Bremen basin.notebooks/01_objective1/19-28_pipeline_comparison.ipynb— canonical pipeline comparison for the spatial configs.notebooks/01_objective1/19-28_training_curves.ipynb— per-epoch train/val curves for all spatial-AIFL runs.
- Python 3.10+
- uv package manager
- Recommended: Access to the aisc-cluster via VSCode Remote-SSH
# Clone the repository
git clone https://github.com/aihpi/pilotproject-OpenRiverSense.git
cd pilotproject-OpenRiverSense
# Install dependencies
uv sync
# Optional: Activate the virtual environment (or run commands with 'uv run')
source .venv/bin/activate # On Unix/macOSThis installs NeuralHydrology from PyPI along with all required dependencies.
Optional — register the venv as a Jupyter kernel:
uv run python -m ipykernel install --user --name=pilotproject-openriversense --display-name="Python (pilotproject-openriversense)"Optional — clone NeuralHydrology locally for code exploration:
git clone https://github.com/neuralhydrology/neuralhydrology.git
rm -rf neuralhydrology/.gitThe folder neuralhydrology/ is in .gitignore. Note: the venv still uses the PyPI-installed package, not your local clone.
All datasets were downloaded from Zenodo. See scripts/data_processing/ for the download scripts.
| Dataset | Zenodo Record | Contents |
|---|---|---|
| Caravan v1.6 (base) | https://zenodo.org/records/15529786 | 8 subdatasets (camels, camelsaus, camelsbr, camelscl, camelsgb, hysets, lamah), NetCDF timeseries + attributes |
| Caravan-DE extension | https://zenodo.org/records/14755229 | 1,887 German basins (camelsde) |
| Caravan-DK extension | https://zenodo.org/records/15200118 | 308 Danish basins (camelsdk) |
| Caravan-GRDC extension | https://zenodo.org/records/14006282 | 5,357 basins across 25 countries (GRDC) |
| Caravan-MultiMet v1.1 | https://zenodo.org/records/14196771 (part 1), https://zenodo.org/records/14196772 (part 2) | ERA5_LAND, HRES, CPC, IMERG zarr stores for 22,485 basins |
Training datasets live in a shared cluster location accessible to all aisc team members. Do not move, change or delete files/folders unless you are sure about the consequences.
Caution
The shared cluster path /sc/projects/sci-aisc/... is read by everyone with aisc access on the HPI cluster. Do not move/change/delete without coordination. Datasets are large — do not copy them into your home directory; point experiments at the shared path or create a symlink (ln -s SOURCEPATH TARGETPATH).
Location: /sc/projects/sci-aisc/pilotproject-openriversense/data/Caravan-nc/
Caravan aggregates global daily hydro-meteorological data in NetCDF format. We use the European subdatasets:
| Subdataset | Basins | Region |
|---|---|---|
camelsde |
1,887 | Germany |
lamah |
859 | Central Europe (Austria, Switzerland, …) |
camelsgb |
671 | Great Britain |
GRDC |
1,218 | Europe (filtered from 5,357 global) |
camelsdk |
308 | Denmark |
Reference: Caravan paper (Kratzert et al., 2023).
Some subdatasets (Caravan-DE, CAMELS-DK, GRDC) ship separately — see scripts/data_processing/download_caravan*.sh.
Location: /sc/projects/sci-aisc/pilotproject-openriversense/data/Caravan-MultiMet-nc/
Caravan-MultiMet extends Caravan with multiple meteorological products including ECMWF HRES weather forecasts. The 5 generic forcing variables (temperature_2m, total_precipitation, surface_net_solar_radiation, surface_net_thermal_radiation, surface_pressure) are shared between ERA5-Land and HRES, which is what enables AIFL pre-train / fine-tune (Taccari et al., 2026). MultiMet ships in Zarr; we convert to per-basin NetCDF via scripts/data_processing/convert_multimet_to_nh.py.
Location: /sc/projects/sci-aisc/pilotproject-openriversense/data/CAMELS_DE/
1,582 German catchments with up to 70 years (1951–2020) of daily data and HYRAS (1×1 km) forcing. Used to verify NeuralHydrology against a published LSTM benchmark (median NSE = 0.84). Reference: CAMELS-DE paper (Loritz et al., 2024).
One-time download instructions (for maintainers)
cd /sc/projects/sci-aisc/pilotproject-openriversense/
wget https://zenodo.org/records/16755906/files/camels_de.zip
unzip camels_de.zip -d data/CAMELS_DE/
rm camels_de.zipSource: CAMELS-DE v1.1.0 on Zenodo
- NeuralHydrology Documentation: https://neuralhydrology.readthedocs.io
- NeuralHydrology GitHub: https://github.com/neuralhydrology/neuralhydrology
- Caravan paper: https://www.nature.com/articles/s41597-023-01975-w
- CAMELS-DE paper: https://doi.org/10.5194/essd-16-5625-2024
- AIFL paper (Taccari et al., 2026): https://arxiv.org/html/2602.16579v1
# Add a new package
uv add package-name
# Add a development dependency
uv add --dev package-nameAfter pulling changes from git:
uv syncWhen contributing, please:
- Create a new branch for your work.
- Write clear commit messages.
- Document experiments in
EXPERIMENT_LOG.mdand updateCHANGELOG.mdon push. - Keep configuration files organized in
configs(numbered) with a matching SLURM script inscripts.
Kratzert, F. et al. (2018) "Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks," Hydrology and Earth System Sciences, 22(11), pp. 6005–6022. https://doi.org/10.5194/hess-22-6005-2018
Kratzert, F. et al. (2023) "Caravan — A global community dataset for large-sample hydrology," Scientific Data, 10(1), p. 61. https://doi.org/10.1038/s41597-023-01975-w
Loritz, R. et al. (2024) "CAMELS-DE: hydro-meteorological time series and attributes for 1582 catchments in Germany," Earth System Science Data, 16(12), pp. 5625–5642. https://doi.org/10.5194/essd-16-5625-2024
Acuña Espinoza, E. et al. (2024) "To bucket or not to bucket? Analyzing the performance and interpretability of hybrid hydrological models with dynamic parameterization," Hydrology and Earth System Sciences, 28(12), pp. 2705–2719. https://doi.org/10.5194/hess-28-2705-2024
Santos, L.B.L. et al. (2023) "A Neural Network-Based Hydrological Model for Very High-Resolution Forecasting Using Weather Radar Data," Eng, 4(3), pp. 1787–1796. https://doi.org/10.3390/eng4030101
Weißenborn, M., Breuer, L. and Houska, T. (2025) "Neural networks in catchment hydrology: a comparative study of different algorithms in an ensemble of ungauged basins in Germany," Hydrology and Earth System Sciences, 29(19), pp. 5131–5164. https://doi.org/10.5194/hess-29-5131-2025
Bao, N. et al. (2025) "RFM_Trans: Runoff forecasting model for catchment flood protection using strategies optimized Transformer," Expert Systems with Applications, 277, p. 127228. https://doi.org/10.1016/j.eswa.2025.127228
Liu, J. et al. (2024) "Probing the limit of hydrologic predictability with the Transformer network," Journal of Hydrology, 637, p. 131389. https://doi.org/10.1016/j.jhydrol.2024.131389
Taccari, M.L. et al. (2026) "AIFL: A Global Daily Streamflow Forecasting Model Using Deterministic LSTM Pre-trained on ERA5-Land and Fine-tuned on IFS" https://arxiv.org/html/2602.16579v1
This project is licensed under the MIT License.
The AI Service Centre Berlin Brandenburg is funded by the Federal Ministry of Research, Technology and Space under the funding code 16IS22092.