pacificclimate · QSparks · Jul 20, 2026 · Jul 24, 2026 · Jul 27, 2026
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # PDP-Next
 
-THREDDS-backed, database-free replacement for the Pacific Climate Data Portal (PDP) with projection support, faster subsetting, additional palettes, and portal-based interactive map viewing.
+THREDDS-backed, database-free replacement for the Pacific Climate Data Portal (PDP) with projection support, additional palettes, and portal-based interactive map viewing.
 
 ## Layout
 
@@ -11,15 +11,39 @@ THREDDS-backed, database-free replacement for the Pacific Climate Data Portal (P
 - `viewer/`: current frontend/viewer assets
 - `thredds/`: local THREDDS config and support files
 
+## Production Files
+
+The production metadata files are deployed under:
+
+```text
+/.../swarm_files/
+```
+
+The active generated metadata is in `portal-meta/*.json`. The production
+min/max CSV and portal membership definitions are in `portal-prep/`:
+
+```text
+portal-meta/<portal>.json
+portal-prep/pdp_min_max.csv
+portal-prep/portal-file-patterns/<portal>.txt
+```
+
+`portal-meta/portal-meta-bu/` contains backup JSON files and is not an active
+metadata output directory.
+
 ## Current Metadata Flow
 
 1. Build or refresh the `/pdp/<portal>/<file>.nc` hardlink mirror from portal file patterns.
-2. Optionally compute min/max rendering metadata for selected files.
-3. Build `portal-meta/<portal>.json` from NetCDF metadata and normalized derived fields.
+2. Export existing min/max ranges from the `modelmeta` database.
+3. Build the ordered min/max CSV, scanning only files missing from the export.
+4. Build `portal-meta/<portal>.json` from NetCDF metadata and normalized derived fields.
 
 The metadata builder reads NetCDF metadata, normalizes common fields into `metadata.derived`, and applies per-portal
 menu rules from `portal_meta_builder/portals.py`.
 
+See [`portal-prep/README.md`](portal-prep/README.md) for the complete,
+copy-pasteable metadata preparation workflow.
+
 ## Scripts
 
 ### Build hardlink mirror
@@ -34,16 +58,26 @@ Uses file-pattern definitions from `portal-prep/portal-file-patterns/`.
 ### Compute rendering min/max CSV
 
 ```bash
-python3 scripts/calculate-portal-minmax.py
-python3 scripts/calculate-portal-minmax.py --portal prism
+export PDP_MODELMETA_URL='postgresql://USERNAME@HOSTNAME:5432/modelmeta'
+
+psql --no-psqlrc --quiet --csv --dbname "$PDP_MODELMETA_URL" \
+  --file portal-prep/min_max_query.sql \
+  > portal-prep/db-export.csv
+
+python3 scripts/calculate-portal-minmax.py \
+  --db-csv portal-prep/db-export.csv \
+  --out-csv portal-prep/pdp_min_max.csv \
+  --all
 ```
 
-Writes `portal-prep/pdp_min_max.csv`.
+Uses `portal-prep/db-export.csv` as its first source and scans remaining files
+in bounded chunks. The reduced export query is
+`portal-prep/min_max_query.sql`. Writes `portal-prep/pdp_min_max.csv`.
 
 ### Build portal metadata JSON
 
 ```bash
-python3 scripts/update-portal-meta.py
+python3 scripts/update-portal-meta.py --prune
 python3 scripts/update-portal-meta.py --portal prism
 ```
 
@@ -54,7 +88,10 @@ Writes `portal-meta/<portal>.json`.
 1. Add a portal pattern file at `portal-prep/portal-file-patterns/<portal>.txt`.
 2. Add a portal definition in `portal_meta_builder/portals.py`.
 3. Run `scripts/gen_hardlinks.py` for the portal.
-4. Run `scripts/update-portal-meta.py --portal <portal>`.
+4. If ModelMeta contains the portal's ensemble, add its database ensemble name
+   to `portal-prep/min_max_query.sql` and regenerate `portal-prep/db-export.csv`.
+5. Run `scripts/calculate-portal-minmax.py --portal <portal>`.
+6. Run `scripts/update-portal-meta.py --portal <portal>`.
 
 If the portal needs custom menu labels or grouping, add a focused menu builder in
 `portal_meta_builder/portals.py`. If it needs additional normalized metadata, add that in

diff --git a/portal-prep/README.md b/portal-prep/README.md
@@ -0,0 +1,147 @@
+# Portal metadata preparation
+
+Run these commands from the repository root. The complete flow is:
+
+1. Refresh the hardlink mirror from the portal file patterns.
+2. Export existing ranges from the `modelmeta` database.
+3. Build the ordered portal min/max CSV, calculating only database misses.
+4. Rebuild the portal metadata JSON consumed by the viewer.
+
+## 1. Refresh the hardlink mirror
+
+The hardlink mirror under `/storage/data/projects/comp_support/thredds/pdp`
+must be current before rebuilding the portal metadata. Refresh it when portal
+membership or source files have changed:
+
+```bash
+python3 scripts/gen_hardlinks.py
+```
+
+## Production locations
+
+The production copies used by PDP are under:
+
+```text
+/.../swarm_files/pdp-next
+```
+
+The active files produced or consumed by this workflow are:
+
+```text
+portal-meta/<portal>.json
+portal-prep/pdp_min_max.csv
+portal-prep/portal-file-patterns/<portal>.txt
+```
+
+## 2. Export ranges from modelmeta
+
+[`min_max_query.sql`](min_max_query.sql) selects only the fields needed by
+the range builder and restricts the result to the relevant database
+ensembles. Run it against the `modelmeta` database and save the CSV here:
+
+```bash
+export PDP_MODELMETA_URL='postgresql://USERNAME@HOSTNAME:5432/modelmeta'
+
+psql \
+  --no-psqlrc \
+  --quiet \
+  --csv \
+  --dbname "$PDP_MODELMETA_URL" \
+  --file portal-prep/min_max_query.sql \
+  > portal-prep/db-export.csv
+```
+
+`PDP_MODELMETA_URL` is the connection string for the `modelmeta` database on
+the DM host. Replace `USERNAME` and `HOSTNAME` with its connection values.
+
+The export must begin with this exact header:
+
+```text
+filename,netcdf_variable_name,range_min,range_max,ensemble_name
+```
+
+Check it before continuing:
+
+```bash
+head -1 portal-prep/db-export.csv
+wc -l portal-prep/db-export.csv
+```
+
+`db-export.csv` is an intermediate snapshot. Regenerate it whenever ranges or
+ensemble membership change in `modelmeta`.
+
+## 3. Build the ordered min/max CSV
+
+Pass the database export to the range builder and request a complete refresh:
+
+```bash
+python3 scripts/calculate-portal-minmax.py \
+  --db-csv portal-prep/db-export.csv \
+  --out-csv portal-prep/pdp_min_max.csv \
+  --all
+```
+
+For every file selected by `portal-file-patterns/*.txt`, the builder:
+
+1. Looks for an unambiguous range in `db-export.csv`, preferring an exact
+   source path and then an unambiguous basename match.
+2. Reads only the NetCDF header when it must choose among multiple variables.
+3. Scans database misses in bounded chunks, avoiding whole-file memory loads.
+4. Writes `pdp_min_max.csv` atomically in deterministic portal/path order.
+5. Saves progress after every completed NetCDF scan, so a later failure does
+   not discard completed work.
+
+The output has no header and uses this layout:
+
+```text
+portal_id,source_path,variable,min,max,source
+```
+
+The default is one scan worker with a 256 MB chunk budget. Keep one worker on
+memory-constrained hosts; every additional worker has its own chunk budget.
+Files already represented by the database do not need to be scanned.
+
+Useful checks:
+
+```bash
+wc -l portal-prep/pdp_min_max.csv
+tail -5 portal-prep/pdp_min_max.csv
+```
+
+## 4. Rebuild portal metadata
+
+Build the JSON documents from the hardlink mirror and attach the ranges from
+the ordered CSV:
+
+```bash
+python3 scripts/update-portal-meta.py \
+  --minmax-csv portal-prep/pdp_min_max.csv \
+  --prune
+```
+
+`--prune` removes cached entries that are no longer present in the mirror. The
+result is one `portal-meta/<portal>.json` file per portal. Each matched file
+receives a `rendering` object containing `min`, `max`, `variable`, `logScale`,
+and the min/max source.
+
+For a focused metadata rebuild, repeat `--portal` as needed:
+
+```bash
+python3 scripts/update-portal-meta.py \
+  --portal vicgl \
+  --minmax-csv portal-prep/pdp_min_max.csv \
+  --prune
+```
+
+## Portal file patterns
+
+`portal-file-patterns/` is the source of truth for portal membership. Each
+`<portal>.txt` accepts:
+
+- An absolute file path
+- A glob containing `*`, `?`, or `[` patterns
+- An exclusion prefixed with `!`
+
+Blank lines and lines beginning with `#` are ignored. Bash extglob syntax such
+as `!(...)` is not supported. Basenames must be unique within a portal because
+the hardlink mirror stores files as `<portal>/<basename>`.
diff --git a/portal-prep/min_max_query.sql b/portal-prep/min_max_query.sql
@@ -0,0 +1,27 @@
+SELECT
+    df.filename,
+    dfv.netcdf_variable_name,
+    dfv.range_min,
+    dfv.range_max,
+    e.ensemble_name
+FROM pcic_meta.data_file_variables AS dfv
+JOIN pcic_meta.data_files AS df
+    ON df.data_file_id = dfv.data_file_id
+JOIN pcic_meta.ensemble_data_file_variables AS edfv
+    ON edfv.data_file_variable_id = dfv.data_file_variable_id
+JOIN pcic_meta.ensembles AS e
+    ON e.ensemble_id = edfv.ensemble_id
+WHERE dfv.range_min IS NOT NULL
+  AND dfv.range_max IS NOT NULL
+  AND e.ensemble_name IN (
+      'bccaq_version_2',
+      'bccaq2_canesm5',
+      'bccaq2_cmip6',
+      'bc_prism_monthly_and_climos',
+      'gridded-obs-met-data',
+      'hydro_test',
+      'mbcn_canesm5',
+      'mbcn_cmip6',
+      'vicgl_cmip5'
+  )
+ORDER BY e.ensemble_name, df.filename, dfv.data_file_variable_id;
diff --git a/portal-prep/portal-file-patterns/bccaqv2_u5.txt b/portal-prep/portal-file-patterns/bccaqv2_u5.txt
@@ -0,0 +1,3 @@
+/storage/data/climate/downscale/BCCAQ2/bccaqv2_with_metadata/*_19500101-21001231.nc
+!/storage/data/climate/downscale/BCCAQ2/bccaqv2_with_metadata/*subset*_19500101-21001231.nc
+
diff --git a/portal-prep/portal-file-patterns/bccaqv2_u6.txt b/portal-prep/portal-file-patterns/bccaqv2_u6.txt
@@ -0,0 +1,22 @@
+# BCCAQv2 downscaled climate scenarios (CMIP6_BCCAQv2)
+# Excludes Derived by only including files in the top-level model directories.
+
+/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/*/*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/AgroClimate/*
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CWEC2020_Factors/*
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/Degree_Climatologies/*
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/Ensemble_Averages/*
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r2i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r3i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r4i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r5i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r6i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r7i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r8i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r9i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*_r10i1p2f1_*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/dtr_monmean_r1_ssp585.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/dtr_r1_ssp585.nc
+
+
+
diff --git a/portal-prep/portal-file-patterns/canada_mosaic.txt b/portal-prep/portal-file-patterns/canada_mosaic.txt
@@ -0,0 +1,6 @@
+# Canada Mosaic 30arcsec portal membership
+/storage/data/climate/observations/gridded/Canada_mosaic_30arcsec/*.nc
+!/storage/data/climate/observations/gridded/Canada_mosaic_30arcsec/*seasann*.nc
+
+
+
diff --git a/portal-prep/portal-file-patterns/canesm5_m6.txt b/portal-prep/portal-file-patterns/canesm5_m6.txt
@@ -0,0 +1,5 @@
+# CanESM5 (Multivariate) — MBCn/CanESM5_10
+# Top-level files only (exclude Derived trees that duplicate basenames)
+/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*txtn*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*370*.nc
diff --git a/portal-prep/portal-file-patterns/canesm5_u6.txt b/portal-prep/portal-file-patterns/canesm5_u6.txt
@@ -0,0 +1,5 @@
+# CanESM5 (Univariate) — BCCAQv2/CanESM5
+# Top-level files only (exclude Derived trees that duplicate basenames)
+/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/*.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/dtr_monmean_r1_ssp585.nc
+!/storage/data/climate/downscale/BCCAQ2/CMIP6_BCCAQv2/CanESM5/dtr_r1_ssp585.nc
diff --git a/portal-prep/portal-file-patterns/gridded_daily.txt b/portal-prep/portal-file-patterns/gridded_daily.txt
@@ -0,0 +1,9 @@
+# Daily Gridded Meteorological Datasets portal membership
+# PCIC Blend diagonal files
+/storage/data/climate/observations/gridded/PCIC_Blend/diagonal/*day_PCIC_Blended_Observations_v1_1950-2012.nc
+
+# VIC gen2 forcing (if present)
+/storage/data/projects/dataportal/data/vic-gen2-forcing/anusplin_*.nc
+/storage/data/projects/dataportal/data/vic-gen2-forcing/PNWNAmet_*.nc
+!/storage/data/projects/dataportal/data/vic-gen2-forcing/PNWNAmet_*invert_lat.nc
+
diff --git a/portal-prep/portal-file-patterns/mbcn.txt b/portal-prep/portal-file-patterns/mbcn.txt
@@ -0,0 +1,16 @@
+# MBCn downscaled climate scenarios (CMIP6_MBCn)
+# Top-level daily files in model folders.
+
+/storage/data/climate/downscale/MBCn/CMIP6_MBCn/*/*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r2i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r3i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r4i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r5i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r6i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r7i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r8i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r9i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*_r10i1p2f1_*.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/TESTING_REGION/template_PCIC-Blend_PNW.nc
+!/storage/data/climate/downscale/MBCn/CMIP6_MBCn/CanESM5_10/*txtn*.nc
+
diff --git a/portal-prep/portal-file-patterns/prism.txt b/portal-prep/portal-file-patterns/prism.txt
@@ -0,0 +1,5 @@
+# PRISM portal membership
+
+/storage/data/climate/observations/gridded/PRISM/split_climos/*ClimMean_PRISM_historical_*.nc
+
+/storage/data/climate/observations/gridded/PRISM/bc_*_monthly_CAI_timeseries_19500101_20071231.nc
diff --git a/portal-prep/portal-file-patterns/vicgl.txt b/portal-prep/portal-file-patterns/vicgl.txt
@@ -0,0 +1,4 @@
+# Gridded Hydrologic Model Output (VICGL)
+/storage/data/projects/hydrology/dataportal/CMIP5/VICGL/**/*.nc
+!/storage/data/projects/hydrology/dataportal/CMIP5/VICGL/*TRANSP_VEG_pfc.nc
+
diff --git a/portal_meta_builder/__init__.py b/portal_meta_builder/__init__.py
@@ -0,0 +1,4 @@
+from .builder import build_portal_payload
+
+__all__ = ["build_portal_payload"]
+
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		/storage/data/climate/downscale/BCCAQ2/bccaqv2_with_metadata/*_19500101-21001231.nc
		!/storage/data/climate/downscale/BCCAQ2/bccaqv2_with_metadata/subset_19500101-21001231.nc
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,4 @@
		from .builder import build_portal_payload

		__all__ = ["build_portal_payload"]