Skip to content

fix: uniform level gating + step-count/UTC/-i ordering bugs in ingest-model.sh#7

Open
mrauhala wants to merge 1 commit into
masterfrom
fix/ingest-model-gating
Open

fix: uniform level gating + step-count/UTC/-i ordering bugs in ingest-model.sh#7
mrauhala wants to merge 1 commit into
masterfrom
fix/ingest-model-gating

Conversation

@mrauhala

Copy link
Copy Markdown
Collaborator

Summary

Four bugs from a focused ingest-model.sh audit. The first two are the gate inconsistency cluster; the last two are small high-confidence correctness fixes that ride along.

F1 — uniform level gating

The SFC/PL/ML blocks had three different gates — only ML checked its input pattern existed. icon.cnf (area=europe) defines MODEL_RAW_SFC and MODEL_RAW_ML but no MODEL_RAW_PL, so every icon-europe run entered the PL block with an empty pattern and fired gribtoqd at a bare directory, failing every cycle. Now every completeness check and run block is guarded by [ -n MODEL_RAW_<lvl> ], matching ML.

F2 — drop eval from PL/ML step counts

eval gribstepcount "$INFILE_PL" expands the glob into separate args; gribstepcount's local FILES=$1 then counts only the first matched file. The SFC form (no eval) passes the glob as one string and lets grib_get $FILES expand it over all files. INFILE_* are already fully resolved upstream, so the eval was both unnecessary and wrong. All three checks now use the SFC form.

F3 — -t now parses as UTC

-t used date +%s (local) while -i and all RT_DATE_* use date -u. On a non-UTC host (e.g. FMI/Helsinki EET) this shifted the requested reference time by the TZ offset. Added -u.

F4 — -i <file> ordering

-i without -t ran the latest() incoming-dir scan first and could exit "No data available" before reading the -i file — exactly the case where incoming may be empty. And even on success, RT_DATE_*/OUTNAME/TMP were derived from the latest() time, then RT was overwritten by the file's time afterwards, so the output filename used the wrong reference time. Reordered: -i determines RT up front, before the derivations; latest() is skipped when -i is given.

Notes

  • No behaviour change for the common models (all define SFC+PL, single combined input file per level), so the matching/completeness logic is identical there.
  • Validated with bash -n and shellcheck (no errors). I can't exercise the grib_get/qdtools pipeline on macOS — the real check is a two-runs-no-change test on a host plus an icon-europe run showing no PL gribtoqd attempt.
  • Touches the shipped script, so it changes RPM content: needs a version bump before merging to master (else the release workflow rebuilds/clobbers v26.5.22). Left out so you pick the version.

Test plan

  • CI builds RPM on Rocky 8/9/10
  • icon area=europe: run logs no PL conversion attempt (PL gated off; no gribtoqd on a directory)
  • A model with multi-file PL split by step range: …_pressure.sqd completeness now matches on the second run (no perpetual re-convert)
  • -t 20260522T12 on a non-UTC host yields reference time 12:00 UTC
  • -i /tmp/file.grib with an empty incoming dir converts the file instead of exiting "No data available"

…ring

Addresses four issues found in a focused ingest-model.sh audit.

F1 — uniform gating. The SFC/PL/ML blocks had three different gates;
only ML checked that its input pattern (MODEL_RAW_ML) was set. A model
that omits a level — e.g. icon area=europe defines SFC and ML but no
MODEL_RAW_PL — entered the PL block with an empty pattern and fired
gribtoqd at a bare directory every cycle. Now every completeness check
and run block is guarded by [ -n MODEL_RAW_<lvl> ], matching ML.

F2 — drop `eval` from the PL/ML step counts. `eval gribstepcount
"$INFILE_PL"` expands the glob into separate args, and gribstepcount's
`local FILES=$1` then counts only the first matched file, whereas the
SFC form (no eval) passes the glob as one string and lets grib_get
expand it over all files. INFILE_* are already fully resolved earlier,
so the eval was unnecessary as well as wrong. All three now use the
SFC form.

F3 — `-t` parsed reference time in local time (`date +%s` without -u)
while the -i branch and every RT_DATE_* derivation use UTC. On a
non-UTC host this shifted the requested reference time. Added -u.

F4 — `-i <file>` without `-t` scanned the incoming dir via latest()
first and could exit "No data available" before ever reading the -i
file; and even when it succeeded, RT_DATE_*/OUTNAME/TMP were derived
from the latest() time and only then overwritten by the -i file's RT.
Reordered so -i determines RT up front, before the RT_DATE_*
derivations, and the latest() scan is skipped entirely when -i is given.

No change in behaviour for the common models (all define SFC+PL, single
combined input file per level).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant