fix: uniform level gating + step-count/UTC/-i ordering bugs in ingest-model.sh#7
Open
mrauhala wants to merge 1 commit into
Open
fix: uniform level gating + step-count/UTC/-i ordering bugs in ingest-model.sh#7mrauhala wants to merge 1 commit into
mrauhala wants to merge 1 commit into
Conversation
…ring Addresses four issues found in a focused ingest-model.sh audit. F1 — uniform gating. The SFC/PL/ML blocks had three different gates; only ML checked that its input pattern (MODEL_RAW_ML) was set. A model that omits a level — e.g. icon area=europe defines SFC and ML but no MODEL_RAW_PL — entered the PL block with an empty pattern and fired gribtoqd at a bare directory every cycle. Now every completeness check and run block is guarded by [ -n MODEL_RAW_<lvl> ], matching ML. F2 — drop `eval` from the PL/ML step counts. `eval gribstepcount "$INFILE_PL"` expands the glob into separate args, and gribstepcount's `local FILES=$1` then counts only the first matched file, whereas the SFC form (no eval) passes the glob as one string and lets grib_get expand it over all files. INFILE_* are already fully resolved earlier, so the eval was unnecessary as well as wrong. All three now use the SFC form. F3 — `-t` parsed reference time in local time (`date +%s` without -u) while the -i branch and every RT_DATE_* derivation use UTC. On a non-UTC host this shifted the requested reference time. Added -u. F4 — `-i <file>` without `-t` scanned the incoming dir via latest() first and could exit "No data available" before ever reading the -i file; and even when it succeeded, RT_DATE_*/OUTNAME/TMP were derived from the latest() time and only then overwritten by the -i file's RT. Reordered so -i determines RT up front, before the RT_DATE_* derivations, and the latest() scan is skipped entirely when -i is given. No change in behaviour for the common models (all define SFC+PL, single combined input file per level). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four bugs from a focused
ingest-model.shaudit. The first two are the gate inconsistency cluster; the last two are small high-confidence correctness fixes that ride along.F1 — uniform level gating
The SFC/PL/ML blocks had three different gates — only ML checked its input pattern existed.
icon.cnf(area=europe) definesMODEL_RAW_SFCandMODEL_RAW_MLbut noMODEL_RAW_PL, so every icon-europe run entered the PL block with an empty pattern and firedgribtoqdat a bare directory, failing every cycle. Now every completeness check and run block is guarded by[ -n MODEL_RAW_<lvl> ], matching ML.F2 — drop
evalfrom PL/ML step countseval gribstepcount "$INFILE_PL"expands the glob into separate args;gribstepcount'slocal FILES=$1then counts only the first matched file. The SFC form (noeval) passes the glob as one string and letsgrib_get $FILESexpand it over all files.INFILE_*are already fully resolved upstream, so theevalwas both unnecessary and wrong. All three checks now use the SFC form.F3 —
-tnow parses as UTC-tuseddate +%s(local) while-iand allRT_DATE_*usedate -u. On a non-UTC host (e.g. FMI/Helsinki EET) this shifted the requested reference time by the TZ offset. Added-u.F4 —
-i <file>ordering-iwithout-tran thelatest()incoming-dir scan first and could exit "No data available" before reading the-ifile — exactly the case where incoming may be empty. And even on success,RT_DATE_*/OUTNAME/TMPwere derived from thelatest()time, thenRTwas overwritten by the file's time afterwards, so the output filename used the wrong reference time. Reordered:-ideterminesRTup front, before the derivations;latest()is skipped when-iis given.Notes
bash -nandshellcheck(no errors). I can't exercise thegrib_get/qdtoolspipeline on macOS — the real check is a two-runs-no-change test on a host plus an icon-europe run showing no PLgribtoqdattempt.v26.5.22). Left out so you pick the version.Test plan
gribtoqdon a directory)…_pressure.sqdcompleteness now matches on the second run (no perpetual re-convert)-t 20260522T12on a non-UTC host yields reference time 12:00 UTC-i /tmp/file.gribwith an empty incoming dir converts the file instead of exiting "No data available"