Skip to content

diagnostics: log inputs, computed max_order, and warn when write_new_parameters produces no output#177

Merged
Sebokolodi merged 2 commits into
masterfrom
fix/rescale-I-model-diagnostics
Jul 17, 2026
Merged

diagnostics: log inputs, computed max_order, and warn when write_new_parameters produces no output#177
Sebokolodi merged 2 commits into
masterfrom
fix/rescale-I-model-diagnostics

Conversation

@Sebokolodi

Copy link
Copy Markdown
Contributor

Motivation

write_new_parameters currently derives the number of output files from a max_order heuristic on new_coeffs:

max_order = (
    np.sum(np.any((new_coeffs != 0.0) & (~np.isnan(new_coeffs)), axis=(1, 2))) - 1
)
for i in range(max_order + 1):
    pf.writeto(...)

When new_coeffs is entirely 0.0 or NaN across every plane, max_order resolves to -1, the write loop becomes range(0), and no files are written, no exception is raised, no log line is emitted.

Downstream consumers that assume newcoeff*.fits exist can be hurt badly by this silent path. In the CIRADA POSSUM pipeline (POSSUM_Polarimetry_Pipeline/pipeline/modules/fracpol3d_Irescale.py), a subsequent delete-then-rename block would destroy the input coeff*.fits files and only crash much later with a FileNotFoundError at pf.getdata(coeff0.fits), far from the root cause. That has now been fixed defensively on the pipeline side (POSSUM_Polarimetry_Pipeline#33), but the underlying silent no-op inside RMtools deserves visibility here too.

Observed: tile 6614, 943MHz, RMtools 1.4.11 in production — the writer produced zero output files with no warning; only became visible when a downstream getdata crashed.

Changes (all diagnostics — no behaviour change on the happy path)

  • Add a module-level logger and a small _summarize_coeffs helper that returns a compact one-line descriptor of a coefficient array (shape, dtype, total elements, nan count, zero count, finite-nonzero count).
  • rescale_I_model_3D: log input coeffs summary at the start and output new_coeffs summary at the end (INFO).
  • write_new_parameters: log input summary and the computed max_order up front; log every file written with its full path; log a final total-files-written count at the end (INFO).
  • write_new_parameters: when max_order < 0, emit both logger.warning and warnings.warn(UserWarning) with a clear message identifying the condition ("every plane of new_coeffs is 0.0 or NaN"), then return explicitly. The historical behaviour of writing no files is preserved; the reason is now visible even to callers that haven't configured logging.

Non-goals

  • No behaviour change on inputs that previously wrote files. Same files, same content, same order.
  • Not changing the max_order heuristic itself. If it turns out the heuristic is wrong (e.g. failing on valid low-order fits under some conditions), that's a separate fix.
  • Not raising instead of warning. warnings.warn(UserWarning) is the loudest visibility change that doesn't break existing callers that treat the empty-output case as valid.

Test plan

  • Existing tests should pass unchanged (only additive log/warning code paths).
  • Manually construct new_coeffs filled with NaN, call write_new_parameters — expect a UserWarning and a logger warning, no files written (as before).
  • Manually construct valid new_coeffs — expect INFO log lines showing the summary, max_order, and per-file paths, plus the normal FITS output files.

Sebokolodi and others added 2 commits July 17, 2026 10:07
…parameters produces no output

write_new_parameters currently derives its output count from a max_order heuristic on new_coeffs (any-non-zero-non-nan planes). When new_coeffs is entirely 0.0 or NaN, max_order = -1, the write loop becomes range(0), and NO files are written — with no exception raised and no log line emitted.

Downstream consumers that assume newcoeff*.fits exist can be caught out badly by this: POSSUM_Polarimetry_Pipeline's fracpol3d_Irescale step, for example, will destroy its input coeff*.fits files in an ill-fated delete-then-rename block and only crash much later with a FileNotFoundError, far from the actual root cause. That has now been fixed defensively on the pipeline side, but the underlying silent no-op deserves visibility here too.

Changes (all logging, no behaviour change on the happy path):

  * Add a module-level logger and a small _summarize_coeffs helper for compact array descriptors (shape, dtype, nan/zero/finite-nonzero counts).

  * rescale_I_model_3D logs input coeffs and output new_coeffs summaries at INFO.

  * write_new_parameters logs input summary, computed max_order, planned file count, and each file it writes at INFO; total file count at end.

  * When max_order < 0, write_new_parameters now emits both logger.warning and warnings.warn(UserWarning) explaining the condition, and returns explicitly. Preserves the historical 'no files written' behaviour but makes the reason loudly traceable.

This surfaces the root cause of the tile 6614 (RMtools 1.4.11) incident observed in the CIRADA POSSUM pipeline, without changing what the function actually writes on any input where it did write something before.
@Sebokolodi
Sebokolodi merged commit bcfd15c into master Jul 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant