enh: warn loudly when no latent variable is significant and reliable - #22
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up: make the null result unmissable
When a run finds no latent variable that is both significant (p < 0.05) and reliable (|bootstrap ratio| > 1.96 on both sides),
run_pipelinecompletes normally — which is correct, a null result is a legitimate outcome. But it was communicated far too quietly:significant_lvs: []inlog.txtand an INFO line,subject_scores.csvcontaining just the subject-ID index with no score columns,A user skimming the
figures/folder just saw missing plots with no explanation. This adds an explicitWARNINGat the end of the run whenfinal_lv_namesis empty, spelling out the criteria and that those plots were skipped. The graceful-completion behaviour is unchanged.Changes
pipeline.run_pipeline: branch the end-of-run summary —INFOlisting the LVs when any survive, aWARNINGdescribing the null result when none do.Scope note
The cross-validation pipeline does not have the same silent-null problem — it always reports accuracy, a permutation p-value, and figures regardless of significance, so there is no skipped output to warn about. Flagging "doesn't beat chance" there would stray into interpretation, so it's deliberately left alone.
Tests (written first, watched fail)
TestNullResultWarning::test_warns_when_no_lvs_survive— forces an all-falsefinal_lvs(monkeypatchingPLS.filter_lvs) and asserts aWARNINGrecord is emitted.test_no_warning_when_lvs_survive— a normal run (which keeps ≥1 LV on the synthetic data) emits no such warning.All 183 tests pass; ruff clean; coverage 98%.
🤖 Generated with Claude Code