Influence-robust jackknife gate for the observational sensory relate#466
Merged
Conversation
Sparse, wide predictor blocks (many descriptors non-zero on only a single product) let a lone high-leverage observation manufacture a "driver": after unit-variance scaling that single non-zero becomes a fixed spike, so its entire apparent relationship to a response is one product. Such an effect passes a bare FDR gate and survives cross-validation, because the column contributes nothing to any out-of-fold prediction. Gate both observational relate paths on an exhaustive leave-one-out jackknife: - Marginal associations: attach a Fisher-z jackknife (`jackknife_se`, `influence_robust`, `n_supporting`); `significant` now requires both Benjamini-Hochberg rejection and jackknife robustness. - Discriminator: reuse the LOO refit already run for the Q-squared gate to read Martens' per-coefficient jackknife interval (`jackknife_significant`); `discriminator_significant` additionally requires that interval to exclude zero. The jackknife adds no threshold of its own - it reuses the existing `alpha` and the number of observations - so a genuine multi-observation driver is unaffected while a single-support spike is demoted. Deterministic (LOO is exhaustive). Tests add a two-attribute case study: a single-support spike whose in-sample correlation is strong (r ~ 0.99, p < 1e-4) is demoted, while a smooth multi-product driver stays significant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018M9u5EpSjHa3mjNaYDJ7qq
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A jackknife standard error of exactly zero means every leave-one-out deletion returns the same correlation, i.e. the association is maximally stable. Treat it as influence-robust when the correlation is non-zero, instead of the previous blanket demotion of the zero-spread case. Add unit coverage for the helper's degenerate branches (too few observations, single-support collapse, zero-spread perfect correlation), addressing the uncovered patch lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018M9u5EpSjHa3mjNaYDJ7qq
The `not np.isfinite(se)` guard in the correlation jackknife was dead code: the correlations are clipped before the Fisher-z transform and the caller guarantees non-constant inputs, so the jackknife standard error is always finite. Remove it and note why. Add a small-sample discriminator test so the branch where the leave-one-out gate is skipped (fewer than five products, jackknife undefined) is exercised and every descriptor correctly defaults to not-confirmed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018M9u5EpSjHa3mjNaYDJ7qq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
process_improve.sensory.analyze_descriptivenow gates both observational relate paths on an exhaustive leave-one-out jackknife:jackknife_se,influence_robust,n_supporting);significantnow requires both Benjamini-Hochberg rejection and jackknife robustness.jackknife_significant);discriminator_significantadditionally requires that interval to exclude zero.alphaand the number of observations - so a genuine multi-observation driver is unaffected while a single-support spike is demoted. It is deterministic (LOO is exhaustive; no RNG). All added record fields are additive; existing keys and their meanings are unchanged.Test plan
uv run pytest tests/test_sensory.py tests/test_sensory_end_to_end.py --no-cov(45 passed). New case study: a single-support spike with a strong in-sample correlation (r ~ 0.99, p < 1e-4) is demoted on both paths, while a smooth multi-product driver stays significant. Verified the demotion is not over-pruning: genuine drivers keepjackknife_significant, and a pre-change comparison shows the discriminator's permutation gate behaviour is unchanged.uv run ruff check .passes.uv run mypy src/process_improvepasses (139 files).Checklist
pyproject.toml(MINOR:1.55.1->1.56.0, behavioural change to relate significance;CITATION.cffsynced)ruff check .passesCHANGELOG.mdupdated🤖 Generated with Claude Code
https://claude.ai/code/session_018M9u5EpSjHa3mjNaYDJ7qq
Generated by Claude Code