Hypergraph backboning + Backbone 3.0 parity (mtc/signed/SDSM-EC) + reference updates - #7
Merged
Merged
Conversation
Evaluate incorporating hypergraph backboning methods (in the spirit of arXiv:2606.00893) into the library. Documents two method families (projection backboning vs. direct hyperedge filtering), shows that projection backboning already works via the existing bipartite incidence machinery, and proposes a phased plan (Phase 0 adapters+docs, Phase 1 a hypergraph module) with a representation decision and testing strategy. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Expand the hypergraph backboning proposal to (1) optional interoperability with xgi, hypergraphx, HyperNetX, and Hypergraph-Analysis-Toolbox, centered on the shared incidence/bipartite substrate and the HIF interchange format (no required dependencies), and (2) a method inventory distinguishing generalizations of existing backbones from genuinely hypergraph-native methods worth porting: statistically validated hypergraphs (SVH) and cores (SVC), toplex/inclusion reduction, s-connectivity/s-line backbones, and order-resolved hyperedge filtering. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
…L method) After reading arXiv:2606.00893 (Kirkley, Felippe, Malizia & Battiston, 'Hypergraph backboning') in full: the method is a parameter-free, information-theoretic (MDL) compression backbone that prunes nested/redundant hyperedges, with a weighted extension (single knob gamma) -- a distinct paradigm from the statistical SVH/SVC family the first draft assumed. Re-scope: document the actual objective (parent/child encoding, reduced mutual information, inverse compression ratio eta), the weighted empirical-Bayes model, the intersection-graph + greedy star-partition optimizer and its complexity, and inputs/outputs. Reframe around three paradigms (projection / statistical / MDL); make the MDL backbone the headline Phase 1 method (networkx + numpy/scipy, no required third-party hypergraph dep); keep SVH/SVC and structural methods as Phase 2. Resolve the prior open questions from the paper. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
Implement Phase 1 of the hypergraph backboning proposal: a new networkx_backbone.hypergraph module providing the parameter-free, information-theoretic (minimum description length) hypergraph backbone of Kirkley, Felippe, Malizia & Battiston (arXiv:2606.00893), which prunes nested and redundant hyperedges and extends to weighted hypergraphs. - mdl_hypergraph_backbone: greedy 'edge' optimizer over the intersection graph (star-partition of parent/child hyperedges); unweighted objective plus the empirical-Bayes weighted model (Poisson/Geometric priors, single knob gamma). - hypergraph_compression_ratio: inverse compression ratio eta. - intersection_graph: hyperedges linked when they share >= 1 node. - HypergraphBackbone result dataclass (backbone, assignment, eta, diagnostics). Implemented with stdlib math (lgamma/log2) + networkx only, so it stays within the core dependency footprint (no numpy/scipy required). Inputs/outputs use plain hyperedge collections since NetworkX has no native hypergraph type. Adds 29 tests reproducing the paper's qualitative claims (recovery of top faces of nested simplices, eta behavior, gamma=1 == unweighted, gamma->0 forcing high-weight hyperedges into the backbone) plus edge cases and validation. Wires the module into the package API and docs (api page, concepts, README). https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
…formula The disparity filter formula was corrected in 73ccc6e to the canonical Serrano et al. (2009) form alpha = (1 - p)^(k-1) (the integral of the null density), replacing the previous alpha = 1 - (k-1)(1-p)^(k-2), which produced negative values clamped to 0 and marked almost every edge significant. The Les Miserables benchmark still encoded the old behavior (247 edges kept at alpha<0.05). Verified independently that the corrected formula keeps 9 edges on les_miserables_graph() at alpha<0.05 (the old formula reproduces 247), so the implementation is correct and the stale expectation is updated to 9. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Add the structural sui generis hypergraph methods from the design proposal, implemented with stdlib + networkx only: - maximal_hyperedges: inclusion (toplex) reduction -- keep hyperedges not contained in any other (cf. HyperNetX toplexes). - order_filter: order-resolved filtering -- select hyperedges by size/order. - s_components: s-connected components (hyperedges sharing at least s nodes). - intersection_graph: generalized with an s parameter, yielding the s-line graph for s > 1 (s=1 preserves prior behavior). These return plain hyperedge collections (no HypergraphBackbone), since they are structural utilities rather than the MDL optimizer. Adds 10 tests and wires the functions into the package API and docs. SVH/SVC (statistical, scipy-based) remain for a follow-up. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
Add the statistical hypergraph backbone methods from Musciotto, Battiston & Mantegna (2021), faithfully re-implemented from Hypergraphx get_svh/get_svc: - statistically_validated_hypergraph (alias svh): validates observed hyperedges that recur more than expected per order, with the binomial-tail p-value P(X >= n), X ~ Binomial(N, prod d_i / N), and a Benjamini-Hochberg FDR corrected for the number of possible order-k hyperedges. - statistically_validated_cores (alias svc): validates significant groups (including sub-groups), processed high-to-low order with sub-combinations of validated cores removed. - ValidatedHypergraph result dataclass. scipy is imported lazily (consistent with the statistical/bipartite modules); the module remains importable without it. Multiplicities are taken from repeated hyperedges or explicit integer weights. P-values verified exactly against scipy.stats.binom; planted over-represented groups validate while expected background does not; SVC drops sub-combinations of validated cores. Adds 14 tests and wires the methods into the package API and docs. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Add networkx_backbone/hypergraph_io.py converting the hyperedge-list representation to and from: - a NetworkX incidence bipartite graph (hypergraph_to_bipartite), so the existing bipartite projection backbones (sdsm/fdsm/fixed*) apply to hypergraphs; - the HIF (Hypergraph Interchange Format) JSON standard (read_hif/write_hif), stdlib-only, cross-checked against xgi in both directions; - the xgi, HyperNetX, HypergraphX, and HAT hypergraph classes via lazy from_*/to_* adapters (no required dependency; friendly ImportError if absent). Adds 12 tests: real round-trips through xgi and hypergraphx, HIF round-trips (dict/file/weights) and an xgi cross-check, sdsm driven from a converted hypergraph, and an availability-aware adapter test that round-trips when a library is installed and asserts a helpful ImportError otherwise. Wires the functions into the package API, docs, and a README hypergraph quick-start. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
Implement the second greedy scheme from Kirkley et al. 2026 (Appendix D) and a method switch on mdl_hypergraph_backbone: - method="edge": greedy parent-child link addition (previous behavior). - method="node": facility-location-style greedy that adds backbone parents by decreasing total parent-child savings, then force-covers the remainder. - method="auto" (new default): run both and keep the lower description length, matching the paper's procedure. On a downward-closed simplex the node scheme compresses better (backbone 8 vs 14), and auto selects it. Refactor the optimizer into _greedy_edge/_greedy_node/_description_length helpers. Add tests for all three methods (nested-simplex recovery, auto == min(edge, node), node-mode weighting) and fix the now-stale invalid-method test. Docs: add a hypergraph backbone tutorial (MDL with the method switch, structural methods, SVH/SVC, interoperability) wired into the tutorials toctree; note the method options in the README quick-start; mark the optimizer choices resolved in the design doc. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Add docs/design/backbone-3.0-coverage.md evaluating networkx-backbone against Neal's Backbone 3.0 (PLOS ONE, 2026), based on the authoritative zpneal/backbone v3.0.4 source. Finding: every backbone *model* in Backbone 3.0 is already implemented here (disparity, mlf, lans, sdsm, fdsm, fixedfill/row/col, bicm, fastball, backbone_from_* wrappers), including its hypergraph-projection input via hypergraph_to_bipartite; the library exceeds it (noise-corrected, ECM, MLA, structural/proximity families, and the hypergraph module). The differences are cross-cutting features, with proposed implementations: multiple-testing correction (mtc), signed backbones, SDSM-EC edge constraints, and narrative methods text. Adds the Backbone 3.0 and SDSM-EC citations to the README references and a coverage note. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
Add adjust_pvalues(pvalues, method) reproducing R's p.adjust for the corrections in Backbone 3.0's mtc option: bonferroni, holm (step-down), hochberg (step-up), bh/fdr (Benjamini-Hochberg), by (Benjamini-Yekutieli), and none. BH/BY verified exactly against scipy.stats.false_discovery_control; pure Python, no new deps. Wire an `mtc` parameter into threshold_filter that adjusts the score values across all tested edges/nodes before thresholding (valid only with mode="below"). Default mtc="none" preserves existing behavior. Adds tests and docs; marks the mtc gap resolved in the coverage evaluation. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Add a `signed` option to the statistical scorers, matching Backbone 3.0: a two-tailed test that retains significantly strong (sign +1) and significantly weak (sign -1) edges, annotating each with a `sign` attribute and storing a two-sided p-value (2*min(upper, lower), clipped to 1). - Weighted filters: disparity_filter, marginal_likelihood_filter, lans_filter (and the disparity/mlf/lans aliases). Lower tails: disparity CDF, binomial cdf, empirical CDF (OR rule across endpoints). - Projection null models: sdsm, fdsm, fixedfill, fixedrow, fixedcol. Lower tails: normal cdf, Monte-Carlo below-count, binomial/hypergeometric cdf. - Wrappers: backbone_from_weighted gains signed (+ mtc) pass-through; backbone_from_projection forwards signed to the fixed* models too. signed=False (default) leaves behavior and stored p-values unchanged. Adds tests verifying strong/weak sign assignment, two-sided p-values, determinism, and that unsigned output is unchanged. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Add optional prohibited/required parameters to sdsm implementing the Stochastic Degree Sequence Model with Edge Constraints (Neal & Neal 2023): cells named as (agent, artifact) pairs are fixed to null probability 0 (prohibited) or 1 (required) before the Poisson-binomial test. Defaults (None) leave behavior unchanged; unknown nodes are ignored; constraints compose with signed and are forwarded by backbone_from_projection via kwargs. Adds tests and the SDSM-EC reference. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
…ncepts - README: update the Backbone 3.0 coverage note (features now implemented) and add a "Significance options" quick-start example (mtc + signed). - statistical tutorial: add "Multiple-testing correction" and "Signed backbones" sections. - bipartite tutorial: add "Signed backbones" and "Edge constraints (SDSM-EC)" sections + SDSM-EC reference. - concepts: note adjust_pvalues/mtc and signed on the statistical methods. - coverage design doc: mark mtc/signed/SDSM-EC implemented; narrative parked. - bump function count (87) for adjust_pvalues. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
Reviewed the Backbone 3.0 bibliography (zpneal/backbone vignette bib) and added the canonical references for implemented methods that were missing from the docs: - Saracco et al. (2015) -- Bipartite Configuration Model (added to bicm docstring and README) - Neal, Domagalski & Sagan (2021) -- FDSM / fixed models (README, bipartite tutorial) - Godard & Neal (2022) -- fastball (added to fastball docstring, README, tutorial) - Foti et al. (2011) -- LANS (README, statistical tutorial) - Dianati (2016) -- MLF (README, statistical tutorial) Also: complete the Neal 2026 citation (PLOS One, 21, e0349258); fix a merged Satuluri/Serrano reference line; cite the multiple-testing corrections (Holm 1979, Hochberg 1988, Benjamini-Hochberg 1995, Benjamini-Yekutieli 2001) in adjust_pvalues and the statistical tutorial; add a References section to the statistical tutorial and expand the bipartite tutorial references. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
brianckeegan
changed the title
The docs CI builds with -W (warnings as errors) and failed with 18 warnings introduced by the new hypergraph API page: - 17x "duplicate object description" for HypergraphBackbone/ValidatedHypergraph attributes: conf.py sets autodoc_default_options members=True globally, so the autoclass directives documented the dataclass fields in addition to numpydoc rendering each class's "Attributes" docstring section. Add :no-members: to both autoclass directives so only the numpydoc Attributes section is rendered. - 1x "py:mod reference target not found: networkx_backbone.hypergraph_io" in concepts.rst: the module had no automodule directive to resolve the :mod: xref. Register it with `.. automodule:: networkx_backbone.hypergraph_io :no-members:` on the hypergraph API page. Verified locally: build now emits 0 structural warnings (only environment-local intersphinx inventory fetch failures remain, which do not occur in CI). https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
GitHub runners deprecate Node.js 20 (forced to Node 24 on 2026-06-16). Update the JavaScript actions across all workflows: - Bump actions/checkout v4 -> v5 and actions/setup-python v5 -> v6 (Node 24), the two actions named in the runner deprecation warning. - Set the GitHub-sanctioned FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true workflow-level env in every workflow so the remaining JS actions still on Node 20 (upload-artifact, download-artifact, upload-pages-artifact, deploy-pages, and conda-incubator/setup-miniconda) run on Node 24 without risky version guesses. YAML validity and top-level env placement verified for all four workflows. https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
networkx_backbone.hypergraphmodule +hypergraph_io, brings the statistical methods to feature-parity with Neal's Backbone 3.0 (mtc / signed / SDSM-EC), fixes a stale disparity benchmark, and fills documentation reference gaps.Hypergraph backbones (
hypergraph.py,hypergraph_io.py)mdl_hypergraph_backbonewith amethodswitch (edge/node/auto), weighted viagamma;hypergraph_compression_ratio;HypergraphBackbone.maximal_hyperedges,order_filter,s_components,intersection_graph(s=...).statistically_validated_hypergraph/_cores(Musciotto et al. 2021; verified vsscipy.stats.binom).hypergraph_to_bipartite,read_hif/write_hif, and lazyfrom_*/to_*for xgi / HyperNetX / HypergraphX / HAT.Backbone 3.0 feature parity (new this round)
mtcmultiple-testing correction:adjust_pvalues(Bonferroni/Holm/Hochberg/BH/BY; BH & BY verified againstscipy.stats.false_discovery_control) + anmtcargument onthreshold_filter.signedbackbones: two-tailed option ondisparity_filter/mlf/lansandsdsm/fdsm/fixedfill/fixedrow/fixedcol, adding a"sign"attribute (+1 strong, −1 weak) and a two-sided p-value; forwarded by thebackbone_from_*wrappers.sdsm(prohibited=, required=)edge constraints (Neal & Neal 2023).signed=False/mtc="none"keep prior behavior.Fixes & docs
α=(1−p)^(k−1)(independently verified); updated the stale Les Misérables expectation (247 → 9).bicm/fastball/adjust_pvaluesdocstrings, and the statistical/bipartite tutorials.backbone-3.0-coverage.md).Tests
Full suite: 287 passed, 1 skipped (was 215 with 1 failing at the start of this work). New coverage for the MDL method switch, mtc, signed, SDSM-EC, and the adapters (real round-trips through xgi/hypergraphx).
Note on versioning
The version is
setuptools-scm-driven (git tags). A0.3.0release bump is deferred until this PR is merged — at that point it is av0.3.0tag on the default branch (and a matchingdocs/conf.pyreleaseupdate), per the maintainer's request.https://claude.ai/code/session_01TEehb7gmfJc8WfUNSjs7eD