Refactor constraint pipeline to per-base architecture with v4.1.1 support - #51
Open
jkgoodrich wants to merge 40 commits into
Open
Refactor constraint pipeline to per-base architecture with v4.1.1 support#51jkgoodrich wants to merge 40 commits into
jkgoodrich wants to merge 40 commits into
Conversation
…iant_expected_ht`
…les. Introduce gene quality metrics computation in the constraint pipeline and refactor resource utility constants.
…ic loss-of-function annotations and streamline resource collection for the constraint pipeline. Remove deprecated code and improve clarity in resource utility functions.
…unused `print_global_struct` function and replace `aggregate_expected_variants_expr` with `aggregate_constraint_metrics_expr` in the aggregation process. Introduce new utility functions for better organization and clarity.
…tion. Replace deprecated functions with updated counterparts, streamline percentile threshold calculations, and enhance the handling of observed and possible variants in the constraint pipeline.
…ring. Introduce new utility functions for filtering test data and reading adjacency data, while updating the context preparation process to streamline annotations for coverage, allele number, and site frequency spectrum. Remove deprecated functions and improve overall code organization.
…get_syn_adj_r_ht` function for reading synonymous DNM adjacency data and update context preparation to include synonymous adjacencies in annotations. Enhance overall clarity and organization of the constraint pipeline.
…raint into jg/switch_constraint_to_per_base
…iltering. Replace population references with genetic ancestry group terminology and improve documentation for downsampling logic. Streamline annotations for observed and possible variants in the constraint pipeline.
…supported versions list. Adjust current version constant accordingly.
…on. Add functionality to compute gene quality metrics and streamline aggregation processes. Implement new command-line arguments for applying models on aggregated data and using aggregated expected outputs. Refactor utility functions for clarity and organization.
…ments to the README and CLAUDE.md files. Revise project overview, structure, and pipeline steps to reflect the current version 4.1.1. Add details on new command-line arguments and resource paths, while improving clarity and organization throughout the documentation.
…. Update keep_regions for both GRCh37 and alternative chromosome formats to include specific genomic intervals for PCSK9.
…filtering. Exclude non-Python files from zip builds to reduce size. Introduce a mapping for sites versioning and refactor gene quality metrics computation to utilize updated data structures. Improve clarity in utility functions and adjust documentation accordingly.
…to include segmental duplication and low-complexity region metrics. Update related utility functions for improved clarity and efficiency. Enhance resource handling by integrating new genomic interval data for more accurate quality assessments.
…eline. Add utility function to retrieve thresholds path and update resource handling to include new TSV export functionality. Refactor related code for clarity and consistency.
…se in the constraint pipeline. Introduce new command-line argument for mutation rate preparation, update resource handling, and implement utility function for mutation rate Table preparation. Enhance logging for export processes.
…raint into jg/switch_constraint_to_per_base
…s, path conventions, and shared structures for version 4.1.1.
ch-kr
added a commit
to broadinstitute/regional_missense_constraint
that referenced
this pull request
Aug 13, 2026
`create_constraint_prep_ht` read the per-variant expected dataset at gnomad-constraint's `CURRENT_VERSION`, which moved from 4.1 to 4.1.1 in the per-base refactor (broadinstitute/gnomad-constraint#51). The coverage corrected dataset only exists for 4.1, so the read resolved to a nonexistent path and `prep-constraint` failed immediately: HailException: No file or directory found at gs://gnomad/v4.1.1/ constraint_coverage_corrected/apply_models/transcript_consequences/ gnomad.v4.1.1.per_variant_expected.coverage_corrected.ht The version is now pinned in `CONSTRAINT_VERSION` and passed explicitly, so the read stops tracking a default that moves underneath it. This affects any freeze, not just freeze 3. Assisted-by: ClaudeCode:claude-opus-5
Bins are computed as hl.int(rank * multiplier / n_rows) over 0-based ranks, so they run from 0 to multiplier - 1. The README documented them as 1-based. Assisted-by: ClaudeCode:claude-opus-5[1m]
Rank and percentile bin annotations are no longer added inside compute_constraint_metrics. They are applied by the now-public compute_constraint_percentile_bins, which runs as a second phase over an intermediate constraint_metrics_pre_rank Table. The ranking moved from third to last in the orchestration; nothing downstream of it reads the rank or bin fields, so the annotations are additive and order-independent. This allows the ranking to be recomputed without rerunning the metrics, needed to reissue v4.1.1 as v4.1.2 with corrected bins (broadinstitute/gnomad_production#2202). --skip-pre-rank-metrics reuses the existing pre-rank Table and recomputes only the ranking, and --use-mane-select-over-canonical exposes the transcript filter that the ranking phase now takes directly. With --skip-pre-rank-metrics the step declares the pre-rank Table as an input and drops its upstream pipeline inputs, since the aggregated constraint group and gene quality metrics Tables are not read in that mode and need not exist for the target version. Without this the input existence check fails on a missing constraint_group.ht, which --overwrite does not suppress. Register 4.1.2 in VERSIONS and SITES_VERSION_MAP. CURRENT_VERSION is left at 4.1.1, so runs pass --version 4.1.2 explicitly. Assisted-by: ClaudeCode:claude-opus-5[1m]
jkgoodrich
force-pushed
the
jg/switch_constraint_to_per_base
branch
from
August 18, 2026 20:07
02f9658 to
0470d69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major refactor of the gnomAD constraint pipeline from a grouped/aggregated approach to a per-base (per-variant) architecture. Adds version 4.1.1 support, introduces new pipeline steps, extracts constants into a dedicated module, and adds release preparation for the mutation rate table.
Architecture changes
apply_models()step is replaced by a three-step per-variant path:--apply-models-per-variant— compute expected counts per variant--aggregate-per-variant-expected— sum per-variant counts by transcript/consequence--aggregate-by-constraint-groups— group into constraint categories (lof, mis, syn)--apply-models-aggregatedchains all three steps without writing per-variant intermediates, for faster iteration--use-aggregated-expectedflag allows--aggregate-by-constraint-groupsto read from the aggregated path output instead of per-variantNew pipeline steps
--create-training-set--apply-models-per-variant--aggregate-per-variant-expected--aggregate-by-constraint-groups--apply-models-aggregated--compute-gene-quality-metrics--prepare-release-mutation-rate--export-release-downsampling-tsvconstraint.pyutility refactoradd_vep_context_annotations->prepare_context_ht;annotate_context_ht->preprocess_data;create_observed_and_possible_ht->create_training_set;apply_models(local) -> usesgnomad.utils.constraint.apply_models+ newcreate_per_variant_expected_htcreate_per_variant_expected_ht,aggregate_per_variant_expected_ht,create_aggregated_expected_ht,aggregate_by_constraint_groups,compute_gene_quality_metrics,prepare_release_mutation_ht,prepare_release_ht, plus helpersassemble_constraint_context_ht,build_constraint_consequence_groups,aggregate_constraint_metrics_expr,calibration_model_group_expr,variant_observed_and_possible_expr,count_observed_and_possible_by_group,compute_percentile_thresholds,rank_array_element_metrics, etc.)resource_utils.pyrefactorconstants.pymodule (VERSIONS, DATA_TYPES, MODEL_TYPES, POPS, COVERAGE_CUTOFF, MU_GROUPING, etc.)get_per_variant_apply_ht,get_aggregated_expected_ht,get_constraint_group_ht,get_gene_quality_metrics_ht,get_release_mutation_ht,get_release_mutation_tsv_path,get_loeuf_thresholds_pathfilter_for_test,get_adj_r_ht,get_syn_adj_r_ht,get_all_sites_anSITES_VERSION_MAPto handle v4.1.1 using v4.1 sites releasekeep_regionsfor test filteringPipelineResourceCollectionandPipelineStepResourceCollectionimports moved fromconstraint_pipeline.pytoresource_utils.pyconstants.pyCentralizes all pipeline constants including:
VERSIONS,CURRENT_VERSION(4.1.1),SITES_VERSION_MAP,DATA_TYPES,MODEL_TYPES,GENOMIC_REGIONSMU_GROUPING,CALIBRATION_GROUPING,AGGREGATE_SUM_FIELDSRELEASE_KEY_ORDER,RELEASE_CG_RENAME,RELEASE_CG_SELECT,RELEASE_PIPELINE_PARAM_GLOBALS,RELEASE_GROUP_NAMES,RELEASE_LOF_FIELDS,CONSTRAINT_GRANULARITIES,PLI_EXPECTED_VALUESCLASSIC_LOF_ANNOTATIONS,SFS_BIN_CUTOFFS,ADJ_FREQ_METAOther changes
CURRENT_VERSION4.1 -> 4.1.1, added toVERSIONSlistCLAUDE.mdwith code style, gotchas, and dataproc submission instructions; expandedREADME.mdwith full pipeline step table, resource paths, and constraint_metrics schema.gitignore: Added.claude/andCLAUDE.local.md