Skip to content

feat(fine-mapping): GWAS Catalog fine-mapping manifest generator#1270

Open
project-defiant wants to merge 4 commits into
devfrom
fine-mapping-manifest
Open

feat(fine-mapping): GWAS Catalog fine-mapping manifest generator#1270
project-defiant wants to merge 4 commits into
devfrom
fine-mapping-manifest

Conversation

@project-defiant

@project-defiant project-defiant commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a FineMappingManifest dataset and a GWASCatalogFineMappingManifestGenerator step that joins the fine-mapping planner output with the GWAS Catalog study index to produce a single TSV manifest file — the input required by the downstream MultiSuSiE Nextflow pipeline.

What's new

  • FineMappingManifest dataset (src/gentropy/dataset/fine_mapping.py) — new Dataset-derived class with columns runId, studyId, route, summarystatsLocation, majorAncestry, traitFromSourceMappedIds (comma-joined, sorted, deduped).
  • GWASCatalogFineMappingManifestGenerator step (src/gentropy/finemapping_manifest.py) — filters the study index to GWAS Catalog studies with sumstats + LD population structure, resolves majorAncestry via LDAnnotator._get_major_population, joins with the planner output, and writes a single TSV via toPandas().to_csv(). Optional summary_statistics_glob overrides summarystatsLocation with live GCS paths.
  • Hydra config (src/gentropy/config.py) — GWASCatalogFineMappingManifestConfig registered as step=gwas_catalog_finemapping_manifest.
  • 4 tests (tests/gentropy/step/) — 2 orchestration (mocked) + 2 integration (real Spark): TSV content, schema, ancestry derivation, trait dedup/sort, missing-study warning.
  • Constraint fixes (constraint.py, constraint_set.py) — HasAllowedMajorAncestry now guards against null/empty ldPopulationStructure before indexing; _compute_major_ancestry uses LDAnnotator._get_major_population directly; traitSet is array_sort(array_distinct(...)) for stable ordering.

Real-data run (GWAS Catalog, release 26.06 sumstats susie study index — 214,716 studies)

The manifest was generated against the full GWAS Catalog study index using the planner output from the preceding step.

Fine-mapping plan (gs://gwas_catalog_multi_ancestry_fine_mapping/plan/):

Metric Value
Total studies in plan 214,716
Eligible runs (non-null runId) 5,288
Ineligible studies 202,640
Run sizes pairs: 3,788 · triplets: 1,500
Ancestry combinations AFR+NFE: 3,617 · EAS+NFE: 166 · AFR+EAS+NFE: 1,500 · AFR+EAS: 5

Manifest (gs://gwas_catalog_multi_ancestry_fine_mapping/manifest.tsv):

Metric Value
Total rows 12,076
Distinct runs 5,288
Sumstat paths resolved via glob gs://gwas_catalog_inputs/harmonised_summary_statistics/GCST*

A test manifest (4 triplet runs, 12 rows, traits EFO_0004309 / EFO_0004541 / EFO_0004611 / EFO_0004833) has been uploaded to gs://ot_orchestration/tests/multi_ancestry_fine_mapping/manifest.tsv for pipeline integration testing.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Dataset documentation Improvements or additions to documentation Feature Method size-L Step

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants