feat(l2g): add EGL and training set generation steps

[addramir]

Context

Closes the automation gap described in opentargets/issues#3783 — the L2G gold standard was being generated outside gentropy from ad-hoc notebooks (EGL_and_training_set/2606/01_EGL_preparation.ipynb and 02_training_set.ipynb). This PR brings that logic into the pipeline as two parametrised steps.

What's added

EffectorGeneListStep (gentropy.effector_gene_list)

Builds the Effector Gene List (EGL): a deduplicated set of (diseaseId, targetId) pairs. It combines up to three optional platform-ETL sources, each with its own parametrised filter:

Source	Input	Filter param
Rare-variant evidence	rare_variant_evidence_paths (list, unioned)	rare_variant_score_threshold (0.75)
ChEMBL clinical precedence	clinical_evidence_path	approved_clinical_phases (PHASE_4/APPROVAL/PHASE_3/PREAPPROVAL)
Legacy OTG gold standard	gold_standard_path (JSON)	gold_standard_confidence (High/Medium)

Output: one clean parquet with distinct diseaseId, targetId.

TrainingSetStep (gentropy.training_set)

Labels the L2G feature matrix against the EGL (positive/negative) and applies a chain of toggleable, parametrised cleaning filters mirroring notebook 02:

• replication (apply_replication_filter, default on; min_replication_studies=2)
• max positives per locus (max_gsp_per_locus=2)
• protein-protein interaction / STRING (apply_interaction_filter, interaction_source, interaction_score_threshold)
• distance-leakage removal (apply_distance_filter)
• protein-coding restriction (protein_coding_only)
• deduplication of identical positive feature profiles (apply_deduplication)

Output: clean JSON matching the notebook example (studyLocusId, geneId, diseaseIds, variantId, studyId, goldStandardSet).

Also

• Config dataclasses EffectorGeneListConfig / TrainingSetConfig registered as effector_gene_list and training_set steps.
• Docs pages under docs/python_api/steps/.
• Unit tests for both steps (EGL end-to-end; training-set filter logic + validation). 11 tests, all green.

Notes for review

• Output format matches the notebook example rather than the L2GGoldStandard schema (kept diseaseIds array), per request. It does not yet plug directly into LocusToGeneTrainTestSplitStep; happy to add a schema-conformant variant if wanted.
• Replication filter defaults to on (notebook's final set used the _no_repl path — it's a single flag to flip).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces two new pipeline steps to generate the L2G gold standard inside gentropy (replacing the prior notebook-based process): an Effector Gene List (EGL) builder and a Training Set generator that labels and filters the L2G feature matrix.

Changes:

• Added EffectorGeneListStep to assemble a deduplicated (diseaseId, targetId) EGL from optional platform/legacy sources with configurable filters.
• Added TrainingSetStep to label the L2G feature matrix against the EGL and apply a configurable chain of cleaning/leakage-reduction filters, writing the final JSON output.
• Registered both steps in Hydra config, added API docs pages, and added unit tests for EGL and training-set filters/validation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/gentropy/effector_gene_list.py	New EGL step implementation combining optional evidence sources into a distinct parquet output.
src/gentropy/training_set.py	New training set step implementation: annotate + label + optional filters + JSON output.
src/gentropy/config.py	Adds EffectorGeneListConfig and TrainingSetConfig and registers both steps.
tests/gentropy/step/test_effector_gene_list_step.py	End-to-end tests validating EGL source combination and filtering.
tests/gentropy/step/test_training_set_step.py	Unit tests for training-set labeling and filter logic + constructor validation.
docs/python_api/steps/effector_gene_list.md	API docs stub for the EGL step.
docs/python_api/steps/training_set.md	API docs stub for the training set step.

Comments suppressed due to low confidence (2)

src/gentropy/effector_gene_list.py:101

• Calling .count() just to log source sizes triggers a full Spark action (and will likely be repeated again during the union/write), which can be very expensive at pipeline scale. Prefer logging that the source was included (or compute counts only under an explicit debug flag with caching if really needed).

            logger.info("Clinical source contributes %d pairs", clinical.count())

src/gentropy/effector_gene_list.py:108

• Calling .count() just to log source sizes triggers a full Spark action (and will likely be repeated again during the union/write), which can be very expensive at pipeline scale. Prefer logging that the source was included (or compute counts only under an explicit debug flag with caching if really needed).

            logger.info("Gold standard source contributes %d pairs", gold.count())

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.