Skip to content

chore: trigger release process#1254

Open
github-actions[bot] wants to merge 8 commits into
mainfrom
dev
Open

chore: trigger release process#1254
github-actions[bot] wants to merge 8 commits into
mainfrom
dev

Conversation

@github-actions

@github-actions github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

⚠️ This PR requires a MERGE COMMIT (Don't squash!)

addramir and others added 2 commits July 6, 2026 11:52
@addramir @claude
@project-defiant
feat(l2g): add LocusToGeneModelTuningStep for pre-split CV and hyperp…
da85030 
…arameter tuning (#1230)

* feat(l2g): add LocusToGeneTrainTestSplitStep pipeline step

Introduce a dedicated step that builds the annotated gold-standard feature
matrix from the full feature matrix + credible set, performs a hierarchical
train/test split, and writes both partitions to parquet. A predefined test
set from a previous run can be supplied to keep the held-out set stable
across feature matrix updates.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): wire LocusToGeneStep to consume presplit train/test parquets

LocusToGeneStep no longer performs gold standard annotation or
train/test splitting internally. Instead it requires presplit parquets
produced by LocusToGeneTrainTestSplitStep, making the two steps
composable in a pipeline without redundant Spark computation.

- Remove gold_standard_curation_path, variant_index_path,
  gene_interactions_path from LocusToGeneStep and LocusToGeneConfig
- Remove prepare_gold_standard() and _annotate_gold_standards_w_feature_matrix()
- run_train() validates presplit paths are provided and loads them directly
- LocusToGeneTrainer.train() accepts presplit_train_df/presplit_test_df
  to bypass internal generate_train_test_split
- Add import pandas as pd at module level in l2g.py
- Add tests for LocusToGeneTrainTestSplitStep static helpers and
  LocusToGeneTrainer presplit bypass path

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): address Copilot review comments on LocusToGeneTrainTestSplitStep

- Write split stats JSON in the fresh-split branch (was only written
  for the predefined-test branch)
- Create parent directories before writing the JSON sidecar locally so
  the step doesn't fail on a fresh output path
- Remove coalesce(1) to avoid forcing a single output partition
- Unpersist annotated_fm after both branches write their parquets
- Replace toPandas() calls in the predefined-test branch with Spark
  left_anti / inner joins and create_map label encoding, keeping all
  data on the cluster and avoiding driver-memory limits

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): address second round of Copilot review comments

- Tighten _parse_gold_standard match: add guard so the unexpected_columns
  branch cannot fire when missing_mandatory_columns is also present
- Persist train_sdf/test_sdf before counting in the predefined-test branch
  to avoid recomputing the joins on the subsequent .write actions
- Raise ValueError in LocusToGeneTrainer.train() when exactly one of
  presplit_train_df / presplit_test_df is provided
- Make credible_set_path / feature_matrix_path optional in LocusToGeneConfig
  and LocusToGeneStep (predict mode validates they are present; train mode
  no longer loads them unnecessarily)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add split_stats_path parameter to LocusToGeneTrainTestSplitStep

Allows callers to specify an explicit output path for the split statistics
JSON instead of deriving it from train_parquet_path.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add per-split breakdown stats to LocusToGeneTrainTestSplitStep log

Adds n_positive, n_negative, n_unique_loci, n_unique_positive_genes, and
(when traitFromSourceMappedId is present) n_unique_positive_gene_disease_pairs
for both train and test splits, nested under "train"/"test" keys in the stats JSON.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix: leakage

* feat(l2g): add LocusToGeneCrossValidationStep for pre-split CV

New dedicated step that runs k-fold cross-validation on pre-built,
annotated train/test feature matrix parquets without any gold-standard
parsing or training set construction.

Trainer upgrades (ported from PR #1228):
- evaluate() extended with TP/FP/TN/FN confusion-matrix components
- cross_validate() now supports cv_results_dir: writes cv_results.json,
  cv_folds.csv and per-config ROC/PR/confusion-matrix PNGs to a local
  or GCS directory; hyperparameter grid is swept explicitly in file mode
- _run_cv_fold() extracts single-fold logic and adds locus-level stats
  (n_loci, n_loci_one_gene_above, n_loci_no_gene_above)
- _expand_grid() expands W&B-style grid to flat config list
- _summarise_and_plot_config(), _write_cv_files(), _upload_dir_to_gcs()
  handle file-mode output and GCS upload
- _plot_roc_curves(), _plot_pr_curves(), _plot_confusion_matrix() PNGs

LocusToGeneStep and its training path are unchanged.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix: tests

* test(l2g): add unit and integration tests for CV step and trainer utilities

Tests cover _expand_grid, _write_cv_files, cross_validate with cv_results_dir,
and LocusToGeneCrossValidationStep end-to-end including hyperparameter grid sweep.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(test): replace method-assign with patch.object for log_to_wandb mock

Direct instance method assignment is rejected by mypy; patch.object is
the correct way to stub a method in tests.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): address Copilot review comments on LocusToGeneCrossValidationStep

- Extract _L2G_FEATURES_LIST and _L2G_HYPERPARAMETERS as module-level
  constants in config.py; LocusToGeneConfig, LocusToGeneTrainTestSplitConfig,
  and LocusToGeneCrossValidationConfig all reference them to eliminate
  copy-paste duplication
- Unpersist train/test feature matrix Spark DataFrames in
  LocusToGeneCrossValidationStep after toPandas() materialises them
- Replace _upload_dir_to_gcs inline GCS client with the shared
  copy_to_gcs helper from gentropy.external.gcs

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add holdout row to CV results for each hyperparameter config

For each config evaluated during cross-validation, train on the full
training set and evaluate on the held-out test set, appending a
"holdout" row to cv_folds.csv alongside the per-fold rows.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): use Any type for list/dict fields in LocusToGeneCrossValidationConfig

OmegaConf cannot handle field(default_factory=lambda: ...) on list[str]
and dict[str, Any] typed fields in structured configs — it tries to
resolve default_factory as a dict key and raises ConfigAttributeError.
Apply the same Any workaround already used for hyperparameter_grid.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): handle OmegaConf DictConfig in hyperparameters_dict property

When LocusToGeneCrossValidationStep is instantiated via Hydra, the
hyperparameters field arrives as an OmegaConf DictConfig rather than a
plain dict. The property fell through to default_factory() which does
not exist on DictConfig. Fall back to dict() conversion instead.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): handle integer-encoded goldStandardSet in CV step

The train-test split step saves goldStandardSet as 0/1 integers.
LocusToGeneCrossValidationStep was unconditionally applying a
string-to-int map, silently producing NaN for all rows and leaving zero
positives for hierarchical_split. Now skips the map when the column is
already numeric. Adds a regression test with integer-encoded labels.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(test): make holdout row conditional on test data being set

_eval_on_test_set is skipped when x_test/y_test/test_df are None so
that unit tests calling cross_validate directly (without a full step
setup) still pass. Extracts _maybe_append_holdout_row helper to keep
cross_validate complexity within the C901 limit. Updates
test_cv_results_json_structure to expect n_splits + 1 fold_metrics
entries when test data is present.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add n_estimators to default hyperparameters

Set n_estimators=300 as the explicit default. With eta=0.05 the previous
implicit default of 100 trees gave a learning capacity (eta × n_estimators)
of 5, versus XGBoost's internal default of ~30. 300 trees brings capacity
closer to what the learning rate implies.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add gamma and max_delta_step to default hyperparameters

gamma=0 (min split gain, explicit default) and max_delta_step=1
(stabilises gradient updates for imbalanced binary classification,
per XGBoost docs recommendation).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* feat(l2g): add holdout_only mode to cross-validation step

When holdout_only=True, CV folds are skipped entirely. Each config is
trained once on the full training set and evaluated directly on the
holdout set. Produces a cv_folds.csv with one row per config (fold=holdout)
and n_splits=0 in cv_results.json.

Use when a large pre-split holdout is available and k-fold CV would add
noise and 5x compute overhead without additional signal.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(l2g): raise ValueError when holdout_only=True without cv_results_dir

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): fix ambiguous variable name and holdout_only guard

- Rename `l` → `label` in _plot_confusion_matrix list comprehension (ruff E741)
- Raise ValueError when holdout_only=True without cv_results_dir

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(l2g): remove W&B from cross_validate and clean up config casts

- Remove wandb_run_name param and all W&B sweep logic from cross_validate
  and _run_cv_fold; CV output is file-based or terminal only
- Drop now-unused wandb imports (_setup, wandb_sweep, wandb_agent)
- Remove redundant list()/dict() casts around _L2G_* constants in
  LocusToGeneCrossValidationConfig

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* refactor(l2g): rename LocusToGeneCrossValidationStep to LocusToGeneModelTuningStep

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <69353402+project-defiant@users.noreply.github.com>
@pre-commit-ci
Merge pull request #1253 from opentargets/pre-commit-ci-update-config
f50204a 
chore: pre-commit autoupdate
@dependabot
build(deps-dev): update mypy requirement from <2.2,>=1.13 to >=1.13,<…
0c5f8b3 
…2.3 (#1255)

---
updated-dependencies:
- dependency-name: mypy
  dependency-version: 2.2.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
dependabot Bot and others added 3 commits July 16, 2026 09:32
@dependabot
build(deps-dev): update mkdocs requirement (#1256)
44eb312 
Updates the requirements on [mkdocs](https://github.com/mkdocs/mkdocs) to permit the latest version.
- [Release notes](https://github.com/mkdocs/mkdocs/releases)
- [Commits](mkdocs/mkdocs@1.5.3...1.6.0)

---
updated-dependencies:
- dependency-name: mkdocs
  dependency-version: 1.6.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@project-defiant
feat(ess): add effective sample size (#1263)
6729cbb
@project-defiant
feat(ess): add effective sample size (#1264)
72d1a45
@github-actions github-actions Bot added documentation Improvements or additions to documentation Datasource labels Jul 21, 2026
@addramir @claude
feat(qc): relax sumstats QC thresholds for exome/wgs-gwas studies (#1269
c648372 
)

Studies whose analysisFlags contain ExWAS or wgsGWAS now use relaxed
thresholds in StudyIndex.annotate_sumstats_qc: higher mean beta bound,
lower min GC lambda, lower min variant count, and the PZ check skipped.
Addresses opentargets/issues#4416.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto-pr Chore Dataset Datasource documentation Improvements or additions to documentation Method size-XL Step

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@addramir @project-defiant