feat(ranking): per-class metric weights, gt.tl.ranking, and dataset-level plot#3
Merged
Merged
Conversation
Add a metric_weights argument to gt.pl.ranking so users can weight the four metric classes (predictive, genomic, literature, mechanistic) when computing the mean F0.1 that determines method ordering. Weights default to 1 each, must be non-negative, keys are restricted to valid classes, and any omitted class is treated as weight 0. The overall score is the weighted mean across present classes. Extract the numeric logic into a new public gt.tl.ranking(df, metric_weights) that returns the exact weighted mean F0.1 per method as a DataFrame (index name, column mean_f01, sorted descending). gt.pl.ranking now delegates to it, keeping a single source of truth. Document both in the eval vignette, add tests for tl/pl ranking, and bump version to 0.0.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Weight metric classes per dataset, then average the per-dataset weighted means across datasets for the final mean_f01. gt.tl.ranking now returns mean_f01 as the first column followed by one column per dataset holding that dataset's weighted mean. gt.pl.ranking gains level='dataset', a heatmap with datasets as columns and per-column ranks, ordered by the final mean_f01 and (like level='task') without the overall barplot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
Pre-release numpy/pandas/sklearn can no longer fancy-index a pandas Index with sklearn's integer position array, raising "only integer scalar arrays can be converted to a scalar index" in the py3.14-pre CI job. Passing a plain list routes sklearn through its list-indexing path, which is robust across versions. Behavior is unchanged on stable deps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds configurable per-class metric weights to GRN method ranking, a new public
gt.tl.rankingfunction as the single source of the ranking math, and alevel='dataset'view forgt.pl.ranking.metric_weightsongt.pl.ranking(defaultdict(predictive=1, genomic=1, literature=1, mechanistic=1)): weights are validated to be non-negative and restricted to the four metric-class keys. A class omitted from the dict is treated as weight 0.mean_f01is the mean of those per-dataset values across datasets (rather than collapsing datasets first).gt.tl.ranking(df, metric_weights=None): returns a DataFrame indexed by method name, withmean_f01as the first column followed by one column per dataset (that dataset's weighted mean F0.1), sorted descending.gt.pl.rankingcalls it internally so plot ordering and exact values stay in sync.gt.pl.ranking(..., level='dataset'): heatmap with datasets as columns, per-column ranks inside each cell, rows ordered by the finalmean_f01, and (likelevel='task') no overall barplot.docs/notebooks/eval.ipynbvignette explaining default equal weighting, how to reweight/filter, and how to read exact values viagt.tl.ranking.Test plan
pytest tests/test_pl_ranking.py tests/test_tl_ranking.py— 54 passing, covering weight validation, per-dataset-then-across-dataset aggregation, zero/missing-class handling, NaN coverage, and the new dataset-level figure.ruff check/ruff format --checkclean.🤖 Generated with Claude Code