Add weighted_choice helper for weighted random selection in TPV configs#202
Open
davelopez wants to merge 5 commits into
Open
Add weighted_choice helper for weighted random selection in TPV configs#202davelopez wants to merge 5 commits into
weighted_choice helper for weighted random selection in TPV configs#202davelopez wants to merge 5 commits into
Conversation
Introduces a utility to select items from a weighted pool, enabling flexible distribution of jobs across multiple storage paths. Refactors existing sampling logic to reuse the new weight resolution utilities, reducing code duplication. Adds necessary type definitions to support the feature. This change improves job routing capabilities by allowing administrators to steer workload traffic based on configurable probabilities.
Covers the new utility with comprehensive unit tests verifying weight resolution, default behaviors, and edge cases like empty inputs or invalid weights. Includes an integration test with a configuration fixture to ensure correct parameter assignment during destination mapping. Guarantees reliable operation across various scenarios and prevents regressions in routing logic.
Adds a dedicated guide explaining how to configure and utilize the new utility for distributing jobs across multiple working directories. Provides a practical configuration example, clarifies weight mechanics, and outlines compatibility requirements. Helps users understand how to leverage the feature for storage balancing and failover scenarios.
Introduces a reference section for the utility module exposed in the evaluation context.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
helpers.weighted_choice(items)function to the TPV evaluation context, enabling weighted random selection of string values from a pool of{value, weight}dictionaries.Problem
Users with multiple storage paths (e.g., job working directory roots) need a way to distribute jobs across them with configurable weights — steering most jobs to a particular path while keeping a fallback available, or draining a path by setting its weight to 0.
Changes
Core (
tpv/core/helpers.py)weighted_choice(items)— selects a singlevaluefrom a weighted pool of dicts. Falls back to unweighted random selection if no non-default weights are defined or all weights are zero._resolve_weights()and_has_weighted_selection()— shared helpers for weight resolution and detection.weighted_random_sampling()— refactored to reuse the shared weight helpers, reducing duplication.Documentation
docs/topics/tpv_by_example.rst— new recipe: "Selecting a job working directory from a weighted pool" with a full YAML example.docs/topics/advanced_topics.rst— new Helper functions reference table documenting all exposed helpers (weighted_choice,weighted_random_sampling,job_args_match,input_size, etc.).Tests
tests/test_helpers.py— 8 unit tests covering unweighted fallback, weighted selection, default weights, zero/negative weights, empty list, and return type.tests/test_mapper_weighted_choice.py— 2 integration tests verifyingweighted_choiceworks inside a params f-string and setsjob_working_directoryon the destination.tests/fixtures/mapping-weighted-choice.yml— mapping fixture with ajwd_poolcontext variable.Usage
/fast/jobsis selected ~75% of the time. Setweight: 0to drain a path.Related
job_working_directoryjob-concern support: galaxyproject/galaxy#23133Notes
job_working_directoryjob-concern support (PR #23133). Without it, the param is set by TPV but Galaxy will use the object-store derived path instead.value— any string-valued pool (cache paths, runner URLs, etc.) works the same way.