Skip to content

Add weighted_choice helper for weighted random selection in TPV configs#202

Open
davelopez wants to merge 5 commits into
galaxyproject:mainfrom
davelopez:add_weighted_choice
Open

Add weighted_choice helper for weighted random selection in TPV configs#202
davelopez wants to merge 5 commits into
galaxyproject:mainfrom
davelopez:add_weighted_choice

Conversation

@davelopez

Copy link
Copy Markdown

Adds a new helpers.weighted_choice(items) function to the TPV evaluation context, enabling weighted random selection of string values from a pool of {value, weight} dictionaries.

Problem

Users with multiple storage paths (e.g., job working directory roots) need a way to distribute jobs across them with configurable weights — steering most jobs to a particular path while keeping a fallback available, or draining a path by setting its weight to 0.

Changes

Core (tpv/core/helpers.py)

  • weighted_choice(items) — selects a single value from a weighted pool of dicts. Falls back to unweighted random selection if no non-default weights are defined or all weights are zero.
  • _resolve_weights() and _has_weighted_selection() — shared helpers for weight resolution and detection.
  • weighted_random_sampling() — refactored to reuse the shared weight helpers, reducing duplication.

Documentation

  • docs/topics/tpv_by_example.rst — new recipe: "Selecting a job working directory from a weighted pool" with a full YAML example.
  • docs/topics/advanced_topics.rst — new Helper functions reference table documenting all exposed helpers (weighted_choice, weighted_random_sampling, job_args_match, input_size, etc.).

Tests

  • tests/test_helpers.py — 8 unit tests covering unweighted fallback, weighted selection, default weights, zero/negative weights, empty list, and return type.
  • tests/test_mapper_weighted_choice.py — 2 integration tests verifying weighted_choice works inside a params f-string and sets job_working_directory on the destination.
  • tests/fixtures/mapping-weighted-choice.yml — mapping fixture with a jwd_pool context variable.

Usage

global:
  context:
    jwd_pool:
      - value: /fast/jobs
        weight: 3
      - value: /slow/jobs
        weight: 1

tools:
  default:
    params:
      job_working_directory: "{helpers.weighted_choice(jwd_pool)}"

/fast/jobs is selected ~75% of the time. Set weight: 0 to drain a path.

Related

Notes

  • This recipe requires Galaxy's job_working_directory job-concern support (PR #23133). Without it, the param is set by TPV but Galaxy will use the object-store derived path instead.
  • The helper is generic over the meaning of value — any string-valued pool (cache paths, runner URLs, etc.) works the same way.

Introduces a utility to select items from a weighted pool, enabling flexible distribution of jobs across multiple storage paths. Refactors existing sampling logic to reuse the new weight resolution utilities, reducing code duplication. Adds necessary type definitions to support the feature. This change improves job routing capabilities by allowing administrators to steer workload traffic based on configurable probabilities.
Covers the new utility with comprehensive unit tests verifying weight resolution, default behaviors, and edge cases like empty inputs or invalid weights. Includes an integration test with a configuration fixture to ensure correct parameter assignment during destination mapping. Guarantees reliable operation across various scenarios and prevents regressions in routing logic.
Adds a dedicated guide explaining how to configure and utilize the new utility for distributing jobs across multiple working directories. Provides a practical configuration example, clarifies weight mechanics, and outlines compatibility requirements. Helps users understand how to leverage the feature for storage balancing and failover scenarios.
Introduces a reference section for the utility module exposed in the evaluation context.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant