Support container image version pinning for services and batch jobs

[cswaney]

Summary

Let users select a specific container image when launching services and batch jobs, record it in the database, and surface it in the CLI and UI.

Note on scope. This issue originally proposed pinning Python package versions (requested_tigerflow_ml_version, on-demand installs, version ranges). That framing predates the move to containers, when Blackfish managed Python venvs. Now that execution is image-based, the equivalent guarantee is pinning the container image. Package-level pinning, auto-install, and version ranges are obsolete and out of scope. The original text is preserved in the comment below.

Problem

The container image is resolved at render time from process-global config (blackfish_config.IMAGES[...]) and is never persisted.

1. No reproducibility. Change BLACKFISH_TIGERFLOW_ML_IMAGE (or ship a new DEFAULT_IMAGES pin) and existing services/jobs silently run a different image on their next restart. For batch jobs this is not hypothetical: _submit → _render_script runs on every resubmission (jobs/base.py:832 in the restart loop, plus start and resume), so a long-running job can change image mid-life.
2. No user control. There is no way to select an image version, and no API surface listing available ones.

BatchJob already records tigerflow_version / tigerflow_ml_version, but those are read out of the container post-hoc — they say what was inside the image, not which image was chosen, and neither is surfaced anywhere.

Requirements

• Users can select a specific container image when running services and batch jobs.
• The image version is recorded in the DB.
• The image version is surfaced in the CLI (blackfish ls, blackfish batch ls) and the UI (service/job details, plus both launchers).

Design

• New nullable image_ref column (full repo:tag) on Service and BatchJob. It cannot be named image: Service.image is the SQLAlchemy polymorphic discriminator holding the service type, and Model.image is an HF pipeline tag.
• Resolution prefers the stored pin, falling back to the configured default. Two choke points: services/base.py:691 and _resolve_image_and_provider in jobs/base.py.
• The pin is persistent — reused on restart/resume, which is what makes this reproducibility rather than a label.
• Backfill at launch: when no pin is given, write the resolved ref back to the row, so every launch records what actually ran.
• Selectable versions come from SIFs staged on the profile ({cache_dir}/images/*.sif), since images are staged out of band via apptainer pull and there is no registry integration. An unrunnable pin is therefore not selectable.
• The pre-selected default is the configured default, never the newest staged tag — it must only change when someone deliberately changes config, not because an admin staged a new SIF.
• Templates need no changes: all six already take an image object and use .sif / .docker_ref.

Known traps

• A job resolves its image twice — once in create_tigerflow_client (the pre-flight/health client, which derives the SIF path it probes) and once in _render_script. Both must honor the pin, or check_health validates a different image than the job runs.
• ImageSpec.sif is lossy: it drops the registry prefix, so a filename yields name+tag but not the repo. Discovery must match staged SIFs against known repos in config and extract only the tag.
• Tag sorting needs real version parsing ("0.1.10" < "0.1.2" as strings), tag styles differ across images (v0.20.0 vs 0.1.2), and non-version tags like latest are legal — listed last, never auto-selected.

Sub-issues

Tracked as sub-issues below. Order: 2 depends on 1; 4 depends on 3 and on 1's request fields; 2 and 3 are independent.

[cswaney]

Original issue text (pre-container design, preserved for history)

Summary

Allow users to specify exact tigerflow-ml versions when creating batch jobs to ensure reproducibility.

Background

BatchJob now records the versions used when a job starts (tigerflow_version, tigerflow_ml_version). This enables:

1. Auditing which versions were used for a job
2. Future support for re-running jobs with the same versions

Proposed Feature

Add optional version pinning to BatchJob:

class BatchJob:
    # Requested versions (user-specified, optional)
    requested_tigerflow_ml_version: Mapped[Optional[str]]
    
    # Actual versions used (set at runtime)
    tigerflow_version: Mapped[Optional[str]]
    tigerflow_ml_version: Mapped[Optional[str]]

Behavior

1. No version specified: Use whatever is installed (current behavior)
2. Version specified:
   • Check if requested version is installed
   • If not, either fail with clear error or auto-install (TBD)
   • Verify compatibility with installed tigerflow version

Considerations

• Should we support installing specific versions on-demand?
• How to handle tigerflow/tigerflow-ml version compatibility?
• Should we support version ranges (e.g., >=0.2.0,<0.3.0)?

Related

Part of #211 batch jobs refactoring.