Pre-stage Spark-NLP fat jar via spark.jars (no per-job Ivy resolution)

[d0choa]

What

Stop the pts / pts_literature Dataproc clusters re-running Spark-NLP Ivy
resolution against Maven Central on every spark-submit. Load Spark-NLP from a
pre-resolved fat jar via spark.jars (staged by orchestration into the
pipelines bucket) instead of resolving the package via spark.jars.packages.

This is the monorepo home of the work previously split across
opentargets-archive/orchestration#217 and opentargets-archive/pts#146 — both now closed
in favour of this PR. Business logic is equivalent, ported onto pipeline and
adapted to its layout / code style (single quotes, current cluster APIs).

Why

A spark: cluster property lands in spark-defaults.conf, so
spark.jars.packages made every pts spark-submit re-run Ivy dependency
resolution against Maven Central before its session started — even though only
~9 of ~40 steps use Spark-NLP. Evidence (run up-pts-38bc6): 17/17 jobs ran
Ivy resolution, ~3–7 s each, with Maven Central on the critical path of every
step. A pre-resolved fat jar is localized from in-region GCS at submit time — no
dependency resolution, no Maven Central on any runtime path. Details:
opentargets/issues#4453.

Changes

orchestration (orchestration: pre-stage Spark-NLP fat jar via spark.jars)

• operators/gcs.py — UploadRemoteFileOperator gains skip_if_exists
  (idempotent staging) and a configurable timeout (hard timeout=10 was unsafe
  for a ~629 MB stream).
• utils/utils.py — resolve_jar_staging(spark_jars, registry, managed_prefix),
  a pure resolver over a cluster's rendered spark.jars: registered jar → staged;
  jar under the managed prefix with no registered source → raises (DAG fails
  at parse time); jar outside the prefix → left alone.
• config/unified_pipeline.py — a staged_jars registry ({dst_uri: src_url})
  • staged_jar_prefix; spark_nlp_version wired into the clusters template
    context. Adding a staged jar is a one-line registry entry.
• dags/unified_pipeline.py — read the real spark:spark.jars, resolve against
  the registry, fan out one UploadRemoteFileOperator(skip_if_exists=True) per
  jar upstream of create_cluster. pts, pts_literature, and any future
  cluster referencing a registered jar are covered automatically.
• clusters.yaml — pts + pts_literature spark.jars.packages → spark.jars.
• unified_pipeline.yaml — spark_nlp_version: '6.1.5'.
• tests/test_jar_staging.py — resolver unit tests (no Airflow needed).

pts (pts: require ontoma>=2.5.2 for classpath-based Spark NLP check)

• pyproject.toml — ontoma>=2.5.1 → >=2.5.2. Because spark.jars doesn't
  populate spark.jars.packages, OnToma ≤2.5.1 (which guarded on that config
  string) would refuse to run. OnToma 2.5.2 (fix: validate Spark NLP by classpath probe, not spark.jars.packages string OnToma#60) validates
  Spark-NLP by probing the JVM classpath, so it accepts jars loaded via
  spark.jars too.

Coordination / merge order

OnToma is a separate repo (not part of pipeline), so its fix stays its own PR:

Repo	PR	Role
opentargets/OnToma	opentargets/OnToma#60	validate Spark-NLP by JVM classpath probe (release 2.5.2)
this (pipeline)	—	stage the fat jar + point the clusters at it + require ontoma>=2.5.2

1. Merge & publish OnToma 2.5.2 to PyPI (fix: validate Spark NLP by classpath probe, not spark.jars.packages string OnToma#60).
2. Merge this PR and regenerate pts/uv.lock.

⚠️ CI note: until OnToma 2.5.2 is on PyPI, pts dependency resolution fails
(ontoma>=2.5.2 unsatisfiable; pts/uv.lock still resolves 2.5.1), so the pts
jobs here will be red. Expected — this PR should not merge before step 1.

Validation

• resolve_jar_staging unit tests pass (registered / external / under-prefix-raises
  / comma-list / empty).
• ruff check and ty check pass on all changed orchestration files.
• tests/test_dag_validation.py passes — the DAG builds with the staging fan-out.
• Rendered the real UnifiedPipelineConfig: pts and pts_literature each
  resolve to exactly the (JSL S3 source → gs://…/spark-nlp-assembly-6.1.5.jar)
  pair; no spark.jars.packages remains.
• Behavioural check on the next Dataproc run (acceptance criteria in
  PTS Dataproc jobs re-run Spark-NLP Ivy resolution on every step issues#4453): no Ivy :: resolution report :: block; OnToma steps
  still map diseases; non-OnToma steps start clean.

Refs opentargets/issues#4453