Skip to content

fix: validate Spark NLP by classpath probe, not spark.jars.packages string#60

Open
d0choa wants to merge 2 commits into
masterfrom
fix/spark-nlp-classpath-check
Open

fix: validate Spark NLP by classpath probe, not spark.jars.packages string#60
d0choa wants to merge 2 commits into
masterfrom
fix/spark-nlp-classpath-check

Conversation

@d0choa

@d0choa d0choa commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What

Replace OnToma's spark.jars.packages string check with a JVM capability probe
that validates Spark NLP is actually on the classpath.

Why

OnToma.__post_init__ guarded on spark.jars.packages containing "spark-nlp":

if not self._check_spark_config(spark, "spark.jars.packages", "spark-nlp"):
    raise ValueError("Spark session is missing configuration required for Spark NLP.")

That inspects how the jars were declared, not whether they are available.
Spark NLP can be loaded in ways that never populate spark.jars.packages:

  • spark.jars — a pre-resolved fat/assembly jar (localized from GCS, no Ivy);
  • a custom Spark image that bundles com.johnsnowlabs.nlp:spark-nlp.

In both cases the classes are on the classpath but the guard raises a false
negative. This is exactly the situation in the Open Targets PTS pipeline, where
we are moving the Dataproc clusters off spark.jars.packages (per-spark-submit
Ivy resolution against Maven Central) onto a pre-staged fat jar via spark.jars
(see opentargets/issues#4453).

How

py4j resolves a class that is on the classpath to a JavaClass and an absent
one to a JavaPackage. The new _spark_nlp_available() looks up the Spark NLP
entrypoint (com.johnsnowlabs.nlp.DocumentAssembler) on the JVM and checks the
resolved type — validating the capability directly, agnostic to how the jars
were loaded. The raised error now names all three valid load mechanisms.

Tests

tests/test_spark_nlp_check.py (TDD — written before the implementation):

  • probe returns True when the entrypoint resolves to a JavaClass (even with
    spark.jars.packages unset);
  • returns False when it resolves to a JavaPackage (class absent);
  • returns False when the JVM lookup raises;
  • integration: against a real Spark session with Spark NLP loaded, the probe
    resolves the class on the live JVM and reports it available.
tests/test_spark_nlp_check.py ....                                       [100%]
4 passed

Also adds setuptools to the dev group: pyspark==3.5.7 imports distutils,
which is absent on the Python >=3.12 this project supports (requires-python),
so the suite could not otherwise be collected there.

Version

Patch bump 2.5.12.5.2 (backward-compatible: strictly widens what passes
the guard).

…tring

OnToma.__post_init__ guarded on spark.jars.packages containing "spark-nlp".
That checks how the jars were declared, not whether they are actually
available, so it reports false negatives when Spark NLP is loaded via
spark.jars (a pre-resolved fat jar) or a custom Spark image — neither of
which populates spark.jars.packages.

Replace the config-string check with a JVM capability probe: py4j resolves a
class on the classpath to a JavaClass and an absent one to a JavaPackage, so
looking up com.johnsnowlabs.nlp.DocumentAssembler validates the capability
directly and is agnostic to how the jars were loaded.

- add _spark_nlp_available() probe; remove _check_spark_config()
- raise a clearer error naming all three valid load mechanisms
- tests/test_spark_nlp_check.py: probe present/absent/error + a live-JVM
  integration test
- add setuptools to the dev group so pyspark's distutils import resolves on
  the Python >=3.12 the project supports (required to run the suite)
d0choa added a commit to opentargets-archive/pts that referenced this pull request Jul 1, 2026
OnToma 2.5.2 (opentargets/OnToma#60) validates Spark NLP by probing the JVM
classpath instead of requiring spark.jars.packages to name spark-nlp. That lets
the Dataproc clusters load Spark NLP from a pre-staged fat jar via spark.jars
(orchestration#217) — no per-job Ivy resolution against Maven Central — while
OnToma-using steps still start.

Supersedes the earlier session.py workaround (setting spark.jars.packages in-app
post-SparkSubmit): with the OnToma fix that hack is unnecessary.

Companion to opentargets/issues#4453.
@d0choa
d0choa requested a review from vivienho July 1, 2026 14:37
@d0choa
d0choa marked this pull request as ready for review July 1, 2026 14:38
Match the pinned spark-nlp==6.1.5 Python dependency; the fixture previously
loaded the 6.1.3 JVM jars, a version skew against the package it exercises.

(uv.lock: editable self-version follows the 2.5.2 bump.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant