fix: validate Spark NLP by classpath probe, not spark.jars.packages string#60
Open
d0choa wants to merge 2 commits into
Open
fix: validate Spark NLP by classpath probe, not spark.jars.packages string#60d0choa wants to merge 2 commits into
d0choa wants to merge 2 commits into
Conversation
…tring OnToma.__post_init__ guarded on spark.jars.packages containing "spark-nlp". That checks how the jars were declared, not whether they are actually available, so it reports false negatives when Spark NLP is loaded via spark.jars (a pre-resolved fat jar) or a custom Spark image — neither of which populates spark.jars.packages. Replace the config-string check with a JVM capability probe: py4j resolves a class on the classpath to a JavaClass and an absent one to a JavaPackage, so looking up com.johnsnowlabs.nlp.DocumentAssembler validates the capability directly and is agnostic to how the jars were loaded. - add _spark_nlp_available() probe; remove _check_spark_config() - raise a clearer error naming all three valid load mechanisms - tests/test_spark_nlp_check.py: probe present/absent/error + a live-JVM integration test - add setuptools to the dev group so pyspark's distutils import resolves on the Python >=3.12 the project supports (required to run the suite)
d0choa
added a commit
to opentargets-archive/pts
that referenced
this pull request
Jul 1, 2026
OnToma 2.5.2 (opentargets/OnToma#60) validates Spark NLP by probing the JVM classpath instead of requiring spark.jars.packages to name spark-nlp. That lets the Dataproc clusters load Spark NLP from a pre-staged fat jar via spark.jars (orchestration#217) — no per-job Ivy resolution against Maven Central — while OnToma-using steps still start. Supersedes the earlier session.py workaround (setting spark.jars.packages in-app post-SparkSubmit): with the OnToma fix that hack is unnecessary. Companion to opentargets/issues#4453.
d0choa
marked this pull request as ready for review
July 1, 2026 14:38
Match the pinned spark-nlp==6.1.5 Python dependency; the fixture previously loaded the 6.1.3 JVM jars, a version skew against the package it exercises. (uv.lock: editable self-version follows the 2.5.2 bump.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace OnToma's
spark.jars.packagesstring check with a JVM capability probethat validates Spark NLP is actually on the classpath.
Why
OnToma.__post_init__guarded onspark.jars.packagescontaining"spark-nlp":That inspects how the jars were declared, not whether they are available.
Spark NLP can be loaded in ways that never populate
spark.jars.packages:spark.jars— a pre-resolved fat/assembly jar (localized from GCS, no Ivy);com.johnsnowlabs.nlp:spark-nlp.In both cases the classes are on the classpath but the guard raises a false
negative. This is exactly the situation in the Open Targets PTS pipeline, where
we are moving the Dataproc clusters off
spark.jars.packages(per-spark-submitIvy resolution against Maven Central) onto a pre-staged fat jar via
spark.jars(see opentargets/issues#4453).
How
py4j resolves a class that is on the classpath to a
JavaClassand an absentone to a
JavaPackage. The new_spark_nlp_available()looks up the Spark NLPentrypoint (
com.johnsnowlabs.nlp.DocumentAssembler) on the JVM and checks theresolved type — validating the capability directly, agnostic to how the jars
were loaded. The raised error now names all three valid load mechanisms.
Tests
tests/test_spark_nlp_check.py(TDD — written before the implementation):Truewhen the entrypoint resolves to aJavaClass(even withspark.jars.packagesunset);Falsewhen it resolves to aJavaPackage(class absent);Falsewhen the JVM lookup raises;resolves the class on the live JVM and reports it available.
Also adds
setuptoolsto the dev group:pyspark==3.5.7importsdistutils,which is absent on the Python
>=3.12this project supports (requires-python),so the suite could not otherwise be collected there.
Version
Patch bump
2.5.1→2.5.2(backward-compatible: strictly widens what passesthe guard).