validation: joblib parallel-backend harness for the V3 discussion (#231) - #248
Open
Pirat83 wants to merge 7 commits into
Open
validation: joblib parallel-backend harness for the V3 discussion (#231)#248Pirat83 wants to merge 7 commits into
Pirat83 wants to merge 7 commits into
Conversation
Not pybroker code. Throwaway validation scaffolding for the V3 design discussion on edtechre#231 — empirically verifies every `joblib.Parallel` backend works against the proposed `StrategyConfig.parallel` shape. Two container images (`pybroker-validator` and `spark-with-joblib`) plus shell orchestration; six backends exercised (`loky`, `threading`, `multiprocessing`, `ray`, `dask`, `spark`). See README for usage and the known joblib 1.5.3 + dask 2026.3.0 API mismatch worked around in the Containerfile. Results committed alongside (results.md) reproduce the table posted on edtechre#231.
mypy 2.0.0 (released between 2026-05-06 and 2026-05-07) tightened ndarray generic narrowing, so `np.sort(...)` on an array of dtype Any is now reported as `ndarray[tuple[Any, ...], dtype[Any]]` instead of being assignable to `Sequence[np.datetime64]`. The unpinned `mypy` in tox.ini's typecheck env picked the new release, which broke the daily Schedule workflow on master without any code change. Cast at the single return site of get_unique_sorted_dates so the `Sequence` contract with downstream callers (IndicatorScope, Logger, ModelsMixin) is preserved with no further signature churn. Also bump [mypy] python_version from 3.9 -> 3.11 in setup.cfg. 3.9 and 3.10 were removed from the test matrix in afc137b, and mypy 2.0 warns the value is unsupported. Co-authored-by: Pirat83 <Pirat83@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to the parallel-backend work you just pushed to
v2_preview(set_parallel,enable_parallel_models) and the discussion on #231.While that discussion was open I put together a throwaway harness to sanity-check that every
joblib.Parallelbackend actually wires up under the API shape being discussed (loky,threading,multiprocessing,ray,dask,spark) — each run in its own container, results inresults.md.Caveat, to be upfront about it: Claude and I did our best here, but the workload is a dummy
sleep(0.2) + return a NumPy array— not a real PyBroker model. I don't have a distributed Ray/Dask/Spark setup to actually train a model on, so this confirms the plumbing works (backend selection, serialization round-trip, one real bug found: ajoblib/dasksignature mismatch documented in the README) but not that real parallel model training holds up correctly on those backends. Take it as a starting point / inspiration for your own validation if it's useful — if it's not, feel free to throw it away, no hard feelings. It's parked undervalidation/, isolated from the actual package, so it shouldn't get in your way either way.