Performance campaign: asv infrastructure + 6 non-invasive vectors (2.5–3× wins; PR#224–#230)

[Pirat83]

Summary

This tracks a structured performance campaign on pybroker that I've been running over the last few days. It resulted in 1 benchmarking-infrastructure PR and 6 non-invasive perf vectors, all measurable via asv (Airspeed Velocity) with per-PR evidence.

Everything listed here is ready to review. All 6 vector PRs pass the existing CI (ruff + mypy + pytest on 3.11 + 3.12). All PRs are independent — you can merge any subset in any order (modulo the one explicit V7 → V2 stack). Merging #224 first makes the other diffs collapse to just the vector change.

I also have a list of invasive follow-ups that the non-invasive set deliberately leaves alone — see the bottom.

---

PRs

PR	Vector	Files	Headline result	Significant?
#224	asv infrastructure + 11 benchmark classes	asv.conf.json, benchmarks/, .github/actions/setup-pybroker/, .github/workflows/asv-pr.yml, docs/source/benchmarking.rst	Infrastructure only; no code path changes	N/A
#225	V1 cache=True on every @njit	src/pybroker/vect.py, src/pybroker/eval.py	peak RSS 281 M → 245 M (-13%)	Yes (mem)
#226	V2 integer indexing in Portfolio.capture_bar	portfolio.py, strategy.py, tests/test_portfolio.py	Walkforward: 413 → 164 ms (2.5×); Cold: 389 → 169 ms (2.3×)	Yes — biggest macro win
#227	V5 in-memory L1 in front of diskcache	src/pybroker/cache.py, tests/test_cache.py, benchmarks/bench_backtest.py	~100× on CacheHit micro; flat on macro	Micro only
#228	V8 fast round + enum cache in PriceScope.fetch	src/pybroker/scope.py, benchmarks/bench_backtest.py	PriceScopeFetch: 208 → 66 ms (3.1×)	Yes (micro)
#229	V9 inline ExecContext.long_pos / short_pos	src/pybroker/context.py, benchmarks/bench_backtest.py	long_pos 232 → 79 ms (3.0×), short_pos 218 → 77 ms (2.8×)	Yes (micro)
#230	V7 fetch batch in capture_bar (stacks on #226)	src/pybroker/portfolio.py	+17% on top of V2; cumulative 2.8× vs baseline	Yes

---

Recommended merge order

None of the vectors strictly require any specific order. The only real dependency is #230 (V7) → #226 (V2) because V7 modifies the capture_bar body that V2 introduces.

Suggested order by risk/reward:

1. Add asv benchmark suite + PR-regression workflow #224 — asv infrastructure. Lightest review. Once merged, every other PR's diff shrinks to just the vector change.
2. V2: integer indexing in Portfolio.capture_bar (2.5x warm, 2.3x cold) #226 V2 — the biggest clean macro win (2.5× on the reference walkforward scenario).
3. V7: batch col_scope.fetch in capture_bar (stacks on #226) #230 V7 — ratchets V2 to 2.8× via a single batched fetch_dict call.
4. V1: cache=True on every @njit decorator #225 V1 — cache=True everywhere. Peak RSS win is the repeatable signal; warm CI time is flat here, but cross-process cold-start benefit is real.
5. V5: in-memory L1 cache in front of diskcache #227 V5 — in-memory L1. Micro-bench shows 100×; macro is flat because the reference scenario is dominated by non-cache work. Still a free win for any cache-heavy workflow.
6. V8: fast round + enum cache in PriceScope.fetch (3.1x micro) #228 V8 — order-pipeline cleanup (fast round + enum cache).
7. V9: inline ExecContext.long_pos / short_pos (3x micro) #229 V9 — method-call inlining on long_pos / short_pos. Cleanup-grade polish.

---

Non-invasive vs invasive

Everything above is non-invasive:

• No Decimal → float conversions. Portfolio / Entry / Position keep their exact-arithmetic semantics.
• No change to public method signatures. The only signature change is Portfolio.capture_bar(date, df) → capture_bar(date, col_scope, sym_end_index), both internal.
• No removed behavior. Full test suite (4074 tests) stays green on every branch.
• No new required dependencies. asv is a dev-only dep; the runtime install_requires is unchanged.

Invasive options I deliberately did not pursue here (happy to discuss in follow-up if interest):

• V3 redesign — parallel model training (see next section).
• V4 — @njit the portfolio hot path. Would port capture_bar + check_stops to Numba. Needs Position / Entry state on float64 (API-visible precision change) + NamedTuple → np.record. Multi-day scope. I attempted a scoped version (float internals, Decimal write-back) locally; it regressed because the Decimal(str(round(x, 10))) reconstruction on every call outweighs the Decimal arithmetic it replaces. Documented as a negative result. Real V4 requires committing to float throughout Portfolio state.
• Columnar batch backtest. Rewrite the backtest as vectorized ops where possible. Fork-scale change; not attempted.

---

Why V3 was left out

My original plan included a V3 "parallel walkforward windows" vector using joblib.Parallel. Before implementing I audited _run_walkforward (strategy.py:1333–1418) and found:

• Portfolio is passed by reference to backtest_executions each window (line 1404) and mutated — cash, open positions, orders, trades accumulate window N → window N+1.
• Sessions dict (line 1348) is per-symbol state the user can stash across windows; also mutated.
• Signals are appended across windows.

Walkforward windows are not independent. Parallelizing the window loop would break portfolio continuity. So V3 as-drafted was dropped.

A redesigned version — V3-rev: parallel model training, sequential backtest — is possible. Pre-compute all (train_idx, test_idx) splits, train models in parallel via joblib.Parallel, then run the test phase sequentially with the pre-trained models. That preserves portfolio continuity while parallelizing the work most likely to dominate for ML users (self.train_models(...)).

I didn't include V3-rev here because:

• It only helps model-heavy strategies. For indicator/signal-only strategies (the reference bench, and likely the common case), training is a no-op and the vector delivers nothing.
• The measurement is harder — the reference walkforward has no models, so a meaningful V3-rev benchmark needs a separate model-based scenario. Happy to add one if you want to pursue V3-rev.

---

What deserves more data

Several vectors are verified per-call wins that don't move the current reference scenario's wall-clock, because that scenario isn't the shape where they shine.

Vector	Shows where	Already covered by added benches?
V1 cache=True	Multi-process cold starts (walkforward × joblib pool)	Partially — WalkforwardProperCold in #224 clears .nbi between iterations
V5 L1 cache	Cache-enabled workflows (indicator/model cache on during dev/grid-search)	Yes — #224's CacheHit micro + #227's WalkforwardWithCache
V8 fast round / enum cache	Order-heavy strategies, many symbols hitting PriceScope.fetch	Yes — PriceScopeFetch micro in #228
V9 inline long_pos	Strategies that call ctx.long_pos()/short_pos() frequently	Yes — ExecContextPos micro in #229

If you want these surfaced at the macro level, possibilities include:

• A dense-position scenario (10+ concurrent symbols with open positions at all times) — would move V2 / V7 further and surface V9 more.
• An ML walkforward scenario with a cheap sklearn model — would surface V3-rev if pursued, plus give V1 a real cold-start amortization.
• A cache-enabled scenario that exercises indicator re-reads — would surface V5 on the macro level.

---

Offers

• Extend benchmarks — glad to add scenarios if the coverage above doesn't match your mental model of "workloads pybroker should stay fast on". Suggest a workflow shape and I'll add a class.
• Rebase / reshape / split / squash any PR per review comments.
• Address the checkbox comment on Add asv benchmark suite + PR-regression workflow #224 if you want to make decisions now on Python matrix / PR-check semantics / historical dashboard.
• Open a design thread on V4 (njit portfolio) or V3-rev (parallel model training) if you want to pursue the invasive ones.

Happy to iterate.

🤖 Generated with Claude Code

[edtechre]

Thank you! This is great work! I have merged the PRs.

For V3: Have you run into any issues training multiple models in parallel with joblib? In particular, have you seen any crashes or worker failures?

Do you have any experience with Ray?

[Pirat83]

Thanks for merging the stack! Happy to take V3 forward — answering both
questions plus surfacing one design decision and one forward-looking
reason #233 matters.

joblib — observed failure modes in parallel model training

Worth noting upfront: pybroker today doesn't train models in parallel
at all. ModelsMixin.train_models in model.py iterates
uncached_model_syms sequentially (one source(sym, ...) call per
model-symbol). joblib is used only in
indicator.py::IndicatorsMixin.compute via default_parallel()
(common.py:403 — Parallel(n_jobs=os.cpu_count(), prefer="processes", backend="loky")). So the real question is "will
moving model training under that same loky pool be safe?"

The recurring failure modes with loky + ML training:

1. Nested parallelism deadlocks. loky forks workers, and if the
   trainer (sklearn / xgboost / lightgbm / numpy with MKL or OpenBLAS)
   spawns its own OpenMP thread pool inside each worker you deadlock
   or oversubscribe N×N cores. Mitigation: OMP_NUM_THREADS=1 /
   MKL_NUM_THREADS=1 in the worker env, or threadpoolctl around
   the fit() call.
2. Unpicklable trainers / closures. Any trainer closing over a
   local class, a logger handle, a DB connection, or a C extension
   without __reduce__ blows up when loky serializes the return
   value. Wrapping the model in a plain dataclass before return
   usually resolves it.
3. BrokenProcessPool on native crashes. xgboost / lightgbm on
   certain BLAS combos segfault under memory pressure; loky surfaces
   that as pool-wide failure. max_nbytes + temp_folder tuning
   plus n_jobs=min(os.cpu_count(), N) rather than unconditional
   cpu_count() helps.
4. Fork + numba cache interaction. Pre-V1: cache=True on every @njit decorator #225, every loky worker
   re-JIT'd on first hit and cold starts dominated. cache=True
   removes that class of issue.
5. Copy-on-write OOM amplification. loky forks, so the training
   DataFrame is shared COW. A worker that mutates (or passes through
   a lib that does) triggers full page copies. Large indicator frames
   x N workers = surprise OOM. Mitigation: narrow the frame slice
   per worker and never mutate inside the worker.

All five are mitigable with known patterns, and #225 already removes
one of them. joblib-based parallel training is viable if (a) BLAS
threads are pinned per worker, (b) trainers are required to be
serializable, (c) n_jobs is exposed so users can dial it down for
memory-constrained environments.

Ray / Dask / Prefect — and where I actually landed

I've evaluated the big three for distribution + caching in
valkyrie-research (my downstream research framework) and ended up
not picking any of them. Links for the record:

• Ray
• Dask
• Prefect

Where I landed instead: Redis Pub/Sub + Redis Queues for
distribution and L1/L2 caching, and TimescaleDB as the persistent
warm cache. Reasoning in short: (a) the dependency surface of Ray /
Dask / Prefect is large relative to the actual distribution I need;
(b) Redis + TimescaleDB are already in my stack for other reasons, so
the marginal cost is zero; (c) I control backpressure, queue
semantics, and failure handling directly instead of fighting a
framework's opinions. This is a provisional choice — I plan to
re-evaluate it soon as the research framework matures, and I'm
deliberately keeping the parallel primitive pluggable so the
re-evaluation doesn't require a rewrite.

That design instinct carries into V3 for pybroker. I'd propose
letting the user choose their level of parallelism and distribution
rather than picking one backend for everyone:

• Default: joblib — matches what pybroker already ships, zero new
  dependency, covers the single-node case that I'd bet dominates
  actual usage.
• Opt-in via a pluggable executor — expose the parallel primitive
  as an Executor (in the concurrent.futures sense) or an
  equivalent callable on the Strategy / StrategyConfig API. Users
  who want Ray / Dask / Prefect / a custom Redis-backed queue slot in
  their own executor without pybroker taking a hard dep on any of
  them.
• Serial falls out naturally as n_jobs=1 or passing a
  synchronous executor.

That keeps pybroker's install footprint tight while giving the
"actually run this at cluster scale" users a sanctioned path.

Why #233 matters here — PEP 734 subinterpreters

Worth connecting this to the open 3.14 PR (#233). Python 3.14 ships
PEP 734 — the interpreters
stdlib module — which builds on
PEP 684's per-interpreter GIL
(landed in 3.12) to expose true in-process parallelism without the
fork / serialization / BrokenProcessPool overhead loky carries.
Each subinterpreter has its own GIL, so CPU-bound Python code
actually runs in parallel within a single process, and "shareable"
objects move across interpreters via channels instead of round-tripping
through disk or pipes.

For pybroker specifically that's a fourth backend option alongside
joblib / Ray / Dask — and plausibly a very good one, because the
current loky pain (serialization cost on large DataFrames, fork-safety
with BLAS, worker startup time) mostly evaporates. Free-threaded
CPython (PEP 703, opt-in from
3.13, maturing through 3.14+) is the other half of this story for
threading-based parallel training once the GIL is gone.

None of this requires work today — but it's a real reason to keep
3.14 in the supported matrix now. That's why #233 matters even
though all 4074 tests happen to pass cleanly on 3.14 as-is. A
pluggable executor gives us a natural slot to drop a subinterpreter
backend into when 3.14 adoption catches up, without another API
change.

The Portfolio-reuse design decision (V3 vs a bigger V3)

While rereading _run_walkforward (strategy.py:1360-1445) to scope
V3-rev, one thing became clearer and worth surfacing explicitly.

Today's behavior. A single Portfolio instance is threaded
through backtest_executions across every walkforward window. Cash,
open positions, orders, trades, sessions — all accumulate window N ->
N+1. This is a single continuous trading simulation chopped into
train/test phases. Parallelizing the window loop is therefore
impossible without changing semantics.

V3-rev as originally framed: parallelize just train_models
within each window (embarrassingly parallel: one job per
model-symbol), keep backtest_executions sequential with the shared
Portfolio. No semantic change to walkforward, helps ML users,
ignores everyone else. This is the safe path.

The bigger question worth your read: do you want windows to
stay strictly sequential long-term, or is there room for an opt-in
"independent windows" mode where each window gets a fresh Portfolio
and windows run in parallel? The trade-off is real on both sides:

• Sequential-only mimics live trading — compounding, consequences of
  drawdowns in window 3 affect window 4's sizing. Arguably the point
  of walkforward as a realism check.
• Independent windows measure "does this model generalize to an
  unseen period, starting from a clean slate?" — the question ML
  folks often actually want answered, without portfolio luck from
  earlier windows bleeding in.

Concretely:

• A. Sequential only. Ship V3-rev (parallel training only).
  Minimal surface, no semantics change, closes out V3.
• B. Add an opt-in parallel_windows=True kwarg that gives each
  window a fresh Portfolio and runs windows in parallel. Requires
  working through: what happens to sessions? How are per-window
  results aggregated? Does signals concatenation still make sense?
• C. Both — A now, B as a follow-up design thread.

My lean is C: V3-rev now, and if independent windows are
in-scope I'd want a separate thread to hash out the sessions /
aggregation semantics before writing code. A is fine if independent
windows is out of scope entirely.

---

To unblock V3, the two things I'd need from you:

1. Backend shape for parallel training. Joblib-default + a
   pluggable executor kwarg on the Strategy API, per my recommendation
   above? Or a different shape you have in mind?
2. A / B / C on Portfolio reuse. And for B/C, whether sessions
   should be per-window-reset or require a new explicit "shared
   across windows" slot.

Either way V3 lands as its own PR against dev with a model-based
walkforward benchmark scenario (the reference bench is indicator-only
and would show nothing meaningful for V3).

[edtechre]

Thanks for the writeup!

(serialization cost on large DataFrames, fork-safety
with BLAS, worker startup time) mostly evaporates

For context, the serialization cost is one reason PyBroker avoids heavy usage of Pandas. Instead, NumPy arrays are used whenever possible. Joblib uses shared memory mapping for NumPy arrays via numpy.memmap to drastically reduce serialization overhead. For that reason, PyBroker avoids serializing DataFrames during parallelization with Joblib.

Backend shape for parallel training. Joblib-default + a
pluggable executor kwarg on the Strategy API, per my recommendation
above? Or a different shape you have in mind?

This sounds good to me, but can you clarify with an example of using Executor with the strategy API?

A / B / C on Portfolio reuse. And for B/C, whether sessions
should be per-window-reset or require a new explicit "shared
across windows" slot.

I would say A. Because conceptually independent windows don't make sense since the windows represent a contiguous time series. The walkforward test simulates running a strategy that must periodically re-train on newer data. As the walkforward proceeds, the previous window's test data is added to the training of the subsequent window. Because the windows are sequential, the portfolio state should be shared throughout.

[Pirat83]

Thanks for the memmap pointer and the clear "A" on Portfolio reuse — that unblocks the design. Before writing V3 I staged a small validation harness on a spare machine (podman, one container per backend) to prove the proposed shape actually holds across every joblib.Parallel backend that exists today. Results at the end.

On memmap — framing correction

Agree my earlier framing overstated the serialization cost. With pybroker deliberately keeping hot paths on NumPy, joblib's auto memmap handles it. Updated residual risk envelope for joblib-parallel model training:

1. Nested BLAS thread deadlocks. Still real. Mitigation: OMP_NUM_THREADS=1 / threadpoolctl around the trainer's fit().
2. Non-picklable trainers / closures. Still real. "Wrap the model in a plain dataclass before return" usually fixes it.
3. BrokenProcessPool on native crashes (xgboost / lightgbm on certain BLAS combos). Still real; max_nbytes + temp_folder tuning + n_jobs=min(os.cpu_count(), N) helps.
4. Numba JIT cold-start cost. Solved by V1: cache=True on every @njit decorator #225 (cache=True).
5. Copy-on-write OOM on DataFrames. Largely neutralized by memmap for NumPy, which is pybroker's hot path.

Items 1–3 are the real envelope.

Concrete API example — pluggable joblib.Parallel

Two additive fields on StrategyConfig (src/pybroker/config.py, @dataclass(frozen=True)):

n_jobs: Optional[int] = field(default=None)     # sugar for the default path; ignored when `parallel` is set
parallel: Optional[Any] = field(default=None)   # joblib.Parallel instance; None -> default loky

User-facing — one snippet per backend, all validated below:

from joblib import Parallel

# 1. Default: loky, os.cpu_count(), memmap for NumPy
Strategy(config=StrategyConfig(...), data_source=...)

# 2. Cap workers (still loky)
Strategy(config=StrategyConfig(..., n_jobs=4), ...)

# 3. threading (GIL-bound today; interesting once PEP 703 free-threading matures)
Strategy(config=StrategyConfig(..., parallel=Parallel(backend="threading", n_jobs=4)), ...)

# 4. Ray cluster
import ray; ray.init(address="ray://ray-head:10001")
from ray.util.joblib import register_ray; register_ray()
Strategy(config=StrategyConfig(..., parallel=Parallel(backend="ray", n_jobs=8)), ...)

# 5. Dask cluster
from dask.distributed import Client
from joblib._dask import DaskDistributedBackend
import joblib
joblib.register_parallel_backend("dask", DaskDistributedBackend)
Client("tcp://dask-scheduler:8786")
Strategy(config=StrategyConfig(..., parallel=Parallel(backend="dask", n_jobs=8)), ...)

# 6. Spark cluster
from pyspark.sql import SparkSession
from joblibspark import register_spark; register_spark()
SparkSession.builder.master("spark://spark-master:7077").appName("pybroker").getOrCreate()
Strategy(config=StrategyConfig(..., parallel=Parallel(backend="spark", n_jobs=8)), ...)

Internal call-site change, mirroring the pattern _run_indicators already uses (src/pybroker/indicator.py:280):

# common.py — generalize default_parallel():
def resolve_parallel(config: StrategyConfig):
    if config.parallel is not None:
        return config.parallel
    n_jobs = config.n_jobs if config.n_jobs is not None else os.cpu_count()
    return Parallel(n_jobs=n_jobs, prefer="processes", backend="loky")

# model.py — ModelsMixin.train_models:
with resolve_parallel(config) as parallel:
    trained = parallel(
        delayed(_train_one_model)(
            model_sym, sym_train_data, sym_test_data, train_dates,
            test_dates, indicator_data, scope, ...
        )
        for model_sym in uncached_model_syms
    )
for model_sym, t in zip(uncached_model_syms, trained):
    models[model_sym] = t
    self._set_cached_model(t.instance, t.input_cols, model_sym, cache_date_fields)

Why this shape:

• joblib.Parallel is the API pybroker already commits to in _run_indicators — V3 is structural reuse, not a new paradigm.
• Memmap is preserved in the default + Ray + Dask paths (they all route through joblib.Parallel).
• Any registered joblib backend works by name — Spark, Ray, Dask today; a future subinterpreter backend (PEP 734 primitives landed in Python 3.14 but no joblib wrapper ships yet) drops in without another API change.
• Purely additive on StrategyConfig; backwards compatible.

Validation — every joblib Parallel backend

Ran a small dummy workload (10 tasks × 0.2 s sleep returning a NumPy array, n_jobs=4) through Parallel(backend=<name>, n_jobs=4)(delayed(fn)(i) for i in range(10)) against each backend. Cluster backends use podman containers on a shared network; the tester and the cluster containers use the same image (pybroker-validator) so joblib is installed on both sides — otherwise workers fail deserializing the submitted callable.

Backend	Wall time	Correct	Notes
loky	0.900 s	yes	default; fork + memmap for NumPy
threading	0.614 s	yes	GIL-bound; relevant post-free-thread (PEP 703)
multiprocessing	0.629 s	yes	legacy; loky supersedes
ray	1.116 s	yes	head @ ray://ray-head:10001
dask	0.516 s	yes	scheduler+worker @ tcp://:8786
spark	4.578 s	yes	master+worker @ spark://:7077 via joblibspark

Versions: joblib 1.5.3, ray 2.55.0, dask/distributed 2026.3.0, pyspark 3.5.8 + joblibspark 0.5, Python 3.10 (image) / 3.14 (host). Wall-times are illustrative on an ~8-core box; distribution overhead dominates at this workload size.

One snag worth flagging: joblib 1.5.3 + dask 2026.3.0 trip an API mismatch in joblib._dask.Batch.__call__ (joblib's Parallel dispatch passes extra kwargs — nesting_level — through submit_kwargs to the Dask worker, and Batch.__call__ has signature (self, tasks=None) with no **kwargs, so dispatch raises TypeError worker-side). I couldn't find a pre-existing tracking issue on joblib/joblib — happy to file one. Unblocked locally by widening the signature to (self, tasks=None, **_kwargs). Calling it out so users don't debug the same thing twice — users on older joblib or newer dask-ml likely wouldn't hit it.

One other deployment-shape note (not an API issue): every cluster worker needs joblib + any module referenced by the trainer installed in its Python env. Pretty standard for Ray/Dask/Spark users. For Spark specifically that means baking joblib + user deps into the Spark executor image — the apache/spark upstream image doesn't ship them. This is unchanged regardless of pybroker's API.

Portfolio reuse — aligned on A

Fully agree with the contiguous-time-series reasoning. Scoping V3 to "parallelize train_models within each window, keep backtest_executions sequential with the shared Portfolio." No semantics change. I'll close the walkforward-portfolio thread I opened in the Discussions tab with a pointer back here.

Next

Opening V3-rev as its own PR against dev shortly, with:

• Additive StrategyConfig.n_jobs and StrategyConfig.parallel fields.
• The resolve_parallel(config) helper + train_models parallelization.
• A new model-based walkforward benchmark in benchmarks/bench_backtest.py — the existing indicator-only reference bench wouldn't move for V3.
• No changes to Portfolio or walkforward semantics.

Ping me if the API shape wants adjusting before I open the PR.

[Pirat83]

If you want to reproduce the table yourself: the validation harness (Containerfile(s), test_backends.py, run_one.sh, run_all.sh, and a README) is pushed at Pirat83/pybroker:validation/v3-parallel-backends.

Requirements: just podman (or docker) — no Python / joblib on the host; everything runs in containers.

git clone -b validation/v3-parallel-backends https://github.com/Pirat83/pybroker.git
cd pybroker/validation/v3-parallel-backends
podman build -t pybroker-validator:latest -f Containerfile .
podman build -t spark-with-joblib:latest -f Containerfile.spark .
./run_all.sh
cat results.md

~2 minutes end-to-end on a recent multi-core box. The branch is kept out of dev/master on purpose — it's scaffolding, not pybroker code.

[edtechre]

Thank you @Pirat83 ! The pluggable joblib.Parallel API looks great to me.

[edtechre]

@Pirat83 I have created a new v2_preview branch for a PyBroker 2.0 release. It's a WIP, but I have added support for:

• Setting a custom Parallel backend via pybroker.set_parallel. See 8021c6e
• Parallel model training. This is disabled by default, but can be turned on via the new enable_parallel_models param on Strategy#backtest/walkforward. See 471446e
• Batch fetch_dict in Portfolio.capture_bar optimization: c85f80b

The branch includes many other features that will accompany the v2 release. It's a WIP but once implementation is completed, I will update the documentation with new notebooks covering the new features (margin trading, HPO, multiple timeframes, multi-symbol models, Claude Code integration etc). Stay tuned!

And let me know what you think of the new APIs and features so far.

[Pirat83]

Saw the v2_preview update — nice, set_parallel looks like a clean way to land this. While the API shape was still being discussed here I put together a small harness (podman containers, one per backend) to check that loky/threading/multiprocessing/ray/dask/spark all actually wire up through joblib.Parallel — opened as #248 in case it's useful.

Full disclosure: Claude and I built this without access to a real distributed cluster or an actual model to train, so it's a dummy workload (sleep + return an array), not a real backtest. It proves the backends connect and round-trip, not that parallel model training is correct end-to-end on all of them. Use it for inspiration if it saves you time reproducing that setup — otherwise just ignore it.

[edtechre]

Thank you @Pirat83. I merged your PRs and will include them in 1.2.14 release on main.

For #248, I will take a look to see what I can reuse for an additional test using those different backends.