Follow-up to #17 (bundle-aware Python output).
A Python developer who wants to import two JLW-wrapped libraries in the same process hits a problem that the current implementation doesn't address.
What happens today
Each bundled library ships its own copy of libjulia.so.1.13 (and the rest of the runtime closure) under pkg/bundle/lib/. The user lib has DT_NEEDED libjulia.so.1.13 and a RUNPATH of \$ORIGIN/../lib[/julia].
When Python loads the first library (say pkgA):
ctypes.CDLL(pkgA/bundle/lib/libA.so) triggers the dynamic linker to walk A's RUNPATH and map pkgA/bundle/lib/libjulia.so.1.13 under SONAME libjulia.so.1.13.
When pkgB is then imported:
- B's
DT_NEEDED libjulia.so.1.13 is satisfied by the already-loaded copy. B's own pkgB/bundle/lib/libjulia.so.1.13 is never mapped. First one wins.
This silently "works" only when both libraries were built against byte-compatible Julia versions and their juliac sysimages don't collide on global runtime state. And jl_init is single-shot per-process anyway, so even in the lucky case the libraries are sharing one runtime, not isolated.
If the two libraries were built against different Julia versions, the second one's calls into the shared runtime may crash or silently miscompile.
What privatize = true does (and doesn't) solve
privatize = true (added in #17, currently off by default) salts the bundled libjulia files with a random prefix (abc12345_libjulia.so.1.13) and rewrites the user lib's DT_NEEDED to match. Different SONAMEs → the dynamic linker loads both copies, so A and B each get their own libjulia.
Mechanically this resolves the "first one wins" issue, but it moves the danger: two Julia runtimes in one process is largely uncharted territory. GC root sets, thread-pool initialization, signal handlers, BLAS trampolines (libblastrampoline), and JIT codegen are all designed assuming one runtime per process. We have no evidence this combination works in practice, and reasons to think it doesn't.
What we tell users today
For the MVP this should be a documented limitation in docs/src/index.md, roughly:
One JLW library per process is the supported configuration today. If you need two libraries together, compile them as a single juliac library that exports both APIs. If you must load two separately built libraries in one process, expect sharp edges: the default layout silently shares a runtime (assumes byte-compatible Julia versions); privatize = true isolates them at the dlopen layer but two Julia runtimes in one process is not a tested configuration.
What might be done about it
Open questions for a future session:
- Does `privatize = true` actually let two runtimes coexist in practice? Worth a stress test — load two privatized JLW libraries built against the same Julia version, call back and forth, look for GC/threading/BLAS collisions.
- Does `privatize = true` help across different Julia versions, or do they collide on shared system state (libstdc++, libgomp, libuv signal handlers)?
- Is there a juliac-side story for "merge two libraries into one image" that we should be advocating for?
- Should JuliaLibWrapping detect-at-import-time that a different JLW library is already loaded in the process and refuse, with a clear error rather than silent UB?
Filed from a Claude Code session exploring the design space after #17 landed.
Follow-up to #17 (bundle-aware Python output).
A Python developer who wants to
importtwo JLW-wrapped libraries in the same process hits a problem that the current implementation doesn't address.What happens today
Each bundled library ships its own copy of
libjulia.so.1.13(and the rest of the runtime closure) underpkg/bundle/lib/. The user lib hasDT_NEEDED libjulia.so.1.13and aRUNPATHof\$ORIGIN/../lib[/julia].When Python loads the first library (say
pkgA):ctypes.CDLL(pkgA/bundle/lib/libA.so)triggers the dynamic linker to walk A'sRUNPATHand mappkgA/bundle/lib/libjulia.so.1.13under SONAMElibjulia.so.1.13.When
pkgBis then imported:DT_NEEDED libjulia.so.1.13is satisfied by the already-loaded copy. B's ownpkgB/bundle/lib/libjulia.so.1.13is never mapped. First one wins.This silently "works" only when both libraries were built against byte-compatible Julia versions and their juliac sysimages don't collide on global runtime state. And
jl_initis single-shot per-process anyway, so even in the lucky case the libraries are sharing one runtime, not isolated.If the two libraries were built against different Julia versions, the second one's calls into the shared runtime may crash or silently miscompile.
What
privatize = truedoes (and doesn't) solveprivatize = true(added in #17, currently off by default) salts the bundled libjulia files with a random prefix (abc12345_libjulia.so.1.13) and rewrites the user lib'sDT_NEEDEDto match. Different SONAMEs → the dynamic linker loads both copies, so A and B each get their own libjulia.Mechanically this resolves the "first one wins" issue, but it moves the danger: two Julia runtimes in one process is largely uncharted territory. GC root sets, thread-pool initialization, signal handlers, BLAS trampolines (libblastrampoline), and JIT codegen are all designed assuming one runtime per process. We have no evidence this combination works in practice, and reasons to think it doesn't.
What we tell users today
For the MVP this should be a documented limitation in
docs/src/index.md, roughly:What might be done about it
Open questions for a future session:
Filed from a Claude Code session exploring the design space after #17 landed.