Replace patchelf (GPL) with pure-Julia ELF patching on Linux by asinghvi17 · Pull Request #150 · JuliaLang/JuliaC.jl

asinghvi17 · 2026-05-28T22:54:50Z

Motivation

The objective here is to remove Patchelf_jll since it's GPL licensed. It's only used for library privatization for libjulia on Linux (to enable juliac-in-julia use), where it rewrites the SONAME and references thereto.

This PR removes Patchelf_jll from the runtime graph by doing the same ELF edits in pure Julia, via ObjectFile.jl, and uses Patchelf_jll to test that this does what we want.

What changed

Pure-Julia ELF ops in PatchVersion (src/patchversion.jl): read_soname, read_needed, set_soname!, replace_needed!, built on the module's existing in-place .dynstr patcher (patch_str!, which already errors if a replacement would grow the string). They work for any ELF class/endianness — all multi-byte field decoding goes through ObjectFile.jl/StructIO, which selects the 32- vs 64-bit struct layout and byte-swaps per the parsed ELF header, while the overwritten string bytes are raw ASCII. The only precondition is an ELFHandle (non-ELF inputs are rejected up front), and the patcher refuses to overwrite a tail-merged neighbour string.
Linux salt scheme: prepend → same-length substitution. In-place .dynstr patching can't grow a string, so instead of prepending (<salt>_libjulia.so.X.Y, which lengthens the SONAME/NEEDED) the Linux path now substitutes the 8-char libjulia token with an 8-char salt (<salt>.so.X.Y), which is length-preserving across filename, SONAME, and DT_NEEDED. This is driven by two new platform hooks (plat_salted_basename, plat_dep_libs_prepend); macOS keeps prepend (its install_name_tool can grow strings) and is otherwise untouched.
privatize_linux.jl now calls the PatchVersion ops instead of shelling out to patchelf. (The SONAME/DT_NEEDED strings carry the 2-component version, e.g. libjulia.so.1.12, while files on disk carry the full version, e.g. libjulia.so.1.12.6; the salt is applied as a length-preserving token substitution on the live SONAME/DT_NEEDED string so it never grows.)
Hook plumbing simplified. The random salt is now threaded into the plat_set_library_id! / plat_install_name_change! hooks instead of being recovered from the leading bytes of a salted basename — this removes the _salt_of helper and the basename round-trips. The Linux privatization wrapper also no longer re-wraps failures into a fresh ErrorException (which discarded the original backtrace); errors now propagate with their stack intact.
Patchelf_jll moved from [deps] to [extras] + the test target (the test suite still uses patchelf --print-soname / --print-rpath purely as an independent read-only oracle). No new runtime dependency is added.

Tests

New unit tests (test/elf_ops.jl) for the four ELF ops against a small gcc-built fixture (a library with a SONAME + a dependent): read agreement with the patchelf oracle, same-length set_soname! / replace_needed!, the grow guard, and an end-to-end dlopen of a privatized library.
The existing "Privatization (Unix salted ids)" testset is branched for macOS (prepend) vs Linux (substitution).
A synthetic check that no unsalted libjulia* SONAME/DT_NEEDED survives Linux privatization.
The existing C/Julia dlopen privatization tests remain green.

Full suite passes locally on Linux (including the real sysimage-building integration tests).

Follow-ups

The ELF32 / big-endian support was verified against real cross-toolchain fixtures (i386, s390x, powerpc) and cross-checked with readelf/patchelf — read/write, the grow guard, and structural re-parse all pass and string content is never byte-swapped — but a patched big-endian library has not been dlopen'd on real BE hardware. A big-endian CI job plus committed ELF32/BE fixtures under test/elf/ would close that one gap.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add read_soname, read_needed, set_soname!, replace_needed!; verify against patchelf as a read-only oracle plus an end-to-end dlopen of renamed libraries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Default to the existing prepend salt scheme; wire the common flow (real-file branch, symlink branch, replace_dep_libs) through the hooks. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

get_dependencies_linux/set-soname/replace-needed now call PatchVersion; Linux salts by equal-length token substitution. The SONAME/DT_NEEDED strings carry the two-component version (libjulia.so.MAJ.MIN) while files on disk carry the full version (libjulia.so.MAJ.MIN.PATCH), so the salt is applied as a length-preserving token substitution on the live SONAME/NEEDED string (salt recovered from the salted basename) rather than reusing the longer salted filename. Guards assert the matched SONAME/NEEDED contains the libjulia token so any surprising form fails loudly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ubstitution The Linux assertions account for JuliaC's bundle layout, where real library files carry the full version (so.MAJ.MIN.PATCH) and their SONAME carries the two-component version (so.MAJ.MIN); both have the leading libjulia token replaced by an equal-length salt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…rivatization Copies follow version symlinks to obtain regular library files, and gives the fabricated product its own SONAME (as a real build's product carries) so the walk only sees clean salted names afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The runtime privatization path no longer shells out to patchelf, so it moves from [deps] to [extras] + the test target. Tests still use patchelf as a read-only oracle. [compat] entry retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Pass the random salt into plat_set_library_id! / plat_install_name_change! rather than recovering it from the leading bytes of a salted basename. The salt is already in scope at every hook call site, so the _salt_of helper, the two basename round-trips, and the misleading `soname` parameter all go away. Behaviorally identical: the recovered salt always equaled the original. - Drop the nested try/catch in privatize_libjulia_linux! that re-wrapped failures into a fresh ErrorException (via error(msg, e)) and discarded the original backtrace; let failures propagate with their stack intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Loosen the in-place .dynstr patch precondition from "ELF64 little-endian only" to "any ELF handle". All multi-byte field reads (d_val, sh_link, section offset/size) go through ObjectFile.jl/StructIO, which select the 32- vs 64-bit struct layout and byte-swap per the parsed ELF header; the string bytes we overwrite are raw ASCII, so they are not class/endianness sensitive. The oh::ELFHandle type restriction is the only remaining guard (rejects MachO/COFF). Verified against real cross-toolchain fixtures for all four corners (ELF32/64 x LE/BE: i386, x86-64, s390x, powerpc), cross-checked with readelf and patchelf: read_soname/read_needed, same-length set_soname!/replace_needed!, the grow guard, and structural re-parse all pass; string content is never byte-swapped. Follow-up: add a big-endian CI job that dlopen()s a patched library on real BE hardware, and commit ELF32/BE fixtures into test/elf/, to close the one gap not exercisable on an x86-64 host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Comment-only: trim the verbose explanatory blocks added with the pure-Julia ELF ops and the salt-substitution hooks down to one line each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

topolarity · 2026-05-29T15:38:20Z

+_dyn_string_offsets(oh) =
+    Int[Int(ObjectFile.deref(d).d_val) for tag in (ELF.DT_SONAME, ELF.DT_NEEDED)
+                                       for d in ELF.ELFDynEntries(oh, [tag])]


I didn't know that tail-merging is an optimization that happens, wild!

That said, this set of _dyn_string_offsets means that _patch_dyn_string! is really only prepared to patch DT_SONAME / DT_NEEDED.

If tail-merging is a real issue, then the rest of this file might need some re-work (iirc, we assume tail-merges don't happen) and also patch_dyn_string! should try to have a generic solution that doesn't work only for SONAME, etc.

topolarity

Hmm I have mixed feeling about renaming libjulia-* to salt-* but it's probably OK

Can you ask Claude 🤖 how much work it would be to support growth, like patchelf / install_name_tool does? Would replacements that require growing the table be reliable?

asinghvi17 · 2026-05-29T15:46:05Z

While doing this, I had first tried to keep the libjulia name. Unfortunately, that requires a bit more surgery, because you have to widen the bit length of each of the strings within the record, which then means sequentially bumping each offset in the file. That's a more complex operation 😅 and I felt the risk wasn't worth it. Could be a follow-on PR though.

topolarity · 2026-05-29T15:48:18Z

Do you know whether it would even be reliable for us to do that, or will we run into sections / strings, that we don't have enough information to relocate?

You may want to point Claude to the patchelf source and ask it to identify failure cases.

topolarity · 2026-05-29T19:07:44Z

 const DEP_LIBS_LENGTH = 512 # This is technically 1024 bytes, but we use 512 to be safe

-function replace_dep_libs(file, salt)
+function replace_dep_libs(file, salt; prepend::Bool)


These conventions are quite messy

It's a bit confusing that plat_salted_basename does a full basename replacement, but replace_dep_libs has its this prefix convention (which is implicitly required to match), and then plat_set_library_id! also demands the raw salt but expects many already-salted inputs.

Feels like a lot of mixed conventions

Can you simplify this to a straightforward remap of libjulia => salted_libjulia, or maybe a full remap of library / SONAMES that the platform-specific privatization logic provides and we pass around everywhere?

Done, just pushed a couple of commits to deal with that

topolarity · 2026-05-29T19:08:39Z

Other than the salting logic and the tail-merging issue, I think these changes are largely good.

If you can fix those up @asinghvi17 I'll be happy to merge!

Scan the whole .dynstr and refuse an in-place patch whose target byte range (string + NUL) overlaps any other record, instead of only checking DT_SONAME / DT_NEEDED offsets. This catches tail-merging — where the target is the shared suffix of a longer string and an in-place overwrite (even equal-length) would corrupt that longer string — and also covers non-dynamic neighbours. Drops the now-unneeded _dyn_string_offsets helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The grow guards interpolated the raw Vector{UInt8} buffers, so a failure printed as a wall of hex (UInt8[0x6c, 0x69, ...]) instead of the offending names. Show the strings as text with their byte lengths and explain that in-place patches can only shrink or keep length. Update the grow-guard test to match the new wording. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the three coexisting name-rewriting conventions in the common privatization logic with one explicit, per-run name map (`SaltMap`) that is built once by the platform and threaded through every rewrite site. Previously the salted form of a library name was derived in several places with conventions that had to match implicitly: - a full-basename transform hook (`plat_salted_basename`), - a separate prepend/substitute reconstruction inside the `dep_libs` rewrite (the `prepend::Bool` + prefix convention), and - library-id / dependency-rewrite hooks that took the raw `salt` and re-derived the salted form from it (plus a `plat_dep_prefix` string convention). Now a `SaltMap` carries the platform's name-salting function (`rename`) and a materialized `original_basename => salted_basename` map for the files actually found on disk. `rename` is the single definition of "how a name is salted" and is applied uniformly to basenames, SONAME/install-id strings, dependency references, and the `dep_libs` stems. The map records "which names were salted" for dependency/NEEDED rewrites and symlink retargeting. The only remaining platform difference is how the map is built: - one platform's renamer prepends a token (may grow; tooling can rewrite longer strings), - the other's renamer substitutes an equal-length token, keeping every in-place patch length-preserving. Dependency-reference framing differs only via a single `plat_dep_ref` hook. Removed: `plat_salted_basename`, `plat_dep_prefix`, `plat_dep_libs_prepend`, the `prepend`/prefix coupling in the `dep_libs` rewrite, and the raw-`salt` arguments on the library-id and dependency-rewrite hooks. Library-id setting and dependency rewrites now receive only fully-resolved names (or the map). Behavior is preserved on both platforms by construction; the `dep_libs` rewrite produces byte-identical output to the prior whole-blob replace. Symbol-version stamping and symlink handling are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Make SaltMap{F} carry a concrete renamer type instead of an abstract `::Function` field, so the salting transform is type-stable. The single constructor infers F; all SaltMap type annotations are the UnionAll and are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

replacements_for pointed each rewritten dependency at the recorded real-file basename (e.g. the 3-component libjulia.so.1.12.6), but DT_NEEDED / soname strings are 2-component (libjulia.so.1.12), so on a real build the rewrite grew the string and the in-place patcher rejected it. Salt the live dependency string itself instead — matching the salted symlink we already create and staying length-preserving on Linux; basenames is now only the bundled-or-not guard. Caught by the full programatic.jl suite (real PackageCompiler build). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Collapse the wrapped # comment blocks in the privatization sources to one line each. Docstrings are left intact. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Drop the garbled multi-line comment in the replace_needed! error test (and rename that testset to match what it actually checks), and collapse the wrapped comments in the synthetic privatization testset to single lines. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gbaraldi · 2026-06-02T13:30:13Z

https://lief.re/blog/2025-07-13-patchelf is maybe a useful ref for this

asinghvi17 · 2026-06-02T14:06:21Z

Ah interesting! We could use that as a stopgap for now

topolarity

The SaltMap is a big improvement, but it's usage is still pretty inconsistent: smap.basenames[...], record!(...), salt_name(...) all do nearly the same thing and are mixed fairly randomly. It's probably possible to rely on just salt!(smap, basename) and get_salt(smap, basename) functions.

It also feels like the name-handling / etc. functionality for plat_... should be separated from the plat_set_library_id!, etc. functionality that does actual library manipulation. Having them mixed together makes the implementation ping-pong confusingly through the platform-specific logic IMO

There may also be a test coverage gap, since it looks like the salt is not always applied (at least on macOS) - or maybe I am mis-reading the logic

Can you do a human read-through of these changes and make sure you're happy with the structure / style before the next round of review?

topolarity · 2026-06-02T13:58:04Z

+salt_name(m::SaltMap, name::AbstractString) = m.rename(String(name))
+
+# Record that real library basename `base` is salted, returning the salted basename.
+function record!(m::SaltMap, base::AbstractString)


Suggested change

function record!(m::SaltMap, base::AbstractString)

function salt!(m::SaltMap, base::AbstractString)

topolarity · 2026-06-02T14:03:51Z

+plat_make_renamer(::MacOSPlatform, salt::String) = name -> _salt_julia_name_macos(name, salt)
+# macOS sets the install_name id to @rpath/<salted basename>.
+plat_set_library_id!(::MacOSPlatform, smap::SaltMap, libpath::String) =
+    install_name_id!(libpath, plat_dep_ref(MacOSPlatform(), basename(libpath)))


Why doesn't this use the salt?

It also salts a different thing (the filename) than on Linux (the SONAME)

topolarity · 2026-06-02T14:07:36Z

-plat_install_name_change!(::PrivatizePlatform, binpath::String, old::String, new::String) = error("Unsupported platform change")
+
+# How a dependency is referenced in load metadata (macOS `@rpath/<name>`, Linux bare `<name>`); applied uniformly to the id and every dependency reference.
+plat_dep_ref(::PrivatizePlatform, name::String) = error("Unsupported platform")


Suggested change

plat_dep_ref(::PrivatizePlatform, name::String) = error("Unsupported platform")

plat_library_loadpath(::PrivatizePlatform, name::String) = error("Unsupported platform")

topolarity · 2026-06-02T14:11:00Z

-        # This is necessary because the DT_NEEDED entries may point to a symlink
-        salted_filenames[link_base] = salted_target_base
+        # Record the symlink basename so DT_NEEDED entries that name the symlink are recognized as bundled (and thus rewritten).
+        smap.basenames[link_base] = salted_target_base


This breaks your contract for SaltMap, since it bypasses the remapper...

topolarity · 2026-06-02T14:13:13Z


+# macOS's renamer prepends `<salt>_` to the libjulia token (install_name_tool can grow strings); names without the token are left untouched.
+_salt_julia_name_macos(name::AbstractString, salt::String) =
+    replace(String(name), "libjulia" => string(salt, "_", "libjulia"))


Suggested change

replace(String(name), "libjulia" => string(salt, "_", "libjulia"))

replace(String(name), "libjulia" => string(salt, "_", "libjulia"); count = 1)

for consistency with Linux

topolarity · 2026-06-02T14:22:27Z

Can you also move all of the plat_* functions into a Privatize and remove the prefix? It's not very idiomatic Julia - it's just vestigial from past AI efforts

topolarity · 2026-06-02T14:25:58Z

+            # Salt the live dep string (not the recorded basename) so a soname like libjulia.so.1.12 maps to <salt>.so.1.12 — length-preserving; basenames is only the "bundled?" guard.
+            new_dep = plat_dep_ref(platform, salt_name(smap, b))
            push!(seen, (dep, new_dep))
        end


Suggested change

end

else

@assert !occursin("libjulia", b) "Any `libjulia-*` library should have been remapped"

end

topolarity · 2026-06-02T14:26:55Z

-        if haskey(salted_filenames, b)
-            new_dep = plat_dep_prefix(platform) * salted_filenames[b]
+        if haskey(smap.basenames, b)
+            # Salt the live dep string (not the recorded basename) so a soname like libjulia.so.1.12 maps to <salt>.so.1.12 — length-preserving; basenames is only the "bundled?" guard.


These details don't seem to have anything to do with the specific logic here. They are commenting on how the SaltMap remapper was chosen - this code doesn't really care about that.

The comments in general in this PR could use a sweep

asinghvi17 and others added 11 commits May 28, 2026 22:31

PatchVersion: add in-place .dynstr patch helpers

ac3cb5e

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

PatchVersion: pure-Julia DT_SONAME/DT_NEEDED ops + unit tests

6ec3bdc

Add read_soname, read_needed, set_soname!, replace_needed!; verify against patchelf as a read-only oracle plus an end-to-end dlopen of renamed libraries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

privatize(linux): rename salt helpers (no longer patchelf)

fce83ff

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Condense multi-line comments to single lines

d390de3

Comment-only: trim the verbose explanatory blocks added with the pure-Julia ELF ops and the salt-substitution hooks down to one line each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

asinghvi17 marked this pull request as ready for review May 28, 2026 23:42

Use open(file; kwargs...) keyword-separator style

fd47edb

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

topolarity reviewed May 29, 2026

View reviewed changes

topolarity requested changes May 29, 2026

View reviewed changes

asinghvi17 force-pushed the as/linux-no-patchelf branch from ee18d7b to c0bf612 Compare May 29, 2026 23:18

asinghvi17 mentioned this pull request May 29, 2026

Spike: unify name salting behind a single SaltMap asinghvi17/JuliaC.jl#1

Merged

asinghvi17 force-pushed the as/linux-no-patchelf branch from f15d2f3 to 07e6510 Compare May 29, 2026 23:47

asinghvi17 and others added 5 commits May 29, 2026 20:09

Condense multi-line comments to single lines

dbb2b2e

Collapse the wrapped # comment blocks in the privatization sources to one line each. Docstrings are left intact. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

topolarity requested changes Jun 2, 2026

View reviewed changes

topolarity reviewed Jun 2, 2026

View reviewed changes

topolarity force-pushed the main branch from a839423 to d00abf5 Compare June 11, 2026 01:55

	function record!(m::SaltMap, base::AbstractString)
	function salt!(m::SaltMap, base::AbstractString)

	plat_dep_ref(::PrivatizePlatform, name::String) = error("Unsupported platform")
	plat_library_loadpath(::PrivatizePlatform, name::String) = error("Unsupported platform")

	replace(String(name), "libjulia" => string(salt, "_", "libjulia"))
	replace(String(name), "libjulia" => string(salt, "_", "libjulia"); count = 1)

-        end
+        else
+            @assert !occursin("libjulia", b) "Any `libjulia-*` library should have been remapped"
+        end

Uh oh!

Uh oh!

Conversation

asinghvi17 commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

What changed

Tests

Follow-ups

Uh oh!

Choose a reason for hiding this comment

Uh oh!

topolarity left a comment

Choose a reason for hiding this comment

Uh oh!

asinghvi17 commented May 29, 2026

Uh oh!

topolarity commented May 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

topolarity commented May 29, 2026

Uh oh!

gbaraldi commented Jun 2, 2026

Uh oh!

asinghvi17 commented Jun 2, 2026

Uh oh!

topolarity left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

topolarity commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

asinghvi17 commented May 28, 2026 •

edited

Loading

topolarity left a comment •

edited

Loading

topolarity commented Jun 2, 2026 •

edited

Loading