Implement Windows privatization of bundled libjulia#151
Draft
asinghvi17 wants to merge 12 commits into
Draft
Conversation
Ship the 88-byte precompiled .rsrc resource-directory header (src/template/rsrc.bin) used to inject an SxS private-assembly RT_MANIFEST resource into Windows PE products. The blob encodes the resource tree RT_MANIFEST(0x18) -> id 0x2 -> langid 0x409 -> data entry at byte 0x48, with the OffsetToData/Size fields zeroed (patched at injection time). It is pure binary (no embedded strings). sha256: 8e50045a2142f22092d43ada239841c192f79ea9ec1e56697cca2e9e4de001b2 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the mingw-w64 artifact bin/ directory lookup out of get_compiler_cmd into reusable mingw_bindir() and mingw_tool(name) helpers so linking and the upcoming Windows privatization share one source of truth for the toolchain location. The Windows branch of get_compiler_cmd now calls mingw_tool, preserving existing behavior. The artifact lookup is wrapped in @static if Sys.iswindows() inside mingw_bindir() because the @artifact_str macro expands at compile time and the artifact entry is os="windows"-gated, so it is unresolvable off-Windows; this keeps the package loadable on non-Windows hosts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Create src/privatize_windows.jl with the module header, ObjectFile / StructIO imports, the TEMPLATE_DIR RelocatableFolders constant pointing at the shipped rsrc.bin, the .rsrc data-entry patch offsets (MANIFEST_ADDRESS_OFFSET 0x48, MANIFEST_SIZE_OFFSET 0x4c), and the generate_manifest_xml / manifest_identity_for helpers plus the LIBJULIA_DLL_CANDIDATES tuple. The identity is JuliaC.PrivateRuntime.<basename>. The file is not yet included by JuliaC.jl (wired in a later step), so the package still loads unchanged. Manifest generator verified in isolation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append inject_private_manifest!(product_path, dll_names) to privatize_windows.jl. It builds the .rsrc section payload (precompiled header ++ generated manifest ++ 4-byte pad), adds it as a .rsrc section via the mingw objcopy tool, then patches the COFF optional header's ResourceTable data directory (VirtualAddress/Size) and the section's internal IMAGE_RESOURCE_DATA_ENTRY OffsetToData/Size fields. The COFF offset math, COFFImageDataDirectory pack, and ResourceTable field offset were verified on Linux against real 32-bit and 64-bit PE binaries (computed datadirs offset reproduces the actual .rsrc section VA). Running objcopy.exe and resolving the mingw artifact are Windows-CI-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append the in-place libpath fix that strips the stale ../bin/ prefix (and @../bin/ -> @) from the bundled libjulia.dll's embedded, colon-separated, NUL-terminated library search path; present_libjulia_dlls to list the libjulia*.dll files actually present in the bundle bin/ dir (so trim builds that drop libjulia-codegen.dll produce a correct manifest); and the standalone privatize_libjulia_windows! entry point that injects the manifest into the product and fixes the libpath, with the try/catch + error(..., e) convention matching the macOS/Linux wrappers. The libpath rewrite algorithm was verified on Linux fixtures: rewrites ../bin/a:../bin/b:@../bin/c -> a:b:@c, leaves prefix-less entries untouched (../bin/x:plainentry:@../bin/y -> x:plainentry:@y), preserves trailing bytes, and errors with "Unable to open" on a missing file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add include("privatize_windows.jl") after privatize_macos.jl in
JuliaC.jl, and replace the Windows @warn no-op in privatize_libjulia!
with an elseif Sys.iswindows() branch dispatching to
privatize_libjulia_windows!. The full package loads with all new symbols
defined, the patch-offset consts correct (0x48/0x4c), and the
TEMPLATE_DIR RelocatableFolders path resolving to the 88-byte rsrc.bin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Sys.iswindows()-guarded testset inside the Library flows (trim) block that builds a privatized library bundle, then asserts the injected .rsrc resource directory is present (read via ObjectFile), the COFF optional-header ResourceTable data directory points at the .rsrc section VA (read independently via raw seek/read, recomputing the datadirs offset from the file), and the bundled libjulia.dll no longer contains the ../bin/ prefix. No objdump/dumpbin dependency. ObjectFile is referenced as JuliaC.ObjectFile since it is a JuliaC dep but not in the test target extras. On Linux the body is skipped; the full suite passes (Pkg.test). The Windows assertions run on windows-latest CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment-only: trim the three verbose explanatory blocks in the Windows privatization file down to one line each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
This looks pretty good to me, but would be good to have a list of changes versus the original implementation and also a test that actually loads the library within Julia and confirms that @asinghvi17 anything that's keeping it in draft? |
Add a Windows-gated test that loads the bundled, privatized product within a fresh Julia process and confirms `Libdl.dllist()` reports two distinct copies of the runtime libraries: the host process's already-loaded copy plus the bundled private copy that the side-by-side (SxS) manifest steers the product to load from its own directory. Also checks that the exported symbol still resolves through the privatized runtime. Gated to `Sys.iswindows()` so the suite is a no-op elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the wrapped explanatory comment blocks in the privatized-load testset to single lines. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add flushed progress markers (M1_PRE_DLOPEN .. M5_POST_CCALL) so the CI log shows whether the hang is at SxS activation (load) or the first-ccall runtime init, and report Libdl.dllist() dirs right after dlopen (before the ccall) to tell whether the product bound its own private libjulia. Run the subprocess under a 180s watchdog so the job can never again consume the 6h CI limit. Diagnostic step toward fixing the Windows-only deadlock; not the final form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JuliaC's
privatize_libjulia!handles macOS and Linux, but on Windows it was a no-op (@warn "Privatization not implemented for this OS"). This PR implements Windows privatization soBundleRecipe(privatize=true)/--privatizeworks on Windows too.Mechanism
Windows PEs have no SONAME, so the Unix salt-rename approach doesn't apply. This uses the standard Windows side-by-side (SxS) isolation mechanism instead:
.rsrcresource into the built product, so a directly-launched product resolves its own bundledlibjulia.dllvia an activation context. The manifest XML is generated in code (identityJuliaC.PrivateRuntime.<product>, with a<file>list built from thelibjulia*.dllactually present — so trim builds that droplibjulia-codegen.dllare handled). Only an 88-byte precompiled resource-directory header (src/template/rsrc.bin, no text content) is shipped as a static asset.../bin/prefix inlibjulia.dll's libpath string is stripped in place.Implementation
src/privatize_windows.jlwithprivatize_libjulia_windows!— standalone (not theplat_*hooks, which model Unix salt-renaming).inject_private_manifest!: build the.rsrcsection, add it viaobjcopy, then patch the COFF optional-header ResourceTable data directory and the section's manifest address/size fields usingObjectFile+StructIO.objcopyis taken from themingw-w64artifact JuliaC already uses for Windows linking (resolution factored into sharedmingw_bindir/mingw_toolhelpers inlinking.jl).privatize_libjulia!dispatch inbundling.jl, replacing the@warn.Isolation contract (v1)
This makes a directly-launched product load its own bundled
libjulia.dll; it does not renamelibjulia.dll, so it does not guarantee multiple privatized bundles coexisting in a single host process. Per-bundle DLL renaming + import-table rewriting is a possible follow-up.Tests
A guarded
if Sys.iswindows()testset intest/programatic.jlbuilds a bundle withprivatize=trueand reads the injected resource directory back from the product's PE headers (rawseek/read, no external tools). Locally (Linux) the COFF parse/patch offset math, manifest generation, and libpath-fix algorithm were verified against real PE binaries (including the 32-bit branch); actually runningobjcopy --add-sectionand the SxS loader honoring the manifest are exercised by this testset on Windows CI.🤖 Generated with Claude Code