perf(rattler): re-sign macOS binaries ad-hoc in-process

[wolfv]

Description

On macOS, prefix replacement rewrites bytes inside Mach-O binaries, which invalidates their code signature. rattler currently re-signs each affected binary by spawning a /usr/bin/codesign process. When a package contains many binaries this is a real bottleneck — conda measured ~9.5s across 186 codesign spawns in a single install (see conda/conda#15975, which motivated this change).

Rather than batching the subprocess calls (conda's approach), this does the signing in-process:

• conda-forge binaries ship with an ad-hoc signature (just a CodeDirectory — no certificate, no CMS blob).
• rattler's binary prefix replacement is length-preserving (copy_and_replace_cstring_placeholder null-pads), so the LC_CODE_SIGNATURE load command and __LINKEDIT segment don't move.

So re-signing collapses to: recompute the CodeDirectory's per-page SHA-256 hashes and overwrite them in place. No subprocess, no new dependencies (SHA-256 comes from rattler_digest, already used here).

A new minimal adhoc_sign module implements this for thin and fat (universal) Mach-O binaries. link_file tries it first and falls back to the /usr/bin/codesign subprocess for anything it doesn't handle — unsigned binaries (which would need inserting a load command + growing __LINKEDIT), big-endian Mach-O, or non-SHA-256 directories — so behavior is unchanged in those cases and this is purely a fast path for the common one.

How Has This Been Tested?

Added a macOS-gated end-to-end test (resign_roundtrip_matches_codesign) that:

1. Compiles a real binary with clang and ad-hoc-signs it with /usr/bin/codesign.
2. Corrupts a byte inside the hashed region and asserts codesign --verify now fails.
3. Re-signs in-process with adhoc_resign.
4. Asserts codesign --verify now passes and the binary still executes.

cargo fmt --check and cargo clippy --all-targets are clean.

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR.

Tools: Claude Code (Opus 4.8)

Prompt:

Is this an improvement that we should also implement? https://github.com/conda/conda/pull/15975
[... discussion concluded that in-process ad-hoc signing is a better fit for
rattler than conda's subprocess batching, since rattler already signs in
parallel ...]
Yeah please show me the minimal, possibly dependency free way, of doing ad-hoc signing :)
Can you rebase this on main and make a PR from it (push to my fork)

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added sufficient tests to cover my changes.

🤖 Generated with Claude Code

[baszalmstra]

Thats insanely cool! Useful for Dagmar as well!!

[jezdez]

I see Claude had its hand in this, has this... been audited? I've contemplated writing an in-process tool for conda/conda#15975 but ultimately worried it would defeat the purpose of using available projects that we know work well and are security reviewed. Hence the focus on batching the processes instead.

[wolfv]

there is nothing particularly security critical in this as it's just replacing a SHA hash (the ad-hoc signature is not really "cryptographic").

I have written this code a few times already :)

[baszalmstra]

Review from Fable:

Review: #2554 — perf(rattler): re-sign macOS binaries ad-hoc in-process

1. High — non-ad-hoc (cert/CMS-signed) binaries are silently corrupted

resign_slice never verifies the signature is actually ad-hoc. It reads the CD version (offset 8) and hashOffset (offset 16) but skips flags (offset 12), and never inspects the SuperBlob for a CMS wrapper (CSSLOT_SIGNATURESLOT, 0x10000). For a binary with a real certificate signature, the CMS blob signs the CodeDirectory hash — rewriting page hashes invalidates that binding, yet the function returns Resigned, so the corrupt signature is written to disk, the codesign fallback is skipped, and no error is raised even under AppleCodeSignBehavior::Fail. The old path handled this case correctly: codesign --force strips the cert signature and replaces it with a working ad-hoc one.

This is rare with conda-forge (builds ad-hoc sign everything), but rattler installs packages from arbitrary channels, and a Developer-ID-signed binary containing the placeholder prefix would go from "works, re-signed ad-hoc" to "killed by the kernel at exec, install reported success."

Fix (cheap): require flags & CS_ADHOC (0x0000_0002) != 0, and/or bail to NeedsFullSign if a CSSLOT_SIGNATURESLOT blob with length > 8 is present. Also worth an accompanying test.

2. Medium — codeLimit64 read at the wrong offset

if version >= 0x0002_0300
    && let Some(cl64) = read_u64_be(data, cd_off + 48)

Per cs_blobs.h, offset 48 is teamOffset (v0x20200); codeLimit64 lives at offset 56, after scatterOffset (44) and spare3 (52). The doc comment above it repeats the mistake. It's benign today only by accident: ad-hoc CDs have teamOffset == 0, so the cl64 != 0 guard falls through to the 32-bit codeLimit. If teamOffset were ever non-zero, code_limit would be garbage — the expected_slots != n_code_slots sanity check would catch it and fall back, so it fails safe, but the read should be corrected to cd_off + 56. (Files > 4 GiB, where codeLimit64 actually matters, currently take the fallback path either way.)

3. Low — scatter vectors not rejected

For version >= 0x20100, a non-zero scatterOffset (offset 44) changes which pages the hash slots cover; recomputing linearly would write wrong hashes and return Resigned. Scatter signatures are essentially extinct, but a 4-byte read + bail is cheap insurance in the same spirit as the other checks.

4. Low — robustness against corrupt/hostile input

The parser runs on untrusted package contents, and while every field read is properly bounds-checked (good!), three spots trust file-supplied values structurally:

• collect_slices does Vec::with_capacity(nfat) with nfat straight from the file (adhoc_sign.rs, fat branch). A crafted 0xcafebabe + nfat = 0xffffffff header requests a ~64 GiB allocation before any entry is validated. Clamp first: 8 + nfat * entry_size <= data.len().
• 1usize << page_shift with page_shift up to 255 overflows the shift — a panic in debug builds (release gets a masked shift, then the slot-count check bails). Validate page_shift (e.g. reject >= 32).
• resign_slice takes _len but ignores it: page hashing bounds-checks against the whole file rather than the slice end, so a corrupt fat header can make one slice's hashing read a neighbor's bytes. Not memory-unsafe, just looser than it needs to be.

Also cosmetic in the same function: a fat file with nfat == 0 returns Resigned vacuously, and collect_slices reads the magic at offset 0 twice.

5. Notes on the link.rs integration (no action strictly required)

• The fast path does a whole-file fs::read per binary. Package linking runs files in parallel, so several multi-hundred-MB dylibs (pytorch-class packages) in flight means a real memory spike where the old path streamed. Probably an acceptable trade for the process-spawn win, but worth being conscious of; the buffer could eventually be shared with the prefix-replacement step that already streamed these bytes.
• fs::write(...).map_err(LinkFileError::FailedToOpenDestinationFile) reuses a misleadingly-named variant for a write failure (an existing pattern in this function, admittedly).
• Err(_) => false on the initial read silently swallows I/O errors — defensible, since the fallback will surface anything real.
• Positive side effect worth calling out in the PR description: in-process signing works on non-macOS hosts, so cross-platform osx-* environment creation can now produce valid binaries where spawning codesign was impossible.
• Good detail: the buffer is only written when every slice resigns, so a fat binary that partially fails leaves the on-disk file untouched for the fallback.

Test coverage

Gaps:

• No test for fat/universal binaries (supported by the code, exercised nowhere).
• No fallback test for cert-signed input (finding 1).
• The parser is pure &[u8] → outcome and compiles everywhere, so hand-crafted-header unit tests (truncated files, NotMachO, huge nfat, wrong hash type) could run on Linux CI too, not just the macOS lane.

[baszalmstra]

I would prefer #2588. Can we close this?