CEP XXXX: Document status quo for prefix replacement - #180
Conversation
chrisburr
changed the title
Record occurrences per encoding ([{"encoding": ..., "ranges": ...}]) so a
single package serves installers with different search semantics. Producers
must now cover all five encodings replaced by existing installers; the
previous rule omitting offsets for wide-encoding files is gone. Reference
the prefix replacement CEP (conda#180) for replacement semantics and
defer padding strategies and mixed text/binary substitution to a future
versioned paths format.
chrisburr
changed the title
| <tr><td> Author(s) </td><td> Chris Burr <christopher.burr@cern.ch> </td></tr> | ||
| <tr><td> Created </td><td> Jul 15, 2026 </td></tr> | ||
| <tr><td> Updated </td><td> Jul 15, 2026 </td></tr> | ||
| <tr><td> Discussion </td><td> NA </td></tr> |
There was a problem hiding this comment.
| <tr><td> Discussion </td><td> NA </td></tr> | |
| <tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/180 </td></tr> |
| <tr><td> Created </td><td> Jul 15, 2026 </td></tr> | ||
| <tr><td> Updated </td><td> Jul 15, 2026 </td></tr> | ||
| <tr><td> Discussion </td><td> NA </td></tr> | ||
| <tr><td> Implementation </td><td> NA (documents existing behavior) </td></tr> |
There was a problem hiding this comment.
You could link here some permalinks to the relevant modules in each package manager.
jaimergp
left a comment
jaimergp
left a comment
There was a problem hiding this comment.
I have some comments about what we should define as a standard, a divergence and a bug. In practice, conda has been the leading de facto standard, so unless the behavior is clearly a bug, their design decision should be recorded as a standard (on which we can iterate in further work). Under that definition, we can standardize the UTF encoding and pyzzer handling divergences as standard. Deprecating that should be done explicitly afterwards, or non-compliant tools should catch up and fix the bug.
If further work like #179 is already in the pipeline addressing some of the limitations (e.g. the ambiguity of UTF handling), even more so.
| Only files whose path entry carries a `prefix_placeholder` are processed. All | ||
| other files are linked or copied unmodified. Which files receive an entry is | ||
| decided at build time and is out of scope here. |
There was a problem hiding this comment.
| Only files whose path entry carries a `prefix_placeholder` are processed. All | |
| other files are linked or copied unmodified. Which files receive an entry is | |
| decided at build time and is out of scope here. | |
| All files whose path entry (in `paths.json` or equivalent) carries a `prefix_placeholder` MUST be processed. All | |
| other files MUST be linked or copied unmodified. Which files receive an entry is | |
| decided at build time and is out of scope here. |
| ### Text files | ||
|
|
||
| Installers MUST replace every occurrence of the placeholder with the target | ||
| prefix. The file length may change. |
There was a problem hiding this comment.
| prefix. The file length may change. | |
| prefix. The file size may change. |
Talking about length is ambiguous here, I think.
| When the target prefix contains backslashes (Windows), installers MUST | ||
| convert them to forward slashes before replacement. This matches the | ||
| separator convention CEP 34 requires of the placeholder itself and avoids | ||
| producing invalid escape sequences in patched source files. |
There was a problem hiding this comment.
This is not clear to me. Will the final contents of the file use forward or back slashes? I'd venture it must use whatever was using originally, but replacement must take into account the slash normalization, correct? As in:
- Read file and find occurrences of the placeholder prefix with both types of slashes. Record found slash pattern.
- Replace placeholder with target prefix, matching the required slash pattern.
Is that what happens?
|
|
||
| ### Binary files | ||
|
|
||
| Binary replacement MUST preserve the file length. Installers replace each |
There was a problem hiding this comment.
| Binary replacement MUST preserve the file length. Installers replace each | |
| Binary replacement MUST preserve the file size. Installers MUST replace each |
| occurrence, shift the remainder of the surrounding NUL-terminated string | ||
| towards the start, and fill the gap before the original NUL terminator with |
There was a problem hiding this comment.
Isn't that "shifting" implied by left-aligned replacement (the most common type)?
| ### Encodings | ||
|
|
||
| conda replaces occurrences of the placeholder encoded as UTF-8, UTF-16-LE/BE | ||
| and UTF-32-LE/BE, in both text and binary mode. This was added by | ||
| [conda/conda#9946][conda-9946] for packages that store paths as wide strings | ||
| ([conda/conda#4298][conda-4298], wxWidgets). libmamba and rattler replace | ||
| UTF-8 only; [mamba-org/mamba#2215][mamba-2215] tracks this but was never | ||
| implemented. Neither conda-build nor rattler-build detects wide-encoding | ||
| occurrences at build time, so affected files only receive a path entry | ||
| through conda-build's `binary_has_prefix_files` or by also containing a UTF-8 | ||
| occurrence. | ||
|
|
||
| The multi-encoding replacement is a sequence of byte-level search passes, | ||
| one per encoding in the order listed above, and byte-level search cannot | ||
| always tell the two endiannesses of a wide encoding apart. An ASCII | ||
| character encodes to its ASCII byte plus zero bytes, on opposite sides | ||
| depending on the endianness, so the big-endian encoding of an ASCII string | ||
| contains the little-endian byte pattern of the same string at a shift of | ||
| one byte (three bytes for UTF-32), with the final zero byte supplied by | ||
| whatever follows the last character: the high bytes of the next character | ||
| or the NUL terminator. For example, a placeholder `/pfx` stored as a | ||
| NUL-terminated UTF-16-BE string, followed by one more zero byte (padding, | ||
| or the high byte of the next big-endian character), admits two complete | ||
| readings, each leaving a single unclaimed zero byte at the opposite end: | ||
|
|
||
| ```text | ||
| bytes: 00 2f 00 70 00 66 00 78 00 00 00 | ||
| read as BE: └'/'┘ └'p'┘ └'f'┘ └'x'┘ └NUL┘ | ||
| read as LE: └'/'┘ └'p'┘ └'f'┘ └'x'┘ └NUL┘ | ||
| ``` | ||
|
|
||
| A big-endian wide string containing the placeholder therefore also matches | ||
| the little-endian pass, which runs first and splices the little-endian | ||
| target at the shifted position. Whether the wrong guess matters depends on | ||
| the characters involved. Below U+0100 every byte the wide encoding adds is | ||
| zero, and the zeros are interchangeable between the two readings: the | ||
| spliced bytes read back correctly at the big-endian alignment, so the | ||
| wrong guess produces exactly the bytes the right one would. At U+0100 and | ||
| above the added bytes are no longer zero, the shift pairs bytes across | ||
| character boundaries, and the string is corrupted instead of replaced (a | ||
| little-endian splice of `ā`, U+0101, reads back big-endian as the pair | ||
| U+0001, U+0170). The ambiguity is symmetric, since a little-endian string | ||
| preceded by a zero byte contains a phantom big-endian match one byte | ||
| earlier, so no pass order resolves it; it merely goes unnoticed while the | ||
| target prefix and the remainder of the affected string stay below U+0100, | ||
| which paths in practice almost always do. A target prefix containing | ||
| characters at or above U+0100, installed over a wide string of the | ||
| opposite endianness, is the combination that corrupts. Metadata that | ||
| records the encoding of each occurrence, as proposed in | ||
| [conda/ceps#179][ceps-179], is not subject to this ambiguity. |
There was a problem hiding this comment.
Since #179 will address this unambiguously:
- I'd trim this to the first paragraph and mention that this behavior is currently unspecified, and should be addressed by further work.
- I'd take the other two paragraphs and use it in the Rationale or Motivation sections for 179.
There was a problem hiding this comment.
The ambiguity is still there for generating the paths.json. The only thing that can truly solve it is metadata in the recipe but that's outside of the scope of #179.
| rattler rewrites Python interpreters to a `/bin/sh` trampoline instead of | ||
| the `#!/usr/bin/env` form. | ||
|
|
||
| ### Format strictness |
There was a problem hiding this comment.
I'd advocate for standardizing around the strict behavior of conda and submitting bug reports to the other projects.
| records the encoding of each occurrence, as proposed in | ||
| [conda/ceps#179][ceps-179], is not subject to this ambiguity. | ||
|
|
||
| ### pyzzer launchers |
There was a problem hiding this comment.
This also sounds like a bug in rattler
| ### Shebang lines | ||
|
|
||
| On Unix target platforms, a text file starting with `#!` gets special | ||
| treatment: the first line is rewritten as a unit rather than by plain | ||
| substitution. If the patched interpreter path fits within the length limit | ||
| and contains no spaces, it is kept. Otherwise the line is rewritten to an | ||
| `#!/usr/bin/env <program>` form, on the assumption that the environment will | ||
| be activated when the script runs. The limits in use are 127 bytes on Linux | ||
| and 512 on macOS. | ||
|
|
||
| The rewritten output differs between installers; see below. Only the intent | ||
| is shared: the first line must remain a working interpreter directive. |
There was a problem hiding this comment.
This is the specification, so we need to use RFC language like MUST or MUST NOT.
| When the target platform is Windows, installers MUST NOT apply C-string | ||
| replacement. The pyzzer rewrite described under divergences is the one | ||
| exception conda and libmamba make. |
There was a problem hiding this comment.
| When the target platform is Windows, installers MUST NOT apply C-string | |
| replacement. The pyzzer rewrite described under divergences is the one | |
| exception conda and libmamba make. | |
| When the target platform is Windows, installers MUST NOT apply C-string | |
| replacement, except for the `pyzzer`-style launchers, which feature a shebang line that MUST be handled as if it was a Unix shebang. |
WDYT about standardizing this? Or is it an edge case that doesn't deserve standardization?
There was a problem hiding this comment.
Alternatively, we can encode this spec as a behavior to follow in this type of case: "when the file features an occurrence of the prefix placeholder and its functionality depends on its correct replacement with the target prefix, install tools SHOULD strive to apply the transformation when feasible (e.g. easy-to-define generalizations like pyzzer launchers).
2 participants
In #179 (comment) I found myself thinking there is a need for an informative CEP to document the status quo for what prefix replacement means.
🤖 the document was written by AI based on my human researched notes 🤖
Checklist for submitter
cep-0000.mdnamedcep-XXXX.mdin the root level.CEP XX: Amend XYZ.## Changelogsection right above the final "Copyright" section with an item that uses syntaxYYYY-MM-DD: Brief explanation of changes.Checklist for CEP approvals
${greatest-number-in-main} + 1.cep-XXXX.mdfile has been renamed accordingly.# CEP XXXX -header has been edited accordingly.pre-commitchecks are passing.