fix(utils): length-prefix paths in compute_content_hash; deprecate bare content_sha* recipe keys

[pb01ka]

Description

Fix a hash-collision bug in compute_content_hash reported in conda/ceps#150 and implement the v2 algorithm specified in the accompanying CEP (supersedes CEP 19).

Bug fix - new v2 algorithm

Root cause. The hash stream for each directory entry was built by concatenating raw bytes in the order <path><type><content><separator>, with no field-length information. Because filenames can contain the same bytes used as type markers (F, D, L) and the entry separator (-), two structurally different trees could produce an identical byte stream.

For example:

• Tree 1: a single file named testFhello-world with content www
• Tree 2: a file test (content hello) plus a file world (content www)

Both produced stream testFhello-worldFwww-, yielding the same SHA-256 digest.

Fix. Each variable-length field is now prefixed with its decimal byte length followed by : before being fed to the hasher:

Field	Before (CEP-19)	After (v2)
Path	<path_bytes>	<len(path_bytes)>:<path_bytes>
Symlink target	<target_bytes>	<len(target_bytes)>:<target_bytes>

This makes field boundaries unambiguous and eliminates the collision.

Backwards compatibility and key naming

The algorithm change produces different digests for the same directory contents, so existing stored hashes are not compatible with the new algorithm. Two mechanisms are provided:

1. Distinct key families. The new fixed algorithm is exposed under the content_sha256_v2 / content_sha384_v2 / content_sha512_v2 recipe keys. New recipes should use these. The original CEP-19 algorithm is retained under the bare content_sha256 / content_sha384 / content_sha512 keys for backwards compatibility, but those keys are now deprecated - using them emits a DeprecationWarning at build time directing recipe authors to migrate to the _v2 keys.

2. legacy parameter. compute_content_hash() accepts a new legacy=True keyword argument that reproduces the original CEP-19 byte stream exactly, for any code that computes or verifies hashes programmatically against pre-existing stored values. Passing legacy=True also emits a DeprecationWarning.

Migration guide

Old key (deprecated)	New key
content_sha256	content_sha256_v2
content_sha384	content_sha384_v2
content_sha512	content_sha512_v2

Re-compute your hashes with compute_content_hash(directory, legacy=False) (the default) and update the recipe keys accordingly.

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation? (no user-facing docs affected)

[conda-bot]

We require contributors to sign our Contributor License Agreement and we don't have one on file for @pb01ka.

In order for us to review and merge your code, please e-sign the Contributor License Agreement PDF. We then need to manually verify your signature, merge the PR (conda/infrastructure#1345), and ping the bot to refresh the PR.

[jaimergp]

UTF-8 byte length as in the number of bytes required to encode the path with UTF-8? And why do we need the colon separator?

[pb01ka]

Added an explanation for it.

[codspeed-hq]

Merging this PR will not alter performance

✅ 5 untouched benchmarks

---

_{Comparing pb01ka:fix-hashing (3f69c32) with main (e2e8091)}

[pb01ka]

Seems like the following test is failing,

FAILED tests/test_api_build.py::test_recipe_builds[source_url] - RuntimeError: content_sha256 mismatch in source item #0: obtained 'fa18683d70b5b776b017ac0c55b1086a70c7a12584fc1c2fd5166b79f568c687' != expected 'a884ace5aa3a7e7f5a8b5adeb5cbfa7209f2ae88134d362c8bbca9c82ad2bb06'

This means the change I have made might be backwards incompatible? Is this a genuine concern?

cc: @jaimergp

[jaimergp]

This means the change I have made might be backwards incompatible? Is this a genuine concern?

That's to be decided in the CEP amendment we need to co-submit along with this fix. If it's a problem, we may need to add a _v2 suffix or similar.

[pb01ka]

@jaimergp The PR is passing all the tests. Please let me know if the changes look good here.

[jaimergp]

Not sure about the v1 suffix here. It makes it look like this is actually the successor, not the deprecated version. We need a different term. Maybe just a leading underscore? Or something else?

• _content_sha256
• _legacy_content_sha256
• _deprecated_content_sha256

Or the other way around, adding something to the new one (which is maybe better for bw compat):

• length_prefixed_content_sha256 (too long?)
• delimited_content_sha256
• normalized_content_sha256 (true in both editions, but is maybe more informative)

[pb01ka]

Good point on the naming. A couple of thoughts:

1. Leading underscore vs _v1 suffix - semantically these are equivalent: both signal "this is the old/legacy variant." The backward-compatibility story is identical either way - users who have stored hashes computed with the original CEP-19 algorithm (no length-prefixing) still need to opt in to the legacy key explicitly, regardless of whether it's called _content_sha256 or content_sha256_v1. I slightly prefer _v1 because I think it makes the relationship between the two explicit (v1 = original, unversioned = current/improved), but I'm open to switching if the team finds _content_sha256 or _legacy_content_sha256 clearer.

2. Adding something to the new key instead - this is actually the cleanest option for backward compatibility. If we keep the existing content_sha256 name for the legacy algorithm and introduce length_prefixed_content_sha256 (or delimited_/normalized_) for the improved one, existing recipes and tooling that already reference content_sha256 continue to work with zero changes and no deprecation warnings needed. The new, more secure algorithm is purely opt-in via the new name. The only downside is that the "plain" name now refers to the weaker algorithm, which could surprise new users - but that's a documentation problem, not a compatibility one. Among the suggestions, normalized_content_sha256 reads the best to me since length-prefixing is an implementation detail whereas "normalized" conveys intent.

[jaimergp]

I think we can find a better adjective than normalized, if it's about intent. Can you propose a few? e.g. I'm thinking of corrected_content_sha256 but I'm worried there's yet another flaw and then we have to come up with something else 😂

[pb01ka]

Fair concern - "corrected" (and "normalized") both imply "this is the fixed version," which ages badly if another issue surfaces later. Here are a few alternatives grouped by strategy:

Describe the mechanism, not the quality

• delimited_content_sha256 - the entries (paths, symlinks) are delimited before hashing, so concatenation collisions can't occur. Specific enough to be meaningful, but makes no claim about overall algorithm quality.
• prefixed_content_sha256 - refers directly to the length-prefixing. Accurate, but does expose the implementation detail (length_prefixed_ has similar concerns).

Describe the property

• distinct_content_sha256 - the inputs are kept distinct; no two different path lists can produce the same hash input. Conveys the intent without implying "fully corrected."
• unambiguous_content_sha256 - similar angle: the serialization is unambiguous. Slightly more self-explanatory but a bit long.

Just version it

• content_sha256_v2 - the most future-proof option. Makes no quality claim at all, just signals "newer algorithm." If a third revision is ever needed, v3 slots in cleanly. The downside is it's less informative on its own, but a short docstring fixes that.

My recommendation: delimited_content_sha256 if you want the name to be self-documenting, content_sha256_v2 if you want it to be future-proof without overloading meaning onto an adjective.

[jaimergp]

Let's go with _v2 then.

[pb01ka]

Already done. We are in good shape as of now.

[pb01ka]

The failing test doesn't seem relevant. AFAICT, they seem to fail due to Github constraints. Please let me know if I am perceiving this wrongly.

Other than this the PR is up to date. Please let me know if any other change is needed.

[pb01ka]

The failing test is due to network issues I think,

FAILED tests/test_post.py::test_pypi_installer_metadata - conda.CondaMultiError: ('Connection broken: IncompleteRead(8124139 bytes read, 4861299 more expected)', IncompleteRead(8124139 bytes read, 4861299 more expected))

Please feel free to re-run the failed checks to be sure. Other than this the PR is ready. If you have any reviews please feel free to share.

I will also send a draft PR with the CEP linking this PR. It will supersede CEP-19.

[pb01ka]

The CEP PR is raised at conda/ceps#174.

@jaimergp @kenodegard

[jaimergp]

Thanks @pb01ka, I'll mark this as draft until the CEP is approved.