feat: normalize line endings to LF in noarch packages

[wolfv]

Summary

Makes noarch packages reproducible across Windows and Unix by normalizing text-file line endings to LF at packaging time. Closes #837.

This is a reworked version of #2193, which had gone stale against main (the packaging code was moved into the rattler_build_core crate) and reimplemented line-ending handling by hand. This version rebases onto main, plugs into the live packaging path (write_to_dest), and uses the line-ending crate that was already a workspace dependency.

What changed

• New helper copy_normalizing_line_endings(src, dst) in crates/rattler_build_core/src/packaging/file_mapper.rs.
• write_to_dest calls it instead of fs::copy when target_platform == Platform::NoArch — this is the canonical noarch check and covers both noarch: generic and noarch: python, while correctly excluding platform-specific ABI3 (version_independent) packages.
• CHANGELOG entry.

Behavior

• UTF-8 text files are rewritten with all line endings (CRLF and bare CR) collapsed to LF, via line_ending::LineEnding::normalize.
• Binary files and non-UTF-8 encodings (e.g. UTF-16/UTF-32) are copied verbatim. This is deliberate: a naive byte-level \r\n → \n rewrite corrupts multi-byte encodings — for example U+0A0D in UTF-16LE is the byte sequence 0D 0A, whose 0D would be wrongly stripped. Restricting normalization to valid UTF-8 avoids this class of corruption.
• Files already using LF (and everything not normalized) go through fs::copy, preserving the fast copy path and original file metadata.
• Permissions (e.g. the executable bit) are preserved when a file is rewritten, on all platforms.

Tests

• test_copy_normalizing_line_endings: CRLF→LF, bare CR→LF, LF-unchanged, mixed endings, binary-verbatim, and a UTF-16LE case proving the multi-byte corruption is avoided.
• test_copy_normalizing_line_endings_preserves_mode (unix): verifies both that CRLF is normalized and that the 0o755 mode is preserved.

Notes for reviewers

• Normalization is applied to all UTF-8 text files in noarch packages, matching the intent of Textfiles for noarch: generic or noarch: python should have the unix line ending #837 ("most or all files ... unix line ending"). If you'd prefer an opt-out or a narrower set of files, that's easy to add.
• Clippy could not be run in the build environment (the pinned 1.95.0 toolchain lacks the clippy component and the stable fallback is too old for the workspace), but the crate compiles warning-free under 1.95.0 and all tests pass.

Prompt

Can you review this PR? We might have to redo it: #2193 — followed by: "Make a new PR for this!"

🤖 Generated with Claude Code

---

Generated by Claude Code

[pb01ka]

I am thinking about the case where a file is the diff between CRLF/CR and LF. Then \r will be normalised there as well but may be the presence of \r was needed there for test purposes (or anything else?). May be it's a rare case, but plausible for tooling/CI-related packages.

I guess we can make this normalisation optional? Just an open question.

conda/ceps#174 - Worth noting here, especially - conda/ceps#174 (comment).

Rest everything looks good. :).

[pb01ka]

I think once the buffer has the full file from src, we can directly copy from buffer to dst instead of falling through to fs::copy(src, dst). What do you say?

Side note - I tried the following change, cargo test -p rattler_build_core --lib packaging::file_mapper and it passed the tests.

Suggested change

	// Only touch files that actually contain a carriage return, and only if the
	// whole file is valid UTF-8 (the leading kilobyte can be misleading).
	if memchr::memchr(b'\r', &buffer).is_some()
	&& let Ok(text) = std::str::from_utf8(&buffer)
	{
	fs::write(dst, LineEnding::normalize(text).as_bytes())?;
	// Preserve the permissions (e.g. the executable bit) that `fs::copy`
	// would otherwise carry over.
	fs::set_permissions(dst, fs::metadata(src)?.permissions())?;
	return Ok(());
	}
	}
	// Binary, non-UTF-8, or already LF-only files are copied unchanged.
	// Only trust the classification once the whole file is confirmed valid
	// UTF-8 (the leading kilobyte can be misleading). We've already paid the
	// cost of reading the whole file into memory at this point, so write it
	// back out directly instead of re-reading/re-copying it from disk via
	// `fs::copy`, whether or not it actually needed normalizing.
	if let Ok(text) = std::str::from_utf8(&buffer) {
	if memchr::memchr(b'\r', &buffer).is_some() {
	fs::write(dst, LineEnding::normalize(text).as_bytes())?;
	} else {
	fs::write(dst, &buffer)?;
	}
	// Preserve the permissions (e.g. the executable bit) that `fs::copy`
	// would otherwise carry over.
	fs::set_permissions(dst, fs::metadata(src)?.permissions())?;
	return Ok(());
	}
	}
	// Binary or non-UTF-8 files are copied unchanged.

[wolfv]

I started to be unsure about this PR actually .. maybe it is OK to tell people that rebuilding a noarch package identically on Windows is not possible.

I think if we want to do this normalization we should probably also have a way to escape it. And then it's usually easy enough to just build on a UNIX platform ...

[pb01ka]

I think if we want to do this normalization we should probably also have a way to escape it. And then it's usually easy enough to just build on a UNIX platform ...

I agree. And the consensus on my CEP is also similar. So we can do this here as well. If it helps (and sounds good to you) I can do it by checking out from your branch and creating a new PR. Please let me know.