fix(sessions): escape undecodable subprocess output bytes

[leongdl]

Fixes: #296

What was the problem/requirement? (What/Why)

The stdout read loop in run_subprocess decoded each line with String::from_utf8_lossy, which replaces every undecodable byte with U+FFFD. The reader survives invalid input (unlike the Python implementation, which had a genuine crash here), but the original byte values are destroyed, so the session log gives no indication of what the subprocess actually emitted.

openjd-sessions-for-python decodes the same output with errors="backslashreplace", so the two implementations logged different text for identical child output. That divergence is the subject of #296.

The motivating case is a production Windows customer-managed fleet where Unreal Engine wrote the cp1252 em dash 0x97 to stdout. In Python that byte crashed the stdout reader thread and lost all subsequent output (fixed in openjd-sessions-for-python#343, which chose backslashreplace so the byte value survives into the log). Here the same byte was silently flattened to �, leaving an operator no way to tell which code page the renderer was using.

What was the solution? (How)

Add decode_backslashreplace, which decodes as UTF-8 and escapes every byte that is not valid UTF-8 as \xNN with lowercase hex, and use it in the stdout read loop.

It is built on slice::utf8_chunks (stable since 1.79.0, comfortably below the 1.94.1 MSRV), which yields (valid prefix, invalid sequence) pairs. Valid text therefore needs no re-validation, and invalid sequences are delimited exactly as UTF-8 validation defines them. CPython escapes each byte of an undecodable sequence individually, so a 3-byte invalid sequence such as an encoded surrogate produces three escapes; this matches that.

Fully valid input returns Cow::Borrowed and does not allocate. This matters because the function runs once per line of subprocess output, and it is a small improvement to the allocation chain flagged as item 4 in reports/sessions-quality-evaluation-report.md (that item remains open; this only removes the decode-step allocation on the valid path).

What is the impact of this change?

• Session logs preserve the original byte values of non-UTF-8 subprocess output instead of showing �, so the emitting code page is identifiable.
• The Rust and Python implementations now log identical text for identical child output.
• Log content changes for processes that emit non-UTF-8: a byte previously rendered as � now renders as \x97.
• Escaping expands each undecodable byte from 1 byte to 4 characters, where from_utf8_lossy expanded an invalid sequence to 3 bytes. A line consisting largely of undecodable bytes can therefore grow up to fourfold before the existing 64KB truncation is applied, rather than threefold. Both are bounded by the same truncation.
• Valid UTF-8 output is unaffected, and is now decoded without allocating.

How was this change tested?

Yes, unit tests were run: cargo test -p openjd-sessions is green (443 tests), as is cargo test -p openjd-cli (155 tests, the dependent crate). cargo clippy --all-features --all-targets --workspace -- -D warnings, cargo fmt --all --check, RUSTDOCFLAGS="-D warnings" cargo doc, and the copyright-header check all pass.

Test-first: the integration test was written before the fix and fails against from_utf8_lossy with "bad \u{fffd} byte!\n".

New tests:

• test_run_subprocess_invalid_utf8_is_escaped — end-to-end through run_subprocess: the customer's 0x97 byte appears as \x97 and no U+FFFD is present.
• test_run_subprocess_valid_utf8_not_escaped — negative control against over-escaping, end to end.
• test_decode_backslashreplace_matches_cpython — a 17-case conformance table whose expected values were generated by CPython's own decoder, covering the 0x97 byte, always-invalid bytes, consecutive invalid bytes, cp1252 text runs, a truncated sequence mid-line and one at EOF (where Utf8Error::error_len() is None), a lone continuation byte, an encoded surrogate, an overlong encoding, an out-of-range lead byte, valid 2/3/4-byte sequences, empty input, an already-present literal backslash, and NUL/tab passthrough.
• test_decode_backslashreplace_borrows_valid_input — pins the no-allocation fast path.
• test_decode_backslashreplace_escapes_use_lowercase_hex — CPython emits lowercase; uppercase would be a visible divergence.

Beyond the table, the algorithm was differential-tested against CPython over 20,000 random and adversarial inputs (uniform random bytes, sequences biased toward UTF-8 lead/continuation bytes, valid text with invalid bytes spliced in, and truncated valid text) with zero divergences.

Every test was mutation-checked; each of these fails at least one test:

Mutant	Caught by
revert to from_utf8_lossy	the end-to-end escape test
uppercase hex digits	lowercase-hex test + conformance table
drop invalid bytes entirely	3 tests
emit U+FFFD instead of escapes	3 tests
swap nibble order (0x97 would render \x79)	3 tests
drop the valid prefix	conformance table + end-to-end
remove the no-allocation fast path	the borrow test
naive escape-all-non-ASCII decoder	both negative controls + conformance table

The last mutant exists to prove the valid-UTF-8 negative controls are not vacuous: no other mutant makes them fail, because this implementation structurally cannot over-escape.

Not verified locally: the full cargo test --workspace (the openjd-snapshots AWS SDK build exhausted disk on my machine) and the conformance suite. Both run in CI. Nothing in this change is reachable from openjd-snapshots, which does not depend on openjd-sessions.

Was this change documented?

Yes. decode_backslashreplace carries a doc comment explaining the failure mode, the Python parity requirement, and the borrow behavior. Per the spec co-evolution convention, specs/sessions/subprocess.md is updated in the same commit: the stdout-loop excerpt now shows the new call, and a new "Decoding Undecodable Output Bytes" section documents the escaping, the CPython parity, and the fourfold expansion bound. The stale from_utf8_lossy reference in item 4 of the sessions quality report is corrected (the item itself stays open).

No public-api.md change: decode_backslashreplace is pub(crate), matching the neighbouring truncate_line.

Is this a breaking change?

No. No public API changes. The only observable difference is the text of logged subprocess output for processes that emit non-UTF-8 bytes.

Does this change impact security?

No new files, directories, or permissions, and no threat model change. Log content now includes escaped byte values in place of replacement characters, derived from the same subprocess output that was already being logged. If anything this is a small improvement: the escaped form is pure ASCII, so raw undecodable bytes are not passed through to the log sink.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[leongdl]

Cross-reference: the Python-side fix this achieves parity with is openjd-sessions-for-python#343. Both originate from the same customer report.