fix(shell): keep valid UTF-8 prefix on fallback - #34
Open
asto18089 wants to merge 1 commit into
Open
Conversation
The legacy-fallback arm of ShellStreamDecoder handed the whole pending_utf8 buffer, including its certified UTF-8 prefix, to the system code-page decoder. One stray byte in an otherwise valid chunk therefore re-mojibaked all preceding output on Windows, worst on the synchronous decode path where a fresh decoder sees the complete output. Split on Utf8Error::valid_up_to first: emit the valid prefix as-is and pass only the invalid remainder to the legacy decoder (or from_utf8_lossy when no legacy encoding is configured). This restores the prefix-preserving behavior reviewed in Hmbown#24 (candidate 6544900) that was dropped when the fix was re-landed in Hmbown#29 from upstream follow-up 0a85b13, and reinstates the regression tests lost with it. Note for the next parent sync: the restored test carries the forkguard_ prefix, so the register's forkguard test inventory count moves from 56 to 57 when this is synced. Signed-off-by: asto <asto18089@126.com>
|
Thanks @asto18089 for taking the time to contribute. This repository is observing a maintainer-managed PR intake gate in dry-run mode, so this pull request is staying open. This note helps maintainers prepare the allowlist before any enforcement is considered. Please read |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
During the r11 review of the Windows shell decoder, one MAJOR finding was verified fixed on candidate
654490026(the prefix-preserving split reviewed in #24): when a chunk contains valid UTF-8 followed by genuinely invalid bytes, only the invalid remainder may go to the system code-page decoder. That fix was lost when the work was re-landed in #29 from upstream follow-up0a85b13ba, which predates it. The publishedpinvou-v0.9.5-r11head therefore still re-mojibakes the entire valid prefix of a chunk once one stray byte appears — worst on the synchronous path, where a fresh decoder sees the complete output of a foreground command (decode_shell_byteswithlast = true), so a single stray byte corrupts all of it on Windows.Changes
crates/tui/src/tools/shell/output.rs: the finalErr(error)arm now splits onUtf8Error::valid_up_to(), emits the certified UTF-8 prefix unchanged, and passes onlypending_utf8[valid_up_to..]to the legacy decoder (orfrom_utf8_lossywhen no legacy encoding is configured). The other two match arms and the flattened control flow from132f8567care unchanged.forkguard_shell_valid_utf8_prefix_survives_legacy_fallback("中文" +0x92through WINDOWS_1252 must yield "中文’") and the deterministic CP1252 finalization assertion inrunning_full_decoder_keeps_incomplete_utf8_private_until_final.Verification
cargo test -p codewhale-tui --lib output::tests: 15 passed (was 14; the new prefix test included). Without the fix, the new test fails by reproducing the mojibake.cargo test -p codewhale-tui --lib shell::tests::: 96 passed.cargo fmt --check -p codewhale-tui: clean.cargo clippy -p codewhale-tui --all-features --lockedwith the CI lint set: no diagnostics inoutput.rs; remaining local diagnostics are pre-existing on a newer toolchain (28 before the change, 27 after) and do not reference the touched file.Notes for the next parent sync (r12)
forkguard_prefix, so the parent register's forkguard test inventory count moves from 56 to 57 and the r12 drift/topology figures must be recomputed.