Decode output incrementally, with a configurable decoder - #52
Merged
Conversation
Command output was decoded per read with a strict `bytes.decode()`, raising `UnicodeDecodeError` in two cases: - A valid multibyte character split across two non-blocking reads (the read loop grabs whatever is in the pipe, with no alignment to char boundaries). - Genuinely invalid UTF-8 bytes in the output (issue #49). Decode with an incremental decoder instead: it holds a partial multibyte char until the next read completes it, and (by default) replaces invalid bytes with the Unicode replacement character. A fresh decoder is created per stream per run, since incremental decoders are stateful. The decoder is configurable via a `decoder_factory` (a zero-arg callable returning a fresh decoder), settable on `Runner(...)` and overridable per `run(...)` call. Defaults to UTF-8 with `errors="replace"`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apakottur
force-pushed
the
fix/utf8-incremental-decode
branch
from
July 10, 2026 13:12
16a3c00 to
7eb9b8d
Compare
Apakottur
changed the title
A single incremental decoder cannot decode stdout and stderr in the same run: it buffers partial multibyte bytes, so interleaved decode() calls on the two streams corrupt each other, and state bleeds across runs. Keep a factory so a fresh decoder is created per stream per run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
collections.abc.Buffer only exists on Python 3.12+, so mypy failed on the 3.10 type-check job. _typeshed.ReadableBuffer is the standard typeshed buffer type across all versions and is only imported under TYPE_CHECKING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Drop `typing.override` (added in 3.12; ImportError at runtime on 3.10/3.11). - Use `_typeshed.ReadableBuffer` instead of `collections.abc.Buffer` (3.12+). - Restore noqa for the stdlib-signature `decode` override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Fix an issue with output decoding which resulted in
UnicodeDecodeErrorin two different cases:€(b"\xe2\x82\xac") can arrive asb"\xe2\x82"on one read andb"\xac"on the next; each half fails to decode. Timing-dependent, so it surfaced as flaky crashes.How?