fix: replace undecodable bytes in read_text to prevent surrogate UnicodeEncodeError - #5648
fix: replace undecodable bytes in read_text to prevent surrogate UnicodeEncodeError#5648ikatyal2110 wants to merge 1 commit into
Conversation
…odeEncodeError When aider is run on a system using a non-UTF-8 locale (e.g. cp936 on Chinese Windows), files opened without an errors= handler can produce Python strings containing lone surrogate characters via surrogateescape. These surrogates cause UnicodeEncodeError when the content is later serialised to JSON and sent to a local model such as Ollama. Adding errors="replace" to the open() call in read_text() ensures that undecodable bytes are substituted with the Unicode replacement character (U+FFFD) instead of being stored as surrogates.
|
Live check on PR head a741b68: read_text currently opens with encoding=self.encoding and no errors= (strict). Invalid UTF-8 is UnicodeDecodeError, not a surrogate string. errors="replace" is a policy change: the file is ingested with U+FFFD instead of failing. The test writes b"hello\xb0world" under utf-8. After replace, the surrogate loop cannot fail (replace never emits D800–DFFF) and encode("utf-8") also cannot fail. Pin the actual contract: content contains "\ufffd" (or assert the full "hello\ufffdworld") so a later strict/surrogateescape rewrite would fail the test. I would not merge as-is unless the intended policy is lossy ingest. If yes, say that in the PR body and tighten the test. If the real bug is a non-utf-8 self.encoding path that uses surrogateescape, show that encoding in the test instead of utf-8. Related: headless --message should not look successful after a swallowed SwitchCoder (#5637). Same class of "don't lie about local bytes." |
3 participants
When aider runs on a system using a non-UTF-8 locale (e.g. cp936 on Chinese Windows), reading a file without an errors= parameter can produce Python strings with lone surrogate characters via surrogateescape. These surrogates later cause UnicodeEncodeError when the file content is JSON-serialised and sent to a local model such as Ollama. Adding errors="replace" to the open() call in read_text() substitutes undecodable bytes with U+FFFD instead of creating surrogates. Fixes #3460