fix(crawl4ai): force UTF-8 I/O in the crwl subprocess on Windows - #1
fix(crawl4ai): force UTF-8 I/O in the crwl subprocess on Windows#1AQQry wants to merge 1 commit into
Conversation
Without PYTHONUTF8/PYTHONIOENCODING the child Python defaults to the ANSI code page (cp1250 on Polish Windows) and crashes with 'charmap' codec errors on non-ANSI characters (m² \xb2, Polish diacritics). Also decode subprocess output with errors="replace" so a stray byte cannot kill a crawl that otherwise succeeded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1a6789c07
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Python defaults to the ANSI code page (cp1250) for stdout and crashes with | ||
| # 'charmap' codec errors on non-ANSI chars (e.g. m² \xb2, Polish content). | ||
| env["PYTHONUTF8"] = "1" | ||
| env["PYTHONIOENCODING"] = "utf-8" |
There was a problem hiding this comment.
On Windows with a non-UTF-8 system code page, venv_env() now forces every child to emit UTF-8, but _browser_fetch() (fetch.py:63-66), the install smoke test (install.py:54-56), and status() (api.py:166-168) still use text=True without encoding="utf-8", so the parent decodes their output using its legacy locale. If a child emits a character whose UTF-8 bytes are invalid in that code page, subprocess.run() raises UnicodeDecodeError; notably, _browser_fetch() then reports failure even if its result file was written successfully, affecting the repository's shared fetch layer. Add explicit UTF-8 decoding (and the intended error policy) to every subprocess receiving this environment, or restrict the forced encoding to callers that already decode it correctly.
AGENTS.md reference: crawl4ai/AGENTS.md:L3-L4
Useful? React with 👍 / 👎.
1 participant
On Windows with a non-UTF-8 ANSI code page (e.g. cp1250 on a Polish system), the
crwlchild process inherits the ANSI code page and crashes withUnicodeEncodeError: 'charmap' codec can't encode characteras soon as a crawled page contains a character outside that page —m²(\xb2), Polish diacritics, most typographic dashes and quotes. The crawl fails even though the fetch itself succeeded.Fix (2 files, 6 lines added):
crawl4ai/resolve.py— setPYTHONUTF8=1andPYTHONIOENCODING=utf-8in the subprocess environment, so the child Python encodes its stdout/stderr as UTF-8 regardless of the host code page.crawl4ai/runner.py— decode subprocess output witherrors="replace", so a single stray byte cannot kill a crawl that otherwise completed.Impact: affects every Windows user whose system code page is not UTF-8 — the failure is silent-ish (it surfaces as a codec traceback, not as a crawl error), and it hits ordinary pages, not edge cases. On Linux/macOS the change is a no-op.
Verified locally against product pages containing
m²and Polish diacritics: fails before the patch, passes after.🤖 Generated with Claude Code
https://claude.ai/code/session_01BtzLRcn4AvLY7Fds54z5pF