Skip to content

fix: strip non-cookie metadata from cookie file to prevent UnicodeEncodeError - #68

Open
cdxiaodong wants to merge 1 commit into
jackwener:mainfrom
cdxiaodong:fix/cookie-metadata-crash
Open

fix: strip non-cookie metadata from cookie file to prevent UnicodeEncodeError#68
cdxiaodong wants to merge 1 commit into
jackwener:mainfrom
cdxiaodong:fix/cookie-metadata-crash

Conversation

@cdxiaodong

Copy link
Copy Markdown

Problem

load_saved_cookies() returns the entire JSON dict from the cookie file, including any metadata fields (e.g. nickname, user_id, tags) that external tools or multi-account managers may have stored alongside real cookies.

When such values contain non-ASCII characters (e.g. a Chinese nickname like "N8N中国"), httpx crashes with UnicodeEncodeError when building the Cookie header:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 470-471: ordinal not in range(128)
  File ".../httpx/_models.py", line 82, in _normalize_header_value
    return value.encode(encoding or "ascii")

Repro

// ~/.xiaohongshu-cli/cookies.json
{
  "a1": "abc123",
  "web_session": "sess456",
  "nickname": "N8N中国",
  "user_id": "670b9380000000001d023ef4"
}
$ xhs hot
UnicodeEncodeError: 'ascii' codec can't encode characters...

This is easy to hit when managing multiple accounts — tools naturally annotate cookie files with metadata (which account is this, nickname, user_id, etc.), and the cookie file format is not documented as cookie-keys-only.

Fix

Add _sanitize_cookie_dict() which filters entries on load:

  1. Strip known metadata keys (_META_COOKIE_KEYS: nickname, user_id, red_id, description, tags, added_at, slug, browser_source)
  2. Drop entries whose key or value is not ASCII-safe — HTTP cookie headers must be ASCII, so any non-ASCII entry is definitionally not a valid cookie
  3. Preserve saved_at for TTL-based refresh in get_cookies()

Non-cookie entries are dropped silently with a logger.debug() message.

Design notes

  • I chose a blacklist + ASCII validation approach rather than a whitelist of known XHS cookie names. XHS may add new cookies at any time, and a whitelist would silently break the client for users who have those new cookies.
  • The blacklist covers common metadata fields that multi-account tools add. The ASCII check is the real safety net — it catches any metadata field regardless of whether it's in the blacklist.
  • This is purely additive — save_cookies() and the cookie file format are unchanged. Only loading is filtered.

Tests

Added 3 test cases to TestLoadSavedCookies:

Test What it covers
test_strips_non_cookie_metadata Known metadata keys (nickname, user_id, tags) are stripped
test_strips_non_ascii_values Non-ASCII values (Chinese nickname) are dropped
test_cookies_to_string_ascii_safe End-to-end: cookies_to_string() on loaded cookies is ASCII-safe (no crash)

All 20 tests pass (17 existing + 3 new). Ruff check passes.

…odeError

load_saved_cookies() returned the entire JSON dict from the cookie file,
including any metadata fields (nickname, user_id, tags, etc.) that
external tools or multi-account managers may have stored alongside real
cookies. When such values contain non-ASCII characters (e.g. a Chinese
nickname), httpx crashes with UnicodeEncodeError when building the
Cookie header.

Fix: add _sanitize_cookie_dict() which filters entries on load:
- Strip known metadata keys (_META_COOKIE_KEYS)
- Drop entries whose key or value is not ASCII-safe
- Preserve saved_at for TTL-based refresh in get_cookies()

Repro before fix:
  # cookies.json contains: {"a1":"abc", "nickname":"N8N中文"}
  xhs hot
  # UnicodeEncodeError: 'ascii' codec can't encode characters

After fix: non-cookie entries are silently dropped with a debug log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants