fix(kb): add traceable web ingest and owners-compatible metadata

[sqhyz55]

Fixes #162 by adding end-to-end trace_id logging, stricter start_url validation, fail-fast crawl errors, and optional JS-rendered crawling (Playwright) with clearer user guidance.

Also adds compatibility for collections requiring an owners field in LanceDB metadata (aligned with #166).

[rogercloud]

Review — PR #179

MAJOR

Dead code: _fetch_html + _fetch_html_static — crawler.py:695-721

The _crawl_page method calls either _fetch_html_rendered(url) (line 575) or _fetch_with_fallback(sessions, url) (line 582) directly. The _fetch_html method and its _fetch_html_static helper are defined but never wired in — zero callers across the entire codebase. ~27 lines that should be removed or wired up.

MINOR

No API-level validation for render_wait_until — kb.py:2314

render_wait_until is an Optional[str] Form field. An invalid value like "invalid" passes through to WebCrawlConfig(render_wait_until=...), where Pydantic's Literal check raises a ValidationError → HTTP 500. Should validate at the API boundary and return a 422 with a clear message.

Unused HTTP sessions when render_js=True — crawler.py:273-301

When render_js=True, _open_sessions at line 297 still creates httpx/curl_cffi sessions, but _crawl_page never uses them in the Playwright path — it calls _fetch_html_rendered directly. The sessions are properly cleaned up by AsyncExitStack, so this is only wasteful, not a leak.

Other

• Playwright lifecycle (_start_playwright / _stop_playwright) with try/finally — correct ✓
• start_url validation with urlparse — handles whitespace, missing scheme/netloc ✓
• asyncio.gather(return_exceptions=True) correctly catches Playwright RuntimeError ✓
• page.goto() returning None handled gracefully with status_code=None ✓
• bool(render_js) wrapper — redundant (FastAPI already coerces), but harmless ✓

Tests: 10/10 passed

[rogercloud]

Additional non-duplicate review comments from static review.

[rogercloud]

Re-reviewed the latest update. The previous inline issues for rendered HTTP status handling, the creation-dialog JS toggle, i18n, and the stripped URL test path look addressed.

Remaining items are non-blocking cleanup / polish:

• src/xagent/core/tools/core/RAG_tools/web_crawler/crawler.py: _fetch_html() / _fetch_html_static() still appear to be dead code; _crawl_page() continues to call _fetch_html_rendered() or _fetch_with_fallback() directly.
• src/xagent/core/tools/core/RAG_tools/web_crawler/crawler.py: render_js=True still opens the normal HTTP session stack even though the Playwright path does not use it. This is wasteful but not a leak.
• src/xagent/web/api/kb.py: invalid render_wait_until values still rely on WebCrawlConfig validation rather than explicit API-boundary validation, so hand-written API calls may get a less clear error than a targeted 422.

I do not consider these production blockers.