Skip to content

fix: preserve case-sensitive source URL identities - #164

Open
er-s-an wants to merge 1 commit into
jordan-gibbs:mainfrom
er-s-an:fix/independence-url-case-20261007
Open

er-s-an wants to merge 1 commit into
jordan-gibbs:mainfrom
er-s-an:fix/independence-url-case-20261007

Conversation

@er-s-an

@er-s-an er-s-an commented Oct 7, 2026

Copy link
Copy Markdown

What this changes

Preserve case-sensitive URL paths and query keys/values in the independence audit, while retaining host normalization and case-insensitive tracking-parameter removal.

Why

canonical_url currently lowercases the entire URL before parsing it. Distinct source URLs such as /Report and /report, ?id=AbC and ?id=abc, or ?ID=123 and ?id=123 therefore become the same canonical identity. compute_independence clusters these distinct sources as copies and incorrectly reduces one source's independence score to 0.5, even when the note bodies are unrelated.

The change only moves lowercasing to the host and tracking-parameter comparison; the existing scheme, www, trailing-slash, query sorting and fragment normalization behavior stays intact. Percent-escape hex digits in the path are normalized without decoding reserved separators.

How it was verified

  • Seven added regressions fail against unmodified main, including three vault-level independence score regressions
  • Two additional regressions preserve percent-escape hex-case equivalence and escaped-vs-literal path separators
  • Independent diff review and repeated focused tests: passed
  • python -m pytest tests/test_core/test_verification.py: 62 passed
  • ruff check src/ tests/: passed
  • Full suite on final code: 1726 passed, 2 failed; pristine main: 1717 passed, the same 2 failed. Both failures are test_crawl4ai_running_loop.py tests that resolve example.com before reaching the mocked browser; DNS resolution is unavailable in this environment
  • mypy src/hyperresearch/: 420 existing errors across 59 files, with diagnostics identical to pristine main
  • Test environment: Python 3.12, writable temporary HOME for crawl4ai cache; optional socksio transport installed for this environment
  • git diff --check: passed

AI assistance: OpenAI Codex assisted with reproduction, implementation and test development.


  • No unrelated reformatting, whitespace, or prose rewriting outside the lines this change needs
  • CHANGELOG.md untouched
  • No unrelated output, docstring, or fixture edits
  • No formatter or automated cleanup changes

@er-s-an
er-s-an marked this pull request as ready for review October 14, 2026 12:27
@er-s-an
er-s-an force-pushed the fix/independence-url-case-20261007 branch from 935efe6 to cc50078 Compare October 16, 2026 01:24
@er-s-an

er-s-an commented Oct 16, 2026

Copy link
Copy Markdown
Author

Rebased onto current main so the branch is no longer behind. No code changes beyond the rebase.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant