Skip to content

fix(engine): preserve Unicode semantic identities - #27

Draft
u235yang wants to merge 1 commit into
swarmclawai:mainfrom
u235yang:codex/fix-unicode-semantic-ids
Draft

fix(engine): preserve Unicode semantic identities#27
u235yang wants to merge 1 commit into
swarmclawai:mainfrom
u235yang:codex/fix-unicode-semantic-ids

Conversation

@u235yang

Copy link
Copy Markdown

Summary

  • preserve the existing semantic slugs for pure ASCII names while deriving normalized, stable SHA-256 slugs for names containing non-ASCII characters
  • use the analyzed concept/entity ID consistently across graph nodes, aggregate pages, source links, backlinks, and candidate paths
  • migrate compatible v8 analysis caches to v9 deterministically without another provider/LLM call, while invalid caches still re-analyze during a full compile
  • deduplicate aggregate provenance and normalized duplicate terms without dropping distinct legacy ASCII terms such as C++ and C#
  • archive obsolete generated pages containing guided-session markers byte-for-byte instead of assigning ambiguous notes to one of several split Unicode pages; compile approvals treat archive creation and old-page deletion as one atomic action

Why

The legacy slugger strips every character outside [a-z0-9]. Chinese concepts and entities therefore collapsed to concept:item / entity:item, causing unrelated terms to share graph nodes, wiki pages, backlinks, and index entries.

Compatibility and migration

  • pure ASCII semantic IDs remain unchanged
  • NFKC normalization makes equivalent Unicode forms resolve to the same identity
  • valid v8 caches migrate locally and preserve producedAt; no provider call is required
  • old many-to-one generated pages are removed from the active graph, while any human guided-session notes are retained in a content-addressed archive for manual reconciliation
  • codeOnly compilation remains LLM-free for existing invalid non-code caches; a later full compile repairs them

Validation

  • pnpm check
  • pnpm --filter @swarmvaultai/engine test — 418 passed, 2 skipped
  • pnpm --filter @swarmvaultai/engine build
  • focused Unicode/cache/path/archive/approval regression coverage
  • independent patch review completed with no remaining blockers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants