From 399d0abb2d7d7dfb373d4658046909c8aaf2575e Mon Sep 17 00:00:00 2001 From: LorenzoGentile Date: Mon, 24 Aug 2026 11:18:29 +0200 Subject: [PATCH] fix(compile): repair wikilinks that name a page too briefly to resolve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generation writes a concept's short canonical name while the page it means carries a longer descriptive title, so `[[Argo CD]]` sits broken beside `argo-cd-image-update-ownership-model`. Extraction picks the titles and generation picks the link text, independently and in that order, and nothing reconciled them — the link stayed dead with its target on disk. A repair pass now runs after interlink resolution and repoints a link when its slug prefixes exactly one page. Only the target is rewritten: `[[Argo CD]]` becomes `[[argo-cd-image-update-ownership-model|Argo CD]]`, so rendered output is byte-identical and a bad match can only misdirect a link, never alter prose. It refuses to guess. Two candidate pages means no repair; zero means no repair either, because a link to a concept the wiki lacks is a signal about what extraction missed rather than noise to hide — lint still reports it. Targets awaiting review are skipped, since they resolve on approval. Measured on five compiles of a mixed corpus (two PDFs plus five prose documents): 21.3% of broken wikilinks repaired on a GPT-5-class model, 16.7% on gpt-4o-mini, with no page's prose changed. --- CHANGELOG.md | 6 ++ docs/cli/compile.mdx | 6 ++ src/compiler/index.ts | 6 ++ src/compiler/link-repair.ts | 166 ++++++++++++++++++++++++++++++++++++ test/link-repair.test.ts | 144 +++++++++++++++++++++++++++++++ 5 files changed, 328 insertions(+) create mode 100644 src/compiler/link-repair.ts create mode 100644 test/link-repair.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 993aefc6..da5cc192 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Wikilinks naming an existing page too briefly never resolved** — page generation writes a concept's short canonical name while the page it means carries a longer descriptive title, so `[[Argo CD]]` sat broken next to `argo-cd-image-update-ownership-model`. Extraction chooses page titles and generation chooses link text, independently and in that order, and nothing reconciled the two. Compile now runs a repair pass after interlink resolution that repoints a link when its slug prefixes exactly one page: `[[Argo CD]]` becomes `[[argo-cd-image-update-ownership-model|Argo CD]]`. + + Only the link target is rewritten, never the text around it, so rendered output is byte-identical and the worst a bad match can do is point a link at the wrong page rather than alter prose. A slug prefixing two pages is left alone rather than guessed at, and so is one prefixing none — a link to a concept the wiki genuinely lacks says something about what is missing and is not noise to hide. Targets awaiting review are skipped too, since they resolve on approval. + + Measured across five compiles of a mixed corpus (two PDFs plus five prose documents), this repaired 21.3% of broken wikilinks on a GPT-5-class model and 16.7% on `gpt-4o-mini`, with no page's prose changed. + - **Windows: profile path validation rejected every declared directory** — on win32, `llmwiki template init` failed for every template with `entity directory must be under 'wiki/'`, any profile declaring a workflow `projectionFile` failed to load, and an entity directory declared as `wiki/` was wrongly accepted despite containing every reserved subtree — on win32 it was the only entity directory that loaded at all. Declared directories canonicalize to `/`-joined repo-relative paths, but the containment check built its prefix with the platform separator (`\` on Windows), so no nested path ever matched. The lexical profile-path checks now compare POSIX paths directly; native path confinement is unchanged. Reported and diagnosed by @squ1ddy (#163). - **Windows: broken links in the generated wiki index** — the same separator bug on the output side. Entity-page links in `wiki/index.md` are built from `path.relative`, which emits `\` on win32, so a NESTED entity directory produced the unusable link `research\papers/foo.md`. Link targets are now normalized to POSIX. Single-level directories were unaffected, which is why this went unnoticed (#163). diff --git a/docs/cli/compile.mdx b/docs/cli/compile.mdx index 11380101..5dc068d7 100644 --- a/docs/cli/compile.mdx +++ b/docs/cli/compile.mdx @@ -25,6 +25,12 @@ Run from your project root. If `sources/` is empty or doesn't exist yet, compile | `--concurrency ` | Maximum LLM calls run in parallel during this compile, across both extraction and page generation. Wins over the `LLMWIKI_COMPILE_CONCURRENCY` environment variable; defaults to `5`. Values above `50` are clamped. | | `--verbose` | Print detailed per-step progress: source sizes, concept merge details, page char counts, embedding batch summary, and total compile time. Equivalent to setting `LLMWIKI_VERBOSE=1`. Quiet mode (`--json`) always wins over verbose. | +## Link repair + +After pages are written and interlinks resolved, compile repairs wikilinks whose target does not exist but unambiguously names a page that does. Generation tends to link a concept by its short canonical name (`[[Argo CD]]`) while the page carries a longer descriptive title (`argo-cd-image-update-ownership-model`); the link is repointed to it and the displayed text is left exactly as written. + +A link whose slug prefixes two pages, or none, is not touched. Guessing between two candidates would be wrong half the time, and a link to a concept your wiki genuinely lacks is worth keeping — `llmwiki lint` still reports it as `broken-wikilink`, which is how you find the pages your sources justify but extraction never created. + ## Incremental behaviour Compile is hash-based and incremental. Each source file's content is fingerprinted on ingest; on subsequent compiles, only sources whose content hash has changed are re-processed through the LLM. Sources that haven't changed are skipped entirely - no API calls, no rewrites. This means re-running `llmwiki compile` after editing a single source touches only the pages that source contributed to, even in a large wiki. diff --git a/src/compiler/index.ts b/src/compiler/index.ts index ac0a18b3..6b9a32ab 100644 --- a/src/compiler/index.ts +++ b/src/compiler/index.ts @@ -36,6 +36,7 @@ import { } from "./deps.js"; import { markOrphaned, orphanUnownedFrozenPages } from "./orphan.js"; import { resolveAndApplyLinks } from "./resolver.js"; +import { repairAndApplyLinks } from "./link-repair.js"; import { generateIndex } from "./indexgen.js"; import { generateMOC } from "./obsidian.js"; import { qualifiedPageId } from "../utils/page-id.js"; @@ -523,6 +524,11 @@ async function finalizeWiki( // Compute + apply the resolution rewrites as ONE journalled batch. compile // holds the project lock for its whole pipeline → the lock-free seam. await resolveAndApplyLinks(root, allChangedSlugs, allNewSlugs); + // Resolution adds links for titles it recognises; repair fixes links the + // model already wrote against a page whose slug is longer than the name it + // used. Ordered after resolution so a link resolution has just created is + // already valid and never looks repairable. + await repairAndApplyLinks(root); } // SINGLE durable state write of the whole compile: the buffered draft is diff --git a/src/compiler/link-repair.ts b/src/compiler/link-repair.ts new file mode 100644 index 00000000..d21c3dce --- /dev/null +++ b/src/compiler/link-repair.ts @@ -0,0 +1,166 @@ +/** + * Repair wikilinks that name an existing page too briefly to resolve. + * + * Page generation writes a link using a concept's short canonical name while + * the page it means carries a longer descriptive title — `[[Argo CD]]` against + * a page slugged `argo-cd-image-update-ownership-model`. Extraction picks the + * page titles and generation picks the link text, independently and in that + * order, so nothing reconciles the two and the link stays broken even though + * its target is sitting right there on disk. + * + * This pass runs once every page is written and rewrites only the link TARGET, + * never the text around it: the example above becomes + * `[[argo-cd-image-update-ownership-model|Argo CD]]`. Rendered output is + * therefore identical, and the worst a mistake here can do is point a link at + * the wrong page — it can never alter prose. + * + * Only an unambiguous prefix match is repaired. A slug that prefixes two pages + * is left alone rather than guessed at, and so is one that prefixes none: a link + * to a concept the wiki genuinely lacks is a signal about what is missing, not + * noise to be hidden. + */ + +import path from "path"; +import { parseFrontmatter } from "../utils/markdown.js"; +import { collectAllPages } from "../linter/rules-shared.js"; +import { listLinkResolvablePendingSlugs } from "./candidate-read.js"; +import { applyCompilePageWritesLocked } from "./compile-write.js"; +import type { CompilePageNamespace, CompilePageWrite } from "./compile-write.js"; +import { QUERIES_DIR } from "../utils/constants.js"; +import * as output from "../utils/output.js"; + +/** `[[target]]` and `[[target|alias]]`, capturing everything between brackets. */ +const WIKILINK_PATTERN = /\[\[([^\]]+)\]\]/g; + +/** Separator joining slug words, and therefore the prefix boundary. */ +const SLUG_SEPARATOR = "-"; + +/** + * Shortest link slug considered for repair. Below this a slug prefixes pages it + * has nothing to do with — `[[a]]` would claim `a-b-c` — and the uniqueness + * check cannot tell that apart from a real abbreviation. + */ +const MIN_REPAIRABLE_SLUG_LENGTH = 3; + +/** + * Slugify a wikilink target the same way page filenames are slugified, so a + * link and the page it names compare on equal terms. + */ +function slugifyTarget(target: string): string { + return target + .toLowerCase() + .replace(/['’]/g, "") + .replace(/[^\p{L}\p{N}\s-]/gu, "") + .replace(/\s+/g, SLUG_SEPARATOR) + .replace(/-+/g, SLUG_SEPARATOR) + .replace(/^-|-$/g, ""); +} + +/** + * Resolve a broken link slug to the single page it prefixes, or null when it + * prefixes none or more than one. + */ +function resolveUniquePrefix(targetSlug: string, slugs: string[]): string | null { + if (targetSlug.length < MIN_REPAIRABLE_SLUG_LENGTH) return null; + const prefix = targetSlug + SLUG_SEPARATOR; + let match: string | null = null; + for (const slug of slugs) { + if (!slug.startsWith(prefix)) continue; + if (match) return null; + match = slug; + } + return match; +} + +/** Split `target|alias` into its parts, preserving an alias that contains a pipe. */ +function splitWikilink(inner: string): { target: string; alias: string } { + const [rawTarget, ...aliasParts] = inner.split("|"); + const target = rawTarget.trim(); + const alias = aliasParts.length > 0 ? aliasParts.join("|").trim() : target; + return { target, alias }; +} + +/** Rewrite every repairable link in a body; returns the body and a repair count. */ +function repairBody( + body: string, + resolve: (targetSlug: string) => string | null, +): { body: string; repaired: number } { + let repaired = 0; + const next = body.replace(WIKILINK_PATTERN, (match, inner: string) => { + const { target, alias } = splitWikilink(inner); + const resolved = resolve(slugifyTarget(target)); + if (!resolved) return match; + repaired += 1; + return `[[${resolved}|${alias}]]`; + }); + return { body: next, repaired }; +} + +/** Derive the compile namespace from a page's absolute file path. */ +function namespaceForPage(filePath: string): CompilePageNamespace { + return path.dirname(filePath).endsWith(path.basename(QUERIES_DIR)) ? "queries" : "concepts"; +} + +/** + * COMPUTE the repair rewrites for every page in the project. + * + * Scans all pages rather than only changed ones: a link broken today becomes + * repairable the moment a later compile creates the page it names, and that + * page's arrival never touches the file holding the link. The pass reads files + * and calls no model, so the cost is the same order as `llmwiki lint`. + * + * @param root - Absolute project root the reads and writes are confined under. + * @returns One write per page whose body actually changed. + */ +export async function repairLinks(root: string): Promise { + const pages = await collectAllPages(root); + if (pages.length === 0) return []; + + const slugs = pages.map((page) => path.basename(page.filePath, ".md").toLowerCase()); + const existing = new Set(slugs); + const pending = await listLinkResolvablePendingSlugs(root); + // A pending target resolves on its own when the candidate is approved, so + // repointing it now would silently redirect the link away from the page the + // author is about to publish. + const resolve = (targetSlug: string): string | null => + existing.has(targetSlug) || pending.has(targetSlug) + ? null + : resolveUniquePrefix(targetSlug, slugs); + + const writes: CompilePageWrite[] = []; + let repairedLinks = 0; + for (const page of pages) { + const { body } = parseFrontmatter(page.content); + const result = repairBody(body, resolve); + if (result.repaired === 0) continue; + repairedLinks += result.repaired; + writes.push({ + namespace: namespaceForPage(page.filePath), + slug: path.basename(page.filePath, ".md"), + body: page.content.replace(body, result.body), + }); + } + + if (repairedLinks > 0) { + output.status( + "🔗", + output.dim(`Repaired ${repairedLinks} wikilink(s) in ${writes.length} page(s)`), + ); + } + return writes; +} + +/** + * COMPUTE the repairs and APPLY them in one step — the single seam callers + * should use, so the "{@link repairLinks} returns writes you MUST apply" + * contract cannot be forgotten. + * + * PRECONDITION: the caller MUST already hold the project lock. This routes + * through the LOCK-FREE {@link applyCompilePageWritesLocked}, matching how + * interlink resolution is applied from the same place in the pipeline. + * + * @param root - Absolute project root the writes are confined under. + */ +export async function repairAndApplyLinks(root: string): Promise { + await applyCompilePageWritesLocked(root, await repairLinks(root)); +} diff --git a/test/link-repair.test.ts b/test/link-repair.test.ts new file mode 100644 index 00000000..25ff0ce8 --- /dev/null +++ b/test/link-repair.test.ts @@ -0,0 +1,144 @@ +/** + * @file test/link-repair.test.ts + * @description Repair of wikilinks whose target names an existing page too + * briefly to resolve — `[[Argo CD]]` against `argo-cd-image-update-ownership-model`. + * The pass must rewrite the TARGET only, never the surrounding prose, and must + * refuse to guess: a slug prefixing two pages or none is left exactly as it is. + */ + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtemp, writeFile, mkdir, readFile, rm } from "fs/promises"; +import path from "path"; +import os from "os"; +import { repairLinks } from "../src/compiler/link-repair.js"; +import { applyCompilePageWritesLocked } from "../src/compiler/compile-write.js"; +import { buildFrontmatter } from "../src/utils/markdown.js"; + +describe("repairLinks", () => { + let tmpDir: string; + let conceptsDir: string; + + beforeEach(async () => { + tmpDir = await mkdtemp(path.join(os.tmpdir(), "llmwiki-link-repair-")); + conceptsDir = path.join(tmpDir, "wiki", "concepts"); + await mkdir(conceptsDir, { recursive: true }); + }); + + afterEach(async () => { + await rm(tmpDir, { recursive: true, force: true }); + }); + + async function writePage(slug: string, body: string): Promise { + const fm = buildFrontmatter({ title: slug, summary: "test" }); + await writeFile(path.join(conceptsDir, `${slug}.md`), `${fm}\n\n${body}\n`, "utf-8"); + } + + async function readPage(slug: string): Promise { + return readFile(path.join(conceptsDir, `${slug}.md`), "utf-8"); + } + + async function repairAndApply(): Promise { + await applyCompilePageWritesLocked(tmpDir, await repairLinks(tmpDir)); + } + + it("repoints a link to the single page its slug prefixes", async () => { + await writePage("argo-cd-image-update-ownership-model", "Details."); + await writePage("deployment", "Managed by [[Argo CD]] end to end."); + + await repairAndApply(); + + expect(await readPage("deployment")).toContain( + "[[argo-cd-image-update-ownership-model|Argo CD]]", + ); + }); + + it("keeps the displayed text byte-identical", async () => { + await writePage("alembic-database-migration-conventions", "Details."); + await writePage("db", "We follow [[Alembic]] here."); + + await repairAndApply(); + + const rendered = (await readPage("db")).replace(/\[\[([^\]|]+)\|([^\]]+)\]\]/g, "$2"); + expect(rendered).toContain("We follow Alembic here."); + }); + + it("preserves an existing alias while repointing it", async () => { + await writePage("tool-registry-integration-surface", "Details."); + await writePage("agents", "See [[Tool Registry|the registry]]."); + + await repairAndApply(); + + expect(await readPage("agents")).toContain( + "[[tool-registry-integration-surface|the registry]]", + ); + }); + + it("leaves a slug that prefixes two pages untouched", async () => { + await writePage("docker-compose-setup", "Details."); + await writePage("docker-image-build", "Details."); + await writePage("ops", "We use [[Docker]] daily."); + + await repairAndApply(); + + expect(await readPage("ops")).toContain("[[Docker]]"); + }); + + it("leaves a link with no candidate page untouched", async () => { + await writePage("deployment", "Runs behind [[Caddy]]."); + + await repairAndApply(); + + expect(await readPage("deployment")).toContain("[[Caddy]]"); + }); + + it("does not touch a link that already resolves", async () => { + await writePage("workspace", "Details."); + await writePage("onboarding", "Create a [[Workspace]] first."); + + await repairAndApply(); + + expect(await readPage("onboarding")).toContain("[[Workspace]]"); + }); + + it("refuses a slug too short to prefix meaningfully", async () => { + await writePage("ai-governance-policy", "Details."); + await writePage("intro", "About [[AI]] generally."); + + await repairAndApply(); + + expect(await readPage("intro")).toContain("[[AI]]"); + }); + + it("writes nothing on a second run", async () => { + await writePage("argo-cd-image-update-ownership-model", "Details."); + await writePage("deployment", "Managed by [[Argo CD]]."); + + await repairAndApply(); + const writes = await repairLinks(tmpDir); + + expect(writes).toEqual([]); + }); + + it("repairs every occurrence across pages in one pass", async () => { + await writePage("external-secrets-operator-runtime-wiring", "Details."); + await writePage("eks", "Uses [[External Secrets Operator]] twice: [[External Secrets Operator]]."); + await writePage("k3s", "Also [[External Secrets Operator]]."); + + await repairAndApply(); + + const eks = await readPage("eks"); + const k3s = await readPage("k3s"); + expect(eks.match(/external-secrets-operator-runtime-wiring/g)).toHaveLength(2); + expect(k3s).toContain("[[external-secrets-operator-runtime-wiring|External Secrets Operator]]"); + }); + + it("leaves frontmatter untouched", async () => { + await writePage("argo-cd-image-update-ownership-model", "Details."); + await writePage("deployment", "Managed by [[Argo CD]]."); + const before = (await readPage("deployment")).split("---")[1]; + + await repairAndApply(); + + expect((await readPage("deployment")).split("---")[1]).toBe(before); + }); +});