From cc8e5bf418a2706c1ddf5feb9f85bb62f0fd3e93 Mon Sep 17 00:00:00 2001 From: LorenzoGentile Date: Thu, 20 Aug 2026 11:48:37 +0200 Subject: [PATCH] feat(compile): make the ## Sources section optional MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Page generation always instructs the model to append a `## Sources` section. For a project that renders source attribution itself this is a third copy of the same list: the `sources:` frontmatter is built by the compiler from the files it actually read, and the inline `^[file.md:1-5]` markers already carry paragraph-level provenance. Stripping it downstream is unreliable. The heading is not a stable string — under `--lang` the model localizes it with the rest of the page, so any consumer matching on `## Sources` silently stops matching as soon as a project sets an output language. Suppressing the request is the only way to reliably not have the section. `LLMWIKI_SOURCES_SECTION=off` (or `--no-sources-section` on compile) drops the instruction line. Unset leaves the prompt byte-for-byte unchanged, and nothing about provenance moves: no linter, exporter, or citation rule parses the section, and the frontmatter and citation markers are untouched. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 8 ++ docs/cli/compile.mdx | 1 + docs/configuration/environment-variables.mdx | 1 + src/cli.ts | 14 +++- src/compiler/prompts.ts | 13 ++- src/utils/sources-section.ts | 50 +++++++++++ test/sources-section.test.ts | 87 ++++++++++++++++++++ 7 files changed, 172 insertions(+), 2 deletions(-) create mode 100644 src/utils/sources-section.ts create mode 100644 test/sources-section.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 993aefc6..12923ab8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 An index written before llmwiki recorded the endpoint carries only its model name. It is preserved while you run without `LLMWIKI_EMBEDDING_PROVIDER` or an endpoint override, so upgrading does not re-embed an existing project; under either override the model name cannot establish where the vectors came from, so the next compile rebuilds the index once and records the full configuration from then on. +- **Optional `## Sources` section** — `LLMWIKI_SOURCES_SECTION=off`, or `--no-sources-section` on `llmwiki compile`, stops page generation from asking the model for a trailing `## Sources` section. Unset preserves the prompt byte-for-byte. + + This is for projects that render source attribution themselves. A page already carries its provenance twice — the `sources:` frontmatter, which the compiler builds from the source files it actually read rather than from anything the model writes, and the inline `^[file.md:1-5]` citation markers — so a consumer that displays either one shows the same list a third time in the prose. Nothing downstream reads the section: it is a prompt instruction only, and no linter, exporter, or citation rule parses it. + + Suppressing the request is the only reliable way to not have the section, because it is not a stable string to strip. Under `--lang` the model localizes that heading along with the rest of the page, so a downstream matcher keyed on `## Sources` silently stops matching the moment a project sets an output language. + + `PROMPT_VERSION` is unchanged. The constant was introduced with the optional language directive already in the page prompt, so `v1` already denotes a contract with user-selected prompt modifiers in it, and the default path here is byte-identical. Say the word if you would rather it move. + ### Fixed - **Windows: profile path validation rejected every declared directory** — on win32, `llmwiki template init` failed for every template with `entity directory must be under 'wiki/'`, any profile declaring a workflow `projectionFile` failed to load, and an entity directory declared as `wiki/` was wrongly accepted despite containing every reserved subtree — on win32 it was the only entity directory that loaded at all. Declared directories canonicalize to `/`-joined repo-relative paths, but the containment check built its prefix with the platform separator (`\` on Windows), so no nested path ever matched. The lexical profile-path checks now compare POSIX paths directly; native path confinement is unchanged. Reported and diagnosed by @squ1ddy (#163). diff --git a/docs/cli/compile.mdx b/docs/cli/compile.mdx index 11380101..5e6b2340 100644 --- a/docs/cli/compile.mdx +++ b/docs/cli/compile.mdx @@ -22,6 +22,7 @@ Run from your project root. If `sources/` is empty or doesn't exist yet, compile |------|-------------| | `--review` | Write generated pages to `.llmwiki/candidates/` instead of `wiki/`. Pages are held for human review and don't appear in the live wiki until you approve them with `llmwiki review approve `. | | `--lang ` | Override the output language for this compile run (e.g. `zh-CN`, `Chinese`, `ja`, `Japanese`). Wins over the `LLMWIKI_OUTPUT_LANG` environment variable. Unset preserves the model's default behaviour. | +| `--no-sources-section` | Stop asking the model for a trailing `## Sources` section in generated pages. Provenance is unaffected: the `sources:` frontmatter and the inline `^[file.md:1-5]` citation markers are produced by the compiler and stay exactly as they are. Equivalent to setting `LLMWIKI_SOURCES_SECTION=off`. | | `--concurrency ` | Maximum LLM calls run in parallel during this compile, across both extraction and page generation. Wins over the `LLMWIKI_COMPILE_CONCURRENCY` environment variable; defaults to `5`. Values above `50` are clamped. | | `--verbose` | Print detailed per-step progress: source sizes, concept merge details, page char counts, embedding batch summary, and total compile time. Equivalent to setting `LLMWIKI_VERBOSE=1`. Quiet mode (`--json`) always wins over verbose. | diff --git a/docs/configuration/environment-variables.mdx b/docs/configuration/environment-variables.mdx index e5520e7d..48755904 100644 --- a/docs/configuration/environment-variables.mdx +++ b/docs/configuration/environment-variables.mdx @@ -154,6 +154,7 @@ new hosts. | Variable | Default | Description | |---|---|---| | `LLMWIKI_OUTPUT_LANG` | *(model default, typically English)* | Language for generated wiki content. Applies to every prompt in the compile and query pipelines. Examples: `zh-CN`, `Chinese`, `ja`, `Japanese`. The `--lang` CLI flag on `llmwiki compile` and `llmwiki query` overrides this for a single invocation | +| `LLMWIKI_SOURCES_SECTION` | *(enabled)* | Set to `off`, `false`, `0`, or `no` to stop asking the model for a trailing `## Sources` section in generated pages. Useful when your own renderer already displays the `sources:` frontmatter, which would otherwise appear twice. Does not change provenance: the frontmatter and the inline `^[file.md:1-5]` citation markers come from the compiler, not from this instruction. The `--no-sources-section` flag on `llmwiki compile` sets it for a single invocation | | `LLMWIKI_PROMPT_BUDGET_CHARS` | `200000` | Character ceiling for combined per-concept source content sent to the LLM. Raise for larger-context models; lower for small-context local models. A stderr warning prints when the cap fires | --- diff --git a/src/cli.ts b/src/cli.ts index 8b49386a..98ec95a5 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -27,6 +27,7 @@ import quickstartCommand from "./commands/quickstart.js"; import contextCommand, { type ContextCommandOptions } from "./commands/context.js"; import { startMCPServer } from "./mcp/server.js"; import { applyLanguageOption } from "./utils/output-language.js"; +import { applySourcesSectionOption } from "./utils/sources-section.js"; import { ensureProviderAvailable } from "./utils/provider-guard.js"; import { setVerbose } from "./utils/output.js"; import { parseConcurrencyFlag } from "./compiler/concurrency.js"; @@ -116,15 +117,26 @@ program "--lang ", "Target language for generated wiki content (e.g. \"Chinese\", \"ja\", \"zh-CN\"). Equivalent to setting LLMWIKI_OUTPUT_LANG.", ) + .option( + "--no-sources-section", + "Omit the trailing ## Sources section from generated pages. Source attribution stays in the sources: frontmatter and the inline ^[...] citation markers. Equivalent to setting LLMWIKI_SOURCES_SECTION=off.", + ) .option( "--concurrency ", "Max concurrent LLM calls during compile (or set LLMWIKI_COMPILE_CONCURRENCY; default 5)", ) .option("--verbose", "Print detailed progress (or set LLMWIKI_VERBOSE=1)") - .action(async (options: { review?: boolean; lang?: string; concurrency?: string; verbose?: boolean }) => { + .action(async (options: { + review?: boolean; + lang?: string; + sourcesSection?: boolean; + concurrency?: string; + verbose?: boolean; + }) => { try { setVerbose(verboseEnabled(options.verbose)); applyLanguageOption(options.lang); + applySourcesSectionOption(options.sourcesSection); requireProvider(); await compileCommand({ review: options.review, concurrency: parseConcurrencyFlag(options.concurrency) }); } catch (err) { diff --git a/src/compiler/prompts.ts b/src/compiler/prompts.ts index e7e1b197..ae35bab0 100644 --- a/src/compiler/prompts.ts +++ b/src/compiler/prompts.ts @@ -12,6 +12,7 @@ import type { } from "../utils/types.js"; import type { PageKindRule, SeedPage } from "../schema/index.js"; import { languageDirective } from "../utils/output-language.js"; +import { sourcesSectionEnabled } from "../utils/sources-section.js"; /** * Build a list of optional prompt lines, omitting empty entries so the @@ -24,6 +25,16 @@ function withLangLine(...lines: string[]): string[] { return lang ? [...lines, lang] : lines; } +/** + * The page-generation instruction for the trailing `## Sources` section, or + * nothing when the project opted out. Spreadable so the default prompt keeps + * its exact wording instead of gaining a blank line where the request was. + */ +function sourcesSectionLines(): string[] { + if (!sourcesSectionEnabled()) return []; + return ["Include a ## Sources section at the end listing the source document."]; +} + /** * Named version of the extraction + page-generation prompt contract. * @@ -174,7 +185,7 @@ export function buildPagePrompt( ...withLangLine( `You are a wiki author. Write a clear, well-structured markdown page about "${concept}".`, "Draw facts only from the provided source material.", - "Include a ## Sources section at the end listing the source document.", + ...sourcesSectionLines(), "Suggest [[wikilinks]] to related concepts where appropriate.", "Write in a neutral, informative tone. Be concise but thorough.", ), diff --git a/src/utils/sources-section.ts b/src/utils/sources-section.ts new file mode 100644 index 00000000..7f4bf897 --- /dev/null +++ b/src/utils/sources-section.ts @@ -0,0 +1,50 @@ +/** + * Controls whether page generation asks the model for a trailing + * `## Sources` section. + * + * A compiled page already carries its provenance twice: the `sources:` + * frontmatter list, which the compiler builds from the source files it + * actually read rather than from anything the model writes, and the inline + * `^[file.md:1-5]` citation markers. A project that renders either of those + * itself ends up showing a third, redundant copy in the prose. + * + * Stripping the section downstream is awkward because it is not a stable + * string: under `--lang` the model localizes that heading along with the rest + * of the page, so a consumer has no fixed text to match on. Suppressing the + * instruction is the only reliable way to not have it. + * + * Opting out is a rendering choice, not a provenance one — the frontmatter and + * the citation markers are untouched. Unset preserves the historical prompt + * byte-for-byte. + */ + +const SOURCES_SECTION_ENV_VAR = "LLMWIKI_SOURCES_SECTION"; + +/** + * Values that switch the section off, compared case-insensitively after + * trimming. Users reach for different spellings of "no" and silently ignoring + * three of the four would be worse than accepting all of them. + */ +const DISABLED_VALUES: ReadonlySet = new Set(["0", "false", "off", "no"]); + +/** True when generated pages should still include a `## Sources` section. */ +export function sourcesSectionEnabled(): boolean { + const raw = process.env[SOURCES_SECTION_ENV_VAR]; + if (raw === undefined) return true; + return !DISABLED_VALUES.has(raw.trim().toLowerCase()); +} + +/** + * Apply the CLI `--no-sources-section` flag into the shared env slot so the + * prompt builder picks it up downstream. + * + * Commander defaults a `--no-x` option to `true`, so only an explicit `false` + * carries the user's intent; leaving the variable alone otherwise keeps it + * authoritative for setups that configure the project rather than a single + * invocation. Mirrors `applyLanguageOption` in output-language.ts. + */ +export function applySourcesSectionOption(enabled: boolean | undefined): void { + if (enabled === false) { + process.env[SOURCES_SECTION_ENV_VAR] = "off"; + } +} diff --git a/test/sources-section.test.ts b/test/sources-section.test.ts new file mode 100644 index 00000000..20f8410d --- /dev/null +++ b/test/sources-section.test.ts @@ -0,0 +1,87 @@ +/** + * Unit tests for the `## Sources` section toggle and its effect on the page + * prompt. + * + * Default behaviour (no env, no flag) must keep the page prompt byte-identical + * to the previous implementation. Opting out must remove only the section + * request — the inline citation contract and the `sources:` frontmatter that + * carry provenance are not part of this switch. + */ + +import { describe, it, expect, afterEach } from "vitest"; +import { + applySourcesSectionOption, + sourcesSectionEnabled, +} from "../src/utils/sources-section.js"; +import { buildPagePrompt } from "../src/compiler/prompts.js"; + +const ENV_KEY = "LLMWIKI_SOURCES_SECTION"; +const SECTION_REQUEST = "Include a ## Sources section"; + +afterEach(() => { + delete process.env[ENV_KEY]; +}); + +describe("sourcesSectionEnabled", () => { + it("defaults to enabled when the env var is unset", () => { + expect(sourcesSectionEnabled()).toBe(true); + }); + + it.each(["0", "false", "off", "no"])("treats %s as disabled", value => { + process.env[ENV_KEY] = value; + expect(sourcesSectionEnabled()).toBe(false); + }); + + it("ignores surrounding whitespace and case", () => { + process.env[ENV_KEY] = " OFF "; + expect(sourcesSectionEnabled()).toBe(false); + }); + + it("stays enabled for any other value", () => { + process.env[ENV_KEY] = "on"; + expect(sourcesSectionEnabled()).toBe(true); + }); +}); + +describe("applySourcesSectionOption", () => { + it("leaves the env var untouched when the flag was not passed", () => { + applySourcesSectionOption(undefined); + expect(process.env[ENV_KEY]).toBeUndefined(); + }); + + it("leaves an existing opt-out in place on commander's default true", () => { + process.env[ENV_KEY] = "off"; + applySourcesSectionOption(true); + expect(sourcesSectionEnabled()).toBe(false); + }); + + it("disables the section when --no-sources-section was passed", () => { + applySourcesSectionOption(false); + expect(sourcesSectionEnabled()).toBe(false); + }); +}); + +describe("buildPagePrompt honours the toggle", () => { + it("requests the section by default", () => { + expect(buildPagePrompt("Concept", "src", "", "")).toContain(SECTION_REQUEST); + }); + + it("omits the request when disabled", () => { + process.env[ENV_KEY] = "off"; + expect(buildPagePrompt("Concept", "src", "", "")).not.toContain(SECTION_REQUEST); + }); + + it("keeps the inline citation contract when disabled", () => { + process.env[ENV_KEY] = "off"; + const out = buildPagePrompt("Concept", "src", "", ""); + expect(out).toContain("^[filename.md:START-END]"); + expect(out).toContain("Draw facts only from the provided source material."); + }); + + it("drops the line rather than blanking it", () => { + const enabled = buildPagePrompt("Concept", "src", "", ""); + process.env[ENV_KEY] = "off"; + const disabled = buildPagePrompt("Concept", "src", "", ""); + expect(disabled.split("\n").length).toBe(enabled.split("\n").length - 1); + }); +});