Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

An index written before llmwiki recorded the endpoint carries only its model name. It is preserved while you run without `LLMWIKI_EMBEDDING_PROVIDER` or an endpoint override, so upgrading does not re-embed an existing project; under either override the model name cannot establish where the vectors came from, so the next compile rebuilds the index once and records the full configuration from then on.

- **Optional `## Sources` section** — `LLMWIKI_SOURCES_SECTION=off`, or `--no-sources-section` on `llmwiki compile`, stops page generation from asking the model for a trailing `## Sources` section. Unset preserves the prompt byte-for-byte.

This is for projects that render source attribution themselves. A page already carries its provenance twice — the `sources:` frontmatter, which the compiler builds from the source files it actually read rather than from anything the model writes, and the inline `^[file.md:1-5]` citation markers — so a consumer that displays either one shows the same list a third time in the prose. Nothing downstream reads the section: it is a prompt instruction only, and no linter, exporter, or citation rule parses it.

Suppressing the request is the only reliable way to not have the section, because it is not a stable string to strip. Under `--lang` the model localizes that heading along with the rest of the page, so a downstream matcher keyed on `## Sources` silently stops matching the moment a project sets an output language.

`PROMPT_VERSION` is unchanged. The constant was introduced with the optional language directive already in the page prompt, so `v1` already denotes a contract with user-selected prompt modifiers in it, and the default path here is byte-identical. Say the word if you would rather it move.

### Fixed

- **Windows: profile path validation rejected every declared directory** — on win32, `llmwiki template init` failed for every template with `entity directory must be under 'wiki/'`, any profile declaring a workflow `projectionFile` failed to load, and an entity directory declared as `wiki/` was wrongly accepted despite containing every reserved subtree — on win32 it was the only entity directory that loaded at all. Declared directories canonicalize to `/`-joined repo-relative paths, but the containment check built its prefix with the platform separator (`\` on Windows), so no nested path ever matched. The lexical profile-path checks now compare POSIX paths directly; native path confinement is unchanged. Reported and diagnosed by @squ1ddy (#163).
Expand Down
1 change: 1 addition & 0 deletions docs/cli/compile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Run from your project root. If `sources/` is empty or doesn't exist yet, compile
|------|-------------|
| `--review` | Write generated pages to `.llmwiki/candidates/` instead of `wiki/`. Pages are held for human review and don't appear in the live wiki until you approve them with `llmwiki review approve <id>`. |
| `--lang <code>` | Override the output language for this compile run (e.g. `zh-CN`, `Chinese`, `ja`, `Japanese`). Wins over the `LLMWIKI_OUTPUT_LANG` environment variable. Unset preserves the model's default behaviour. |
| `--no-sources-section` | Stop asking the model for a trailing `## Sources` section in generated pages. Provenance is unaffected: the `sources:` frontmatter and the inline `^[file.md:1-5]` citation markers are produced by the compiler and stay exactly as they are. Equivalent to setting `LLMWIKI_SOURCES_SECTION=off`. |
| `--concurrency <n>` | Maximum LLM calls run in parallel during this compile, across both extraction and page generation. Wins over the `LLMWIKI_COMPILE_CONCURRENCY` environment variable; defaults to `5`. Values above `50` are clamped. |
| `--verbose` | Print detailed per-step progress: source sizes, concept merge details, page char counts, embedding batch summary, and total compile time. Equivalent to setting `LLMWIKI_VERBOSE=1`. Quiet mode (`--json`) always wins over verbose. |

Expand Down
1 change: 1 addition & 0 deletions docs/configuration/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ new hosts.
| Variable | Default | Description |
|---|---|---|
| `LLMWIKI_OUTPUT_LANG` | *(model default, typically English)* | Language for generated wiki content. Applies to every prompt in the compile and query pipelines. Examples: `zh-CN`, `Chinese`, `ja`, `Japanese`. The `--lang` CLI flag on `llmwiki compile` and `llmwiki query` overrides this for a single invocation |
| `LLMWIKI_SOURCES_SECTION` | *(enabled)* | Set to `off`, `false`, `0`, or `no` to stop asking the model for a trailing `## Sources` section in generated pages. Useful when your own renderer already displays the `sources:` frontmatter, which would otherwise appear twice. Does not change provenance: the frontmatter and the inline `^[file.md:1-5]` citation markers come from the compiler, not from this instruction. The `--no-sources-section` flag on `llmwiki compile` sets it for a single invocation |
| `LLMWIKI_PROMPT_BUDGET_CHARS` | `200000` | Character ceiling for combined per-concept source content sent to the LLM. Raise for larger-context models; lower for small-context local models. A stderr warning prints when the cap fires |

---
Expand Down
14 changes: 13 additions & 1 deletion src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import quickstartCommand from "./commands/quickstart.js";
import contextCommand, { type ContextCommandOptions } from "./commands/context.js";
import { startMCPServer } from "./mcp/server.js";
import { applyLanguageOption } from "./utils/output-language.js";
import { applySourcesSectionOption } from "./utils/sources-section.js";
import { ensureProviderAvailable } from "./utils/provider-guard.js";
import { setVerbose } from "./utils/output.js";
import { parseConcurrencyFlag } from "./compiler/concurrency.js";
Expand Down Expand Up @@ -116,15 +117,26 @@ program
"--lang <code>",
"Target language for generated wiki content (e.g. \"Chinese\", \"ja\", \"zh-CN\"). Equivalent to setting LLMWIKI_OUTPUT_LANG.",
)
.option(
"--no-sources-section",
"Omit the trailing ## Sources section from generated pages. Source attribution stays in the sources: frontmatter and the inline ^[...] citation markers. Equivalent to setting LLMWIKI_SOURCES_SECTION=off.",
)
.option(
"--concurrency <n>",
"Max concurrent LLM calls during compile (or set LLMWIKI_COMPILE_CONCURRENCY; default 5)",
)
.option("--verbose", "Print detailed progress (or set LLMWIKI_VERBOSE=1)")
.action(async (options: { review?: boolean; lang?: string; concurrency?: string; verbose?: boolean }) => {
.action(async (options: {
review?: boolean;
lang?: string;
sourcesSection?: boolean;
concurrency?: string;
verbose?: boolean;
}) => {
try {
setVerbose(verboseEnabled(options.verbose));
applyLanguageOption(options.lang);
applySourcesSectionOption(options.sourcesSection);
requireProvider();
await compileCommand({ review: options.review, concurrency: parseConcurrencyFlag(options.concurrency) });
} catch (err) {
Expand Down
13 changes: 12 additions & 1 deletion src/compiler/prompts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import type {
} from "../utils/types.js";
import type { PageKindRule, SeedPage } from "../schema/index.js";
import { languageDirective } from "../utils/output-language.js";
import { sourcesSectionEnabled } from "../utils/sources-section.js";

/**
* Build a list of optional prompt lines, omitting empty entries so the
Expand All @@ -24,6 +25,16 @@ function withLangLine(...lines: string[]): string[] {
return lang ? [...lines, lang] : lines;
}

/**
* The page-generation instruction for the trailing `## Sources` section, or
* nothing when the project opted out. Spreadable so the default prompt keeps
* its exact wording instead of gaining a blank line where the request was.
*/
function sourcesSectionLines(): string[] {
if (!sourcesSectionEnabled()) return [];
return ["Include a ## Sources section at the end listing the source document."];
}

/**
* Named version of the extraction + page-generation prompt contract.
*
Expand Down Expand Up @@ -174,7 +185,7 @@ export function buildPagePrompt(
...withLangLine(
`You are a wiki author. Write a clear, well-structured markdown page about "${concept}".`,
"Draw facts only from the provided source material.",
"Include a ## Sources section at the end listing the source document.",
...sourcesSectionLines(),
"Suggest [[wikilinks]] to related concepts where appropriate.",
"Write in a neutral, informative tone. Be concise but thorough.",
),
Expand Down
50 changes: 50 additions & 0 deletions src/utils/sources-section.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
/**
* Controls whether page generation asks the model for a trailing
* `## Sources` section.
*
* A compiled page already carries its provenance twice: the `sources:`
* frontmatter list, which the compiler builds from the source files it
* actually read rather than from anything the model writes, and the inline
* `^[file.md:1-5]` citation markers. A project that renders either of those
* itself ends up showing a third, redundant copy in the prose.
*
* Stripping the section downstream is awkward because it is not a stable
* string: under `--lang` the model localizes that heading along with the rest
* of the page, so a consumer has no fixed text to match on. Suppressing the
* instruction is the only reliable way to not have it.
*
* Opting out is a rendering choice, not a provenance one — the frontmatter and
* the citation markers are untouched. Unset preserves the historical prompt
* byte-for-byte.
*/

const SOURCES_SECTION_ENV_VAR = "LLMWIKI_SOURCES_SECTION";

/**
* Values that switch the section off, compared case-insensitively after
* trimming. Users reach for different spellings of "no" and silently ignoring
* three of the four would be worse than accepting all of them.
*/
const DISABLED_VALUES: ReadonlySet<string> = new Set(["0", "false", "off", "no"]);

/** True when generated pages should still include a `## Sources` section. */
export function sourcesSectionEnabled(): boolean {
const raw = process.env[SOURCES_SECTION_ENV_VAR];
if (raw === undefined) return true;
return !DISABLED_VALUES.has(raw.trim().toLowerCase());
}

/**
* Apply the CLI `--no-sources-section` flag into the shared env slot so the
* prompt builder picks it up downstream.
*
* Commander defaults a `--no-x` option to `true`, so only an explicit `false`
* carries the user's intent; leaving the variable alone otherwise keeps it
* authoritative for setups that configure the project rather than a single
* invocation. Mirrors `applyLanguageOption` in output-language.ts.
*/
export function applySourcesSectionOption(enabled: boolean | undefined): void {
if (enabled === false) {
process.env[SOURCES_SECTION_ENV_VAR] = "off";
}
}
87 changes: 87 additions & 0 deletions test/sources-section.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
/**
* Unit tests for the `## Sources` section toggle and its effect on the page
* prompt.
*
* Default behaviour (no env, no flag) must keep the page prompt byte-identical
* to the previous implementation. Opting out must remove only the section
* request — the inline citation contract and the `sources:` frontmatter that
* carry provenance are not part of this switch.
*/

import { describe, it, expect, afterEach } from "vitest";
import {
applySourcesSectionOption,
sourcesSectionEnabled,
} from "../src/utils/sources-section.js";
import { buildPagePrompt } from "../src/compiler/prompts.js";

const ENV_KEY = "LLMWIKI_SOURCES_SECTION";
const SECTION_REQUEST = "Include a ## Sources section";

afterEach(() => {
delete process.env[ENV_KEY];
});

describe("sourcesSectionEnabled", () => {
it("defaults to enabled when the env var is unset", () => {
expect(sourcesSectionEnabled()).toBe(true);
});

it.each(["0", "false", "off", "no"])("treats %s as disabled", value => {
process.env[ENV_KEY] = value;
expect(sourcesSectionEnabled()).toBe(false);
});

it("ignores surrounding whitespace and case", () => {
process.env[ENV_KEY] = " OFF ";
expect(sourcesSectionEnabled()).toBe(false);
});

it("stays enabled for any other value", () => {
process.env[ENV_KEY] = "on";
expect(sourcesSectionEnabled()).toBe(true);
});
});

describe("applySourcesSectionOption", () => {
it("leaves the env var untouched when the flag was not passed", () => {
applySourcesSectionOption(undefined);
expect(process.env[ENV_KEY]).toBeUndefined();
});

it("leaves an existing opt-out in place on commander's default true", () => {
process.env[ENV_KEY] = "off";
applySourcesSectionOption(true);
expect(sourcesSectionEnabled()).toBe(false);
});

it("disables the section when --no-sources-section was passed", () => {
applySourcesSectionOption(false);
expect(sourcesSectionEnabled()).toBe(false);
});
});

describe("buildPagePrompt honours the toggle", () => {
it("requests the section by default", () => {
expect(buildPagePrompt("Concept", "src", "", "")).toContain(SECTION_REQUEST);
});

it("omits the request when disabled", () => {
process.env[ENV_KEY] = "off";
expect(buildPagePrompt("Concept", "src", "", "")).not.toContain(SECTION_REQUEST);
});

it("keeps the inline citation contract when disabled", () => {
process.env[ENV_KEY] = "off";
const out = buildPagePrompt("Concept", "src", "", "");
expect(out).toContain("^[filename.md:START-END]");
expect(out).toContain("Draw facts only from the provided source material.");
});

it("drops the line rather than blanking it", () => {
const enabled = buildPagePrompt("Concept", "src", "", "");
process.env[ENV_KEY] = "off";
const disabled = buildPagePrompt("Concept", "src", "", "");
expect(disabled.split("\n").length).toBe(enabled.split("\n").length - 1);
});
});
Loading