Skip to content

LLMs don't discover /llms.txt — page-reading agents miss the LLM index #90

Description

@sn0wm1ku

Status (updated): The branch/PR for this issue — #91 — now also
delivers the handbook restructure tracked by epic #92 (Quality Gate,
AI Usage Guideline, Planning & Shaping, Design Guidelines, Ethical Design
Policy). This issue is linked as a sub-issue of #92. The original
llms.txt problem statement below is unchanged and remains what closes #90
covers.


Problem

An LLM / agent that reads a handbook page has no reliable way to learn that
the site publishes LLM-optimized versions (/llms.txt, /llms-full.txt, and
per-page .md twins). In short: a page-reading LLM does not reliably discover
that llms.txt exists.

vitepress-plugin-llms already injects a hint, but as hidden DOM:

<div style="display:none;" hidden aria-hidden="true" data-nosnippet>Are you an LLM? View /llms.txt ...</div>

What we actually found

  • WebFetch is not a raw-HTML dump. It runs a small model that summarizes
    the page against your prompt
    . What surfaces is therefore prompt-dependent,
    not a faithful copy of the DOM.
  • Neutral prompt → hint missing. A "summarize, omit nothing" WebFetch of the
    landing page returned only the hero text and the three feature cards. The
    Are you an LLM? View /llms.txt line was not in the output — which is
    exactly why a casual reader (and the next LLM) misses it.
  • Targeted prompt → hint appears. A prompt that explicitly asks "does it
    mention llms.txt?" does coax the hidden line out. That does not mean agents
    discover it — it only proves the summarizer is prompt-dependent, so a hidden
    hint is unreliable for discovery.
  • What we did NOT prove. We can't firmly claim the pipeline strips <head>
    (meta / link) or HTML comments. The fetch that reported "no meta" was itself a
    model summary, and this site has no HTML comment to test against. So: no
    mechanical claim about head/comment stripping — only that hidden content is
    not reliably surfaced by a neutral read.

Fix (layered — the primary one is load-bearing)

  1. Primary (load-bearing): a visible, non-JS, server-rendered line in
    the landing page body. Put it in VitePress markdown / frontmatter so SSG
    bakes it into the static HTML — not a hydrated component, not display:none.
    e.g. a small footer: "AI agents: machine-readable handbook at /llms.txt and
    /llms-full.txt."
    Visible text is what a neutral summary reliably surfaces.
  2. Convention: keep root /llms.txt + /llms-full.txt for agents that
    probe by path. (Already generated by the plugin.)
  3. Head (belt and suspenders): <link rel="alternate" type="text/plain" href="/llms.txt">
    in <head> for agents that parse head. A matching HTTP Link response
    header would help too, but GitHub Pages is a static host and cannot set
    custom headers
    — only feasible if the site moves to Cloudflare Pages /
    Netlify.
  4. robots.txt: a visible # LLM index: /llms.txt line for anyone/anything
    inspecting it.

Acceptance criterion

A neutral-prompt WebFetch of the landing page returns the /llms.txt
line
. One call, checkable after deploy — if a neutral fetch surfaces it,
page-reading agents won't miss it.


🤖 Posted by Claude Code

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions