Problem Statement
AI assistants and their crawlers currently see a stale, partial version of the docs site:
docs.rocketride.org/llms.txt returns 404 in production even though the generator (packages/docs/scripts/lib/llms.js, wired into docs:build as gather → index → compile) already exists in-tree. Production appears to have been last deployed before it landed.docs.rocketride.org/sitemap.xml has 162 URLs and zero <lastmod> entries, so crawlers cannot tell what changed. Observed crawler caches of RocketRide properties run 2–3 months stale (e.g. cached GitHub metadata showed 2.0k stars vs 5.1k live).- There is no
robots.txt, so AI crawler access is undeclared.
This hurts AI/answer-engine visibility of the docs at exactly the moment developers evaluate tools through AI assistants.
Proposed Solution
Verified locally with a full gather → index → docusaurus build run (Docusaurus 3.8.1; built site served and probed):
- Add
packages/docs/static/robots.txt (allow-only: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot, …) + a .gitignore exception, and preserve it in clearGeneratedStatic() (today anything in static/ except .gitkeep/img/ is wiped on every gather).
docusaurus.config.ts: add showLastUpdateTime: true and a sitemap block (lastmod: 'date', drop meaningless changefreq/priority) — both required for <lastmod> emission.gather.js: stamp each staged page's front matter with last_update: {date} from the SOURCE file's last git commit. The assembled content tree under BUILD_ROOT is not git-tracked, so without this the sitemap gets zero dates even with (2) — verified 0/162 without, 159/162 with (the 3 undated are generated pages with no git source).- Redeploy docs — this alone ships the already-implemented
/llms.txt, /llms-full.txt, /nodes catalog and per-page .md siblings.
CI note: the docs build job needs git history for real dates (fetch-depth: 0); a shallow clone collapses every date to the clone day.
PR with all of the above incoming (branch docs/RR-<this issue>-agent-readiness).
Alternatives Considered
- Hand-maintained llms.txt: rejected — the in-tree generator already produces a better one; it just is not deployed.
- Symlink gather mode for git dates: tested, does not work (git only sees tracked paths).
- Post-build sitemap rewriting: works but is a hack compared to front-matter stamping at the source.
Affected Modules
Problem Statement
AI assistants and their crawlers currently see a stale, partial version of the docs site:
docs.rocketride.org/llms.txtreturns 404 in production even though the generator (packages/docs/scripts/lib/llms.js, wired intodocs:buildas gather → index → compile) already exists in-tree. Production appears to have been last deployed before it landed.docs.rocketride.org/sitemap.xmlhas 162 URLs and zero<lastmod>entries, so crawlers cannot tell what changed. Observed crawler caches of RocketRide properties run 2–3 months stale (e.g. cached GitHub metadata showed 2.0k stars vs 5.1k live).robots.txt, so AI crawler access is undeclared.This hurts AI/answer-engine visibility of the docs at exactly the moment developers evaluate tools through AI assistants.
Proposed Solution
Verified locally with a full gather → index → docusaurus build run (Docusaurus 3.8.1; built site served and probed):
packages/docs/static/robots.txt(allow-only: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot, …) + a.gitignoreexception, and preserve it inclearGeneratedStatic()(today anything instatic/except.gitkeep/img/is wiped on every gather).docusaurus.config.ts: addshowLastUpdateTime: trueand asitemapblock (lastmod: 'date', drop meaningless changefreq/priority) — both required for<lastmod>emission.gather.js: stamp each staged page's front matter withlast_update: {date}from the SOURCE file's last git commit. The assembled content tree under BUILD_ROOT is not git-tracked, so without this the sitemap gets zero dates even with (2) — verified 0/162 without, 159/162 with (the 3 undated are generated pages with no git source)./llms.txt,/llms-full.txt,/nodescatalog and per-page.mdsiblings.CI note: the docs build job needs git history for real dates (
fetch-depth: 0); a shallow clone collapses every date to the clone day.PR with all of the above incoming (branch
docs/RR-<this issue>-agent-readiness).Alternatives Considered
Affected Modules