docs: agent/crawler readiness — robots.txt, sitemap lastmod, ship the LLM surface#1616
Open
EdwardLien0426 wants to merge 9 commits into
Open
docs: agent/crawler readiness — robots.txt, sitemap lastmod, ship the LLM surface#1616EdwardLien0426 wants to merge 9 commits into
EdwardLien0426 wants to merge 9 commits into
Conversation
Added robots.txt to allow various AI crawlers and declare sitemap.
Updated .gitignore to include robots.txt in committed assets.
Updated the clearGeneratedStatic function to keep committed files and added functionality to stamp last update dates on staged pages.
EdwardLien0426
requested review from
Rod-Christensen,
jmaionchi and
stepmikhaylov
as code owners
July 16, 2026 22:27
Contributor
📝 WalkthroughWalkthroughDocumentation gathering now adds page descriptions and git-derived update dates, Docusaurus emits sitemap dates, ChangesDocumentation metadata and sitemap dates
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant DocsWorkflow
participant Gather
participant GitHistory
participant Docusaurus
participant Crawler
DocsWorkflow->>Gather: build documentation content
Gather->>GitHistory: read source commit dates
GitHistory-->>Gather: return page dates
Gather->>Docusaurus: provide descriptions and last_update metadata
Docusaurus->>Crawler: publish sitemap lastmod and llms.txt URLs
Crawler->>Docusaurus: read robots.txt and sitemap.xml
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
CI note for reviewers (edited — my original diagnosis was wrong, correcting it below) The red checks are not caused by this PR, and not by it being a fork PR. GitHub's REST API is currently returning 503 across the board — the HTML error body in the job logs is GitHub's own The pattern in CI is consistent — every job that calls the REST API fails, every job that doesn't passes:
Corrections to my original comment, for the record:
githubstatus.com still reports "All Systems Operational", so this isn't acknowledged upstream yet. No action needed from maintainers. Nothing to re-run with elevated permissions and nothing to re-push — a re-run once the API recovers should go green. The change itself is docs-pipeline-only (4 files, +102/−7) and verified locally; details in the description. |
EdwardLien0426
force-pushed
the
docs/RR-1615-agent-readiness
branch
from
July 16, 2026 23:22
4cbf05a to
2898818
Compare
Contributor
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/docs/scripts/lib/gather.js`:
- Around line 359-360: Update stampLastUpdate so its last_update detection is
limited to the document’s front matter rather than scanning the entire content.
Preserve the existing early return for missing dates, and only skip stamping
when a last_update field exists within the front-matter block.
- Around line 231-234: Update the front-matter description extraction in the
gather logic to parse fm[1] with the existing YAML/front-matter parser instead
of the regex matching description. Read the parsed description value, then apply
the existing quote normalization while preserving support for block scalar
descriptions such as > and |.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7e1d9d70-7a2f-4a5f-889b-ca405db38fc4
📒 Files selected for processing (9)
.github/workflows/docs.ymlpackages/docs/.gitignorepackages/docs/content-static/integrations/neo4j.mdpackages/docs/docusaurus.config.tspackages/docs/scripts/lib/gather.jspackages/docs/scripts/lib/llms.jspackages/docs/src/lib/gather.test.mjspackages/docs/src/lib/llms.test.mjspackages/docs/static/robots.txt
EdwardLien0426
force-pushed
the
docs/RR-1615-agent-readiness
branch
from
July 17, 2026 00:04
50f9508 to
7cf333a
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/docs/scripts/lib/gather.js`:
- Around line 240-243: Update the block-scalar line condition in the frontmatter
parsing loop to continue through empty lines while still requiring subsequent
non-empty lines to be indented. Preserve the existing trimming and block
termination behavior for non-empty, non-indented lines.
In `@packages/docs/src/lib/gather.test.mjs`:
- Around line 16-19: Add a test alongside the existing folded block-scalar case
in the test suite, using an empty line within the indented description block and
asserting that pageDescription preserves the expected paragraph-break behavior.
Keep the existing folded scalar test unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: efedf5ee-364d-4b67-979f-d3e1e06a87e7
📒 Files selected for processing (2)
packages/docs/scripts/lib/gather.jspackages/docs/src/lib/gather.test.mjs
added 5 commits
July 16, 2026 18:16
Per RFC 9309 a crawler obeys only the ONE most specific User-agent group matching it and ignores all others, including `*`. Every bot named in this file therefore reads its own two lines and never sees the `*` group, so a `Disallow:` added to `*` alone would bind Googlebot but not GPTBot, ClaudeBot or any other named bot — silently, as robots.txt has no linting. Verified with urllib.robotparser: adding `Disallow: /internal` to `*` leaves GPTBot and ClaudeBot still able to fetch it while Googlebot is blocked. The groups are kept as the issue specifies them, and because they state the intent in a form a human reads. Record the trap next to them instead, so whoever first adds a rule here sees it.
docs:gather stamps every page with its source file's last commit date, and that date becomes the page's <lastmod> in the sitemap. actions/checkout defaults to fetch-depth: 1, so the docs build saw a single-commit history and every one of the 162 pages would have dated to the clone day — the sitemap would claim the whole site changed on each deploy. Google ignores lastmod it finds untrustworthy, so shipping it that way is worse than shipping no lastmod at all.
stampLastUpdate() detected front matter with startsWith('---\\n'), which is
false on a CRLF checkout. Rather than merging into the existing block it
prepended a second one, dropping title (and every other key) out of the
front matter and into the body as literal text.
.gitattributes sets '* text=auto' and pins eol=lf only for .patch/.sh/.py,
so .md files are CRLF in a Windows working tree. CI builds the docs on
ubuntu, so deploys were unaffected — this corrupted local Windows builds.
llms.txt listed every page as a bare relative link, so an agent had to fetch all 161 pages to find out what any of them covered. The gather manifest already carried a description for node pages — it fed the /nodes catalog table but never llms.txt — and doc pages carried none at all. - Add pageDescription(): front-matter 'description:' when declared, else the first prose sentence, skipping MDX imports, JSX, headings, tables and fenced blocks. Mirrors what Docusaurus derives for its meta description. - Attach it to the spine and module-doc manifest entries. - Emit '- [title](url): description', and make the URL absolute — the llms.txt format specifies URLs and an agent that fetched the file on its own has no base to resolve against. - extractDescription() joined the services*.json description array bare, producing 'a node.Can be invoked' and defeating its own '. ' sentence split, so node blurbs ran on untruncated. Join with a space. Verified end to end (registry discover -> gather -> index): 161 links, 161 absolute, 158 described; the 3 without are generated or dateless pages. Unit-covers pageDescription and stampLastUpdate, both pure. The fenced-block case is there because the first draft returned 'code();' as a description.
Two defects in helpers this branch added, both confirmed by reproducing them
before changing anything:
- pageDescription() read 'description:' to end of line, so a YAML block scalar
('description: >' / '|' / '>-') yielded the indicator itself — llms.txt would
have shown '- [Page](url): >'. Read the indented run beneath it instead.
- stampLastUpdate() tested the whole document for 'last_update:', so a page
whose BODY mentioned it (a YAML sample in a fenced block) silently lost its
sitemap date. Scope the guard to the front-matter block.
CodeRabbit suggested parsing the front matter with a YAML library. Not taken:
gather.js has no YAML parser and 'yaml' is not a declared dependency of
packages/docs — frontMatterTitle() right above uses the same regex approach.
Adding a dependency for a case that occurs zero times in-tree would be a
heavier change than the bug warrants, and inconsistent with the file.
Regression tests for both, plus the block-scalar variants. 35/35 pass; the
real gather->index run is unchanged at 161 links / 161 absolute / 158 described
and 159 pages stamped.
EdwardLien0426
force-pushed
the
docs/RR-1615-agent-readiness
branch
from
July 17, 2026 01:16
b509130 to
5fb6877
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/docs/scripts/lib/gather.js`:
- Around line 252-269: Update the description extraction loop to track
admonition state alongside inFence, toggling it on ::: markers and skipping all
lines inside the admonition, including nested content until the closing marker.
Preserve the existing paragraph termination behavior and ensure introductory
prose after the admonition becomes the selected description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ae0a5c7a-a8be-470a-abd1-8ea66e02db8c
📒 Files selected for processing (6)
.github/workflows/docs.ymlpackages/docs/scripts/lib/gather.jspackages/docs/scripts/lib/llms.jspackages/docs/src/lib/gather.test.mjspackages/docs/src/lib/llms.test.mjspackages/docs/static/robots.txt
asclearuc
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes that make
docs.rocketride.orgreadable by AI assistants and crawlers.1. Add
packages/docs/static/robots.txt— a singleUser-agent: */Allow: /group plus the sitemap declaration. Needs a.gitignoreexception and aKEEPset inclearGeneratedStatic(), which otherwise wipes everything instatic/except.gitkeep/img/on every gather.2. Emit
<lastmod>in the sitemap —showLastUpdateTime: true+sitemapoptions indocusaurus.config.ts, andgather.jsstamps each staged page's front matter with the SOURCE file's last git commit date. The assembled content tree is not git-tracked, so Docusaurus cannot date pages without this (0/162 URLs dated without, 159/162 with — the 3 undated are generated pages with no git source). The live sitemap today has no<lastmod>at all and carrieschangefreq/priority, which Docusaurus' own source comments call "useless".3. Make llms.txt actually useful — it listed every page as a bare relative link, so an agent had to fetch all 161 pages to learn what any of them covered. Now each entry is
- [title](absolute-url): description. The manifest already carried descriptions for node pages (they fed the/nodescatalog table but never llms.txt); doc pages had none, sopageDescription()derives one — front-matterdescription:when declared, else the first prose sentence, skipping MDX imports, JSX, headings, tables and fenced blocks, mirroring what Docusaurus derives for its meta description. URLs are absolute because the llms.txt format specifies URLs and an agent that fetched the file standalone has no base to resolve against.Correction to an earlier version of this description: it claimed production 404s on the LLM surface. That is wrong —
/llms.txt,/llms-full.txtand the per-page.mdsiblings are all already live and servingtext/plain. Only/robots.txt404s. Item 3 above is therefore about the quality of a file that already ships, not about shipping it.On the robots.txt shape
The per-crawler groups are exactly as the issue specifies them. Kept, but with a warning recorded inline (a097b68), because they carry a trap worth knowing about before anyone edits this file:
Per RFC 9309 a crawler obeys only the single most specific matching
User-agentgroup and ignores all others, including*. Every bot named here therefore reads its own two lines and never sees the*group. Today that is harmless — the groups are behaviourally identical to*alone, which already allows everything. But aDisallow:added later to*alone would bind Googlebot and not the named AI bots, silently, since robots.txt has no linting. Demonstrated withurllib.robotparser, addingDisallow: /internalto the*group:/internal/internalCollapsing to a single wildcard group would remove the trap, but the groups also state the intent — we welcome AI crawlers — in a form a human reads, which is what the issue asked for. So they stay and the trap is documented next to them; any rule meant for everyone must be repeated in every group.
Worth a conscious ✅ from reviewers: allow-all includes
Google-ExtendedandCCBot, which are not ordinary crawlers — they are training-data controls (Gemini/Vertex, and Common Crawl, which feeds many corpora). This PR opts the docs into AI training. For public docs that is normally the goal, but it is a product decision, not a technicality.Two defects found while reviewing this branch, fixed here
fetch-depth: 0for the docs build (d84f264) — originally written up as a note asking a maintainer to configure it, but.github/workflows/docs.ymlis in this repo and had nofetch-depth, soactions/checkoutdefaulted to1. The headline feature would have shipped broken: with a single-commit history all 162 pages date to the clone day, so the sitemap would claim the whole site changed on every deploy. Google ignoreslastmodit does not trust, making that strictly worse than emitting none. This pulls.github/workflows/**into the diff, so the CIchangesfilter now classifies this PR as code and runs the full build matrix instead of skipping it.stampLastUpdate()detected front matter withstartsWith('---\n'), false on a CRLF checkout. It then prepended a second front-matter block instead of merging, droppingtitleout of the front matter and into the body as literal text..gitattributessets* text=autoand pinseol=lfonly for.patch/.sh/.py, so.mdis CRLF in a Windows working tree. CI builds on ubuntu, so deploys were unaffected — this corrupted local Windows builds. One-line fix (/^---\r?\n/), which also brings it in line withfrontMatterTitle()right above it, already CRLF-tolerant.Also fixed in passing:
extractDescription()joined theservices*.jsondescription array bare, producinga node.Can be invokedand defeating its own'. 'sentence split, so node blurbs ran on untruncated. Joining with a space fixes both. Invisible before, visible now that these strings are llms.txt descriptions.Heads-up: the docs deploy is red on develop, unrelated to this PR
Noticed while verifying this branch — flagging only, deliberately not fixed here since it is outside this PR's scope and deserves its own issue.
builder docs:buildfails on develop right now, so nothing has shipped to docs.rocketride.org since the graph migration (457d137) landed. The migration renameddb_neo4j→graph_neo4jand correctly added a/nodes/db_neo4j→/nodes/graph_neo4jredirect inredirects.ts, butcontent-static/integrations/neo4j.mdstill links the old path. Node page routes come from the node's directory name (gather.js), so/nodes/db_neo4jno longer exists as a route, andonBrokenLinks: 'throw'fails the build. Docusaurus' broken-link check does not consider client redirects, so the redirect does not save it:Same error in the failing develop run (29535879279). Note this never affects PR CI —
docs.ymlonly triggers on push to develop. It does mean this PR's output will not appear on the site until that is fixed, whoever picks it up.Type
docs
Testing
./builder testpassespageDescription()andstampLastUpdate()are pure and now unit-covered (packages/docs/src/lib/gather.test.mjs), alongside the updatedllms.test.mjs. 35/35 pass. Several cases exist because they caught real defects rather than confirming intent: the fenced-block case (the first draft returnedcode();as a page description), and the two CodeRabbit found — a YAML block scalar (description: >) yielding the bare indicator, and a body mention oflast_update:silently costing a page its<lastmod>. Both were reproduced failing before being fixed in b509130.Verified end to end with a real
registry discover→gather→index→docusaurus build(3.8.1) — build green, and the built site checked directly:robots.txtshippedclearGeneratedStatic()— seededstatic/with a stalellms.txt/quickstart.mdalongside it; the stale files were cleared,robots.txt/img//.gitkeepkept<lastmod>fetch-depth: 0buys; without it all 16 collapse to onechangefreq/priorityAlso:
pageDescriptionspot-checked across all 31 spine pages (31 clean, 0 empty, 0 malformed);robots.txtvalidated with Python'surllib.robotparser; the threesitemapoptions checked against the installed@docusaurus/plugin-sitemapJoi schema, which explicitly permitsnullfor each;prettier --checkclean../builder testnot run — happy to if required.docs:gather-dev(symlink mode) does not stamp dates, so the dev preview shows none; the production build uses copy mode and does. Intentional — a symlink cannot carry the front-matter edit.Checklist
Linked Issue
Fixes #1615
Summary by CodeRabbit
Documentation
Bug Fixes