Enhance llms.txt discovery by injecting HTTP Link headers and <link> …#724
Enhance llms.txt discovery by injecting HTTP Link headers and <link> …#724Thushani-Jayasekera wants to merge 1 commit into
Conversation
…tags in root and portal view pages
📝 WalkthroughChangesThis PR enhances llms.txt discovery for AI agents by introducing server-side signals through HTTP headers and HTML links, along with an updated robots.txt file. robots.txt EnhancementUpdated the Root Page Discovery MiddlewareAdded response-intercepting middleware for the root path (
Organization/View Page Discovery MiddlewareAdded response-intercepting middleware for portal view pages (
Both middleware implementations monkey-patch the Implementation Details
WalkthroughThe change updates the 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
src/app.js (2)
375-375: ⚡ Quick winPrefer
res.append('Link', ...)instead ofres.set('Link', ...).Current code overwrites any existing
Linkheader on these responses. Appending is safer for compatibility with other middleware/handlers.Suggested change
- res.set('Link', '</llms.txt>; rel="describedby"; type="text/plain"'); + res.append('Link', '</llms.txt>; rel="describedby"; type="text/plain"'); ... - res.set('Link', `<${llmsUrl}>; rel="describedby"; type="text/plain"`); + res.append('Link', `<${llmsUrl}>; rel="describedby"; type="text/plain"`);As per coding guidelines, "Provide concise, actionable feedback focused on correctness and best practices."
Also applies to: 391-391
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/app.js` at line 375, Replace the use of res.set('Link', ...) with res.append('Link', ...) wherever the Link header is added (e.g., the occurrences using res.set('Link', '</llms.txt>; rel="describedby"; type="text/plain"') and the similar call later) so existing Link headers aren’t overwritten by this middleware; update those calls to call res.append('Link', ...) instead to preserve other handlers' Link values.
378-379: ⚡ Quick winRestrict HTML injection to HTML responses.
The current rewrite triggers on any string body containing
</head>. Add a content-type check (for example,text/html) to avoid unintended mutation of non-HTML responses.As per coding guidelines, "Provide concise, actionable feedback focused on correctness and best practices."
Also applies to: 394-395
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/app.js` around lines 378 - 379, The middleware currently rewrites any string variable named body whenever it contains '</head>'; restrict this to only HTML responses by checking the response Content-Type before modifying body—e.g., verify the response header (res.getHeader('content-type') or headers['content-type']) includes 'text/html' and only then perform the body.replace call that injects before '</head>'; apply the same Content-Type guard to both locations where body is rewritten (the block around the shown body.replace at lines ~378-379 and the similar replacement at ~394-395).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/app.js`:
- Around line 389-396: The route builds llmsUrl using raw req.params (orgName,
viewName) which can produce invalid headers/HTML; encode the parameters before
composing the URL and using it in res.set and the injected href. Replace direct
uses of orgName and viewName with encoded values (e.g., encodedOrgName =
encodeURIComponent(orgName), encodedViewName = encodeURIComponent(viewName)),
then build llmsUrl from those encoded variables and use that llmsUrl in
res.set('<Link...>') and inside the res.send replacement so both the Link header
and the injected <link rel="alternate"... href="..."> use the encoded URL (refer
to symbols: orgName, viewName, llmsUrl, res.set, res.send, originalSend).
---
Nitpick comments:
In `@src/app.js`:
- Line 375: Replace the use of res.set('Link', ...) with res.append('Link', ...)
wherever the Link header is added (e.g., the occurrences using res.set('Link',
'</llms.txt>; rel="describedby"; type="text/plain"') and the similar call later)
so existing Link headers aren’t overwritten by this middleware; update those
calls to call res.append('Link', ...) instead to preserve other handlers' Link
values.
- Around line 378-379: The middleware currently rewrites any string variable
named body whenever it contains '</head>'; restrict this to only HTML responses
by checking the response Content-Type before modifying body—e.g., verify the
response header (res.getHeader('content-type') or headers['content-type'])
includes 'text/html' and only then perform the body.replace call that injects
before '</head>'; apply the same Content-Type guard to both locations where body
is rewritten (the block around the shown body.replace at lines ~378-379 and the
similar replacement at ~394-395).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
| const { orgName, viewName } = req.params; | ||
| const llmsUrl = `/${orgName}/views/${viewName}/llms.txt`; | ||
| res.set('Link', `<${llmsUrl}>; rel="describedby"; type="text/plain"`); | ||
| const originalSend = res.send.bind(res); | ||
| res.send = function (body) { | ||
| if (typeof body === 'string' && body.includes('</head>')) { | ||
| body = body.replace('</head>', | ||
| ` <link rel="alternate" type="text/plain" href="${llmsUrl}">\n</head>`); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, check if src/app.js exists and examine the relevant lines
if [ -f "src/app.js" ]; then
echo "=== Lines 380-410 of src/app.js ==="
sed -n '380,410p' src/app.js | cat -n
else
echo "File not found, searching for app.js"
find . -name "app.js" -type f | head -20
fiRepository: wso2/api-developer-portal-core
Length of output: 1688
🏁 Script executed:
# Search for the specific code pattern to verify location and context
rg "orgName.*viewName" -A 10 -B 5 --context=15Repository: wso2/api-developer-portal-core
Length of output: 50387
🏁 Script executed:
# Check if encodeURIComponent is used elsewhere in the codebase for similar purposes
rg "encodeURIComponent" -B 2 -A 2Repository: wso2/api-developer-portal-core
Length of output: 50387
Encode route parameters before building the URL for headers and HTML output.
The orgName and viewName parameters should be encoded before insertion into the HTTP Link header and HTML href attribute to ensure well-formed and safe output.
Suggested change
const { orgName, viewName } = req.params;
+const safeOrgName = encodeURIComponent(orgName);
+const safeViewName = encodeURIComponent(viewName);
-const llmsUrl = `/${orgName}/views/${viewName}/llms.txt`;
+const llmsUrl = `/${safeOrgName}/views/${safeViewName}/llms.txt`;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const { orgName, viewName } = req.params; | |
| const llmsUrl = `/${orgName}/views/${viewName}/llms.txt`; | |
| res.set('Link', `<${llmsUrl}>; rel="describedby"; type="text/plain"`); | |
| const originalSend = res.send.bind(res); | |
| res.send = function (body) { | |
| if (typeof body === 'string' && body.includes('</head>')) { | |
| body = body.replace('</head>', | |
| ` <link rel="alternate" type="text/plain" href="${llmsUrl}">\n</head>`); | |
| const { orgName, viewName } = req.params; | |
| const safeOrgName = encodeURIComponent(orgName); | |
| const safeViewName = encodeURIComponent(viewName); | |
| const llmsUrl = `/${safeOrgName}/views/${safeViewName}/llms.txt`; | |
| res.set('Link', `<${llmsUrl}>; rel="describedby"; type="text/plain"`); | |
| const originalSend = res.send.bind(res); | |
| res.send = function (body) { | |
| if (typeof body === 'string' && body.includes('</head>')) { | |
| body = body.replace('</head>', | |
| ` <link rel="alternate" type="text/plain" href="${llmsUrl}">\n</head>`); |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/app.js` around lines 389 - 396, The route builds llmsUrl using raw
req.params (orgName, viewName) which can produce invalid headers/HTML; encode
the parameters before composing the URL and using it in res.set and the injected
href. Replace direct uses of orgName and viewName with encoded values (e.g.,
encodedOrgName = encodeURIComponent(orgName), encodedViewName =
encodeURIComponent(viewName)), then build llmsUrl from those encoded variables
and use that llmsUrl in res.set('<Link...>') and inside the res.send replacement
so both the Link header and the injected <link rel="alternate"... href="...">
use the encoded URL (refer to symbols: orgName, viewName, llmsUrl, res.set,
res.send, originalSend).
1 participant
Problem
AI agents given just a portal URL (e.g. https://localhost:3000/aiorg/views/default) had no way to automatically discover the llms.txt file at
/{orgName}/views/{viewName}/llms.txt. The root /llms.txt already existed but wasn't signaled anywhere.
What We Discussed
Why agents couldn't auto-discover llms.txt:
that respects those conventions
already check automatically
What we ruled out:
Fixes Applied
User-agent: *
Allow: /
AI agent entry point: https://your-domain/llms.txt
Any agent hitting the root URL now gets:
Any page under /:orgName/views/:viewName/* now gets:
Both 2 and 3 are handled by response-intercepting middleware in app.js — no template or controller changes needed.
Discovery Chain for an Agent
GET / → Link header + tag → /llms.txt
GET /llms.txt → root index, instructs agent to fetch /{orgName}/views/{viewName}/llms.txt
GET /aiorg/views/default → Link header + tag → /aiorg/views/default/llms.txt
GET /aiorg/views/default/llms.txt → full API catalog
Claude (claude.ai / Claude Code)
Does NOT auto-discover. When given a URL, I fetch the page and read the HTML text content — but I don't automatically follow Link headers or
tags, and I don't check /llms.txt first by default. The user needs to either: - Explicitly tell me to read the llms.txt URL, OR - Have the llms.txt URL visible as readable text in the page contentThat's exactly what your home-discover.js does — it generates a prompt that tells Claude where to find llms.txt. That's the right workaround
for now.
Cursor
Partial support. Cursor's @docs feature can index documentation sites. If you add a portal URL, it crawls it. Whether it automatically picks up
llms.txt depends on the version — newer versions have added llms.txt awareness for their docs indexer. The tag and Link header we added
would help here.
Codex (OpenAI)
No auto-discovery. Similar to Claude — it doesn't check /llms.txt automatically. Needs an explicit prompt or system instruction pointing to the
URL.
Bottom Line
Tool | Auto-checks /llms.txt | Follows Link header | Needs explicit prompt -- | -- | -- | -- Claude | No | No | Yes Cursor (@docs) | Partially | Possibly | Sometimes Codex | No | No | YesThe llms.txt standard is still early. The most reliable path for all three today is what you already have — the "Run in Claude" / copy prompt
flow in home-discover.js that gives the agent the exact URL. The signals we added (Link header, tag, root /llms.txt) are future-proofing
for when tools adopt the standard more broadly.