Version: 2.0 | License: MIT | Status: Draft
A human-readable, machine-parseable file that websites place at their root (/agents.txt) to declare their identity, terms of use for AI agents, service catalog, and agentic endpoints.
It fills the gap between:
robots.txt— tells crawlers whether they can access pages (network level)llms.txt— gives LLMs context about the site (read-only)agents.txt— tells AI agents who you are, what you offer, and what they can do (identity + permissions + execution)
# agents.txt — example.com
# Version: 2.0
# Updated: 2026-05-12
# License: MIT
## Identity
---yaml
identity:
name: Example Business
owner: Jane Doe
website: https://example.com
email: info@example.com
---
## Terms of Use
| Agent | Owner | Allowed usage | Conditions | Data usage |
|-------|-------|---------------|------------|------------|
| Googlebot | Google | Full indexing | — | — |
| GPTBot | OpenAI | Answer-engine | Attribute source | rag, answer-engine |
| Bytespider | ByteDance | Denied | — | — |
### Data usage defaults
---yaml
terms:
data_usage:
training: false
rag: true
derivative: true
---
## Services
---yaml
services:
- id: consulting
name: "Consulting"
starting_price: "$200/hour"
url: https://example.com/consulting/
---
## Agentic Endpoints
---yaml
api_schema:
type: openapi # or "mcp" for Model Context Protocol
version: 3.1.0
url: https://example.com/openapi.json
---When an autonomous AI agent visits your website, it needs to answer:
- "Who runs this business?" → Identity section
- "Can I use this data?" → Terms of Use section
- "What do they offer?" → Services section
- "Can I interact programmatically?" → Agentic Endpoints (OpenAPI)
No other standard provides all four.
| Section | Purpose | Required |
|---|---|---|
| Header | Version, date, license | Yes |
| Discovery | Cross-references to llms.txt, openapi.json via HTML <link> tags |
Recommended |
| Terms of Use | What AI agents can/cannot do with your data | Yes |
| Identity | Business name, owner, contact, location | Yes |
| Brand Voice | How agents should represent your brand | Optional |
| Services | What you offer, pricing, URLs | Recommended |
| Agentic Endpoints | OpenAPI spec for tool calling | Optional |
- Markdown with embedded YAML blocks (
---yaml...---) - Human-readable first, machine-parseable second
- YAML blocks contain all structured data; everything else is prose
Add <link> tags in your HTML <head> so agents discover your AI-related files. This follows standard HTML semantics (like <link rel="icon">) and is RFC 9309 compliant — no invented directives in robots.txt:
<link rel="agent" href="/agents.txt" />
<link rel="llms" href="/llms.txt" />Any crawler or LLM that parses HTML will find these. robots.txt should only contain standard directives (User-agent, Allow, Disallow, Sitemap).
Serve the same data as structured JSON at /api/agents. The json-schema.json defines the structure:
| Object | Description | Required |
|---|---|---|
identity |
Business owner, contact, web presence | Yes |
terms |
Agent permissions (allowed/denied) and conditions | No |
voice |
Behavioral directives for AI agents | No |
services |
Service catalog with pricing and URLs | No |
api_schema |
OpenAPI or MCP reference for tool calling | No |
The terms.data_usage object provides granular control over how agents use your content:
| Permission | Description | Default |
|---|---|---|
training |
Allow content to train foundation models | false |
rag |
Allow content in retrieval-augmented generation | true |
derivative |
Allow derivative works (translations, summaries) | true |
Per-agent overrides are available via agents[].data_usage array (values: answer-engine, rag, training, derivative).
| Field | Format | Example |
|---|---|---|
website, url, contact |
URI | https://example.com |
identity.email |
Email (RFC 5322) | info@example.com |
agents[].access |
Enum: full, answer-engine, denied |
answer-engine |
updated |
Date (YYYY-MM-DD) |
2026-05-12 |
# Using ajv-cli
npx ajv-cli validate -s json-schema.json -d your-api-response.json
# Using the included validator script
./tools/validate.sh https://example.com/api/agentsAsturWebs — the first production implementation:
- agents.txt — Markdown version
- /api/agents — JSON version
- openapi.json — OpenAPI 3.1.0 spec
Blog posts:
- agents.txt: the standard I created for AI agent discovery — what it is and why, SSOT architecture, the three-layer stack
- BytIA: the chatbot that knows where you are and how you feel — production chatbot powered by agents.txt data
Source: github.com/asturwebs/asturwebs-v2
agents.txt is not only for external AI agents — your own chatbot should use it as its Single Source of Truth (SSOT).
Most chatbots have business data (pricing, services, contact info) hardcoded in the system prompt. This creates duplication: change a price and you must update both your website and your prompt. agents.txt eliminates this.
- Define your business data once in agents.txt (identity, services, voice, terms)
- Serve it programmatically via
/api/agents(JSON endpoint) - Inject it as context into your chatbot's system prompt at runtime
// 1. Fetch agents data (or import from your SSOT module)
const agentsData = await fetch('https://asturwebs.es/api/agents').then(r => r.json());
// 2. Build business context from the data
const businessContext = `
## Identity
${agentsData.identity.name} — ${agentsData.identity.description}
Contact: ${agentsData.identity.email} | ${agentsData.identity.phone}
## Services
${agentsData.services.map(s => `- ${s.name}: ${s.description}`).join('\n')}
`;
// 3. Inject as system message, separate from behavior
const messages = [
{ role: 'system', content: systemPrompt }, // behavior only
{ role: 'system', content: businessContext }, // data from agents.txt
{ role: 'user', content: userMessage },
];- Single source of truth — change data in agents.txt, your chatbot reflects it instantly
- Cleaner prompts — separate behavior instructions from business data
- Reusable across clients — for multi-tenant chatbots, each client's agents.txt feeds their own instance
- Future-proof — as your business evolves, both external agents and your internal chatbot stay in sync
Real-world implementation: AsturWebs uses this pattern. The chatbot BytIA (asturwebs.es) gets its business context from the same
/api/agentsthat serves external AI agents. See the asturwebs-v2 source.
| Site | agents.txt | JSON API | OpenAPI |
|---|---|---|---|
| asturwebs.es | ✅ | ✅ | ✅ |
To add your site, open a PR editing this table.
| File | Purpose |
|---|---|
| docs/ecosystem.md | Competitive landscape, positioning strategy, adoption flywheel |
| docs/research/ | Background research on related AI web standards |
- Open an issue with your use case or feedback
- Submit a PR with spec improvements or translations
- Add your implementation to the Adopters table
If your site implements the standard, show it:
[](https://github.com/asturwebs/agents-txt)MIT — copy, adapt, integrate into any commercial or open-source project.
agents.txt is part of a growing ecosystem of machine-readable web standards for AI:
| Standard | Purpose | Scope |
|---|---|---|
| robots.txt (RFC 9309) | Crawler access control | Network level — what to crawl |
| llms.txt | LLM context provider | Knowledge — what to understand |
| agents.txt (this) | Agent discovery + permissions | Identity + actions + execution |
| agents-brief.txt | Agent mission brief | Alternative approach to agent instructions |
| ai.txt (Spawning) | Training consent | Opt-in/opt-out for model training |
| ai.txt (DSL paper) | Granular AI control | Per-element HTML control |
| operate.txt | UI operation guide | Browser automation behavior |
| AI Manifest (IETF draft) | Workflow instructions | Step-by-step task execution |
| Model Context Protocol | Tool discovery & execution | Runtime tool loading for agents |