agent-docs

NotebookLM for coding agents. Ingest API references and product guides from documentation websites into structured, token-optimized markdown that coding agents (Claude Code, Codex, Cursor) can reference across sessions.

The Problem

Coding agents lose documentation context between sessions. Every new conversation starts from zero — the agent has to re-read docs, re-discover API shapes, and re-learn product conventions. Copy-pasting docs into prompts is fragile, eats tokens, and goes stale.

agent-docs solves this by creating a persistent, git-tracked .agent-docs/ directory in your project. It fetches live documentation, transforms it into a canonical format optimized for agent consumption, and writes it as plain markdown files. Agents read these files like any other code — no special tooling, no databases, no infrastructure.

Quick Start

# Install globally
npm install -g @neogp/agent-docs

# Or run without installing
npx @neogp/agent-docs ingest <url>

# Ingest an API reference (single-page docs with endpoint listings)
agent-docs ingest https://fintechprimitives.com/docs/api/ \
  --type api-reference --name fp-api

# Ingest product guides (multi-page docs with sidebar navigation)
agent-docs ingest https://docs.fintechprimitives.com/ \
  --type guide --name fp-guides

# Check what's been ingested
agent-docs status

This creates a .agent-docs/ directory in your current working directory:

.agent-docs/
├── manifest.yaml           # Index of all ingested doc sets
├── fp-api/
│   ├── _meta.yaml          # API overview, auth pattern, resource index
│   └── api/
│       ├── purchases.md    # One file per resource group
│       ├── redemptions.md
│       └── ...
└── fp-guides/
    ├── _meta.yaml          # Guide overview, topic index
    └── guides/
        ├── going-live-checklist.md
        ├── identity-overview.md
        └── ...

Usage

Ingest documentation

agent-docs ingest <url> [options]

Option	Description	Default
-t, --type <type>	api-reference, guide, or auto	auto
-n, --name <name>	Name for the doc source	Derived from URL
-o, --output <dir>	Output directory	.agent-docs
-f, --force	Overwrite existing output	false
--dry-run	Preview without writing files	false
-v, --verbose	Debug logging	false

Auto-detection (--type auto) examines the page structure:

• Many anchor links (#section) and few sidebar page links → single-page → treated as API reference
• Sidebar with links to multiple pages → multi-page → treated as guide

Check status

agent-docs status [--output .agent-docs]

Prints a table of all ingested sources with type, date, and stats. Sources older than 30 days are flagged as stale.

Re-ingest

agent-docs ingest <url> --name <existing-name> --force

The --force flag removes the existing output for that source before re-ingesting. The manifest is updated in place — other sources are preserved.

Use from any project

Run from inside your project directory. Docs land in .agent-docs/ by default:

cd ~/my-project
agent-docs ingest https://api-docs.example.com/ --name example-api
# Creates ~/my-project/.agent-docs/example-api/

Or specify a custom output directory:

agent-docs ingest https://api-docs.example.com/ --output docs/.agent-docs

Output Format

Every generated file includes YAML frontmatter so agents can quickly assess relevance:

API endpoint file:

---
source: https://example.com/docs/api/
type: api-reference
resource: purchases
last_ingested: 2026-03-11
---

# Purchases

## POST /v2/purchases

Create a new purchase order.

### Parameters

| Param | Type | Required | Description |
|-------|------|----------|-------------|
| amount | number | yes | Amount in INR |

### Request

​```json
{ "amount": 5000 }
​```

### Response

​```json
{ "id": "pur_xxx", "status": "pending" }
​```

Guide file:

---
source: https://docs.example.com/identity/overview/
type: guide
topic: identity-overview
last_ingested: 2026-03-11
---

# Identity Overview

Content in clean markdown...

_meta.yaml per source provides a structured index:

name: fp-api
base_url: https://fintechprimitives.com
auth: Bearer token in Authorization header
resources:
  - name: Purchases
    file: api/purchases.md
  - name: Redemptions
    file: api/redemptions.md

manifest.yaml at the root lists all sources:

version: "1.0"
note: >-
  The source of truth for each API or product guide is its official website
  (see source_url). However, coding agents MUST NOT fetch docs from the
  internet. Use the pre-ingested local files in this directory instead.
  Re-run agent-docs ingest to update.
sources:
  - name: fp-api
    type: api-reference
    path: fp-api
    source_url: https://fintechprimitives.com/docs/api/
    last_ingested: 2026-03-11
    endpoint_count: 245
  - name: fp-guides
    type: guide
    path: fp-guides
    source_url: https://docs.fintechprimitives.com/
    last_ingested: 2026-03-11
    guide_count: 78

Pointing Your Agent at the Docs

Copy the documentation retrieval protocol into your project's CLAUDE.md, .cursorrules, or equivalent agent instructions file:

# The AGENT_DOCS_INSTRUCTIONS.md file is included in the npm package
cat node_modules/@neogp/agent-docs/AGENT_DOCS_INSTRUCTIONS.md >> CLAUDE.md

Or manually add:

Reference `.agent-docs/manifest.yaml` for available API documentation.
Read the relevant files under `.agent-docs/` before making API calls.
Do NOT fetch documentation from the internet — use the local .agent-docs/ files only.

The agent will discover the manifest, find the right resource file, and use the structured endpoint definitions, parameter tables, and examples to write accurate code.

---

How agent-docs Works

Why it exists

Large language models are stateless between sessions. When you use a coding agent to build against an external API, every new session starts with zero knowledge of that API. The common workarounds all have problems:

• Paste docs into the prompt — manual, eats context window, goes stale
• Let the agent fetch docs live — slow, unreliable, agent may misparse the HTML
• Summarize docs yourself — time-consuming, loses detail, hard to maintain

agent-docs automates the transformation of live documentation into a format that is both human-readable and agent-optimized. The output is plain markdown files tracked in git — no runtime dependencies, no databases, no API keys required.

Architecture

URL ──► Fetcher ──► Crawler ──► Parser ──► Extractor ──► Writer ──► .agent-docs/

1. Fetcher (src/utils/fetcher.ts) HTTP client wrapping Node's native fetch with retry logic (3 attempts, exponential backoff) and concurrency control via p-limit (max 3 parallel requests). Handles transient failures without overwhelming the target server.

2. Crawler (src/crawlers/) Two strategies based on auto-detected doc type:

• Single-page crawler — For API references where all endpoints live on one page (hash-anchored sections). Parses the DOM with Cheerio, finds all headings with id attributes, and splits the page into sections by collecting content between consecutive headings.

• Multi-page crawler — For product guides with sidebar navigation. Extracts all internal links from sidebar/nav elements, deduplicates them, then fetches each page in parallel (rate-limited).

3. Parser (src/parsers/) Transforms raw HTML into clean, structured content:

• HTML Cleaner — Strips navigation, footers, scripts, styles, sidebars, and ad elements. Extracts the main content area (<main>, <article>, .content).
• Markdown Converter — Turndown with custom rules for code blocks (preserves language hints from <pre> class attributes), images (replaced with [Image: alt] placeholders), and tables.
• Section Splitter — Splits markdown on heading boundaries while respecting code fences (won't split inside ``` blocks).
• Table Parser — Detects parameter tables by column headers (Name/Type/Required/Description and variants), extracts structured parameter data.

4. Extractor (src/extractors/) Derives structured data from parsed content:

• Endpoint Extractor — Scans sections for HTTP method + path patterns (GET /v2/... or POST https://...), extracts description, parameters, request/response JSON examples, and curl snippets. Groups endpoints by resource (derived from URL path segments).
• Guide Extractor — Extracts page title, converts content to clean markdown, pulls out code examples with language tags, identifies prerequisite sections, and replaces mermaid diagrams with placeholders.

5. Writer (src/writers/) Writes the canonical output:

• Canonical Writer — Generates one .md file per resource group (API) or per page (guide), with YAML frontmatter and consistent heading structure.
• Meta Writer — Generates _meta.yaml per source with resource/guide index, detected auth pattern (from curl examples), and base URL.
• Manifest Writer — Creates or updates the top-level manifest.yaml, merging new entries without overwriting existing sources.

Design decisions

Heuristic-first, no API keys required. The core pipeline uses DOM pattern matching (Cheerio) and regex-based extraction. It works out of the box without any LLM API calls. An optional --enhance pass (not yet implemented) can use an LLM to improve parameter descriptions and fill gaps.

One file per resource group, not per endpoint. An API with 150 endpoints produces ~15-80 resource files instead of 150 individual files. This keeps the file count manageable while letting agents read all endpoints for a resource in a single file read.

Cheerio over Puppeteer/Playwright. Both target doc sites are server-rendered HTML. No JavaScript execution needed. Cheerio is orders of magnitude faster, has zero system dependencies, and works in any environment.

YAML frontmatter on every file. Agents can scan frontmatter to assess relevance without reading full content. The type, resource/topic, and source fields enable targeted lookups.

Git-tracked output. The .agent-docs/ directory is plain files. Commit it to your repo and every developer (and every agent session) has the same documentation baseline. Diffs show exactly what changed between re-ingests.

Testing

77 tests across 15 test suites, using real HTML fixtures saved from the target documentation sites. Tests run against these snapshots — no network calls during testing.

npm test