diff --git a/.gitignore b/.gitignore index 008c364..d24fe9f 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,6 @@ .cursorignorerules .cursorignoreconfig /mcpb/stage + +# Benchmark artifacts (regenerated by benchmarks/*.mjs) +/benchmarks/*.json diff --git a/README.md b/README.md index 5cd8eec..22babbb 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Give your AI full read/write access to Notion with **one token and one paste**. Three reasons it exists when Notion ships its own MCP: - **Built for agents, not humans-in-the-loop.** Notion's hosted MCP is OAuth-only — it cannot run headless. This server authenticates with a token, so it works in **CI, cron jobs, background agents, and self-hosted deployments** where nobody can click "Authorize". -- **~90% less context overhead.** Two MCP tools (`notion_execute` + `notion_describe`) dispatch **43 operations**, instead of one tool schema per endpoint flooding your agent's context. +- **97% smaller tool footprint at connection.** Two MCP tools (**422 tokens**) instead of one schema per endpoint — the official open-source server loads **17,163 tokens** of tool schemas before you do anything. Operation schemas load on demand via `notion_describe`, so even a typical multi-operation task stays 85–95% lighter. [Measured, reproducible →](./benchmarks) - **The operational stuff is built in.** Batched mutations with atomic rollback, idempotency keys, automatic retry on rate limits, slim token-efficient responses, full markdown round-trip, and self-healing validation errors that let the model fix its own bad payloads in one turn. @@ -123,7 +123,7 @@ If you just want to chat with your Notion in claude.ai's web UI, use Notion's ho | Capability | Official Notion MCP (open source) | **This server** | | --- | --- | --- | -| **Tool surface** | ~24 tools (one per endpoint) loaded into context | **2 tools** — the LLM loads ~90% less schema | +| **Tool surface** | 24 tools (one per endpoint), 17,163 tokens loaded into context | **2 tools**, 422 tokens — [97% less schema at connection](./benchmarks) | | **Operations covered** | ~24 endpoints | **43 operations** (plus a `trash_page` alias) across pages, blocks, databases, data sources, views, templates, comments, users, files | | **Batch mutations** | Not documented | ✅ Universal `{ items: [...] }` envelope; up to **10 in parallel** | | **Atomic batches + rollback** | Not documented | ✅ `atomic: true` aborts on first failure, best-effort archives entities created earlier | diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..0197d76 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,57 @@ +# Context-overhead benchmark + +How much of an agent's context window does the Notion tool surface consume **before it does any work**? Every MCP client sends the server's `tools/list` payload — tool names, descriptions, and JSON input schemas — into the model's context on connection, and it stays there for the whole session. Fewer, smaller schemas = more room for the actual task. + +This benchmark measures that payload for **this server** (2 tools) against the **official open-source server** [`@notionhq/notion-mcp-server`](https://www.npmjs.com/package/@notionhq/notion-mcp-server) (one tool per REST endpoint). + +## Method + +1. Drive each server through the MCP stdio handshake and capture its real `tools/list` response (`list-tools.mjs`). No Notion token needed — schema listing is unauthenticated. +2. Serialize each tool into the shape a client forwards to the model's tool-use API (`{name, description, input_schema}`) and count tokens (`count.py`). +3. Tokenizer: `o200k_base` (GPT-4o/4.1) via `tiktoken` — a public, modern proxy for LLM context cost. Anthropic's tokenizer isn't public; absolute counts shift slightly by model, the **ratio** is stable. + +## Results + +Measured against `@notionhq/notion-mcp-server` (Notion-Version `2022-06-28`), this server at v2.10.1. + +### Static footprint — always in context, every request + +| Server | Tools | Tool-schema tokens | +| --- | --- | --- | +| Official open-source server | 24 | **17,163** | +| This server | 2 | **422** | + +**97.5% smaller — 40.7× less** context spent on tool schemas at connection. + +The official server front-loads all 24 endpoint schemas whether or not you use them. This server exposes two tools — `notion_execute` (dispatches 44 operations) and `notion_describe` (returns any operation's schema on demand) — so the full operation catalog never sits in context. + +### Realistic sessions — static 422 + `notion_describe` only for operations actually used + +| Task | Operations described | Tokens | vs. 17,163 | +| --- | --- | --- | --- | +| Read a page | `get_page` | 582 | 97% less | +| Search + read | `search_pages`, `get_page` | 802 | 95% less | +| Query a database | `query_database` | 796 | 95% less | +| Write: page + blocks | `create_page`, `append_blocks` | 2,270 | 87% less | +| Typical mixed (4 ops) | `get_page`, `search_pages`, `append_blocks`, `query_database` | 1,424 | 92% less | +| Heavy (8 ops) | 8 distinct operations | 3,578 | 79% less | + +`notion_describe` output averages ~650 tokens/operation (69 for a trivial op like `delete_comment`, up to ~4,500 for `batch_mixed_blocks`). Often the agent skips `describe` entirely — `notion_execute` returns self-healing validation errors that let the model correct its own payload in one turn. + +### Honest worst case + +Describing **all 44 operations** would cost ~29,000 tokens — more than the official server's 17,163. You would never do this: the design pays only for what a task touches, while the official server pays its full 17,163 on every connection regardless. Note also this server covers **44 operations vs. the official 24 endpoints**, with richer per-operation schemas (batch semantics, idempotency), so even per-operation the payloads aren't strictly like-for-like. + +## Reproduce + +```bash +# From the repo root, with the server built (npm run build): +cd benchmarks +NOTION_TOKEN=ntn_dummy node list-tools.mjs awkoy node ../build/index.js > awkoy.json +OPENAPI_MCP_HEADERS='{"Authorization":"Bearer ntn_dummy","Notion-Version":"2022-06-28"}' \ + node list-tools.mjs notion-official npx -y @notionhq/notion-mcp-server > official.json +python3 count.py # static footprint + reduction +node describe-all.mjs > all-describe.json # on-demand describe costs +``` + +Requires `tiktoken` (`pip install tiktoken`). diff --git a/benchmarks/count.py b/benchmarks/count.py new file mode 100644 index 0000000..7c322e9 --- /dev/null +++ b/benchmarks/count.py @@ -0,0 +1,35 @@ +import json, tiktoken +enc = tiktoken.get_encoding("o200k_base") # GPT-4o/4.1 tokenizer, modern proxy for LLM context + +def tool_to_api_shape(t): + # What an MCP client forwards to the model's tool-use API: + return {"name": t.get("name",""), "description": t.get("description",""), + "input_schema": t.get("inputSchema", {})} + +def measure(path): + d = json.load(open(path)) + tools = d["tools"] + payload = [tool_to_api_shape(t) for t in tools] + blob = json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + total = len(enc.encode(blob)) + per = [] + for t in payload: + b = json.dumps(t, ensure_ascii=False, separators=(",", ":")) + per.append((t["name"], len(enc.encode(b)))) + return d["label"], len(tools), total, len(blob), per + +results = {} +for f in ["official.json", "awkoy.json"]: + label, n, tok, chars, per = measure(f) + results[label] = (n, tok, chars, per) + print(f"\n=== {label} ===") + print(f"tools: {n} | total tokens: {tok:,} | chars: {chars:,}") + for name, pt in sorted(per, key=lambda x:-x[1])[:6]: + print(f" {pt:>6,} {name}") + +o = results["notion-official"]; a = results["awkoy"] +print("\n" + "="*50) +print(f"official: {o[1]:,} tokens across {o[0]} tools") +print(f"awkoy: {a[1]:,} tokens across {a[0]} tools") +red = 100*(1 - a[1]/o[1]) +print(f"reduction: {red:.1f}% ({o[1]/a[1]:.1f}x smaller)") diff --git a/benchmarks/describe-all.mjs b/benchmarks/describe-all.mjs new file mode 100644 index 0000000..277a37e --- /dev/null +++ b/benchmarks/describe-all.mjs @@ -0,0 +1,19 @@ +import { spawn } from "node:child_process"; +const OPS="add_discussion_comment add_page_comment append_blocks archive_page batch_mixed_blocks create_database create_page create_view delete_block delete_comment delete_view get_block get_block_children get_bot_user get_comment get_data_source get_file_upload get_page get_page_markdown get_self get_user get_view list_comments list_data_source_templates list_data_sources list_file_uploads list_users list_views move_page query_database query_view restore_page search_pages set_page_properties set_page_property set_page_title trash_page update_block update_comment update_data_source update_database update_page_markdown update_view upload_file".split(" "); +const child = spawn("node",["../build/index.js"],{env:{...process.env,NOTION_TOKEN:process.env.NOTION_TOKEN||"ntn_dummy"},stdio:["pipe","pipe","pipe"]}); +let buf=""; const pending=new Map(); let id=0; +const rpc=(m,p)=>{const i=++id;child.stdin.write(JSON.stringify({jsonrpc:"2.0",id:i,method:m,params:p})+"\n");return new Promise(r=>pending.set(i,r));}; +const notify=(m,p)=>child.stdin.write(JSON.stringify({jsonrpc:"2.0",method:m,params:p})+"\n"); +child.stdout.on("data",d=>{buf+=d;let nl;while((nl=buf.indexOf("\n"))>=0){const l=buf.slice(0,nl).trim();buf=buf.slice(nl+1);if(!l)continue;let m;try{m=JSON.parse(l)}catch{continue}if(m.id&&pending.has(m.id)){pending.get(m.id)(m);pending.delete(m.id)}}}); +child.stderr.on("data",()=>{}); +await rpc("initialize",{protocolVersion:"2025-06-18",capabilities:{},clientInfo:{name:"b",version:"0"}}); +notify("notifications/initialized",{}); +const out={}; +for(const op of OPS){ + const r=await rpc("tools/call",{name:"notion_describe",arguments:{operation:op}}); + const text=(r.result?.content||[]).map(c=>c.text||"").join(""); + out[op]=text; +} +child.kill(); +process.stdout.write(JSON.stringify(out)); +process.exit(0); diff --git a/benchmarks/list-tools.mjs b/benchmarks/list-tools.mjs new file mode 100644 index 0000000..7d204b8 --- /dev/null +++ b/benchmarks/list-tools.mjs @@ -0,0 +1,47 @@ +// Drives an MCP server over stdio through the handshake and captures tools/list. +// Usage: node list-tools.mjs