docs(benchmarks): measured context-overhead comparison vs official server

.gitignore

@@ -14,3 +14,6 @@
 .cursorignorerules
 .cursorignoreconfig
 /mcpb/stage
+
+# Benchmark artifacts (regenerated by benchmarks/*.mjs)
+/benchmarks/*.json

README.md

@@ -12,7 +12,7 @@ Give your AI full read/write access to Notion with **one token and one paste**.
 Three reasons it exists when Notion ships its own MCP:
 
 - **Built for agents, not humans-in-the-loop.** Notion's hosted MCP is OAuth-only — it cannot run headless. This server authenticates with a token, so it works in **CI, cron jobs, background agents, and self-hosted deployments** where nobody can click "Authorize".
-- **~90% less context overhead.** Two MCP tools (`notion_execute` + `notion_describe`) dispatch **43 operations**, instead of one tool schema per endpoint flooding your agent's context.
+- **97% smaller tool footprint at connection.** Two MCP tools (**422 tokens**) instead of one schema per endpoint — the official open-source server loads **17,163 tokens** of tool schemas before you do anything. Operation schemas load on demand via `notion_describe`, so even a typical multi-operation task stays 85–95% lighter. [Measured, reproducible →](./benchmarks)
 - **The operational stuff is built in.** Batched mutations with atomic rollback, idempotency keys, automatic retry on rate limits, slim token-efficient responses, full markdown round-trip, and self-healing validation errors that let the model fix its own bad payloads in one turn.
 
 <a href="https://glama.ai/mcp/servers/zrh07hteaa">
@@ -123,7 +123,7 @@ If you just want to chat with your Notion in claude.ai's web UI, use Notion's ho
 
 | Capability | Official Notion MCP (open source) | **This server** |
 | --- | --- | --- |
-| **Tool surface** | ~24 tools (one per endpoint) loaded into context | **2 tools** — the LLM loads ~90% less schema |
+| **Tool surface** | 24 tools (one per endpoint), 17,163 tokens loaded into context | **2 tools**, 422 tokens — [97% less schema at connection](./benchmarks) |
 | **Operations covered** | ~24 endpoints | **43 operations** (plus a `trash_page` alias) across pages, blocks, databases, data sources, views, templates, comments, users, files |
 | **Batch mutations** | Not documented | ✅ Universal `{ items: [...] }` envelope; up to **10 in parallel** |
 | **Atomic batches + rollback** | Not documented | ✅ `atomic: true` aborts on first failure, best-effort archives entities created earlier |

benchmarks/README.md

@@ -0,0 +1,57 @@
+# Context-overhead benchmark
+
+How much of an agent's context window does the Notion tool surface consume **before it does any work**? Every MCP client sends the server's `tools/list` payload — tool names, descriptions, and JSON input schemas — into the model's context on connection, and it stays there for the whole session. Fewer, smaller schemas = more room for the actual task.
+
+This benchmark measures that payload for **this server** (2 tools) against the **official open-source server** [`@notionhq/notion-mcp-server`](https://www.npmjs.com/package/@notionhq/notion-mcp-server) (one tool per REST endpoint).
+
+## Method
+
+1. Drive each server through the MCP stdio handshake and capture its real `tools/list` response (`list-tools.mjs`). No Notion token needed — schema listing is unauthenticated.
+2. Serialize each tool into the shape a client forwards to the model's tool-use API (`{name, description, input_schema}`) and count tokens (`count.py`).
+3. Tokenizer: `o200k_base` (GPT-4o/4.1) via `tiktoken` — a public, modern proxy for LLM context cost. Anthropic's tokenizer isn't public; absolute counts shift slightly by model, the **ratio** is stable.
+
+## Results
+
+Measured against `@notionhq/notion-mcp-server` (Notion-Version `2022-06-28`), this server at v2.10.1.
+
+### Static footprint — always in context, every request
+
+| Server | Tools | Tool-schema tokens |
+| --- | --- | --- |
+| Official open-source server | 24 | **17,163** |
+| This server | 2 | **422** |
+
+**97.5% smaller — 40.7× less** context spent on tool schemas at connection.
+
+The official server front-loads all 24 endpoint schemas whether or not you use them. This server exposes two tools — `notion_execute` (dispatches 44 operations) and `notion_describe` (returns any operation's schema on demand) — so the full operation catalog never sits in context.
+
+### Realistic sessions — static 422 + `notion_describe` only for operations actually used
+
+| Task | Operations described | Tokens | vs. 17,163 |
+| --- | --- | --- | --- |
+| Read a page | `get_page` | 582 | 97% less |
+| Search + read | `search_pages`, `get_page` | 802 | 95% less |
+| Query a database | `query_database` | 796 | 95% less |
+| Write: page + blocks | `create_page`, `append_blocks` | 2,270 | 87% less |
+| Typical mixed (4 ops) | `get_page`, `search_pages`, `append_blocks`, `query_database` | 1,424 | 92% less |
+| Heavy (8 ops) | 8 distinct operations | 3,578 | 79% less |
+
+`notion_describe` output averages ~650 tokens/operation (69 for a trivial op like `delete_comment`, up to ~4,500 for `batch_mixed_blocks`). Often the agent skips `describe` entirely — `notion_execute` returns self-healing validation errors that let the model correct its own payload in one turn.
+
+### Honest worst case
+
+Describing **all 44 operations** would cost ~29,000 tokens — more than the official server's 17,163. You would never do this: the design pays only for what a task touches, while the official server pays its full 17,163 on every connection regardless. Note also this server covers **44 operations vs. the official 24 endpoints**, with richer per-operation schemas (batch semantics, idempotency), so even per-operation the payloads aren't strictly like-for-like.
+
+## Reproduce
+
+```bash
+# From the repo root, with the server built (npm run build):
+cd benchmarks
+NOTION_TOKEN=ntn_dummy node list-tools.mjs awkoy node ../build/index.js > awkoy.json
+OPENAPI_MCP_HEADERS='{"Authorization":"Bearer ntn_dummy","Notion-Version":"2022-06-28"}' \
+  node list-tools.mjs notion-official npx -y @notionhq/notion-mcp-server > official.json
+python3 count.py                       # static footprint + reduction
+node describe-all.mjs > all-describe.json   # on-demand describe costs
+```
+
+Requires `tiktoken` (`pip install tiktoken`).

benchmarks/count.py

@@ -0,0 +1,35 @@
+import json, tiktoken
+enc = tiktoken.get_encoding("o200k_base")   # GPT-4o/4.1 tokenizer, modern proxy for LLM context
+
+def tool_to_api_shape(t):
+    # What an MCP client forwards to the model's tool-use API:
+    return {"name": t.get("name",""), "description": t.get("description",""),
+            "input_schema": t.get("inputSchema", {})}
+
+def measure(path):
+    d = json.load(open(path))
+    tools = d["tools"]
+    payload = [tool_to_api_shape(t) for t in tools]
+    blob = json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
+    total = len(enc.encode(blob))
+    per = []
+    for t in payload:
+        b = json.dumps(t, ensure_ascii=False, separators=(",", ":"))
+        per.append((t["name"], len(enc.encode(b))))
+    return d["label"], len(tools), total, len(blob), per
+
+results = {}
+for f in ["official.json", "awkoy.json"]:
+    label, n, tok, chars, per = measure(f)
+    results[label] = (n, tok, chars, per)
+    print(f"\n=== {label} ===")
+    print(f"tools: {n} | total tokens: {tok:,} | chars: {chars:,}")
+    for name, pt in sorted(per, key=lambda x:-x[1])[:6]:
+        print(f"   {pt:>6,}  {name}")
+
+o = results["notion-official"]; a = results["awkoy"]
+print("\n" + "="*50)
+print(f"official: {o[1]:,} tokens across {o[0]} tools")
+print(f"awkoy:    {a[1]:,} tokens across {a[0]} tools")
+red = 100*(1 - a[1]/o[1])
+print(f"reduction: {red:.1f}%  ({o[1]/a[1]:.1f}x smaller)")

benchmarks/describe-all.mjs

@@ -0,0 +1,19 @@
+import { spawn } from "node:child_process";
+const OPS="add_discussion_comment add_page_comment append_blocks archive_page batch_mixed_blocks create_database create_page create_view delete_block delete_comment delete_view get_block get_block_children get_bot_user get_comment get_data_source get_file_upload get_page get_page_markdown get_self get_user get_view list_comments list_data_source_templates list_data_sources list_file_uploads list_users list_views move_page query_database query_view restore_page search_pages set_page_properties set_page_property set_page_title trash_page update_block update_comment update_data_source update_database update_page_markdown update_view upload_file".split(" ");
+const child = spawn("node",["../build/index.js"],{env:{...process.env,NOTION_TOKEN:process.env.NOTION_TOKEN||"ntn_dummy"},stdio:["pipe","pipe","pipe"]});
+let buf=""; const pending=new Map(); let id=0;
+const rpc=(m,p)=>{const i=++id;child.stdin.write(JSON.stringify({jsonrpc:"2.0",id:i,method:m,params:p})+"\n");return new Promise(r=>pending.set(i,r));};
+const notify=(m,p)=>child.stdin.write(JSON.stringify({jsonrpc:"2.0",method:m,params:p})+"\n");
+child.stdout.on("data",d=>{buf+=d;let nl;while((nl=buf.indexOf("\n"))>=0){const l=buf.slice(0,nl).trim();buf=buf.slice(nl+1);if(!l)continue;let m;try{m=JSON.parse(l)}catch{continue}if(m.id&&pending.has(m.id)){pending.get(m.id)(m);pending.delete(m.id)}}});
+child.stderr.on("data",()=>{});
+await rpc("initialize",{protocolVersion:"2025-06-18",capabilities:{},clientInfo:{name:"b",version:"0"}});
+notify("notifications/initialized",{});
+const out={};
+for(const op of OPS){
+  const r=await rpc("tools/call",{name:"notion_describe",arguments:{operation:op}});
+  const text=(r.result?.content||[]).map(c=>c.text||"").join("");
+  out[op]=text;
+}
+child.kill();
+process.stdout.write(JSON.stringify(out));
+process.exit(0);

benchmarks/list-tools.mjs

@@ -0,0 +1,47 @@
+// Drives an MCP server over stdio through the handshake and captures tools/list.
+// Usage: node list-tools.mjs <label> <command> [args...]
+import { spawn } from "node:child_process";
+
+const [, , label, cmd, ...args] = process.argv;
+const child = spawn(cmd, args, { env: { ...process.env }, stdio: ["pipe", "pipe", "pipe"] });
+
+let buf = "";
+const pending = new Map();
+let idc = 0;
+const rpc = (method, params) => {
+  const id = ++idc;
+  child.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
+  return new Promise((res) => pending.set(id, res));
+};
+const notify = (method, params) =>
+  child.stdin.write(JSON.stringify({ jsonrpc: "2.0", method, params }) + "\n");
+
+child.stdout.on("data", (d) => {
+  buf += d.toString();
+  let nl;
+  while ((nl = buf.indexOf("\n")) >= 0) {
+    const line = buf.slice(0, nl).trim();
+    buf = buf.slice(nl + 1);
+    if (!line) continue;
+    let msg;
+    try { msg = JSON.parse(line); } catch { continue; }
+    if (msg.id && pending.has(msg.id)) { pending.get(msg.id)(msg); pending.delete(msg.id); }
+  }
+});
+
+child.stderr.on("data", () => {}); // swallow banner/logs
+
+const fail = (m) => { console.error("ERR:", m); child.kill(); process.exit(1); };
+setTimeout(() => fail("timeout"), 60000);
+
+await rpc("initialize", {
+  protocolVersion: "2025-06-18",
+  capabilities: {},
+  clientInfo: { name: "bench", version: "0.0.0" },
+});
+notify("notifications/initialized", {});
+const res = await rpc("tools/list", {});
+const tools = res.result?.tools ?? [];
+process.stdout.write(JSON.stringify({ label, count: tools.length, tools }));
+child.kill();
+process.exit(0);