From 3370f1c53ce32660c6ef9fb233e3ec2e8ded27b4 Mon Sep 17 00:00:00 2001 From: Grady Dillon Date: Thu, 23 Jul 2026 20:18:49 -0400 Subject: [PATCH] Sync README with the current SDK surface Document the methods the client already ships but the README omitted, and fix two stale pricing-table facts: - Add sections for search(), output formats + contentOf(), scoped keys (createKey/listKeys/revokeKey), WellMarked.register(), getLogs(), and the compliance overrides. Expand the error table and TypeScript imports. - Fix the pricing table: Free CAN bulk (up to 5/request), and the priority-queue order is Free=Last, Pro=Standard, Growth=High, Enterprise=Highest (was Free=Standard, Pro=High). Docs only. Co-Authored-By: Claude Opus 4.8 --- README.md | 164 +++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 145 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index ee7d680..0e1cb9d 100644 --- a/README.md +++ b/README.md @@ -28,20 +28,22 @@ console.log("retrieved at", result.metadata.retrievedAt); The API key can also be picked up from the `WELLMARKED_API_KEY` environment variable, in which case `new WellMarked()` is enough. -Get a key at [wellmarked.io](https://wellmarked.io). +Get a key at [wellmarked.io](https://wellmarked.io) — or [self-register](#self-registration) with an email. ## Pricing | | Free | Pro | Growth | Enterprise | |-----------------------|-----------|----------------------|----------------------|---------------| | **Monthly Price** | $0 | $29/mo | $79/mo | $199/mo | -| **Annual Price** | — | $299/yr | $799/yr | $1,999/yr | | **Included Requests** | 1,000/mo | 10,000/mo | 40,000/mo | 250,000/mo | -| **Bulk Requests** | ❌ | ✅ (up to 50/request) | ✅ (up to 200/request) | ✅ (Unlimited) | +| **Annual Price** | — | $299/yr | $799/yr | $1,999/yr | +| **Bulk Requests** | ✅ (up to 5/request) | ✅ (up to 50/request) | ✅ (up to 200/request) | ✅ (Unlimited) | +| **Search** | ❌ | ✅ | ✅ | ✅ | | **Crawl** | ❌ | ✅ (2,000 pages/job) | ✅ (10,000 pages/job) | ✅ (Unlimited) | +| **`json`/`chunks` formats** | ❌ | ✅ | ✅ | ✅ | | **Overage Rate** | — | $0.0035/req | $0.0020/req | $0.0012/req | | **JS Rendering** | ❌ | ✅ | ✅ | ✅ | -| **Priority Queue** | Standard | High | High | Highest | +| **Priority Queue** | Last | Standard | High | Highest | See additional pricing information at [wellmarked.io/#pricing](https://wellmarked.io/#pricing). @@ -56,9 +58,52 @@ const wm = new WellMarked({ apiKey: "wm_..." }); const result = await wm.extract("https://example.com/article"); ``` +## Output formats + +`extract`, `search`, `bulk`, and `crawl` all take a `format` option. Exactly one content field is populated per result; the rest stay `null`. `contentOf(result)` returns whichever one came back. + +```typescript +import { contentOf } from "wellmarked"; + +const r = await wm.extract(url, { format: "chunks" }); +for (const chunk of r.chunks!) console.log(chunk.startToken, chunk.text.slice(0, 40)); + +(await wm.extract(url, { format: "json" })).blocks; // typed heading/paragraph/list/code blocks +(await wm.extract(url, { format: "html" })).html; // cleaned main-content HTML +(await wm.extract(url, { format: "links" })).links; // every http(s) link in the content +contentOf(await wm.extract(url)); // whichever field the format populated +``` + +| `format` | Field | Plan | +|--------------|-------------|-------| +| `"markdown"` | `.markdown` | All | +| `"html"` | `.html` | All | +| `"links"` | `.links` | All | +| `"json"` | `.blocks` | Pro+ | +| `"chunks"` | `.chunks` | Pro+ | + +Every extraction result also carries `.metrics` (`contentBytes`, `inputTokens`, `outputTokens`, `tokensSaved`, `reductionPct`) — the token-savings ROI of the extraction, or `null` when the tokenizer is unavailable. + +## Search + +Search the web and extract every result in one call (Pro plan and above). Costs `1 + results.length` requests. Results come back partial — a slow or blocked page is an error item, never sinks the call. + +```typescript +const results = await wm.search("retrieval augmented generation", { numResults: 5 }); +for (const hit of results.results) { + if (hit.ok) { + console.log(hit.title, "→", hit.markdown!.slice(0, 80)); + } else { + console.log(`${hit.url} failed: ${hit.error} (snippet: ${hit.snippet})`); + } +} +``` + +`numResults` is clamped to 1..10 server-side. `search` takes the full extraction parameter set — `format` and the [compliance overrides](#compliance-overrides) apply to every result. (It does not take `retry`; its per-result deadline can't absorb one.) + ## Bulk extraction -Submit many URLs at once (Pro: up to 50; Growth: up to 200; Enterprise: unlimited). The call returns immediately with a `jobId`. Poll with `getJob` or block until done with `waitForJob`. +Submit many URLs at once (Free: up to 5; Pro: up to 50; Growth: up to 200; Enterprise: unlimited). The call returns immediately with a `jobId`. Poll with `getJob` or block until done with `waitForJob`. ```typescript let job = await wm.bulk([ @@ -76,8 +121,6 @@ for (const item of job.results) { } ``` -Pass `retry: N` to any of `extract`, `bulk`, or `crawl` to re-attempt timed-out fetches server-side (`target_timeout` only, fresh connection per attempt, default 0). On the synchronous `extract` each timed-out attempt takes 20–30s before the next fires, so aggressive values belong here on `bulk`, where the async workers absorb the wait. `search` doesn't take `retry` — its 15-second per-result deadline can't absorb one. - `getJob` and `waitForJob` are **polymorphic** — they work for both bulk and crawl `jobId`s. The SDK reads a `kind` discriminator from the API response and returns either a `BulkJob` or a `CrawlJob`. Use the `isCrawlJob(job)` type guard (or check `job.kind === "crawl"`) before reading crawl-specific fields like `job.truncated` or `item.depth`. ```typescript @@ -89,12 +132,14 @@ if (isCrawlJob(job)) { } ``` +Bulk submissions carry an automatically generated `Idempotency-Key` (see [Retries & idempotency](#retries--idempotency)), so an internal retry replays the original job rather than enqueuing a second one. + ## Crawl Crawl a site BFS-style from a root URL — same-site links only, with per-plan depth and page caps (Pro: depth 5, up to 2,000 pages; Growth: depth 10, up to 10,000 pages; Enterprise: unlimited). Like `bulk`, this returns a queued job; poll with `getJob` or block until done with `waitForJob` — the same two methods work on both kinds. Pass `renderJs: true` to fetch each page through Playwright instead of httpx; a single shared browser is launched at the start of the crawl and reused across pages. ```typescript -let job = await wm.crawl("https://docs.example.com", { depth: 2 }); +let job = await wm.crawl("https://docs.example.com", { depth: 2, maxPages: 500 }); job = await wm.waitForJob(job.jobId); // works for crawl AND bulk jobIds for (const page of job.results) { @@ -110,7 +155,73 @@ if (job.kind === "crawl" && job.truncated) { } ``` -Pass `maxPages: N` to stop the crawl after N successful pages — it can only narrow your plan's page cap, never widen it. Each successful page consumes one request from your monthly quota — failed pages (timeouts, robots-disallowed, no-content) are not billed. If you run out of quota mid-crawl the job finishes with `truncated: true`, `truncatedReason: "quota_exhausted"`. +`maxPages` caps how many pages this crawl may bill — it can only narrow your plan's page cap, never widen it. Each successful page consumes one request from your monthly quota — failed pages (timeouts, robots-disallowed, no-content) are not billed. If you run out of quota mid-crawl the job finishes with `truncated: true`, `truncatedReason: "quota_exhausted"`. + +## Retries on timeout + +`extract`, `bulk`, and `crawl` take a `retry` option — server-side re-attempts when the *target* URL times out (`target_timeout`), each on a fresh connection. Default `0` (one attempt); no upper bound, though the server stops re-attempting once a job's 6-hour lifetime is spent. + +```typescript +await wm.extract("https://slow.example.com", { retry: 2 }); // up to 3 fetch attempts +await wm.bulk(urls, { retry: 5 }); // async workers absorb the wait +``` + +On the synchronous `extract` each timed-out attempt takes 20–30s, so aggressive values belong on `bulk`/`crawl`. This is distinct from the client's own transport `maxRetries` (see [Retries & idempotency](#retries--idempotency)). + +## Compliance overrides + +`extract`, `search`, `bulk`, and `crawl` accept three narrow-only overrides on your key's compliance policy — they can tighten it for a single request but never loosen it: + +```typescript +await wm.extract("https://example.com/article", { + allowDomains: ["example.com"], // subset of what the key already allows + denyPatterns: ["*/private/*"], // added to the key's deny list + respectRobots: "strict", // can upgrade to strict, never down to lax +}); +``` + +On `extract` a policy denial throws `PermissionDeniedError` (`domain_not_allowed` / `domain_denied` / `robots_disallowed`); on `bulk`/`crawl` it surfaces as a per-item `error` instead. + +## Self-registration + +Get a free, extract-only key from an email — no existing key needed. The zero-to-first-call path for an agent that discovered WellMarked programmatically. + +```typescript +const account = await WellMarked.register("agent@example.com"); +const wm = new WellMarked({ apiKey: account.apiKey }); // Free plan, scopes: ["extract"] +console.log((await wm.extract("https://example.com")).markdown); +``` + +The key is deliberately weak (Free plan, extract only, no billing). To unlock bulk, crawl, search, or paid quota, a human claims the account on [wellmarked.io](https://wellmarked.io). + +## Scoped keys + +Mint and manage additional keys programmatically (requires a calling key with the `keys` scope). A key can only grant scopes it holds. + +```typescript +const created = await wm.createKey(["extract", "bulk"], { name: "ci-pipeline" }); +console.log(created.apiKey); // shown once — store it + +for (const k of await wm.listKeys()) { // metadata only, never raw values + console.log(k.id, k.name, k.scopes, k.revokedAt); +} + +await wm.revokeKey(created.id); // stops authenticating immediately +``` + +Scope vocabulary: `extract`, `bulk`, `crawl`, `search`, `keys`. + +## Audit log + +`getLogs()` returns your own request history, newest first — which key ran each call and how its compliance policy decided. + +```typescript +const page = await wm.getLogs({ limit: 50, offset: 0 }); +for (const entry of page.logs) { + console.log(entry.timestamp, entry.statusCode, entry.targetUrl, entry.policyDecision); +} +// if (page.hasMore) ... fetch offset += limit +``` ## Webhooks @@ -253,6 +364,16 @@ console.log("New key:", rotated.apiKey); // shown once — store it before the After `rotateKey()` the client automatically switches to the new key for subsequent calls; you still need to persist `rotated.apiKey` somewhere durable, because the previous key stops working immediately and there is no recovery flow. +## Retries & idempotency + +The client retries requests it knows are safe to replay — connection failures and 5xx, and only for GETs and POSTs carrying an `Idempotency-Key`. `bulk()` and `crawl()` send one automatically and reuse it across the SDK's internal retries, so a connection blip replays the original job rather than enqueuing a second one. `POST /extract` is never retried (the API bills it on arrival). + +```typescript +new WellMarked({ apiKey: "wm_...", maxRetries: 2 }); // 3 attempts total; 0 disables +``` + +Pass your own `idempotencyKey` to `bulk()`/`crawl()` to also survive a process crash or your own catch-and-resubmit — same key + same body replays; same key + a different body throws `idempotency_key_reuse`. Records expire after 6 hours. + ## Errors Every non-2xx response is translated into a typed error class. Catch the base class to handle anything, or the specific subclass to handle one failure mode: @@ -275,7 +396,7 @@ try { if (err instanceof RateLimitError) { console.log(`Quota hit. Resets in ${err.retryAfter}s.`); } else if (err instanceof UnprocessableEntityError) { - // err.code is one of: no_content, target_timeout, ... + // err.code is one of: no_content, target_timeout, target_http_error, ... console.log(`Extraction failed (${err.code}): ${err.message}`); } else { throw err; @@ -286,22 +407,22 @@ try { | Error class | HTTP | Typical `code` values | |----------------------------|------|------------------------------------------------------------------------------------------------------| | `AuthenticationError` | 401 | `missing_api_key`, `invalid_api_key` | -| `PermissionDeniedError` | 403 | `account_inactive`, `plan_not_supported`, `forbidden` | -| `NotFoundError` | 404 | `job_not_found` | -| `UnprocessableEntityError` | 422 | `no_content`, `target_timeout`, `bulk_cap_exceeded`, `crawl_depth_exceeded` | -| `RateLimitError` | 429 | `rate_limit_too_fast` *(per-second cap; `retryAfterMs` carries the sub-second back-off)* · `rate_limit_exceeded` *(monthly quota; `retryAfter` in seconds)* | -| `InternalServerError` | 5xx | — | +| `PermissionDeniedError` | 403 | `account_inactive`, `plan_not_supported`, `insufficient_scope`, `forbidden`, `domain_not_allowed`, `domain_denied`, `robots_disallowed` | +| `NotFoundError` | 404 | `job_not_found`, `key_not_found` | +| `UnprocessableEntityError` | 422 | `no_content`, `target_timeout`, `target_http_error`, `unsafe_url`, `bulk_cap_exceeded`, `crawl_depth_exceeded`, `idempotency_key_reuse` | +| `RateLimitError` | 429 | `rate_limit_too_fast` *(per-second cap; `retryAfterMs` carries the sub-second back-off)* · `rate_limit_exceeded` *(monthly quota; `retryAfter` in seconds)* · `register_rate_limited` | +| `InternalServerError` | 5xx | `service_unavailable` *(retryable)* | | `APIConnectionError` | — | DNS / TCP / TLS / timeout failures, raised before any HTTP round-trip | -All inherit from `WellMarkedError`. +`APIStatusError` is the shared base of every HTTP-status error above; all inherit from `WellMarkedError`. ## Configuration ```typescript new WellMarked({ apiKey: "wm_...", // or set WELLMARKED_API_KEY - timeoutMs: 30_000, // per request, default 30s - maxRetries: 2, // retries for safely replayable requests + timeoutMs: 30_000, // per attempt, default 30s + maxRetries: 2, // transport retries for safely replayable requests headers: { "X-Trace-Id": "..." }, // optional: extra headers on every request }); ``` @@ -318,6 +439,11 @@ import type { CrawlJob, ExtractResult, ExtractionMeta, + SearchResults, + CreatedKey, + ApiKeyInfo, + LogsPage, + RegisteredAccount, RotatedKey, RotatedWebhookSecret, Usage, @@ -325,7 +451,7 @@ import type { } from "wellmarked"; ``` -Webhook-side types and helpers (`verifyWebhook`, `WebhookVerificationError`, `WebhookPayload`, `JobWebhookOptions`) are all exported from the package root. +The `contentOf`, `isBulkJob`, and `isCrawlJob` helpers, and the webhook helpers (`verifyWebhook`, `WebhookVerificationError`, `WebhookPayload`, `JobWebhookOptions`), are all exported from the package root. ## For Agents