Skip to content

Sync README with the current SDK surface (search, formats, keys, register, logs) #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gradill22 merged 1 commit into from
Jul 24, 2026
Merged

Sync README with the current SDK surface (search, formats, keys, register, logs) #11

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 145 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,20 +28,22 @@ console.log("retrieved at", result.metadata.retrievedAt);

The API key can also be picked up from the `WELLMARKED_API_KEY` environment variable, in which case `new WellMarked()` is enough.

Get a key at [wellmarked.io](https://wellmarked.io).
Get a key at [wellmarked.io](https://wellmarked.io) — or [self-register](#self-registration) with an email.

## Pricing

| | Free | Pro | Growth | Enterprise |
|-----------------------|-----------|----------------------|----------------------|---------------|
| **Monthly Price** | $0 | $29/mo | $79/mo | $199/mo |
| **Annual Price** | — | $299/yr | $799/yr | $1,999/yr |
| **Included Requests** | 1,000/mo | 10,000/mo | 40,000/mo | 250,000/mo |
| **Bulk Requests** | ❌ | ✅ (up to 50/request) | ✅ (up to 200/request) | ✅ (Unlimited) |
| **Annual Price** | — | $299/yr | $799/yr | $1,999/yr |
| **Bulk Requests** | ✅ (up to 5/request) | ✅ (up to 50/request) | ✅ (up to 200/request) | ✅ (Unlimited) |
| **Search** | ❌ | ✅ | ✅ | ✅ |
| **Crawl** | ❌ | ✅ (2,000 pages/job) | ✅ (10,000 pages/job) | ✅ (Unlimited) |
| **`json`/`chunks` formats** | ❌ | ✅ | ✅ | ✅ |
| **Overage Rate** | — | $0.0035/req | $0.0020/req | $0.0012/req |
| **JS Rendering** | ❌ | ✅ | ✅ | ✅ |
| **Priority Queue** | Standard | High | High | Highest |
| **Priority Queue** | Last | Standard | High | Highest |

See additional pricing information at [wellmarked.io/#pricing](https://wellmarked.io/#pricing).

Expand All @@ -56,9 +58,52 @@ const wm = new WellMarked({ apiKey: "wm_..." });
const result = await wm.extract("https://example.com/article");
```

## Output formats

`extract`, `search`, `bulk`, and `crawl` all take a `format` option. Exactly one content field is populated per result; the rest stay `null`. `contentOf(result)` returns whichever one came back.

```typescript
import { contentOf } from "wellmarked";

const r = await wm.extract(url, { format: "chunks" });
for (const chunk of r.chunks!) console.log(chunk.startToken, chunk.text.slice(0, 40));

(await wm.extract(url, { format: "json" })).blocks; // typed heading/paragraph/list/code blocks
(await wm.extract(url, { format: "html" })).html; // cleaned main-content HTML
(await wm.extract(url, { format: "links" })).links; // every http(s) link in the content
contentOf(await wm.extract(url)); // whichever field the format populated
```

| `format` | Field | Plan |
|--------------|-------------|-------|
| `"markdown"` | `.markdown` | All |
| `"html"` | `.html` | All |
| `"links"` | `.links` | All |
| `"json"` | `.blocks` | Pro+ |
| `"chunks"` | `.chunks` | Pro+ |

Every extraction result also carries `.metrics` (`contentBytes`, `inputTokens`, `outputTokens`, `tokensSaved`, `reductionPct`) — the token-savings ROI of the extraction, or `null` when the tokenizer is unavailable.

## Search

Search the web and extract every result in one call (Pro plan and above). Costs `1 + results.length` requests. Results come back partial — a slow or blocked page is an error item, never sinks the call.

```typescript
const results = await wm.search("retrieval augmented generation", { numResults: 5 });
for (const hit of results.results) {
if (hit.ok) {
console.log(hit.title, "→", hit.markdown!.slice(0, 80));
} else {
console.log(`${hit.url} failed: ${hit.error} (snippet: ${hit.snippet})`);
}
}
```

`numResults` is clamped to 1..10 server-side. `search` takes the full extraction parameter set — `format` and the [compliance overrides](#compliance-overrides) apply to every result. (It does not take `retry`; its per-result deadline can't absorb one.)

## Bulk extraction

Submit many URLs at once (Pro: up to 50; Growth: up to 200; Enterprise: unlimited). The call returns immediately with a `jobId`. Poll with `getJob` or block until done with `waitForJob`.
Submit many URLs at once (Free: up to 5; Pro: up to 50; Growth: up to 200; Enterprise: unlimited). The call returns immediately with a `jobId`. Poll with `getJob` or block until done with `waitForJob`.

```typescript
let job = await wm.bulk([
Expand All @@ -76,8 +121,6 @@ for (const item of job.results) {
}
```

Pass `retry: N` to any of `extract`, `bulk`, or `crawl` to re-attempt timed-out fetches server-side (`target_timeout` only, fresh connection per attempt, default 0). On the synchronous `extract` each timed-out attempt takes 20–30s before the next fires, so aggressive values belong here on `bulk`, where the async workers absorb the wait. `search` doesn't take `retry` — its 15-second per-result deadline can't absorb one.

`getJob` and `waitForJob` are **polymorphic** — they work for both bulk and crawl `jobId`s. The SDK reads a `kind` discriminator from the API response and returns either a `BulkJob` or a `CrawlJob`. Use the `isCrawlJob(job)` type guard (or check `job.kind === "crawl"`) before reading crawl-specific fields like `job.truncated` or `item.depth`.

```typescript
Expand All @@ -89,12 +132,14 @@ if (isCrawlJob(job)) {
}
```

Bulk submissions carry an automatically generated `Idempotency-Key` (see [Retries & idempotency](#retries--idempotency)), so an internal retry replays the original job rather than enqueuing a second one.

## Crawl

Crawl a site BFS-style from a root URL — same-site links only, with per-plan depth and page caps (Pro: depth 5, up to 2,000 pages; Growth: depth 10, up to 10,000 pages; Enterprise: unlimited). Like `bulk`, this returns a queued job; poll with `getJob` or block until done with `waitForJob` — the same two methods work on both kinds. Pass `renderJs: true` to fetch each page through Playwright instead of httpx; a single shared browser is launched at the start of the crawl and reused across pages.

```typescript
let job = await wm.crawl("https://docs.example.com", { depth: 2 });
let job = await wm.crawl("https://docs.example.com", { depth: 2, maxPages: 500 });
job = await wm.waitForJob(job.jobId); // works for crawl AND bulk jobIds

for (const page of job.results) {
Expand All @@ -110,7 +155,73 @@ if (job.kind === "crawl" && job.truncated) {
}
```

Pass `maxPages: N` to stop the crawl after N successful pages — it can only narrow your plan's page cap, never widen it. Each successful page consumes one request from your monthly quota — failed pages (timeouts, robots-disallowed, no-content) are not billed. If you run out of quota mid-crawl the job finishes with `truncated: true`, `truncatedReason: "quota_exhausted"`.
`maxPages` caps how many pages this crawl may bill — it can only narrow your plan's page cap, never widen it. Each successful page consumes one request from your monthly quota — failed pages (timeouts, robots-disallowed, no-content) are not billed. If you run out of quota mid-crawl the job finishes with `truncated: true`, `truncatedReason: "quota_exhausted"`.

## Retries on timeout

`extract`, `bulk`, and `crawl` take a `retry` option — server-side re-attempts when the *target* URL times out (`target_timeout`), each on a fresh connection. Default `0` (one attempt); no upper bound, though the server stops re-attempting once a job's 6-hour lifetime is spent.

```typescript
await wm.extract("https://slow.example.com", { retry: 2 }); // up to 3 fetch attempts
await wm.bulk(urls, { retry: 5 }); // async workers absorb the wait
```

On the synchronous `extract` each timed-out attempt takes 20–30s, so aggressive values belong on `bulk`/`crawl`. This is distinct from the client's own transport `maxRetries` (see [Retries & idempotency](#retries--idempotency)).

## Compliance overrides

`extract`, `search`, `bulk`, and `crawl` accept three narrow-only overrides on your key's compliance policy — they can tighten it for a single request but never loosen it:

```typescript
await wm.extract("https://example.com/article", {
allowDomains: ["example.com"], // subset of what the key already allows
denyPatterns: ["*/private/*"], // added to the key's deny list
respectRobots: "strict", // can upgrade to strict, never down to lax
});
```

On `extract` a policy denial throws `PermissionDeniedError` (`domain_not_allowed` / `domain_denied` / `robots_disallowed`); on `bulk`/`crawl` it surfaces as a per-item `error` instead.

## Self-registration

Get a free, extract-only key from an email — no existing key needed. The zero-to-first-call path for an agent that discovered WellMarked programmatically.

```typescript
const account = await WellMarked.register("agent@example.com");
const wm = new WellMarked({ apiKey: account.apiKey }); // Free plan, scopes: ["extract"]
console.log((await wm.extract("https://example.com")).markdown);
```

The key is deliberately weak (Free plan, extract only, no billing). To unlock bulk, crawl, search, or paid quota, a human claims the account on [wellmarked.io](https://wellmarked.io).

## Scoped keys

Mint and manage additional keys programmatically (requires a calling key with the `keys` scope). A key can only grant scopes it holds.

```typescript
const created = await wm.createKey(["extract", "bulk"], { name: "ci-pipeline" });
console.log(created.apiKey); // shown once — store it

for (const k of await wm.listKeys()) { // metadata only, never raw values
console.log(k.id, k.name, k.scopes, k.revokedAt);
}

await wm.revokeKey(created.id); // stops authenticating immediately
```

Scope vocabulary: `extract`, `bulk`, `crawl`, `search`, `keys`.

## Audit log

`getLogs()` returns your own request history, newest first — which key ran each call and how its compliance policy decided.

```typescript
const page = await wm.getLogs({ limit: 50, offset: 0 });
for (const entry of page.logs) {
console.log(entry.timestamp, entry.statusCode, entry.targetUrl, entry.policyDecision);
}
// if (page.hasMore) ... fetch offset += limit
```

## Webhooks

Expand Down Expand Up @@ -253,6 +364,16 @@ console.log("New key:", rotated.apiKey); // shown once — store it before the

After `rotateKey()` the client automatically switches to the new key for subsequent calls; you still need to persist `rotated.apiKey` somewhere durable, because the previous key stops working immediately and there is no recovery flow.

## Retries & idempotency

The client retries requests it knows are safe to replay — connection failures and 5xx, and only for GETs and POSTs carrying an `Idempotency-Key`. `bulk()` and `crawl()` send one automatically and reuse it across the SDK's internal retries, so a connection blip replays the original job rather than enqueuing a second one. `POST /extract` is never retried (the API bills it on arrival).

```typescript
new WellMarked({ apiKey: "wm_...", maxRetries: 2 }); // 3 attempts total; 0 disables
```

Pass your own `idempotencyKey` to `bulk()`/`crawl()` to also survive a process crash or your own catch-and-resubmit — same key + same body replays; same key + a different body throws `idempotency_key_reuse`. Records expire after 6 hours.

## Errors

Every non-2xx response is translated into a typed error class. Catch the base class to handle anything, or the specific subclass to handle one failure mode:
Expand All @@ -275,7 +396,7 @@ try {
if (err instanceof RateLimitError) {
console.log(`Quota hit. Resets in ${err.retryAfter}s.`);
} else if (err instanceof UnprocessableEntityError) {
// err.code is one of: no_content, target_timeout, ...
// err.code is one of: no_content, target_timeout, target_http_error, ...
console.log(`Extraction failed (${err.code}): ${err.message}`);
} else {
throw err;
Expand All @@ -286,22 +407,22 @@ try {
| Error class | HTTP | Typical `code` values |
|----------------------------|------|------------------------------------------------------------------------------------------------------|
| `AuthenticationError` | 401 | `missing_api_key`, `invalid_api_key` |
| `PermissionDeniedError` | 403 | `account_inactive`, `plan_not_supported`, `forbidden` |
| `NotFoundError` | 404 | `job_not_found` |
| `UnprocessableEntityError` | 422 | `no_content`, `target_timeout`, `bulk_cap_exceeded`, `crawl_depth_exceeded` |
| `RateLimitError` | 429 | `rate_limit_too_fast` *(per-second cap; `retryAfterMs` carries the sub-second back-off)* · `rate_limit_exceeded` *(monthly quota; `retryAfter` in seconds)* |
| `InternalServerError` | 5xx | — |
| `PermissionDeniedError` | 403 | `account_inactive`, `plan_not_supported`, `insufficient_scope`, `forbidden`, `domain_not_allowed`, `domain_denied`, `robots_disallowed` |
| `NotFoundError` | 404 | `job_not_found`, `key_not_found` |
| `UnprocessableEntityError` | 422 | `no_content`, `target_timeout`, `target_http_error`, `unsafe_url`, `bulk_cap_exceeded`, `crawl_depth_exceeded`, `idempotency_key_reuse` |
| `RateLimitError` | 429 | `rate_limit_too_fast` *(per-second cap; `retryAfterMs` carries the sub-second back-off)* · `rate_limit_exceeded` *(monthly quota; `retryAfter` in seconds)* · `register_rate_limited` |
| `InternalServerError` | 5xx | `service_unavailable` *(retryable)* |
| `APIConnectionError` | — | DNS / TCP / TLS / timeout failures, raised before any HTTP round-trip |

All inherit from `WellMarkedError`.
`APIStatusError` is the shared base of every HTTP-status error above; all inherit from `WellMarkedError`.

## Configuration

```typescript
new WellMarked({
apiKey: "wm_...", // or set WELLMARKED_API_KEY
timeoutMs: 30_000, // per request, default 30s
maxRetries: 2, // retries for safely replayable requests
timeoutMs: 30_000, // per attempt, default 30s
maxRetries: 2, // transport retries for safely replayable requests
headers: { "X-Trace-Id": "..." }, // optional: extra headers on every request
});
```
Expand All @@ -318,14 +439,19 @@ import type {
CrawlJob,
ExtractResult,
ExtractionMeta,
SearchResults,
CreatedKey,
ApiKeyInfo,
LogsPage,
RegisteredAccount,
RotatedKey,
RotatedWebhookSecret,
Usage,
WebhookPayload,
} from "wellmarked";
```

Webhook-side types and helpers (`verifyWebhook`, `WebhookVerificationError`, `WebhookPayload`, `JobWebhookOptions`) are all exported from the package root.
The `contentOf`, `isBulkJob`, and `isCrawlJob` helpers, and the webhook helpers (`verifyWebhook`, `WebhookVerificationError`, `WebhookPayload`, `JobWebhookOptions`), are all exported from the package root.

## For Agents

Expand Down