Add compliance overrides, scoped-key CRUD, register, and search#10
Merged
Conversation
Mirrors the Python SDK so both stay in lockstep with the API: - Compliance policy overrides (allowDomains / denyPatterns / respectRobots) on extract/bulk/crawl. JSON.stringify drops undefined, so unset overrides are omitted and can never clobber the key's policy. - Scoped API keys: createKey / listKeys / revokeKey, plus getLogs, with CreatedKey / ApiKeyInfo / RevokedKey / LogEntry / LogsPage. - Static register(): self-provision a free, extract-only key. - search(): web search plus extraction in one synchronous call, with SearchResult / SearchResults carrying a per-page status. Error mapping is by HTTP status, so the new API error codes needed no new error classes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo pins typescript ^6.0.3, but a stale local node_modules had 5.9.3, so both of these passed locally and failed in CI: - lib ES2020 doesn't declare Array.prototype.at, which the tests use. Raised the type surface to ES2022 (emit stays at the ES2020 target); the package supports Node >=18, which has it. - The CJS build passes --moduleResolution node (node10), deprecated in TS 6 and a hard error. Silenced with ignoreDeprecations for now; moving the CJS output to node16 resolution is a separate change. Verified against the pinned toolchain (npm ci) this time: typecheck, test (66), and build all exit 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vitest 4 imports node:util styleText, added in Node 20.12, so the runner dies at startup on 18. The library itself still supports Node >=18.17 per package.json engines, and that floor stays covered by install + typecheck + build; only the runner is skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
extract/bulk/crawl accept `format` (markdown | json | chunks | html | links). ExtractResult, BulkItem and CrawlItem now extend a shared `Content` interface carrying the five format-dependent fields plus `metrics`; exactly one is populated per result and `markdown` still holds the default. New exports: OutputFormat, Content, ContentBlock, Chunk, ContentMetrics, and `contentOf()` for reading whichever field came back. `ok` keys on `error` + contentOf rather than on `markdown` — the old form reported every successful links/html/chunks item as failed. The API's snake_case chunk offsets (start_token/end_token) map to camelCase (startToken/endToken); a test pins that contiguity survives the translation. typecheck, 70 tests and build all clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Product decision: users get no capability to point WellMarked anywhere but
the hosted API. baseUrl leaves both option interfaces, and the injected
fetch goes with it (a custom fetch is the same capability by another door).
The client now resolves globalThis.fetch lazily at call time — the
constructor can't fail in odd environments, and test runners stub the
global exactly as the runtime resolves it.
Tests migrate from the fetch option to vi.stubGlobal("fetch", ...), plus a
negative control: a baseUrl smuggled past the types with a cast is ignored
and the request still hits api.wellmarked.io.
BREAKING vs the published package: both options were public.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SearchOptions extends PolicyOverrideOptions and gains format; SearchResult extends Content so non-markdown formats land in their own fields (with metrics) via the shared contentFromDict path. Tool docs note the Pro+ gate on json/chunks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
retry: server-side re-attempts on target_timeout, fresh connection per
attempt, default 0, no upper bound. Not offered on search() — the API's
15s per-hit deadline can't absorb one. Distinct from the client's own
maxRetries (transport retries between the SDK and the API).
crawl({ maxPages: N }) narrows the plan's page cap (sent as max_pages);
omitted = plan cap alone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
retry's value stays uncapped, but the server stops re-attempting once a job's 6-hour lifetime is spent — state it where the option is documented so the JSDoc promise matches the API's behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirrors the Python SDK so both stay in lockstep with the API.
What's here
allowDomains/denyPatterns/respectRobots) onextract/bulk/crawl.JSON.stringifydropsundefined, so unset overrides are omitted and can only narrow the key's policy — never clobber it. (A real semantic difference from Python, whereNonewould serialize tonulland would clobber.)createKey/listKeys/revokeKey, plusgetLogs, with the matching types.register(): self-provision a free, extract-only key.search(): web search + extraction in one synchronous call.SearchResultcarries a per-pagestatus, so a page that failed extraction is marked rather than silently dropped.Notes
Verification
npm run typecheckclean and 66 tests pass (63 + 3 new forsearch). CI runs the Node 18/20/22 matrix with typecheck + test + build.Merge/release order matters: this package must be published to npm before the
MCPrepo, whose CI installs the publishedwellmarkedand needssearch()/SearchResultsto typecheck.🤖 Generated with Claude Code