feat: opt-in query-scoped extraction (BM25) for GET /api and MCP read_url - #43
Merged
Conversation
Captured from the unmodified server (base eab7bd4); capture.mjs documents the exact requests and normalization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2sCeV2sJ2k4CGwr5mnXr2
…rkdown-sections.js) Pure, dependency-free line-based parser (parseBlocks) plus heading-tree builder (buildSectionTree) - the first building block for the opt-in query-extract feature, which will later BM25-score these sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
findFenceClose hardcoded the closing-fence leading-indent tolerance to
{0,3} even when the fence was opened via FENCE_OPEN_ANY_INDENT_RE
(unlimited indent, used for fences nested in list items). A fence
indented 4+ spaces inside a list item never found its closer, so the
list block silently swallowed everything after it, including headings.
Parameterize the close-tolerance on which open regex was used: keep
{0,3} for top-level fences, allow unlimited leading whitespace for
list-nested fences.
Also add a one-line regression test asserting 7+ '#' chars is not a
heading (ATX caps at 6) - already correctly handled, test added for
coverage per review.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dependency-free tokenizer (letters/digits, hyphen/dot-joined identifiers survive as single tokens) and Okapi BM25 scorer with Lucene-style per-call IDF, for the opt-in query-extract feature's section-scoring step (Task 2 of the query-extract SDD plan). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add lib/query-extract.js: estimateTokens + queryExtract, the pure orchestrating module for the opt-in query-extract feature. Given full converted markdown and a query, returns only the BM25-relevant sections (section mode, >=2 headings) or blocks (block mode / oversized-section descent), reassembled in document order with an elision marker between non-contiguous regions. Whole-page short-circuit for small inputs and whole-page fallback (exact input string) on empty/no-match queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gates the BM25-based query-extract feature on a non-empty `query` param across cache-hit, Reddit, HN, and web response paths: extraction runs on the response copy only, after cache.put/cache read and before frontmatter merge and format branching, so the byte-identical no-query default and cache purity (cache.put always receives the full markdown) are preserved. Adds X-Extracted/-Confidence/-Sections/-Original-Tokens/-Returned-Tokens headers, a JSON `extract` object, and five allowlisted frontmatter fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gates on the new `query`/`max_tokens` params (Task 1-4's queryExtract, same frontmatter field names as GET /api). Wired into all three read_url body-producing paths (cache-hit, Reddit fresh, web fresh); get_share and list_recent are untouched. Since MCP has no response headers, a no-match whole-page fallback is signaled in-band via a marker comment instead of server.js's X-Extracted/X-Extract-* headers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cross-cutting suite mapping 1:1 to the query-extract handoff checklist: byte-identity against pre-feature golden files (no query), well-formed subset (balanced fences, intact table, breadcrumb heading), heading-less page block-mode fallback, non-contiguous elision-marker well-formedness, no-match confidence-low fallback, measured token reduction on the fixture page (1751 -> 499 tokens, ratio ~0.285), and single-extractor-call on repeated queried requests to the same URL. 921/921 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds README (API table, response headers, new "Query-scoped extraction" section, architecture bullet) and help.html (DE/EN parameter row) coverage for the `query`/`max_tokens` params shipped in Tasks 1-6, plus a CHANGELOG entry. Default /api behavior without `query` is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… gate, telemetry length, docs dash - lib/query-extract.js: cap deduped query terms at 64 to bound BM25 scoring cost (CPU DoS on huge heading-poor pages with adversarial queries) - lib/mcp.js: applyQueryExtract now treats a whitespace-only query as inactive, mirroring server.js's trimmedQuery/queryActive gate - server.js: cache-hit extraction telemetry now logs the full pre-extraction body length (baseMd), matching the fresh-path logging convention - public/help.html: use en-dash consistently in the EN max_tokens row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opt-in extract mode: a non-empty
queryparameter onGET /apiand the MCPread_urltool returns only the BM25-relevant markdown sections of the converted page instead of the whole document (typically an ~70-85% token reduction on documentation-style pages; measured 1751 -> 499 tokens on the bundled fixture).Without
query, behavior is byte-identical to v3.3.0 - verified against pre-change golden fixtures committed before the feature was built (test/fixtures/query-extract/golden/, reproduced byte-for-byte at the final commit).Design
lib/markdown-sections.js), BM25 with per-page IDF (lib/bm25.js), selection/reassembly core (lib/query-extract.js).<!-- … -->elision markers between non-contiguous regions; paragraph-level fallback for pages with <2 headings; whole-page fallback withconfidence: lowwhen nothing matches.query,max_tokens(64..20000, default 600). New response metadata:X-Extract-*headers,extractobject informat=json, five optional frontmatter fields.Tests
922/922 in a clean worktree (794 baseline + 128 new), including a byte-identity acceptance gate against pre-feature goldens and 1:1 coverage of the spec's acceptance checklist.
🤖 Generated with Claude Code
https://claude.ai/code/session_01J2sCeV2sJ2k4CGwr5mnXr2