fix(#44): recover split article bodies, and add select.content to the recipe engine

[syswave-dev]

Closes #44.

Root cause

The reported page's body is split across two separate containers, with an in-article call-to-action wedged between them. Readability scores a single top candidate and keeps that candidate plus its direct siblings — the second container is in a different branch, so the entire lead section (7 paragraphs plus a section heading) was dropped. The output still looked well-formed, which is what made this hard to spot.

Verified that vanilla Readability behaves identically on the raw HTML, so this is not caused by PullMD's preprocess/cleanDom layers.

The Trafilatura auto-pick could not rescue it either: Trafilatura's output for this page carried the full text (9.8k vs 7.8k chars) but no markdown headings, and pickBest requires t.headings >= 1 before preferring the longer candidate (lib/scoring.js). That guard is correct — it prevents nav-soup from winning — it just cannot help here.

What this PR does

The reported page could have been patched with a two-line unwrap recipe, and the first commit did exactly that. But the underlying gap is more general, so the PR closes it properly.

1. select.content — recipes can name the article body

The recipe engine was purely subtractive. select.remove and preprocess say what to throw away; nothing said what the article is, so the final choice always stayed with Readability's scoring. For a split body, removal cannot help — the problem is not a surplus element, it is that half the article was never selected.

"select": { "content": [".article-body", ".article-body-continued"] }

Every match is joined in document order into one document and used as the body, skipping both Readability scoring and the Trafilatura auto-pick. Output carries source: recipe-content.

• Nested matches collapse to the outermost, so [".a", ".a p"] cannot emit prose twice
• Invalid selectors skip themselves and never break the page
• select.remove still applies first, so the two compose
• Works on the rendered DOM too, via the existing two-pass Playwright path

Safety net. A stale remove selector is harmless; a stale content selector would yield an empty article. So a selection under 200 chars falls back to the normal pipeline and records why in metadata.extractorReason. A site redesign degrades the recipe to generic behavior instead of breaking the page.

2. The shipped recipe uses it

{
  "name": "claude-blog-split-body",
  "host": ["claude.com", "www.claude.com"],
  "path": "/blog/**",
  "select": { "content": [".u-rich-text-blog"] }
}

This replaces the unwrap detour. The unwrap worked, but it required knowing why Readability fails in order to understand it; content says what it means, and it is what a contributor should copy.

Verification

End to end against the live page through a local server (nocache=1):

	before	after
output	7,782 bytes	10,509 bytes
X-Source	readability	recipe-content
lead section	missing	present
section heading	missing	present
links / images	intact	intact
trailing content	ok	ok (ends at the author line, no newsletter noise)

No regression on unaffected sites: Wikipedia still extracts via readability unchanged. GET /api/recipes/status reports 7 loaded, 0 rejected.

node --test: 995/995 pass (13 new).

Tests

test/recipes-select-content.test.js covers schema and merge-by-concatenation, document-order emission, nested-match dedupe, both fallback paths and their extractorReason, invalid-selector isolation, and select.remove composing with select.content.

test/recipes-claude-blog.test.js asserts the bug without the recipe, the fix with it, and that the recipe does not apply outside the matched path. The fixture is the real page's DOM skeleton with all prose replaced by same-length filler — the structure is what triggers the bug, the wording is not, so no third-party article text is committed.

Docs

SITE-RECIPES.md gets a full section on select.content (semantics, when to reach for it versus remove, the safety net), plus updated schema and merge tables and an entry in the built-in examples. CHANGELOG under Added and Fixed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EF9FFG9xYHpyFexQQpe8CJ