fix(#44): recover split article bodies, and add select.content to the recipe engine

CHANGELOG.md

@@ -9,6 +9,18 @@ Self-hosters should consult [`MIGRATION.md`](./MIGRATION.md) when upgrading acro
 
 ---
 
+## [Unreleased]
+
+### Added
+
+- **`select.content` — recipes can now name the article body outright.** Until now the recipe engine was purely subtractive (`select.remove`, `preprocess`): a recipe could say what to throw away, but never what the article *is*, leaving the final choice to Readability's candidate scoring. `select.content` takes a list of CSS selectors, joins every match in document order into a single document, and uses that as the body — skipping both the Readability scoring and the Trafilatura auto-pick. Nested matches collapse to the outermost, invalid selectors skip themselves, and `select.remove` still applies first, so the two compose. Output carries `source: recipe-content`. Because a stale `content` selector would otherwise yield an empty article, a selection under 200 characters falls back to the normal pipeline and records the reason in `metadata.extractorReason`. Documented in [`SITE-RECIPES.md`](./SITE-RECIPES.md).
+
+### Fixed
+
+- **Blog posts whose body is split across sibling containers lost everything outside the winning block** (closes #44). Some CMS templates wedge an in-article call-to-action between two separate body containers; Readability scores a single top candidate and keeps only that candidate plus its direct siblings, so the entire lead section was dropped while the output still looked well-formed. A shipped recipe (`claude-blog-split-body`) now names both containers via the new `select.content`. The Trafilatura auto-pick could not catch this on its own: its output carried the full text but no markdown headings, and `pickBest` requires at least one heading before preferring the longer candidate.
+
+---
+
 ## [3.5.0] - 2026-07-10
 
 ### Added

SITE-RECIPES.md

@@ -22,7 +22,7 @@ document and the repository.
 6. [Structured data (JSON-LD) into frontmatter](#6-structured-data-json-ld-into-frontmatter)
 7. [Cleaning noisy output](#7-cleaning-noisy-output)
 8. [A complete annotated example](#8-a-complete-annotated-example)
-9. [The four built-in recipes](#9-the-four-built-in-recipes)
+9. [The five built-in recipes](#9-the-five-built-in-recipes)
 10. [Testing your recipe](#10-testing-your-recipe)
 11. [Contributing checklist](#11-contributing-checklist)
 
@@ -40,8 +40,9 @@ would not address. Typical signals:
 - A recommendation rail, "you may also like" block, or paywall scaffold keeps
   leaking into the Markdown. → remove those elements.
 - The extractor picks the wrong container as the article (e.g. it grabs a
-  paragraph-dense inner block and drops the lead image). → unwrap or restructure
-  the offending element.
+  paragraph-dense inner block and drops the lead image, or keeps only one half
+  of a body split across two containers). → name the body with `select.content`,
+  or unwrap the offending element.
 - The page embeds clean structured data (author, publish date, rating) that you
   want promoted into the frontmatter. → map JSON-LD or selectors into frontmatter
   fields.
@@ -176,6 +177,7 @@ match is collected and **merged** into one effective recipe. The merge rules:
 | ----- | -------------- |
 | `preprocess` | Arrays **concatenate**, in recipe order. |
 | `select.remove` | Arrays **concatenate**. |
+| `select.content` | Arrays **concatenate**. |
 | `extractor` | Scalar, **last-wins**. |
 | each `fetch.*` key (`render`, `wait_for`, `wait_timeout_ms`, `mobile_ua`, `pdf`) | Merged **per key**, last-wins on a per-key collision. Setting `wait_for` in one recipe and `mobile_ua` in another gives you both. |
 | `frontmatter.jsonld` | Scalar, **last-wins**. |
@@ -206,7 +208,7 @@ unknown key rejects the whole recipe.
 | `host` | string, or non-empty array of strings | — (required) | Host glob(s) the recipe applies to. See [§3](#3-matching-and-merge-semantics). |
 | `path` | string (non-empty) | `/**` | Restrict to a path glob (e.g. articles only). |
 | `preprocess` | array of action objects | `[]` | Structural HTML edits before extraction (see below). |
-| `select` | object `{ remove: string[] }` | `{ remove: [] }` | CSS selectors of elements to delete before extraction. |
+| `select` | object `{ remove: string[], content: string[] }` | `{ remove: [], content: [] }` | `remove`: CSS selectors of elements to delete before extraction. `content`: CSS selectors that *are* the article body (see [below](#naming-the-article-body-with-selectcontent)). |
 | `extractor` | `"readability"` \| `"trafilatura"` \| `"playwright"` | (unset) | Force a specific extractor and skip the quality auto-pick. |
 | `fetch` | object (see below) | `{}` | Control fetching / rendering. |
 | `frontmatter` | object (see [§6](#6-structured-data-json-ld-into-frontmatter)) | (unset) | Inject custom frontmatter fields from JSON-LD / selectors. |
@@ -445,7 +447,12 @@ There is no Markdown post-processing step by design — content-noise removal
 happens before extraction, so choose your selectors against the (rendered, if
 applicable) DOM.
 
-Two tools, pick by intent:
+Three tools, pick by intent:
+
+- **`select.content`** — names the article body outright; see
+  [below](#naming-the-article-body-with-selectcontent) below. Reach for it when
+  the problem is "the extractor picked the wrong block", not "this block is
+  noise".
 
 - **`select.remove`** — a flat list of CSS selectors whose elements are deleted.
   This is the go-to for "delete these boilerplate blocks": ad slots, related-
@@ -493,6 +500,53 @@ selectors for noise that survives that generic pass on your specific site.
 
 ---
 
+### Naming the article body with `select.content`
+
+`select.remove` and `preprocess` are subtractive: they tell the extractor what
+to throw away, then leave it to guess what remains. `select.content` is the
+positive statement — **this** is the article:
+
+```json
+"select": { "content": [".article-body", ".article-body-continued"] }
+```
+
+Every match is collected and joined into one document, which becomes the
+article. Readability's candidate scoring and the Trafilatura auto-pick are both
+skipped, so the extractor cannot pick the wrong block: you already named the
+right one. Output carries `source: recipe-content`.
+
+Reach for it when the diagnosis is "the extractor picked the wrong container",
+not "this block is noise". The case that motivated it: a CMS that splits the
+body across two containers with a call-to-action wedged between them.
+Readability scores a single top candidate and keeps only that candidate plus its
+direct siblings, so the entire lead section vanished while the output still
+looked well-formed. No amount of `remove` fixes that — the problem is not a
+surplus element, it is that half the article was never selected.
+
+Semantics worth knowing:
+
+- **Document order.** Matches are emitted in the order they appear on the page,
+  regardless of the order you list the selectors in.
+- **Nested matches collapse.** If one match contains another, only the outermost
+  survives, so `[".article", ".article p"]` will not emit the prose twice.
+- **Invalid selectors skip themselves** and never break the rest of the page.
+- **`select.remove` still applies first.** Combining them is normal: name the
+  body with `content`, then delete a widget that lives *inside* it with `remove`.
+- **It runs on the rendered DOM too** when the page goes through Playwright, so
+  author selectors against the rendered markup (see
+  [§5](#5-js-heavy-and-bot-protected-sites-rendered-dom)).
+
+**The safety net.** A `remove` selector that goes stale is harmless — it removes
+nothing. A `content` selector that goes stale would yield an *empty* article, so
+PullMD guards it: if the selected body comes out under 200 characters, the page
+falls back to the normal Readability/Trafilatura pipeline and says so in
+`metadata.extractorReason` (`recipe select.content matched nothing, fell back to
+readability: …`). A site redesign therefore degrades your recipe to the generic
+behavior instead of breaking the page — but check that field if a recipe
+mysteriously stops taking effect.
+
+---
+
 ## 8. A complete annotated example
 
 A generic template exercising host arrays, a path glob, forced rendering with a
@@ -550,7 +604,7 @@ the output begins with a YAML block including `author:`, `published:`, and
 
 ---
 
-## 9. The four built-in recipes
+## 9. The five built-in recipes
 
 The shipped `site-recipes.default.json` is the best reference for real, working
 recipes. Each demonstrates a different feature:
@@ -561,6 +615,7 @@ recipes. Each demonstrates a different feature:
 | `future-plc-recommendations` | `select.remove` — deletes recommendation rails on the same hosts. Split from the paywall recipe to keep one concern per recipe (they merge at match time). |
 | `github-issues` | `fetch.render: "force"` + `wait_for` + `wait_timeout_ms` and a multi-segment `path` glob (`/*/*/issues/*`) — renders JS-loaded issue comments. |
 | `sciencedaily-lead-image` | `preprocess` with `unwrap` and a `path` glob (`/releases/**`) — unwraps a `#text` container so Readability keeps the lead image instead of dropping it. |
+| `claude-blog-split-body` | `select.content` — the article body is split across two containers with a call-to-action between them, and Readability keeps only one of them. The recipe names both outright instead of nudging the scoring. |
 
 ---
 

lib/recipes.js

@@ -20,8 +20,12 @@ const FetchSchema = z.object({
   pdf:             z.enum(['ocr']).optional(),
 }).strict();
 
+// `remove` is subtractive (drop these elements); `content` is positive (THIS is
+// the article). A non-empty `content` bypasses Readability's candidate scoring
+// entirely — see selectRecipeContent in lib/web.js.
 const SelectSchema = z.object({
-  remove: z.array(z.string().min(1)).default([]),
+  remove:  z.array(z.string().min(1)).default([]),
+  content: z.array(z.string().min(1)).default([]),
 }).strict();
 
 // A single frontmatter field descriptor: exactly one of `jsonld` (dot-path into
@@ -73,7 +77,7 @@ export const RecipeSchema = z.object({
   host:       z.union([z.string().min(1), z.array(z.string().min(1)).min(1)]),
   path:       z.string().min(1).default('/**'),
   preprocess: z.array(ActionSchema).default([]),
-  select:     SelectSchema.default({ remove: [] }),
+  select:     SelectSchema.default({ remove: [], content: [] }),
   extractor:  z.enum(['readability', 'trafilatura', 'playwright']).optional(),
   fetch:      FetchSchema.default({}),
   frontmatter: FrontmatterSchema.optional(),
@@ -226,6 +230,7 @@ export function mergeRecipes(recipes) {
   const result = {
     preprocess: [],
     removeSelectors: [],
+    contentSelectors: [],
     extractor: undefined,
     fetch: {},
   };
@@ -234,6 +239,7 @@ export function mergeRecipes(recipes) {
   for (const r of recipes) {
     result.preprocess = result.preprocess.concat(r.preprocess || []);
     result.removeSelectors = result.removeSelectors.concat(r.select?.remove || []);
+    result.contentSelectors = result.contentSelectors.concat(r.select?.content || []);
     if (r.extractor !== undefined) result.extractor = r.extractor;
     if (r.fetch) {
       for (const key of ['render', 'wait_for', 'wait_timeout_ms', 'mobile_ua', 'pdf']) {

lib/web.js

@@ -309,6 +309,58 @@ function cleanDom(document, extraRemoveSelectors = []) {
   }
 }
 
+// Minimum length a recipe-selected body must reach before it is trusted. Below
+// this the selectors are treated as stale (CMS redesign renamed the classes)
+// and the caller falls back to the normal Readability/Trafilatura pipeline.
+// Same threshold Readability itself falls back at, further down.
+const RECIPE_CONTENT_MIN_CHARS = 200;
+
+/**
+ * Recipe `select.content`: positive article selection. Collects every match in
+ * document order and joins them into one synthetic <article>, which lets a
+ * recipe state "THIS is the body" instead of nudging Readability's scoring
+ * until it happens to guess right.
+ *
+ * Nested matches are dropped: a selector list like `.article, .article p`
+ * would otherwise emit the same prose twice, once via the container and once
+ * via each child. Only the outermost match of any nesting chain survives.
+ *
+ * Selectors are evaluated one at a time so a single invalid selector (recipes
+ * are contributor-editable) skips only itself.
+ *
+ * @returns {string|null} joined HTML, or null if nothing usable matched
+ */
+function selectRecipeContent(document, selectors) {
+  const matches = [];
+  for (const sel of selectors) {
+    try {
+      matches.push(...document.querySelectorAll(sel));
+    } catch { /* invalid recipe selector must never break extraction */ }
+  }
+  if (matches.length === 0) return null;
+
+  // De-duplicate identical nodes matched by several selectors, then drop any
+  // node contained in another match. `contains` returns true for the node
+  // itself, hence the identity guard.
+  const unique = [...new Set(matches)];
+  const outermost = unique.filter(
+    (el) => !unique.some((other) => other !== el && other.contains(el)),
+  );
+  if (outermost.length === 0) return null;
+
+  // Document order: querySelectorAll is ordered per selector, but matches from
+  // a second selector can precede those from the first.
+  outermost.sort((a, b) => {
+    // eslint-disable-next-line no-bitwise
+    const pos = a.compareDocumentPosition(b);
+    if (pos & 4 /* DOCUMENT_POSITION_FOLLOWING */) return -1;
+    if (pos & 2 /* DOCUMENT_POSITION_PRECEDING */) return 1;
+    return 0;
+  });
+
+  return `<article>${outermost.map((el) => el.outerHTML).join('\n')}</article>`;
+}
+
 // Recipe `select.remove` selectors, applied one at a time so a single
 // invalid selector (recipes are contributor-editable) skips only itself
 // instead of throwing out of the joined querySelectorAll and killing
@@ -425,6 +477,31 @@ async function convertWithReadability(url, html, comments, statusCode, fetchFn,
     return { markdown: formatHeader(title, url, date, filename) + markdown, title, source: 'readability', metadata };
   }
 
+  let recipeContentFellBack = null;
+
+  // Recipe `select.content`: the recipe names the article outright, so neither
+  // Readability's candidate scoring nor the Trafilatura auto-pick runs. Guarded
+  // by a length floor — if the selectors have gone stale the page degrades to
+  // the normal pipeline instead of returning a near-empty document.
+  if (recipe?.contentSelectors?.length) {
+    const contentHtml = selectRecipeContent(document, recipe.contentSelectors);
+    const selectedMd = contentHtml ? nhm.translate(contentHtml) : '';
+    if (selectedMd.trim().length >= RECIPE_CONTENT_MIN_CHARS) {
+      metadata.quality = qualityScore(selectedMd, { rawHtml: cleanedHtml });
+      metadata.extractorReason = `recipe select.content (${recipe.contentSelectors.length} selector(s))`;
+      metadata.contentLength = selectedMd.trim().length;
+      return {
+        markdown: formatHeader(title, url, date, filename) + selectedMd,
+        title,
+        source: 'recipe-content',
+        metadata,
+      };
+    }
+    recipeContentFellBack = selectedMd.trim().length === 0
+      ? 'matched nothing'
+      : `matched only ${selectedMd.trim().length}c`;
+  }
+
   const reader = new Readability(document);
   const article = reader.parse();
 
@@ -476,7 +553,11 @@ async function convertWithReadability(url, html, comments, statusCode, fetchFn,
   }
 
   metadata.quality = qualityScore(chosenMd, { rawHtml: cleanedHtml });
-  metadata.extractorReason = extractorReason;
+  // A stale select.content must never fail silently — the operator wrote those
+  // selectors expecting them to win, so say that they did not.
+  metadata.extractorReason = recipeContentFellBack
+    ? `recipe select.content ${recipeContentFellBack}, fell back to ${source}: ${extractorReason}`
+    : extractorReason;
   metadata.contentLength = chosenMd.trim().length;
 
   return {

site-recipes.default.json

@@ -96,6 +96,14 @@
       "wait_timeout_ms": 10000
     }
   },
+  {
+    "name": "claude-blog-split-body",
+    "host": ["claude.com", "www.claude.com"],
+    "path": "/blog/**",
+    "select": {
+      "content": [".u-rich-text-blog"]
+    }
+  },
   {
     "name": "booking-hotel-frontmatter",
     "host": ["www.booking.com", "booking.com"],

test/recipes-claude-blog.test.js

@@ -0,0 +1,74 @@
+import { describe, it } from 'node:test';
+import assert from 'node:assert/strict';
+import { RecipeSchema } from '../lib/recipes.js';
+import { extractHtml } from '../lib/web.js';
+import { fileURLToPath } from 'node:url';
+import path from 'node:path';
+import fs from 'node:fs';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const defaultFile = path.join(here, '..', 'site-recipes.default.json');
+
+// The CMS wraps the article body in two separate branches with an in-article
+// CTA card wedged between them. Readability scores one branch as its top
+// candidate; the other is not a sibling, so the whole lead section is silently
+// dropped (issue #44). The recipe names both body blocks outright via
+// select.content. Fixture is the real page's DOM skeleton with the prose
+// replaced by same-length filler — the structure is what triggers the bug, the
+// wording is not.
+const FIXTURE = fs.readFileSync(
+  path.join(here, 'fixtures', 'claude-blog-split-body.html'),
+  'utf8',
+);
+
+const POST_URL = 'https://claude.com/blog/the-new-rules-of-context-engineering';
+
+function shippedRecipes() {
+  const raw = JSON.parse(fs.readFileSync(defaultFile, 'utf8'));
+  return raw.map((entry) => {
+    const parsed = RecipeSchema.safeParse(entry);
+    assert.equal(parsed.success, true, `shipped recipe "${entry.name}" must validate`);
+    return parsed.data;
+  });
+}
+
+describe('claude.com blog built-in recipe (issue #44)', () => {
+  it('ships in the default recipe file and validates', () => {
+    const recipe = shippedRecipes().find((r) => r.name === 'claude-blog-split-body');
+    assert.ok(recipe, 'claude-blog-split-body must ship in site-recipes.default.json');
+    assert.deepEqual(recipe.host, ['claude.com', 'www.claude.com']);
+    assert.equal(recipe.path, '/blog/**');
+    assert.deepEqual(recipe.select.content, ['.u-rich-text-blog']);
+  });
+
+  it('without the recipe Readability drops the lead block', async () => {
+    const result = await extractHtml(FIXTURE, {
+      url: POST_URL,
+      recipes: [],
+      extractor: 'readability',
+    });
+    assert.match(result.markdown, /REST BLOCK/);
+    assert.doesNotMatch(result.markdown, /LEAD BLOCK/);
+  });
+
+  it('with the recipe both body blocks survive', async () => {
+    const result = await extractHtml(FIXTURE, {
+      url: POST_URL,
+      recipes: shippedRecipes(),
+      extractor: 'readability',
+    });
+    assert.equal(result.source, 'recipe-content');
+    assert.match(result.markdown, /LEAD BLOCK/);
+    assert.match(result.markdown, /REST BLOCK/);
+    assert.match(result.markdown, /^## Then and now$/m);
+  });
+
+  it('does not apply outside /blog/**', async () => {
+    const result = await extractHtml(FIXTURE, {
+      url: 'https://claude.com/pricing',
+      recipes: shippedRecipes(),
+      extractor: 'readability',
+    });
+    assert.doesNotMatch(result.markdown, /LEAD BLOCK/);
+  });
+});