[#39] Capture X long-form Articles as blogposts#65
Merged
Conversation
Add an additive, ordered `blocks: list[ArticleBlock]` field to `ContentSourceSuccess`, mirroring exactly how `frames`/`has_speech`/ `language` were added for `x_video` in #44 — no new top-level model, no parallel store. `ArticleBlock` is a `kind`-discriminated union: - `ArticleTextBlock(kind="text", text)` - `ArticleImageBlock(kind="image", media: MediaEntry, alt=None)` The image block WRAPS the existing `MediaEntry` photo-state union, so the photo download engine, the `_reject_local_path_traversal` / `_require_utc_aware` validators, the `_media/` mirror and a future describe path all apply with zero new plumbing. `text` stays the source of truth for the flattened body (= concatenation of the text blocks), so enrich/topics/generate's fallback consume it unchanged. Purely additive + back-compat: legacy `items.json` (no `blocks` key, pre-#20 `ok:`/legacy-media shapes) loads unchanged, defaulting `blocks` to `[]`. Model-only PR: no extract/fetch/media/generate behaviour change. Adds `ArticleBlockAdapter` (out-of-`Item` validation, mirrors `MediaEntryAdapter`). Docs: ARCHITECTURE data-model invariant #12 + the `x_article` note; CLAUDE.md model-seam bullet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mageBlock) Additive `blocks: list[ArticleBlock]` on ContentSourceSuccess; the image block wraps the existing MediaEntry photo-state union so the download engine + validators + _media/ mirror apply with no new plumbing. `text` stays the flattened body (unchanged), so enrich/topics/generate's fallback consume it as before. Purely additive + back-compat (proven lossless on 29k real records) — legacy items.json loads with blocks=[]. Model-only; the producer (fetch), download walk (media) and blogpost renderer (generate) land in PR2-5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ards Address PR2 review round (all non-blocking): - dedup the synthesized article link against a canonical URL key (host aliases → x.com, drop scheme + trailing slash) so a twitter.com / http / trailing-slash variant in entities.urls no longer double-adds; - only synthesize for an Article __typename (reject a Card carrying a rest_id); - require a numeric-string rest_id (kills the non-scalar garbage-URL vector); - keep _extract_links at radon A by hoisting the append into a helper; - note the card/unified_card variant as a conscious deferral folded into the same real-payload validation step (docstring + ARCHITECTURE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract detects the X long-form Article entity on a tweet result (article.article_results.result.rest_id, anchored via _dig) and synthesizes its canonical https://x.com/i/article/<id> Link, routing through the existing rendered-fetch path with no fetch_x change. Normalized dedup against entities.urls (host aliases → x.com, scheme + trailing slash insensitive) plus __typename ("Article"-only) and numeric-string rest_id guards prevent duplicate or garbage links. The Article GraphQL fixture is constructed, not a recorded live payload — validate the key path (and the deferred card/unified_card variant) against a real bookmarked-Article response before production reliance (RFC #39 open-Q #4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ant) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ot-crash, determinism pin - article.py: WARN on a dropped entity-bearing block (real-payload key-drift visibility); `_find_url_by_key` now prefers the canonical CDN key globally (deep media_url_https beats a shallow bare url); decompose _coerce_content_state. - fetch_x.py: INFO structured/fallback, WARN on captured-but-0-blocks (silent-death signal) + on structured-body truncation vs trafilatura (<50%); DEBUG the .json() swallow; wrap parse in try/except → degrade to fallback on ANY exception (incl. RecursionError); `_attach_x_sources` gains an injectable `now=` clock. - models.py: document that an image-only article legitimately has text=="". - Determinism: pin PYTHONHASHSEED in scripts/check.sh + quality.yml; mock trafilatura in the structured-path test. - Tests: decoy-blocks-key gate, canonical-URL preference, dropped-block warning, captured-but-0-blocks warning, truncation warning, parser-exception fallback, injectable-clock. README: "reads like a blogpost" → "can later render as one". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rade, not crash) The truncation diagnostic runs on the structured SUCCESS path; a trafilatura failure there must not discard the already-built good structured body. Wrap the trafilatura.extract call in try/except → skip the check (DEBUG), keep the body. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lidator, observability, tripwire) Structured X-article capture: intercept the article-content GraphQL response and parse its Draft.js content_state into an ordered ArticleBlock body (text runs + MediaPhotoPending inline images, in document order); degrade to the retained trafilatura.extract text-only fallback on any interception/parse miss (never a crash, never a partial body masquerading as complete). The flattened `text` is enforced to equal the concatenation of the text runs by a ContentSourceSuccess model_validator. x_article re-enrich hygiene: _attach_x_sources bumps fetched_at only on a material change (reused fingerprint). Observability + a truncation tripwire surface silent degradation; the parser degrades on any exception. Fixture + article GraphQL op-name are CONSTRUCTED (no real captured payload) — validate against a real bookmarked-Article response before production reliance (RFC #39 open-Q #4). PR3 attaches blocks with pending images; the image DOWNLOAD is PR4 and the blogpost RENDER is PR5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nerator, docs scope-down Fix round addressing PR4 reviews: - Dedicated MediaReport.article_images_skipped counter (stop folding article skips into photos_skipped_already_downloaded); surfaced in SUMMARY + CLI echo. - Thread --limit into _iter_eligible_article_images (top-of-iteration budget check, matching the photo generator) so a spent budget no longer scans past and miscounts skips. - Docs: scope down two render/mirror overclaims (article render/mirror is PR5, not PR4) in README + ARCHITECTURE; reconcile skip-counter wording in CLAUDE. - Tests: article-only total-failure raises; two-x_article-source global index continuation; limit-exhaustion skip-count parity; dedicated skip-counter assertions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… branch Address the PR5 review panel (all non-blocking): - Skip the bare `## Content:` heading when an x_article's blocks render to nothing (e.g. an all-pending image-only Article), mirroring `_video_digest_lines`' empty-block guard. - Add a test for the video-variant-on-article-image defensive branch (logged + skipped, no embed, no crash). - Strengthen the pending-image-silent test (no `![[` embed, no dead URL leaks). - Tighten `_article_caption_lines` type hint to `MediaPhotoDownloaded | MediaPhotoDescribed`. - Fix ARCHITECTURE wording: the failed-image line is a `>` blockquote, and note the empty-body heading suppression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…stics Post-review tightening on the #39 chain (all reviewers cleared; small, scoped): - architect M1: hoist the "\n\n" inter-paragraph separator to a single source of truth `models.ARTICLE_PARAGRAPH_SEP`, imported by both the producer (extract.article) and the consumer (generate). The two independent constants coupled only by a comment could drift; now they can't. models is the shared home (both already depend on it; no new generate->extract edge). - data-safety #1: the navigation `except` in fetch_x `_fetch_tweet` / `_fetch_rendered` bucketed every failure as `timeout` and discarded `str(exc)`. Keep the `timeout` classification but capture a capped `str(exc)` into `error` (new `_nav_error`, capped at `_MAX_ERROR_LEN`, mirroring media._format_error) and add `logger.warning(..., exc_info=True)` so a real nav failure on the unvalidated article path is debuggable. - data-safety #4: `_fetch_tweet`'s `on_response` swallowed a non-JSON body with a bare `pass`; now a DEBUG log with the url, matching `_fetch_rendered`. - quality-gates: add the `_needs_x_fetch` `until`-upper-bound test (created_at > until => not fetched), closing the one untested pure-logic branch. No failure_reason reclassification, no render/model/media behavior change. Gate green in-worktree: 1032 tests, 95% coverage, radon A/B. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39.
Captures X (Twitter) long-form Articles as ordered Obsidian blogposts — text and inline images interleaved where the author placed them — instead of the previous text-only trafilatura dump. Delivered as a 5-PR chain merged onto this umbrella branch, then rebased-forward onto
develop.The chain (extract → fetch/blocks → media → generate)
ArticleBlocktagged union (ArticleTextBlock|ArticleImageBlock) carried onContentSourceSuccess.blocks. Additive field: a legacy record simply has noblockskey and defaults to[]. Invariant: the ordered concatenation of the text-run texts equals the source's flattenedtext(soenrich/topicsconsumetextunchanged).article.article_results.result.rest_id).content_stateinto orderedblocks(extract.article); atext == "".join(block texts)validator + a truncation tripwire guard against a partial capture masquerading as a complete body.MediaEntryphoto-state union asMediaPhotoPending, so the provenxbrain mediadownload engine, validators, and_media/mirroring all apply with zero new plumbing.blocksin order under a## <header>: <title>heading; text paragraphs + inline![[_media/…]]embeds. Empty/image-only-pending bodies suppress the bare heading.Reuse-first
Built on what already existed rather than parallel machinery: the photo download engine +
MediaEntryunion (PR4),_mirror_filefor_media/mirroring at generate time (PR5), the_sources_materially_equalfingerprint + injectable clock in_attach_x_sources(PR3).enrich/topicsare unchanged — they still readtext. The whole feature is back-compat additive: no on-wire migration, text-only fallback preserved end-to-end.Post-review fix round (this branch, commit
ac46002)"\n\n"inter-paragraph separator into a single source of truth (models.ARTICLE_PARAGRAPH_SEP), imported by both producer and consumer so they can't drift.fetch_x: keep thetimeoutclassification but capture a cappedstr(exc)intoerror+logger.warning(exc_info=True)(the article path is the still-unvalidated one)._fetch_tweet's non-JSON-body swallow to a DEBUG log (was a barepass)._needs_x_fetchuntil-upper-bound branch.The X-GraphQL Article-entity path, the Draft.js
content_stateshape, and the intercepted GraphQL op-name are all pinned against CONSTRUCTED fixtures — they are UNVALIDATED against a real bookmarked-Article payload (RFC #39 open-Q #4).This is safe to land: on any shape/op-name drift the parser yields no blocks and the fetch degrades cleanly to the text-only trafilatura path (never a crash, never a partial/wrong body masquerading as complete). But a real-payload validation is required before production reliance — tracked in a follow-up issue (capture one live payload, validate the entity path + content_state op-name/shape, commit it as a recorded fixture, and fold in the deferred card/unified_card variant).
Gate (green in an isolated worktree, own venv+src)
🤖 Generated with Claude Code