Skip to content

[#39] Capture X long-form Articles as blogposts#65

Merged
VGonPa merged 17 commits into
developfrom
feat/39-x-articles-blogpost
Jul 2, 2026
Merged

[#39] Capture X long-form Articles as blogposts#65
VGonPa merged 17 commits into
developfrom
feat/39-x-articles-blogpost

Conversation

@VGonPa

@VGonPa VGonPa commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Closes #39.

Captures X (Twitter) long-form Articles as ordered Obsidian blogposts — text and inline images interleaved where the author placed them — instead of the previous text-only trafilatura dump. Delivered as a 5-PR chain merged onto this umbrella branch, then rebased-forward onto develop.

The chain (extract → fetch/blocks → media → generate)

  1. PR1 — model. New ArticleBlock tagged union (ArticleTextBlock | ArticleImageBlock) carried on ContentSourceSuccess.blocks. Additive field: a legacy record simply has no blocks key and defaults to []. Invariant: the ordered concatenation of the text-run texts equals the source's flattened text (so enrich/topics consume text unchanged).
  2. PR2 — extract the Article-entity link. Resolve the article entity path in the X GraphQL payload (article.article_results.result.rest_id).
  3. PR3 — fetch structured ordered blocks. Intercept the article-content GraphQL response and parse its Draft.js content_state into ordered blocks (extract.article); a text == "".join(block texts) validator + a truncation tripwire guard against a partial capture masquerading as a complete body.
  4. PR4 — media: inline-image download. Each inline image wraps the existing MediaEntry photo-state union as MediaPhotoPending, so the proven xbrain media download engine, validators, and _media/ mirroring all apply with zero new plumbing.
  5. PR5 — generate: blogpost render. Walk blocks in order under a ## <header>: <title> heading; text paragraphs + inline ![[_media/…]] embeds. Empty/image-only-pending bodies suppress the bare heading.

Reuse-first

Built on what already existed rather than parallel machinery: the photo download engine + MediaEntry union (PR4), _mirror_file for _media/ mirroring at generate time (PR5), the _sources_materially_equal fingerprint + injectable clock in _attach_x_sources (PR3). enrich/topics are unchanged — they still read text. The whole feature is back-compat additive: no on-wire migration, text-only fallback preserved end-to-end.

Post-review fix round (this branch, commit ac46002)

  • Consolidated the duplicated "\n\n" inter-paragraph separator into a single source of truth (models.ARTICLE_PARAGRAPH_SEP), imported by both producer and consumer so they can't drift.
  • Navigation-failure diagnostics in fetch_x: keep the timeout classification but capture a capped str(exc) into error + logger.warning(exc_info=True) (the article path is the still-unvalidated one).
  • Aligned _fetch_tweet's non-JSON-body swallow to a DEBUG log (was a bare pass).
  • Closed the untested _needs_x_fetch until-upper-bound branch.

⚠️ CAVEAT — read before relying on this in production

The X-GraphQL Article-entity path, the Draft.js content_state shape, and the intercepted GraphQL op-name are all pinned against CONSTRUCTED fixtures — they are UNVALIDATED against a real bookmarked-Article payload (RFC #39 open-Q #4).

This is safe to land: on any shape/op-name drift the parser yields no blocks and the fetch degrades cleanly to the text-only trafilatura path (never a crash, never a partial/wrong body masquerading as complete). But a real-payload validation is required before production reliance — tracked in a follow-up issue (capture one live payload, validate the entity path + content_state op-name/shape, commit it as a recorded fixture, and fold in the deferred card/unified_card variant).

Gate (green in an isolated worktree, own venv+src)

  • 1053 tests passed
  • ~95% coverage (gate floor 78%)
  • Radon all A/B; Ruff (lint+format), Mypy, Bandit, Detect-secrets, Deptry all PASS

🤖 Generated with Claude Code

VGonPa and others added 17 commits July 2, 2026 11:06
Add an additive, ordered `blocks: list[ArticleBlock]` field to
`ContentSourceSuccess`, mirroring exactly how `frames`/`has_speech`/
`language` were added for `x_video` in #44 — no new top-level model, no
parallel store. `ArticleBlock` is a `kind`-discriminated union:

  - `ArticleTextBlock(kind="text", text)`
  - `ArticleImageBlock(kind="image", media: MediaEntry, alt=None)`

The image block WRAPS the existing `MediaEntry` photo-state union, so the
photo download engine, the `_reject_local_path_traversal` /
`_require_utc_aware` validators, the `_media/` mirror and a future
describe path all apply with zero new plumbing. `text` stays the source
of truth for the flattened body (= concatenation of the text blocks), so
enrich/topics/generate's fallback consume it unchanged.

Purely additive + back-compat: legacy `items.json` (no `blocks` key,
pre-#20 `ok:`/legacy-media shapes) loads unchanged, defaulting `blocks`
to `[]`. Model-only PR: no extract/fetch/media/generate behaviour change.

Adds `ArticleBlockAdapter` (out-of-`Item` validation, mirrors
`MediaEntryAdapter`). Docs: ARCHITECTURE data-model invariant #12 + the
`x_article` note; CLAUDE.md model-seam bullet.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mageBlock)

Additive `blocks: list[ArticleBlock]` on ContentSourceSuccess; the image
block wraps the existing MediaEntry photo-state union so the download
engine + validators + _media/ mirror apply with no new plumbing. `text`
stays the flattened body (unchanged), so enrich/topics/generate's
fallback consume it as before. Purely additive + back-compat (proven
lossless on 29k real records) — legacy items.json loads with blocks=[].
Model-only; the producer (fetch), download walk (media) and blogpost
renderer (generate) land in PR2-5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ards

Address PR2 review round (all non-blocking):
- dedup the synthesized article link against a canonical URL key
  (host aliases → x.com, drop scheme + trailing slash) so a twitter.com /
  http / trailing-slash variant in entities.urls no longer double-adds;
- only synthesize for an Article __typename (reject a Card carrying a rest_id);
- require a numeric-string rest_id (kills the non-scalar garbage-URL vector);
- keep _extract_links at radon A by hoisting the append into a helper;
- note the card/unified_card variant as a conscious deferral folded into the
  same real-payload validation step (docstring + ARCHITECTURE).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract detects the X long-form Article entity on a tweet result
(article.article_results.result.rest_id, anchored via _dig) and synthesizes its
canonical https://x.com/i/article/<id> Link, routing through the existing
rendered-fetch path with no fetch_x change. Normalized dedup against
entities.urls (host aliases → x.com, scheme + trailing slash insensitive) plus
__typename ("Article"-only) and numeric-string rest_id guards prevent duplicate
or garbage links. The Article GraphQL fixture is constructed, not a recorded
live payload — validate the key path (and the deferred card/unified_card
variant) against a real bookmarked-Article response before production reliance
(RFC #39 open-Q #4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ant)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ot-crash, determinism pin

- article.py: WARN on a dropped entity-bearing block (real-payload key-drift
  visibility); `_find_url_by_key` now prefers the canonical CDN key globally
  (deep media_url_https beats a shallow bare url); decompose _coerce_content_state.
- fetch_x.py: INFO structured/fallback, WARN on captured-but-0-blocks (silent-death
  signal) + on structured-body truncation vs trafilatura (<50%); DEBUG the .json()
  swallow; wrap parse in try/except → degrade to fallback on ANY exception
  (incl. RecursionError); `_attach_x_sources` gains an injectable `now=` clock.
- models.py: document that an image-only article legitimately has text=="".
- Determinism: pin PYTHONHASHSEED in scripts/check.sh + quality.yml; mock
  trafilatura in the structured-path test.
- Tests: decoy-blocks-key gate, canonical-URL preference, dropped-block warning,
  captured-but-0-blocks warning, truncation warning, parser-exception fallback,
  injectable-clock. README: "reads like a blogpost" → "can later render as one".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rade, not crash)

The truncation diagnostic runs on the structured SUCCESS path; a trafilatura
failure there must not discard the already-built good structured body. Wrap the
trafilatura.extract call in try/except → skip the check (DEBUG), keep the body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lidator, observability, tripwire)

Structured X-article capture: intercept the article-content GraphQL response and
parse its Draft.js content_state into an ordered ArticleBlock body (text runs +
MediaPhotoPending inline images, in document order); degrade to the retained
trafilatura.extract text-only fallback on any interception/parse miss (never a
crash, never a partial body masquerading as complete). The flattened `text` is
enforced to equal the concatenation of the text runs by a ContentSourceSuccess
model_validator. x_article re-enrich hygiene: _attach_x_sources bumps fetched_at
only on a material change (reused fingerprint). Observability + a truncation
tripwire surface silent degradation; the parser degrades on any exception.

Fixture + article GraphQL op-name are CONSTRUCTED (no real captured payload) —
validate against a real bookmarked-Article response before production reliance
(RFC #39 open-Q #4). PR3 attaches blocks with pending images; the image DOWNLOAD
is PR4 and the blogpost RENDER is PR5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nerator, docs scope-down

Fix round addressing PR4 reviews:
- Dedicated MediaReport.article_images_skipped counter (stop folding article
  skips into photos_skipped_already_downloaded); surfaced in SUMMARY + CLI echo.
- Thread --limit into _iter_eligible_article_images (top-of-iteration budget
  check, matching the photo generator) so a spent budget no longer scans past
  and miscounts skips.
- Docs: scope down two render/mirror overclaims (article render/mirror is PR5,
  not PR4) in README + ARCHITECTURE; reconcile skip-counter wording in CLAUDE.
- Tests: article-only total-failure raises; two-x_article-source global index
  continuation; limit-exhaustion skip-count parity; dedicated skip-counter
  assertions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… branch

Address the PR5 review panel (all non-blocking):
- Skip the bare `## Content:` heading when an x_article's blocks render to
  nothing (e.g. an all-pending image-only Article), mirroring
  `_video_digest_lines`' empty-block guard.
- Add a test for the video-variant-on-article-image defensive branch (logged +
  skipped, no embed, no crash).
- Strengthen the pending-image-silent test (no `![[` embed, no dead URL leaks).
- Tighten `_article_caption_lines` type hint to `MediaPhotoDownloaded | MediaPhotoDescribed`.
- Fix ARCHITECTURE wording: the failed-image line is a `>` blockquote, and note
  the empty-body heading suppression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ages

Closes the render stage. The full #39 chain (extract→fetch/blocks→media→
generate) is now complete end-to-end. Text-only fallback preserved. The
constructed-fixture caveat (open-Q #4) still stands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…stics

Post-review tightening on the #39 chain (all reviewers cleared; small, scoped):

- architect M1: hoist the "\n\n" inter-paragraph separator to a single source
  of truth `models.ARTICLE_PARAGRAPH_SEP`, imported by both the producer
  (extract.article) and the consumer (generate). The two independent constants
  coupled only by a comment could drift; now they can't. models is the shared
  home (both already depend on it; no new generate->extract edge).
- data-safety #1: the navigation `except` in fetch_x `_fetch_tweet` /
  `_fetch_rendered` bucketed every failure as `timeout` and discarded
  `str(exc)`. Keep the `timeout` classification but capture a capped `str(exc)`
  into `error` (new `_nav_error`, capped at `_MAX_ERROR_LEN`, mirroring
  media._format_error) and add `logger.warning(..., exc_info=True)` so a real
  nav failure on the unvalidated article path is debuggable.
- data-safety #4: `_fetch_tweet`'s `on_response` swallowed a non-JSON body with
  a bare `pass`; now a DEBUG log with the url, matching `_fetch_rendered`.
- quality-gates: add the `_needs_x_fetch` `until`-upper-bound test (created_at >
  until => not fetched), closing the one untested pure-logic branch.

No failure_reason reclassification, no render/model/media behavior change.
Gate green in-worktree: 1032 tests, 95% coverage, radon A/B.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant