Long-form media digest: transcribe + summarize bookmarked videos (and podcasts) into the vault

[VGonPa]

Status: PROPOSAL / RFC — a sketch to align on direction, not a committed
spec. Names, boundaries and phasing are open for the maintainer to decide.

Summary

xbrain captures long-form media bookmarks — talks, interviews, tutorials,
podcasts — but does nothing with what they actually contain. The natural next
step after the #40 video-capture work is to turn an unwatched video (or audio
podcast) into a readable, searchable digest in the vault, so the consumption
side of the brain actually gets consumed.

This is the same "bookmark graveyard" the README already names for text, but
sharper. A saved tweet is one scroll from being re-read. A bookmarked 72-minute
talk is, in practice, never reopened — the cost of re-entry is too high. The
text half of xbrain solves "I saved it but never read it." This proposal extends
the same cure to "I saved it but never watched it."

Problem

• After Capture playable video URL (video_info.variants), not just the poster #40, video bookmarks now carry a real playable mp4 URL (refresh-media
  backfilled ~225 existing items; download-videos fetches the bytes). But the
  pipeline stops at capture — the content of the video is still locked inside a
  file nobody opens.
• A long video is the worst-case graveyard item: high value (you saved it for a
  reason), highest re-entry cost (you have to find 72 minutes and a quiet room).
  The probability it gets revisited is far lower than for a saved thread.
• The corpus makes "just download and keep them" a non-starter. Diagnostic
  finding from the Capture playable video URL (video_info.variants), not just the poster #40 work: 225 mp4 videos, 0 HLS, ~140 GB total. Keeping
  that on disk to browse is absurd. Processing has to be ephemeral — one
  video at a time, bytes discarded after the digest is written.

Proposed pipeline (ephemeral, one video at a time)

1. Fetch the video to a temp dir — the planned fetch-video --to <dir> (the
   agent-driven selection/fetch surface), or the existing download-videos.
   Reuses video_media.py.
2. Transcribe with local ASR on Apple Silicon.
   • Best-quality local option observed: NVIDIA Parakeet TDT 0.6b (the model
     the FluidVoice app uses via FluidAudio/CoreML). From Python it runs through
     parakeet-mlx (MLX/Metal).
   • whisper / faster-whisper is the cross-platform fallback.
   • Real-world caveat already observed in the corpus: many X videos have no
     audio track or no speech (silent clips, screen-only loops). The pipeline
     MUST detect this and skip gracefully — no spurious empty digest, no
     crash, just a recorded "no speech" outcome.
3. (Phase 2 — visual key-frames, explicitly deferrable.) Use ffmpeg
   scene-change detection to grab frames where the screen changes (new slide,
   code, diagram, demo), then a vision model describes each. For slide- and
   screen-heavy talks the visual carries as much signal as the audio. The MVP
   is transcript-only highlights; key-frames + vision are a clearly-flagged later
   enhancement, not part of the first cut.
4. Synthesize a structured digest note: key points with timestamps,
   written into the vault like the other xbrain notes. In Phase 2 each highlight
   also carries the key screenshot + what was said + what was shown.
5. Discard the video bytes. Keep the transcript + digest (+ key frames in
   Phase 2).

Dedup by video (one video → one digest → many items)

The same underlying video is often bookmarked by multiple posts. Real example
in the corpus: the Andrew Ng talk is bookmarked twice —
2068763235587694769 (@0xmovez) and 2069122453641523495 (@AnatoliKopadze) —
same video, two items.

The module MUST key on the video identity, not the post. Today the closest
stable handle is the resolved mp4 URL from video_info.variants (ideally the
underlying amplify_video / media_key, if we start capturing it in extract).
Keyed that way, the module:

1. fetches + transcribes the video once, not once per referencing post;
2. produces one digest;
3. links that one digest to all the bookmark items that reference the same
   video — and each of those item notes back-links the digest.

Concretely, the shared transcript is the content source of record, and each
referencing item points at it, so generate renders every one of those notes
with the same digest (and cross-links them in the graph) without re-doing any ML.

Architecture note (the important decision)

Keep xbrain mechanical — list and fetch the videos. The heavy ML (ASR,
vision) and the LLM synthesis are agent-side / external tooling, not baked
into xbrain core. A small CLI should not drag in a heavy MLX/CoreML/ffmpeg
dependency chain.

• Option A (recommended): xbrain stays fetch/list only. An external tool or
  agent does transcribe + digest, consuming xbrain's list-videos /
  fetch-video surface. Mirrors the existing worksheet hand-off, where the LLM
  work lives outside the CLI and needs no paid API.
• Option B: a thin xbrain digest-video command that shells out to an
  external transcriber — convenience wrapper, ML still external.

Recommendation: A, or A-then-maybe-B once the agent-side flow is proven.
Maintainer's call — see open questions.

Generalize: long-form media digest, video first

The same fetch → transcribe → digest shape applies to audio podcasts and
any long-form media bookmark, not just video. Frame the module as a long-form
media digest with video as the first target; audio is a near-free
extension (skip the visual phase, transcribe + digest only).

Integration with the existing pipeline (transcript as a content source)

This is the heart of the proposal: the video module manufactures text, and
everything downstream is xbrain's existing machinery, reused unchanged.

• The module's output — the transcript (plus the optional Phase-2 visual digest)
  — attaches to the bookmarked item as a content source: a new
  ContentKind (e.g. "x_video") added to the ContentKind literal in
  models.py, carried as a ContentSourceSuccess(kind="x_video", text=...) on
  the item's Content.sources. This is exactly how fetch already attaches
  external_article / x_article body text today, and it is the same pattern
  as Phase B: Describe images with vision LLM and feed into enrich #34 (Phase B), which injects image descriptions into the enrich input.
  This proposal is, structurally, "Phase B for video".
• Once the transcript is attached, the existing pipeline runs unchanged:
  vocab → enrich → topics now see the full transcript instead of a 2-line
  tweet, so they assign a real primary_topic and a real summary; generate
  renders the note with the digest (and, in Phase 2, the key-frame screenshots
  embedded inline, the same way downloaded photos already render).
• This is also why video items currently show topic "—": today enrich only
  sees the ~2-line tweet text accompanying the video, so there is nothing to
  topic. Attaching the transcript fixes the topic-faceting gap that is listed
  below as an open question — they are the same problem, resolved by the same
  move.
• Net framing: a bookmarked video becomes "a long post" — a long-text,
  topic-linked, searchable vault note produced by the same machinery as an
  article. The video module only manufactures the text; topics, summaries,
  note rendering, and graph links are all reused as-is. Minimal new surface,
  maximal reuse.
• This also settles the "where does the transcript/digest live" open question:
  on the item, as a content source (rendered into that item's note by
  generate) — not a separate transcripts store.

Scope

In (MVP — Phase 1)

• Ephemeral fetch → local ASR transcript → transcript-only highlights digest note
  in the vault, with timestamps.
• Graceful no-audio / no-speech detection and skip.
• One video at a time; bytes discarded after digest.
• xbrain stays mechanical (Option A): list + fetch only.

Phase 2 (deferred, flagged as enhancement)

• ffmpeg scene-change key-frames + vision-model descriptions.
• Highlights enriched with screenshot + said/shown.

Out / later

• Keeping the video corpus on disk (explicitly rejected — ~140 GB).
• Baking ML into xbrain core.
• Audio podcast pass (trivial once Phase 1 lands; do video first).

Dependencies / relation

• Builds directly on Capture playable video URL (video_info.variants), not just the poster #40 (video capture/download — merged, PRs [#40] Capture playable video URL from video_info.variants (part 1) #41/[#40] Add xbrain refresh-media: backfill playable video on existing items #42/[#40] Download mp4 video bytes: xbrain download-videos (part 2) #43).
• Reuses video_media.py for fetching.
• Complements the planned list-videos + fetch-video (the agent-driven
  selection/fetch surface this digest flow would consume).
• Context: corpus diagnostic from Capture playable video URL (video_info.variants), not just the poster #40 — 225 mp4, 0 HLS, ~140 GB — is why
  ephemeral processing is non-negotiable.

Open questions

• Storage (leaning resolved — see Integration above): attach the transcript
  to the item as a content source so it renders into the item's existing note,
  rather than a separate transcripts store. Open sub-question: do we also keep
  the raw verbatim transcript somewhere, or only the synthesized digest text on
  the item?
• ASR engine: parakeet-mlx (best local quality, Apple-Silicon-only) vs
  whisper/faster-whisper (portable)? And wire it agent-side (Option A) or
  as a thin xbrain digest-video command (Option B)?
• Visual phase: are ffmpeg key-frames + vision worth the complexity, or is a
  transcript-only digest already enough signal for most saved talks?
• Dedup key: is the resolved mp4 URL a stable-enough video identity, or
  should extract start capturing the amplify_video / media_key to key on
  (URLs can rotate signing params)? See Dedup by video above.
• Topics / faceting (mechanism resolved by Integration above — attaching
  the transcript lets enrich/topics assign a real primary_topic):
  remaining question is operational — do we re-run enrich/topics over
  digested video items in the same pass, or as a follow-up backfill (mirroring
  how refresh-media backfilled Capture playable video URL (video_info.variants), not just the poster #40)?

---

Drafted as an RFC from the #40 video work + a read-only code audit, 2026-06-30.