From 5d1730b8fc130024e8c148ea439976e692506d36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 08:03:17 +0200 Subject: [PATCH 1/6] test(#39): add real-payload fixtures + failing inline-MEDIA test Ground-truth fixtures trimmed verbatim from the 3 real captured Articles (article_results.result subtree). First failing test: art-OpenWiki must resolve its atomic MEDIA block to the inline CDN image via the LIST entityMap + media_entities indirection (currently yields zero images). Co-Authored-By: Claude Opus 4.8 --- tests/fixtures/art-Headcount_AI.json | 189 ++++++++++ tests/fixtures/art-OpenWiki.json | 529 +++++++++++++++++++++++++++ tests/fixtures/art-Wiki_Memory.json | 437 ++++++++++++++++++++++ tests/test_article_real.py | 58 +++ 4 files changed, 1213 insertions(+) create mode 100644 tests/fixtures/art-Headcount_AI.json create mode 100644 tests/fixtures/art-OpenWiki.json create mode 100644 tests/fixtures/art-Wiki_Memory.json create mode 100644 tests/test_article_real.py diff --git a/tests/fixtures/art-Headcount_AI.json b/tests/fixtures/art-Headcount_AI.json new file mode 100644 index 0000000..e15f2f6 --- /dev/null +++ b/tests/fixtures/art-Headcount_AI.json @@ -0,0 +1,189 @@ +{ + "content_state": { + "blocks": [ + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "espbg", + "text": "By Leo Bashlykov, Head of Product · Crypto", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "agevm", + "text": "Everyone's writing about how AI is shrinking product teams and killing product jobs. In my case, I'm fighting to hire more.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2ajug", + "text": "The dominant narrative is clear by now: Brian Chesky (Airbnb) says pure people managers have no future. Tomer Cohen killed LinkedIn's APM program and replaced it with Full Stack Builders. Tobias Lütke told Shopify to prove AI can't do the job before asking for more headcount.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "5rcp7", + "text": "All real. All worth taking seriously. Proving AI can't do it before asking for headcount is already part of Revolut's process.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1vbq7", + "text": "But here's what AI actually did to my teams in Crypto at Revolut: it didn't reduce my need for people. It exposed how badly under-resourced my ambition was.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "cgpjm", + "text": "We're running 5 new big bets in parallel right now. If I had one more strong PO tomorrow, I'd start a 6th and a 7th. The constraint was never \"do we have enough work to keep teams busy\" — it was always \"do we have enough strong POs to point at the next opportunity.\"", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "bskeh", + "text": "What AI actually changed:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "5iplo", + "text": "PRD (product requirements document) prep / review / iteration — 5x faster.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "8q2j5", + "text": "Regulatory research — every PO can now dig into the depth of each regime and navigate the product accordingly.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "dnhll", + "text": "Coding — talented engineers shipping 10x faster, no comments needed.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2ri2v", + "text": "Automations — nearly in every workflow → less routine nonsense.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "7ivrt", + "text": "Concrete output — launching our MCP (Model Context Protocol) server for Revolut X as a side project. Without AI-driven release capacity and simple building tools, it wouldn't have shipped.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "fafc8", + "text": "What we haven't cracked yet — governance. And this is non-negotiable in a regulated business like Revolut. Anyone selling you \"AI replaces compliance review\" is selling you a future lawsuit.", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "3o5p0", + "text": "So, my take:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "5rteg", + "text": "→ AI fluency is now the baseline. Agree with Tobias Lütke here — stagnation is slow-motion failure. 70% of skills in tech jobs will change by 2030, no argument there.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "f4h82", + "text": "→ But the narrative that AI = smaller teams is mostly a story from consumer SaaS and dev tools. From where I sit, AI is doing something different: it's enabling top talent to operate at 3-5x capacity, which means I can finally chase the bets that were previously beyond reach.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "fpv2d", + "text": "→ We still have 18 months of strong items in the backlog. I'm all in on AI to compress that to 3 months, so we can move on to brainstorming what comes next and drive the next revolution in the space.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 0, + "length": 1, + "offset": 0 + } + ], + "inlineStyleRanges": [], + "key": "8eu8q", + "text": "💡 The real question isn't \"how many people will AI replace?\"", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2e15a", + "text": "It's: how big is your ambition, and do you have the builders to match it?", + "type": "unstyled" + } + ], + "entityMap": [ + { + "key": "0", + "value": { + "data": { + "url": "https://abs.twimg.com/emoji/v2/svg/1f4a1.svg" + }, + "mutability": "Immutable", + "type": "TWEMOJI" + } + } + ] + }, + "id": "QXJ0aWNsZUVudGl0eToyMDcxODkyMjg2MzE3NzI3NzQ0", + "is_grok_summary_eligible": true, + "lifecycle_state": { + "modified_at_secs": 1782816104 + }, + "media_entities": [], + "metadata": { + "first_published_at_secs": 1782816104 + }, + "preview_text": "By Leo Bashlykov, Head of Product · Crypto\nEveryone's writing about how AI is shrinking product teams and killing product jobs. In my case, I'm fighting to hire more.\nThe dominant narrative is clear", + "rest_id": "2071892286317727744", + "summary_text": "- I’m fighting to hire more in Crypto at Revolut as AI exposes how under-resourced my ambition was.\n- AI makes PRD prep 5x faster, regulatory research deeper, coding 10x quicker, and launches side projects like MCP server possible.\n- We run 5 big bets in parallel; one strong PO would unlock a 6th and 7th.\n- AI enables top talent to operate at 3-5x capacity, compressing backlog and chasing previously unreachable bets.\n- The real question is how big your ambition is and whether you have the builders to match it.", + "title": "The case for headcount in the age of AI" +} diff --git a/tests/fixtures/art-OpenWiki.json b/tests/fixtures/art-OpenWiki.json new file mode 100644 index 0000000..dacc037 --- /dev/null +++ b/tests/fixtures/art-OpenWiki.json @@ -0,0 +1,529 @@ +{ + "content_state": { + "blocks": [ + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1a3kl", + "text": "Today we're releasing OpenWiki, an open source agent and CLI for generating and maintaining documentation for codebases.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "63p07", + "text": "Agents write better code when they understand the repo they're working in. They need to know where key logic lives, how files connect, and which patterns the codebase expects. Good documentation gives agents that context, which leads to more informed code changes and fewer avoidable mistakes.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "14csu", + "text": "The problem is that documentation is hard to keep current. Writing the initial docs takes time, and updating them every time the code changes is even harder. In large repos with frequent PRs, docs can fall out of date quickly.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "bh2cd", + "text": "OpenWiki handles that work automatically. It creates a wiki for your repo, connects that wiki to your coding agent, and keeps it updated as your code changes.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 20, + "offset": 0, + "style": "Bold" + } + ], + "key": "8186u", + "text": "Why wikis for agents", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 0, + "length": 8, + "offset": 67 + }, + { + "key": 1, + "length": 8, + "offset": 77 + }, + { + "key": 2, + "length": 19, + "offset": 91 + } + ], + "inlineStyleRanges": [], + "key": "8oqju", + "text": "We were inspired by existing work around codebase wikis, including DeepWiki, AutoWiki, and Karpathy’s LLM Wiki concept. The shared idea is simple. A wiki gives humans and agents a structured way to understand a codebase without forcing all context into one giant file.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2h0ij", + "text": "That matters because most coding agents already read files like AGENTS.md or CLAUDE.md for instructions. Those files are useful, but they’re not the right place to store hundreds of pages of repo documentation. They should point the agent toward the right context, then let the agent retrieve what it needs.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "8vn9b", + "text": "OpenWiki follows that model. It generates a repo wiki, then updates your agent instruction files with a reference to that wiki. From there, your coding agent can discover and use the docs automatically.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 15, + "offset": 0, + "style": "Bold" + } + ], + "key": "f2dt1", + "text": "Getting started", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "4kbn4", + "text": "OpenWiki is designed to be easy to run from the command line.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "31inp", + "text": "Install it with NPM:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2usuh", + "text": "npm install -g openwiki", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "162a4", + "text": "then run:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "4dlft", + "text": "openwiki --init", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 3, + "length": 1, + "offset": 0 + } + ], + "inlineStyleRanges": [], + "key": "6ene2", + "text": " ", + "type": "atomic" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1fdqh", + "text": "The init command asks for a model provider and API key, then generates documentation for your repo.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1qi44", + "text": "OpenWiki supports both open and closed model providers, including OpenRouter, Fireworks, Baseten, OpenAI, and Anthropic. By default, it uses OpenRouter with an open model, but you can configure the provider that works best for your setup.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 4, + "length": 10, + "offset": 36 + }, + { + "key": 5, + "length": 9, + "offset": 76 + } + ], + "inlineStyleRanges": [], + "key": "1ve3e", + "text": "Because OpenWiki is built on top of DeepAgents, it also supports tracing to LangSmith. If you provide a LangSmith API key, OpenWiki will trace runs to a LangSmith project so you can inspect exactly what the agent did while generating or updating your docs.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 42, + "offset": 0, + "style": "Bold" + } + ], + "key": "dc1ji", + "text": "How OpenWiki connects to your coding agent", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "9jjqd", + "text": "After generating the wiki, OpenWiki updates your repo’s agent instruction files. If your repo uses AGENTS.md, CLAUDE.md, or both, OpenWiki adds a reference to the generated wiki and explains when the agent should use it.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "ecq9o", + "text": "We chose this approach because putting the entire wiki inside an instruction file would add too much context. In a large repo, the wiki can span hundreds of files. Loading all of that into every agent run would be wasteful and hard to maintain.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "ilp0", + "text": "A short reference works better. Your coding agent already reads the instruction file. Once OpenWiki adds the reference, the agent can find the wiki when it needs repo context, without requiring you to change your workflow.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 27, + "offset": 0, + "style": "Bold" + } + ], + "key": "nvu", + "text": "Keeping the wiki up to date", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "7gqs4", + "text": "Generating docs once is useful. Keeping them current is where OpenWiki becomes more valuable.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 6, + "length": 40, + "offset": 20 + } + ], + "inlineStyleRanges": [], + "key": "97flt", + "text": "OpenWiki includes a GitHub Action that can run on a schedule, for example once a day. The action runs OpenWiki with the update flag. OpenWiki checks which commits landed since the last run, uses git diffs to understand what changed, then updates the wiki with the relevant context.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "hf0f", + "text": "That means the workflow can run in the background. As your codebase changes, OpenWiki updates the documentation. Your coding agent keeps picking up the latest wiki through the existing instruction file reference.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 25, + "offset": 0, + "style": "Bold" + } + ], + "key": "f4ueu", + "text": "Built for codebases first", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "7v1vg", + "text": "This first release focuses on wikis for codebases. The goal is to make it easier for agents to understand the repos they work in, without asking developers to manually write and maintain detailed docs.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "dq2jp", + "text": "Over time, we think the OpenWiki concept can apply more broadly. Agents need durable context for many kinds of work, not just coding. Codebase documentation is the first use case, but the same pattern can help agents maintain useful context across other workflows too.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 6, + "offset": 0, + "style": "Bold" + } + ], + "key": "eluqq", + "text": "Try it", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "di84i", + "text": "OpenWiki is open source and available now.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "4nd7p", + "text": "You can install it, run `openwiki --init`, and generate a wiki for your repo in a few minutes.", + "type": "unstyled" + }, + { + "data": { + "urls": [ + { + "fromIndex": 25, + "text": "https://github.com/langchain-ai/openwiki", + "toIndex": 65 + }, + { + "fromIndex": 99, + "text": "https://youtu.be/nIVu3zfYprI", + "toIndex": 127 + } + ] + }, + "entityRanges": [ + { + "key": 7, + "length": 40, + "offset": 25 + } + ], + "inlineStyleRanges": [], + "key": "1a3cd", + "text": "Check out the repo here: https://github.com/langchain-ai/openwiki and watch the walkthrough video: https://youtu.be/nIVu3zfYprI", + "type": "unstyled" + } + ], + "entityMap": [ + { + "key": "4", + "value": { + "data": { + "url": "https://docs.langchain.com/oss/python/deepagents/overview" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "5", + "value": { + "data": { + "url": "https://langsmith.com/" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "6", + "value": { + "data": { + "url": "https://github.com/langchain-ai/openwiki/blob/main/examples/openwiki-update.yml" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "1", + "value": { + "data": { + "url": "https://docs.factory.ai/cli/features/wiki/overview" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "0", + "value": { + "data": { + "url": "https://deepwiki.com/" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "2", + "value": { + "data": { + "url": "https://x.com/karpathy/status/2040470801506541998" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "7", + "value": { + "data": { + "url": "https://github.com/langchain-ai/openwiki" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "3", + "value": { + "data": { + "entityKey": "c07613ca-f39c-4d08-8262-0046a1fad70f", + "mediaItems": [ + { + "localMediaId": "1", + "mediaCategory": "DraftTweetImage", + "mediaId": "2072374092653867008" + } + ] + }, + "mutability": "Immutable", + "type": "MEDIA" + } + } + ] + }, + "cover_media": { + "id": "QXBpTWVkaWE6DAAFCgABHMKNwxAbUAEKAAIAAAAAL4hatQAA", + "media_id": "2072374647472214017", + "media_info": { + "__typename": "ApiImage", + "color_info": { + "palette": [ + { + "percentage": 99.83, + "rgb": { + "blue": 1, + "green": 1, + "red": 1 + } + } + ] + }, + "original_img_height": 893, + "original_img_url": "https://pbs.twimg.com/media/HMKNwxAbUAEMrOF.jpg", + "original_img_width": 2232 + }, + "media_key": "3_2072374647472214017" + }, + "id": "QXJ0aWNsZUVudGl0eToyMDcyMzYxNDUwOTI1MDk2OTYx", + "is_grok_summary_eligible": true, + "lifecycle_state": { + "modified_at_secs": 1782927710 + }, + "media_entities": [ + { + "id": "QXBpTWVkaWE6DAAFCgABHMKNQeJbMAAKAAIAAAAAL4hatQAA", + "media_id": "2072374092653867008", + "media_info": { + "__typename": "ApiImage", + "color_info": { + "palette": [ + { + "percentage": 98.8, + "rgb": { + "blue": 0, + "green": 0, + "red": 0 + } + }, + { + "percentage": 1.01, + "rgb": { + "blue": 105, + "green": 105, + "red": 105 + } + }, + { + "percentage": 0.17, + "rgb": { + "blue": 35, + "green": 65, + "red": 49 + } + } + ] + }, + "original_img_height": 946, + "original_img_url": "https://pbs.twimg.com/media/HMKNQeJbMAA9ljZ.jpg", + "original_img_width": 1582 + }, + "media_key": "3_2072374092653867008" + } + ], + "metadata": { + "first_published_at_secs": 1782927710 + }, + "preview_text": "Today we're releasing OpenWiki, an open source agent and CLI for generating and maintaining documentation for codebases.\nAgents write better code when they understand the repo they're working in. They", + "rest_id": "2072361450925096961", + "summary_text": "- We're releasing OpenWiki, an open source agent and CLI for generating and maintaining codebase documentation.\n- Agents need repo context on key logic, file connections, and patterns to make informed code changes with fewer mistakes.\n- OpenWiki creates a wiki, connects it to coding agents via instruction file references, and keeps docs current automatically.\n- Install with npm, run openwiki --init to generate using supported models, and use GitHub Action for scheduled updates via git diffs.\n- Built first for codebases to give agents durable context without manual maintenance.", + "title": "Introducing OpenWiki, an open source agent for repo documentation" +} diff --git a/tests/fixtures/art-Wiki_Memory.json b/tests/fixtures/art-Wiki_Memory.json new file mode 100644 index 0000000..1a7c8ca --- /dev/null +++ b/tests/fixtures/art-Wiki_Memory.json @@ -0,0 +1,437 @@ +{ + "content_state": { + "blocks": [ + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 11, + "offset": 147, + "style": "Bold" + } + ], + "key": "6krvg", + "text": "Memory for agents is still early, with little to no standards. “Memory” means something different to everyone. But one common pattern is emerging: wiki memory.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "8v4se", + "text": "The idea is simple: use an agent to turn raw source data into a compact, persistent, agent-readable knowledge layer.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "8vi2h", + "text": "Why wikis?", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "bkgit", + "text": "Raw data contains a lot of knowledge, but it is often inefficient to expose directly to an agent. Logs, notes, code, docs, experiments, Slack threads, and transcripts are too noisy and too large. So instead, we run a process over that data and transform it into a denser representation.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1ia2h", + "text": "This is different from basic RAG. RAG usually retrieves raw chunks at query time. A wiki precomputes and maintains a higher-level synthesis, so the agent does not have to rediscover the structure every time.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2pp95", + "text": "This desire exists almost everywhere. When speaking to a friend at a research company, he talked about all the knowledge in their researchers' heads. He wanted to “clone their brain” so that if they left, the knowledge still remained with the company. His hope was that by looking at the experiments they ran, the notes they wrote, and the actions they took, they could approximate this “brain clone.”", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "639m0", + "text": "A wiki is one practical way to do that: not by storing everything, but by compressing what matters into a reusable knowledge base.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "a62ce", + "text": "What is a “wiki”?", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "3nfvl", + "text": "A wiki is an agent-maintained data structure that represents source knowledge in an agent-friendly way.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "aodgn", + "text": "In practice, this often means running an agent over some source material and asking it to create a set of files that future agents can use to understand the domain faster.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "abju9", + "text": "The important bit is not that it literally looks like Wikipedia. The important bit is that it is persistent, structured, inspectable, and updated over time.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "dgbn0", + "text": "Examples of wikis", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 0, + "length": 21, + "offset": 0 + } + ], + "inlineStyleRanges": [], + "key": "9p8s8", + "text": "DeepWiki by Cognition is probably the first example of this I remember seeing. DeepWiki creates AI-generated documentation for GitHub repositories. It is intended to give humans and coding agents a higher-level mental map of a codebase, making it easier to understand and navigate.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 1, + "length": 29, + "offset": 0 + } + ], + "inlineStyleRanges": [], + "key": "2s5pb", + "text": "Karpathy recently wrote about what he called an “LLM Wiki” or “LLM knowledge base.” This is a more general form of the same pattern: instead of only working over code, it can work over arbitrary source files. His framing is that the LLM incrementally builds and maintains a persistent markdown wiki that sits between the user and the raw sources.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 2, + "length": 25, + "offset": 0 + } + ], + "inlineStyleRanges": [], + "key": "d541l", + "text": "Factory launched AutoWiki as a similar offering to DeepWiki. AutoWiki analyzes a codebase and generates structured, browsable documentation that stays current as the repo changes.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [ + { + "key": 3, + "length": 7, + "offset": 55 + }, + { + "key": 4, + "length": 5, + "offset": 64 + }, + { + "key": 5, + "length": 4, + "offset": 71 + }, + { + "key": 6, + "length": 3, + "offset": 81 + } + ], + "inlineStyleRanges": [], + "key": "dtjma", + "text": "This pattern also sits adjacent to memory systems like LangMem, Letta, Mem0, and Zep. Those systems attack the broader agent-memory problem, while wiki memory is notable because it often uses the simplest possible substrate: files.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1qbbv", + "text": "A wiki for every domain", + "type": "header-two" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [ + { + "length": 3, + "offset": 125, + "style": "Bold" + } + ], + "key": "7vg9u", + "text": "I would argue that for every domain there exists a knowledge base you would be well served to create. This knowledge base is not just the raw data. It is an intelligently compressed version of the raw data.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "d532p", + "text": "There are a bunch of open questions here:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "75ilc", + "text": "What is the raw data?", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "ev8hd", + "text": "What is the best format for the compressed data?", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "9uh6d", + "text": "How should that data be compressed?", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "1v5fb", + "text": "How should the compressed representation stay up to date?", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "58j4o", + "text": "We are starting to see some common answers emerge:", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "afedk", + "text": "What is the raw data? → anything an agent can read or access", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "2ve0i", + "text": "What is the best format for the compressed data? → files", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "4mrn4", + "text": "How do you compress that data? → an agent", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "9o7j6", + "text": "How do you maintain it? → an agent", + "type": "unordered-list-item" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "8ehuq", + "text": "Files are attractive because they are inspectable, editable, versionable, and easy for agents to read and write.", + "type": "unstyled" + }, + { + "data": {}, + "entityRanges": [], + "inlineStyleRanges": [], + "key": "6a1b8", + "text": "Wikis are not all of memory. They are best for durable domain knowledge, not necessarily short-term conversation state, user preferences, or high-frequency event logs. But for many domains, wiki memory may be the simplest useful long-term memory pattern we have.", + "type": "unstyled" + } + ], + "entityMap": [ + { + "key": "4", + "value": { + "data": { + "url": "https://www.letta.com/blog/agent-memory" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "5", + "value": { + "data": { + "url": "https://arxiv.org/html/2504.19413v1" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "6", + "value": { + "data": { + "url": "https://www.getzep.com/" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "1", + "value": { + "data": { + "url": "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "0", + "value": { + "data": { + "url": "https://cognition.ai/blog/deepwiki" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "2", + "value": { + "data": { + "url": "https://factory.ai/news/wiki" + }, + "mutability": "Mutable", + "type": "LINK" + } + }, + { + "key": "3", + "value": { + "data": { + "url": "https://docs.langchain.com/oss/python/concepts/memory" + }, + "mutability": "Mutable", + "type": "LINK" + } + } + ] + }, + "cover_media": { + "id": "QXBpTWVkaWE6DAAFCgABHMEX45LWoAAKAAIAAAAAoqCtagAA", + "media_id": "2071963569755693056", + "media_info": { + "__typename": "ApiImage", + "color_info": { + "palette": [ + { + "percentage": 91.76, + "rgb": { + "blue": 0, + "green": 0, + "red": 0 + } + }, + { + "percentage": 7.29, + "rgb": { + "blue": 107, + "green": 106, + "red": 106 + } + }, + { + "percentage": 0.26, + "rgb": { + "blue": 187, + "green": 228, + "red": 228 + } + }, + { + "percentage": 0.21, + "rgb": { + "blue": 83, + "green": 169, + "red": 185 + } + }, + { + "percentage": 0.12, + "rgb": { + "blue": 84, + "green": 130, + "red": 86 + } + } + ] + }, + "original_img_height": 534, + "original_img_url": "https://pbs.twimg.com/media/HMEX45LWoAA7btu.jpg", + "original_img_width": 1334 + }, + "media_key": "3_2071963569755693056" + }, + "id": "QXJ0aWNsZUVudGl0eToyMDcxOTYzMjcyNzI3OTI4ODMz", + "is_grok_summary_eligible": true, + "lifecycle_state": { + "modified_at_secs": 1782829597 + }, + "media_entities": [], + "metadata": { + "first_published_at_secs": 1782829597 + }, + "preview_text": "Memory for agents is still early, with little to no standards. “Memory” means something different to everyone. But one common pattern is emerging: wiki memory.\nThe idea is simple: use an agent to turn", + "rest_id": "2071963272727928833", + "summary_text": "- Memory for agents lacks standards but wiki memory emerges as a pattern: agents turn raw noisy data like logs, notes, and code into compact persistent agent-readable knowledge layer. \n- Unlike RAG retrieving raw chunks, wiki precomputes higher-level synthesis so agents avoid rediscovering structure every time. \n- Wiki is agent-maintained persistent structured files representing source knowledge for faster domain understanding. \n- Examples include DeepWiki and AutoWiki for codebases plus LLM Wiki for arbitrary sources using markdown files. \n- For every domain create intelligently compressed knowledge base with agent compression and maintenance on simple file substrate for durable knowledge.", + "title": "Wiki Memory" +} diff --git a/tests/test_article_real.py b/tests/test_article_real.py new file mode 100644 index 0000000..cdc3142 --- /dev/null +++ b/tests/test_article_real.py @@ -0,0 +1,58 @@ +# tests/test_article_real.py +"""Ground-truth tests for the X-article parser against REAL captured payloads. + +Unlike `tests/test_article.py` (which pins CONSTRUCTED Draft.js shapes), every +fixture here is the `article_results.result` subtree trimmed VERBATIM from a real +bookmarked-Article GraphQL response (`payload[1]`), committed under +`tests/fixtures/art-*.json`. They are the ground truth for #39/#66: X's real +`entityMap` is a LIST keyed by `entry.key`, a `MEDIA` entity resolves its CDN URL +indirectly via `media_entities[].media_info.original_img_url`, and the lead image +lives in a separate `cover_media` sibling. The parser must resolve inline + cover +images here, render `## ` headings and `- ` bullets, and keep the flattened-text +invariant — degrading to text-only (never crashing) on any shape miss. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from xbrain.extract.article import parse_article_content_state +from xbrain.models import ArticleImageBlock, ArticleTextBlock, ContentSourceSuccess + +_FIXTURES = Path(__file__).parent / "fixtures" + +# Ground-truth image URLs (from the real captured payloads). +_OPENWIKI_COVER = "https://pbs.twimg.com/media/HMKNwxAbUAEMrOF.jpg" +_OPENWIKI_INLINE = "https://pbs.twimg.com/media/HMKNQeJbMAA9ljZ.jpg" +_WIKI_MEMORY_COVER = "https://pbs.twimg.com/media/HMEX45LWoAA7btu.jpg" + + +def _load(name: str) -> dict[str, Any]: + """The trimmed `article_results.result` subtree of a real captured Article.""" + return json.loads((_FIXTURES / f"art-{name}.json").read_text(encoding="utf-8")) + + +def _image_urls(blocks: list) -> list[str]: + """The `media.url` of every `ArticleImageBlock`, in document order.""" + return [b.media.url for b in blocks if isinstance(b, ArticleImageBlock)] + + +def test_openwiki_resolves_inline_media_from_entitymap_list(): + """art-OpenWiki: the atomic MEDIA block resolves to the inline CDN image. + + The entity is found in the LIST `entityMap` by `entry.key`, and its + `mediaItems[0].mediaId` resolves against the sibling `media_entities[]`. + The 7 LINK entities must NOT be mistaken for images. + """ + _title, blocks = parse_article_content_state(_load("OpenWiki")) + urls = _image_urls(blocks) + assert _OPENWIKI_INLINE in urls + # The inline image sits among the text runs, not at the very front (that is + # the cover's slot), and is preceded + followed by text. + inline_idx = next( + i for i, b in enumerate(blocks) if isinstance(b, ArticleImageBlock) and b.media.url == _OPENWIKI_INLINE + ) + assert any(isinstance(b, ArticleTextBlock) for b in blocks[:inline_idx]) + assert any(isinstance(b, ArticleTextBlock) for b in blocks[inline_idx + 1 :]) From 75906e56a4afd44a513cee525cd6ce5824a0835b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 08:14:37 +0200 Subject: [PATCH 2/6] fix(#39): resolve X-article inline + cover images on the real X payload shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #39 parser was pinned to a constructed fixture and resolved ZERO images on live X Articles (validated against 3 real bookmarked Articles, #66). Three real-shape divergences fixed in extract/article.py: - entityMap is a LIST of {key, value} keyed by entry.key (not a dict) — index by key, with the old dict shape kept as a defensive path. - a MEDIA entity's URL is indirect: data.mediaItems[].mediaId resolves against the sibling media_entities[] (media_id -> media_info.original_img_url). - cover_media is a separate lead image, prepended as the first ArticleImageBlock (deduped against inline). Also render header-two -> "## " and unordered-list-item -> "- " by baking the markdown prefix into the text run (flattened-text invariant preserved). Degrades to text-only on any shape miss; no model change. Co-Authored-By: Claude Opus 4.8 --- src/xbrain/extract/article.py | 187 +++++++++++++++++++++++++++++----- 1 file changed, 163 insertions(+), 24 deletions(-) diff --git a/src/xbrain/extract/article.py b/src/xbrain/extract/article.py index dca0f95..1881823 100644 --- a/src/xbrain/extract/article.py +++ b/src/xbrain/extract/article.py @@ -58,14 +58,16 @@ def parse_article_content_state(payload: Any) -> tuple[str | None, list[ArticleB `title` may be `None` even when blocks are found (a title-less shape still yields a body). """ - content_state = _find_content_state(payload) + container, content_state = _find_article_container(payload) if content_state is None: return None, [] raw_blocks = content_state.get("blocks") if not isinstance(raw_blocks, list): return None, [] - entity_map = _entity_map(content_state) - blocks = _build_blocks(raw_blocks, entity_map) + entity_by_key = _entity_by_key(content_state) + media_index = _media_index(container) + blocks = _build_blocks(raw_blocks, entity_by_key, media_index) + blocks = _prepend_cover(blocks, container) return _find_title(payload), blocks @@ -106,12 +108,18 @@ def _looks_like_draftjs_blocks(blocks: Any) -> bool: return any(isinstance(b, dict) and ("type" in b or "text" in b) for b in blocks) -def _find_content_state(node: Any) -> dict[str, Any] | None: - """Locate the content_state dict anywhere in `node` (BFS, key-anchored). +def _find_article_container(node: Any) -> tuple[dict[str, Any] | None, dict[str, Any] | None]: + """Locate the article container + its content_state anywhere in `node` (BFS). - Prefers an explicit `content_state` / `contentState` key at any level, but - also accepts a node that IS a content_state (the response being the body - itself). Null-safe: a missing/renamed path degrades to None. + Returns `(container, content_state)` where `container` is the dict that HOLDS + the `content_state` — on the real X shape it is the `article_results.result` + node, so its sibling `media_entities` (inline-image CDN URLs) and + `cover_media` (the lead image) are readable off it. When the response IS the + content_state itself (a title-less body passed directly), the container is + that same dict — its media siblings are simply absent (null-safe reads). + + Prefers an explicit `content_state` / `contentState` key at any level. Both + elements are `None` on a missing/renamed path (degrade to the fallback). """ queue: list[Any] = [node] while queue: @@ -121,49 +129,149 @@ def _find_content_state(node: Any) -> dict[str, Any] | None: if key in current: coerced = _coerce_content_state(current[key]) if coerced is not None: - return coerced + return current, coerced coerced = _coerce_content_state(current) if coerced is not None: - return coerced + return current, coerced queue.extend(current.values()) elif isinstance(current, list): queue.extend(current) - return None + return None, None -def _entity_map(content_state: dict[str, Any]) -> dict[str, Any]: - for key in ("entityMap", "entity_map"): - value = content_state.get(key) - if isinstance(value, dict): - return value +def _entity_by_key(content_state: dict[str, Any]) -> dict[str, Any]: + """Index the Draft.js `entityMap` by entity key, LIST or dict shape. + + The REAL X shape is a LIST of `{"key": , "value": {type, data}}`; + a block's `entityRanges[0].key` matches an element's `key` (NOT its list + index), so we key by `str(entry["key"])`. The older CONSTRUCTED shape is a + plain `{key: value}` dict — accepted verbatim as the defensive path. Any + other shape yields an empty map (every atomic block then resolves to no + image, and the body degrades to text). + """ + raw = content_state.get("entityMap") + if raw is None: + raw = content_state.get("entity_map") + if isinstance(raw, list): + indexed: dict[str, Any] = {} + for entry in raw: + if isinstance(entry, dict) and "key" in entry and isinstance(entry.get("value"), dict): + indexed[str(entry["key"])] = entry["value"] + return indexed + if isinstance(raw, dict): + return raw return {} -def _build_blocks(raw_blocks: list[Any], entity_map: dict[str, Any]) -> list[ArticleBlock]: +def _media_info_url(node: Any) -> str | None: + """The `media_info.original_img_url` CDN URL on a media node, or None. + + Shared by the inline `media_entities[]` index and the `cover_media` reader. + Null-safe: a missing `media_info` / URL (or a non-http value) yields None. + """ + if not isinstance(node, dict): + return None + info = node.get("media_info") + if isinstance(info, dict): + url = info.get("original_img_url") + if isinstance(url, str) and url.startswith("http"): + return url + return None + + +def _media_index(container: dict[str, Any] | None) -> dict[str, str]: + """Map `str(media_id)` → CDN URL from the container's `media_entities[]`. + + A `MEDIA` entity carries only a `mediaId`; the CDN URL lives on the sibling + `media_entities[]` array keyed by `media_id`. This builds that lookup so + `_resolve_media_url` can turn a `mediaId` into a real image URL. Null-safe: + a missing / non-list `media_entities` yields an empty index. + """ + index: dict[str, str] = {} + if not isinstance(container, dict): + return index + entities = container.get("media_entities") + if not isinstance(entities, list): + return index + for entity in entities: + if not isinstance(entity, dict): + continue + media_id = entity.get("media_id") + url = _media_info_url(entity) + if media_id is not None and url: + index[str(media_id)] = url + return index + + +# Draft.js block `type`s whose text run carries a markdown prefix, baked into +# the run AFTER the `\n\n` separator so the flattened-text invariant still holds +# (`generate` strips only the leading separator, leaving the prefix to render). +_BLOCK_PREFIXES = { + "header-one": "# ", + "header-two": "## ", + "header-three": "### ", + "unordered-list-item": "- ", + "ordered-list-item": "1. ", +} + + +def _block_prefix(block_type: Any) -> str: + """The markdown prefix for a heading / list block `type` (else "").""" + if isinstance(block_type, str): + return _BLOCK_PREFIXES.get(block_type, "") + return "" + + +def _build_blocks( + raw_blocks: list[Any], entity_by_key: dict[str, Any], media_index: dict[str, str] +) -> list[ArticleBlock]: """Turn Draft.js blocks into ordered `ArticleBlock`s (images + text runs).""" blocks: list[ArticleBlock] = [] have_text = False for raw in raw_blocks: if not isinstance(raw, dict): continue - image = _image_block(raw, entity_map) + image = _image_block(raw, entity_by_key, media_index) if image is not None: blocks.append(image) continue text = raw.get("text") if isinstance(text, str) and text.strip(): separator = ARTICLE_PARAGRAPH_SEP if have_text else "" - blocks.append(ArticleTextBlock(text=separator + text)) + prefix = _block_prefix(raw.get("type")) + blocks.append(ArticleTextBlock(text=separator + prefix + text)) have_text = True continue # Neither an image nor a text run: if it referenced an entity, it is a # DROPPED media/atomic block (an embed/divider, or an image whose URL did # not resolve). Log it so a real-payload key drift is visible rather than # silently losing content (data-safety observability, #39 PR3 review). - _log_dropped_block(raw, entity_map) + _log_dropped_block(raw, entity_by_key) return blocks +def _prepend_cover( + blocks: list[ArticleBlock], container: dict[str, Any] | None +) -> list[ArticleBlock]: + """Prepend the article's `cover_media` lead image as an `ArticleImageBlock`. + + The cover lives outside `content_state.blocks` entirely (a `cover_media` + sibling), so it is added as the FIRST block — the lead image, before any text + run. Null-safe: no cover (or no URL) leaves `blocks` untouched. Dedup: if the + cover URL already appears inline, it is not emitted twice. + """ + if not isinstance(container, dict): + return blocks + url = _media_info_url(container.get("cover_media")) + if not url: + return blocks + for block in blocks: + if isinstance(block, ArticleImageBlock) and block.media.url == url: + return blocks + cover = ArticleImageBlock(media=MediaPhotoPending(url=url), alt=None) + return [cover, *blocks] + + def _log_dropped_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> None: """WARN when a non-text block references an entity we could not render. @@ -183,9 +291,18 @@ def _log_dropped_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> None: ) -def _image_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> ArticleImageBlock | None: - """An `ArticleImageBlock` when `raw` references an inline image, else None.""" - entity = _first_entity(raw, entity_map) +def _image_block( + raw: dict[str, Any], entity_by_key: dict[str, Any], media_index: dict[str, str] +) -> ArticleImageBlock | None: + """An `ArticleImageBlock` when `raw` references an inline image, else None. + + Resolves the CDN URL in two ways, in order: (1) the REAL X indirection — + `data.mediaItems[i].mediaId` looked up in `media_index` (built from the + sibling `media_entities[]`); (2) the defensive fallback — a URL stored + directly on the entity `data` (`media_url_https`, `mediaUrl`, … the old + CONSTRUCTED shape). A non-image entity (LINK/TWEET/…) is never an image. + """ + entity = _first_entity(raw, entity_by_key) if entity is None: return None if str(entity.get("type", "")).upper() not in _IMAGE_ENTITY_TYPES: @@ -193,12 +310,34 @@ def _image_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> ArticleImag data = entity.get("data") if not isinstance(data, dict): return None - url = _find_url_by_key(data) + url = _resolve_media_url(data, media_index) or _find_url_by_key(data) if not url: return None return ArticleImageBlock(media=MediaPhotoPending(url=url), alt=_alt_text(data)) +def _resolve_media_url(data: dict[str, Any], media_index: dict[str, str]) -> str | None: + """Resolve a `MEDIA` entity's CDN URL via `data.mediaItems[].mediaId`. + + Each `mediaItems[i]` carries a `mediaId` (never the URL itself); the URL is + looked up in `media_index` (`str(media_id)` → CDN URL, from the sibling + `media_entities[]`). Returns the first resolvable URL, or None when no item + resolves (the caller then tries the defensive URL-in-entity fallback). + """ + items = data.get("mediaItems") + if not isinstance(items, list): + return None + for item in items: + if not isinstance(item, dict): + continue + media_id = item.get("mediaId") or item.get("media_id") + if media_id is not None: + url = media_index.get(str(media_id)) + if url: + return url + return None + + def _first_entity(raw: dict[str, Any], entity_map: dict[str, Any]) -> dict[str, Any] | None: """Resolve the first entity referenced by `raw`'s entityRanges, or None.""" ranges = raw.get("entityRanges") or raw.get("entity_ranges") From 8f1cc1058e1b5b63a32a147074889f265b6a6061 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 08:14:37 +0200 Subject: [PATCH 3/6] test(#39): lock ground-truth image inventory + degradation paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real-payload acceptance tests (tests/test_article_real.py): exact image inventory per Article (OpenWiki cover+inline, Wiki Memory cover-only, Headcount none), cover-is-lead-block, LINK-never-image, headings/bullets markdown, flattened-text invariant via the model validator, and degradation (missing media siblings drop images but keep all text; foreign payloads -> (None, [])). Baseline the 3 fixtures' base64 node-ids: each decodes to "ArticleEntity:" — benign GraphQL global IDs in real-payload ground-truth fixtures, not credentials. Co-Authored-By: Claude Opus 4.8 --- .secrets.baseline | 39 +++++++++++++++-- tests/test_article_real.py | 89 +++++++++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 5 deletions(-) diff --git a/.secrets.baseline b/.secrets.baseline index 5a1d8e0..5f72953 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -90,6 +90,10 @@ { "path": "detect_secrets.filters.allowlist.is_line_allowlisted" }, + { + "path": "detect_secrets.filters.common.is_baseline_file", + "filename": ".secrets.baseline" + }, { "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies", "min_level": 2 @@ -129,23 +133,50 @@ "filename": "scripts/check.sh", "hashed_secret": "e22586930a5b2f196cd9070b9a4af5c47c1380fa", "is_verified": false, - "line_number": 43 + "line_number": 53 }, { "type": "Secret Keyword", "filename": "scripts/check.sh", "hashed_secret": "9d4e1e23bd5b727046a9e3b4b7db57bd8d6ee684", "is_verified": false, - "line_number": 127 + "line_number": 414 }, { "type": "Secret Keyword", "filename": "scripts/check.sh", "hashed_secret": "a602e216eb44a3ac5e096036eeaaef6bb9159677", "is_verified": false, - "line_number": 415 + "line_number": 418 + } + ], + "tests/fixtures/art-Headcount_AI.json": [ + { + "type": "Base64 High Entropy String", + "filename": "tests/fixtures/art-Headcount_AI.json", + "hashed_secret": "935c40c735878a3f840936083023bb1457a8f152", + "is_verified": false, + "line_number": 176 + } + ], + "tests/fixtures/art-OpenWiki.json": [ + { + "type": "Base64 High Entropy String", + "filename": "tests/fixtures/art-OpenWiki.json", + "hashed_secret": "cf1534db4a3700e7356f37e5b15459808dc52cb9", + "is_verified": false, + "line_number": 476 + } + ], + "tests/fixtures/art-Wiki_Memory.json": [ + { + "type": "Base64 High Entropy String", + "filename": "tests/fixtures/art-Wiki_Memory.json", + "hashed_secret": "b8cf2aa50cede3f008a49f610a8c90d5135530e8", + "is_verified": false, + "line_number": 424 } ] }, - "generated_at": "2026-05-18T17:05:58Z" + "generated_at": "2026-07-04T06:13:06Z" } diff --git a/tests/test_article_real.py b/tests/test_article_real.py index cdc3142..01b858e 100644 --- a/tests/test_article_real.py +++ b/tests/test_article_real.py @@ -18,6 +18,8 @@ from pathlib import Path from typing import Any +import pytest + from xbrain.extract.article import parse_article_content_state from xbrain.models import ArticleImageBlock, ArticleTextBlock, ContentSourceSuccess @@ -52,7 +54,92 @@ def test_openwiki_resolves_inline_media_from_entitymap_list(): # The inline image sits among the text runs, not at the very front (that is # the cover's slot), and is preceded + followed by text. inline_idx = next( - i for i, b in enumerate(blocks) if isinstance(b, ArticleImageBlock) and b.media.url == _OPENWIKI_INLINE + i + for i, b in enumerate(blocks) + if isinstance(b, ArticleImageBlock) and b.media.url == _OPENWIKI_INLINE ) assert any(isinstance(b, ArticleTextBlock) for b in blocks[:inline_idx]) assert any(isinstance(b, ArticleTextBlock) for b in blocks[inline_idx + 1 :]) + + +# Ground-truth image inventory per real Article (cover first, then inline, in +# document order). This is the #66 acceptance table, locked as a test. +_EXPECTED_IMAGES = [ + ("OpenWiki", [_OPENWIKI_COVER, _OPENWIKI_INLINE]), + ("Wiki_Memory", [_WIKI_MEMORY_COVER]), + ("Headcount_AI", []), +] + + +@pytest.mark.parametrize("name, expected", _EXPECTED_IMAGES) +def test_real_payload_image_inventory(name: str, expected: list[str]) -> None: + """Every real Article resolves EXACTLY its ground-truth images, in order.""" + _title, blocks = parse_article_content_state(_load(name)) + assert _image_urls(blocks) == expected + + +@pytest.mark.parametrize( + "name, cover", [("OpenWiki", _OPENWIKI_COVER), ("Wiki_Memory", _WIKI_MEMORY_COVER)] +) +def test_cover_media_is_the_lead_block(name: str, cover: str) -> None: + """When a `cover_media` exists, it is prepended as the FIRST block.""" + _title, blocks = parse_article_content_state(_load(name)) + assert isinstance(blocks[0], ArticleImageBlock) + assert blocks[0].media.url == cover + + +def test_link_entities_never_become_images() -> None: + """art-OpenWiki has 7 LINK entities + 1 MEDIA; only MEDIA + cover are images.""" + _title, blocks = parse_article_content_state(_load("OpenWiki")) + assert len(_image_urls(blocks)) == 2 + + +def test_headings_and_bullets_render_as_markdown() -> None: + """`header-two` → `## ` and `unordered-list-item` → `- `, baked into runs.""" + _title, blocks = parse_article_content_state(_load("Wiki_Memory")) + texts = [b.text for b in blocks if isinstance(b, ArticleTextBlock)] + assert any(t.lstrip("\n").startswith("## ") for t in texts) + assert any(t.lstrip("\n").startswith("- ") for t in texts) + + +@pytest.mark.parametrize("name", ["OpenWiki", "Wiki_Memory", "Headcount_AI"]) +def test_flattened_text_invariant_holds(name: str) -> None: + """`text` == concat of the `ArticleTextBlock` texts — enforced by the model. + + Building a `ContentSourceSuccess` from the parsed blocks + flattened text + exercises the `_text_matches_blocks` validator, so a broken separator/prefix + that desynced the two would raise here. + """ + _title, blocks = parse_article_content_state(_load(name)) + flat = "".join(b.text for b in blocks if isinstance(b, ArticleTextBlock)) + source = ContentSourceSuccess( + kind="x_article", + url="https://x.com/i/article/1", + text=flat, + blocks=blocks, + http_status=200, + attempts=1, + ) + assert source.text == flat + + +def test_missing_media_entities_drops_image_but_keeps_text() -> None: + """No `media_entities`/`cover_media` → images unresolved, text still complete. + + The degrade-not-crash guarantee: a real body whose media siblings are absent + (or renamed) yields zero images but loses no text run — never a crash. + """ + payload = _load("OpenWiki") + payload.pop("media_entities", None) + payload.pop("cover_media", None) + _title, blocks = parse_article_content_state(payload) + assert _image_urls(blocks) == [] + assert any(isinstance(b, ArticleTextBlock) for b in blocks) + + +@pytest.mark.parametrize( + "payload", [{}, {"foo": "bar"}, [], None, {"content_state": {"blocks": []}}] +) +def test_foreign_payload_degrades_to_none_empty(payload: Any) -> None: + """Any non-Article / empty-body shape degrades to `(None, [])`, never a crash.""" + assert parse_article_content_state(payload) == (None, []) From f185b57061bf8eb191d812e8596d76f4677aa59a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 08:57:53 +0200 Subject: [PATCH 4/6] =?UTF-8?q?fix(#39):=20address=20review=20=E2=80=94=20?= =?UTF-8?q?galleries,=20media=20tripwire,=20richest-payload=20selection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review panel (silent-failure + test-quality + spec-conformance) hardening of the real-payload image fix: - Multi-image gallery: a MEDIA block's `mediaItems` now yields ONE ArticleImageBlock per resolvable item (was first-only), so a gallery is no longer silently truncated to its first image; a partial resolve WARNs for the lost remainder. - Image tripwire: WARN when the content_state carries media indicators (atomic blocks / media_entities / cover_media) but resolves 0 images — the exact regression the original #39 defect produced silently. Text-only articles are not flagged. - Richest-payload selection (fetch_x._structured_article): parse ALL captured payloads and pick the one with the most blocks (was first-with-blocks), so a preview/skeleton article response can't ship as the complete body; WARN if >1 parsed. - Atomic-ness decided from the entity, not blank text: a caption-bearing atomic image that fails to resolve is logged as a drop, not demoted to a text run. - Failed gallery no longer falls back to a stray click-through `url` on the entity data (only the no-mediaItems defensive shape uses the URL-in-entity fallback). - Per-item alt text; DEBUG log on an unparseable content_state string; blockquote/ header-one/three/four prefixes; TypeGuard on _is_image_entity. - Rewrote the stale module docstring (was "pinned against a constructed fixture, validate before production") to reflect the now-completed real-payload validation. Co-Authored-By: Claude Opus 4.8 --- src/xbrain/extract/article.py | 296 +++++++++++++++++++++------------- src/xbrain/fetch_x.py | 50 ++++-- 2 files changed, 219 insertions(+), 127 deletions(-) diff --git a/src/xbrain/extract/article.py b/src/xbrain/extract/article.py index 1881823..8424b22 100644 --- a/src/xbrain/extract/article.py +++ b/src/xbrain/extract/article.py @@ -1,32 +1,41 @@ """Parse an X long-form Article's GraphQL payload into ordered `ArticleBlock`s. X serialises a long-form Article body as a Draft.js `ContentState`: an ordered -list of `blocks` (paragraphs, plus `atomic` blocks that reference inline media) -and an `entityMap` that resolves each media reference. This module turns that -payload into the ordered `list[ArticleBlock]` carried on -`ContentSourceSuccess.blocks` (#39 PR3): text runs become `ArticleTextBlock`, -inline images become `ArticleImageBlock` wrapping a `MediaPhotoPending` (so the -existing `xbrain media` engine downloads them later, PR4), IN DOCUMENT ORDER. - -FIXTURE PROVENANCE / RESILIENCE: the exact key path is pinned against a -CONSTRUCTED fixture (see `tests/test_article.py` / `tests/test_fetch_x.py`), not -a recorded live payload — validate against a real bookmarked-Article GraphQL -response before production reliance (RFC #39 open-Q #4). The parser therefore -anchors ONLY on stable key names and degrades to `(None, [])` on ANY shape drift -so the caller (`fetch_x._fetch_rendered`) falls back to trafilatura text rather -than crash — never a partial/wrong block set masquerading as a complete body. +list of `blocks` (paragraphs, headings, list items, plus `atomic` blocks that +reference inline media) and an `entityMap` that resolves each media reference. +This module turns that payload into the ordered `list[ArticleBlock]` carried on +`ContentSourceSuccess.blocks` (#39 PR3): text runs become `ArticleTextBlock` +(with `## `/`- ` markdown prefixes baked in for headings/lists), inline images +become `ArticleImageBlock`s wrapping a `MediaPhotoPending` (so the existing +`xbrain media` engine downloads them later, PR4), IN DOCUMENT ORDER. The lead +`cover_media` image is prepended as the first block. + +VALIDATION / RESILIENCE: the key path is validated against three REAL captured +bookmarked-Article GraphQL payloads (`tests/test_article_real.py` + +`tests/fixtures/art-*.json`, #66). On the live shape the `entityMap` is a LIST +keyed by `entry.key`, a `MEDIA` entity resolves its CDN URL INDIRECTLY via a +sibling `media_entities[]` (`mediaItems[].mediaId` -> `media_id` -> +`media_info.original_img_url`), and the lead image lives in a separate +`cover_media` sibling. The parser anchors ONLY on stable key names and degrades +safely: a partial shape miss yields an image-less but text-complete body (with a +WARN), a wholesale miss yields `(None, [])`, so the caller +(`fetch_x._fetch_rendered`) falls back to trafilatura rather than crash — never a +partial/wrong block set masquerading as a complete body. The older CONSTRUCTED +shape (dict `entityMap`, URL directly on the entity `data`) is retained as a +defensive path (`tests/test_article.py`). FLATTENED-BODY INVARIANT: the inter-paragraph separator (`\\n\\n`) is baked into -each non-first text run, so the source's flattened `text` is the EXACT -`"".join(b.text for b in blocks if isinstance(b, ArticleTextBlock))` (the PR1 -contract) AND still reads naturally for `enrich`/`topics`. +each non-first text run (after any `## `/`- ` prefix), so the source's flattened +`text` is the EXACT `"".join(b.text for b in blocks if isinstance(b, +ArticleTextBlock))` (the PR1 contract) AND still reads naturally for +`enrich`/`topics`. """ from __future__ import annotations import json import logging -from typing import Any +from typing import Any, TypeGuard from xbrain.models import ( ARTICLE_PARAGRAPH_SEP, @@ -42,13 +51,26 @@ # LINK / TWEET / MENTION entity is explicitly NOT one, so an inline-link # paragraph keeps its text rather than being mistaken for an image. _IMAGE_ENTITY_TYPES = frozenset({"IMAGE", "MEDIA"}) -# Ordered key names that may carry an image's CDN URL, at any nesting depth of -# the entity `data` (X shapes vary: a flat `url`, a `mediaUrl`, or a nested -# `mediaItems[]`). Anchoring on the key name keeps the resolver drift-tolerant. +# Ordered key names that may carry an image's CDN URL directly on the entity / +# media-item `data` (the defensive/constructed shape; the live shape resolves +# via `media_entities` instead). Anchoring on the key name keeps it drift-tolerant. _IMAGE_URL_KEYS = ("media_url_https", "mediaUrl", "media_url", "mediaURL", "url") -# Ordered key names that may carry an image's alt text (top-level of `data`). +# Ordered key names that may carry an image's alt text. _ALT_KEYS = ("altText", "alt_text", "alt", "description") +# Draft.js block `type`s whose text run carries a markdown prefix, baked into +# the run AFTER the `\n\n` separator so the flattened-text invariant still holds +# (`generate` strips only the leading separator, leaving the prefix to render). +_BLOCK_PREFIXES = { + "header-one": "# ", + "header-two": "## ", + "header-three": "### ", + "header-four": "#### ", + "unordered-list-item": "- ", + "ordered-list-item": "1. ", + "blockquote": "> ", +} + def parse_article_content_state(payload: Any) -> tuple[str | None, list[ArticleBlock]]: """Map an X article GraphQL `payload` to `(title, ordered_blocks)`. @@ -56,7 +78,8 @@ def parse_article_content_state(payload: Any) -> tuple[str | None, list[ArticleB Returns `(None, [])` when no usable `content_state` is found (missing / renamed / malformed) — the caller then routes to the trafilatura fallback. `title` may be `None` even when blocks are found (a title-less shape still - yields a body). + yields a body). On a partial media-shape miss the body is returned WITH its + text runs but WITHOUT the unresolved images (a WARN is logged), never a crash. """ container, content_state = _find_article_container(payload) if content_state is None: @@ -68,6 +91,7 @@ def parse_article_content_state(payload: Any) -> tuple[str | None, list[ArticleB media_index = _media_index(container) blocks = _build_blocks(raw_blocks, entity_by_key, media_index) blocks = _prepend_cover(blocks, container) + _warn_if_media_unresolved(raw_blocks, container, media_index, blocks) return _find_title(payload), blocks @@ -88,10 +112,16 @@ def _coerce_content_state(value: Any) -> dict[str, Any] | None: def _load_json(value: str) -> Any: - """Parse a JSON string, or None on failure (never raises).""" + """Parse a JSON string, or None on failure (never raises). + + A content_state that arrives as a string but does not parse is a real + serialization drift; log it at DEBUG so the specific "present but + unparseable" case is diagnosable rather than indistinguishable from absent. + """ try: return json.loads(value) except (ValueError, TypeError): + logger.debug("article: a content_state string was not valid JSON; treating as absent") return None @@ -184,8 +214,9 @@ def _media_index(container: dict[str, Any] | None) -> dict[str, str]: A `MEDIA` entity carries only a `mediaId`; the CDN URL lives on the sibling `media_entities[]` array keyed by `media_id`. This builds that lookup so - `_resolve_media_url` can turn a `mediaId` into a real image URL. Null-safe: - a missing / non-list `media_entities` yields an empty index. + `_item_url` can turn a `mediaId` into a real image URL. Keys are stringified + so an int `media_id` and a str `mediaId` still match. Null-safe: a missing / + non-list `media_entities` yields an empty index. """ index: dict[str, str] = {} if not isinstance(container, dict): @@ -203,53 +234,108 @@ def _media_index(container: dict[str, Any] | None) -> dict[str, str]: return index -# Draft.js block `type`s whose text run carries a markdown prefix, baked into -# the run AFTER the `\n\n` separator so the flattened-text invariant still holds -# (`generate` strips only the leading separator, leaving the prefix to render). -_BLOCK_PREFIXES = { - "header-one": "# ", - "header-two": "## ", - "header-three": "### ", - "unordered-list-item": "- ", - "ordered-list-item": "1. ", -} - - -def _block_prefix(block_type: Any) -> str: - """The markdown prefix for a heading / list block `type` (else "").""" - if isinstance(block_type, str): - return _BLOCK_PREFIXES.get(block_type, "") - return "" - - def _build_blocks( raw_blocks: list[Any], entity_by_key: dict[str, Any], media_index: dict[str, str] ) -> list[ArticleBlock]: - """Turn Draft.js blocks into ordered `ArticleBlock`s (images + text runs).""" + """Turn Draft.js blocks into ordered `ArticleBlock`s (images + text runs). + + A block that references an image entity is resolved to one or more + `ArticleImageBlock`s (a `mediaItems` gallery yields one per resolvable item) + and NEVER falls through to the text branch — so a caption-bearing atomic + block whose media fails to resolve is logged as a drop rather than silently + demoted to text. Headings/list items get their markdown prefix baked in. + """ blocks: list[ArticleBlock] = [] have_text = False for raw in raw_blocks: if not isinstance(raw, dict): continue - image = _image_block(raw, entity_by_key, media_index) - if image is not None: - blocks.append(image) + entity = _first_entity(raw, entity_by_key) + if _is_image_entity(entity): + images, unresolved = _resolve_image_blocks(entity, media_index) + if images: + blocks.extend(images) + if unresolved: + _log_partial_gallery(len(images), unresolved) + else: + _log_dropped_block(raw, entity_by_key) continue text = raw.get("text") if isinstance(text, str) and text.strip(): separator = ARTICLE_PARAGRAPH_SEP if have_text else "" - prefix = _block_prefix(raw.get("type")) - blocks.append(ArticleTextBlock(text=separator + prefix + text)) + blocks.append(ArticleTextBlock(text=separator + _block_prefix(raw.get("type")) + text)) have_text = True continue - # Neither an image nor a text run: if it referenced an entity, it is a - # DROPPED media/atomic block (an embed/divider, or an image whose URL did - # not resolve). Log it so a real-payload key drift is visible rather than - # silently losing content (data-safety observability, #39 PR3 review). _log_dropped_block(raw, entity_by_key) return blocks +def _is_image_entity(entity: dict[str, Any] | None) -> TypeGuard[dict[str, Any]]: + """True when `entity` is an inline-image entity (`IMAGE`/`MEDIA` type). + + A `TypeGuard` so callers narrow `entity` to a non-optional dict afterwards. + """ + return isinstance(entity, dict) and str(entity.get("type", "")).upper() in _IMAGE_ENTITY_TYPES + + +def _resolve_image_blocks( + entity: dict[str, Any], media_index: dict[str, str] +) -> tuple[list[ArticleImageBlock], int]: + """Resolve an image entity to `(image_blocks, unresolved_item_count)`. + + Two shapes, tried in order: (1) the REAL X gallery — `data.mediaItems[]`, + each item resolved via `_item_url` (its `mediaId` looked up in `media_index`, + or a URL stored on the item), yielding ONE `ArticleImageBlock` per resolvable + item so a multi-image gallery is not truncated to its first image; a + `mediaItems` present but partially/wholly unresolvable does NOT fall through + to a stray `data`-level URL (which could be a click-through link) — it reports + the miss instead. (2) the defensive shape — no `mediaItems`, so a single URL + stored directly on the entity `data` (`media_url_https`, `mediaUrl`, …). + """ + data = entity.get("data") + if not isinstance(data, dict): + return [], 0 + items = data.get("mediaItems") + if isinstance(items, list) and items: + images: list[ArticleImageBlock] = [] + unresolved = 0 + for item in items: + if not isinstance(item, dict): + continue + url = _item_url(item, media_index) + if url: + images.append( + ArticleImageBlock( + media=MediaPhotoPending(url=url), alt=_alt_text(item) or _alt_text(data) + ) + ) + else: + unresolved += 1 + return images, unresolved + url = _find_url_by_key(data) + if url: + return [ArticleImageBlock(media=MediaPhotoPending(url=url), alt=_alt_text(data))], 0 + return [], 0 + + +def _item_url(item: dict[str, Any], media_index: dict[str, str]) -> str | None: + """One `mediaItems[i]`'s CDN URL: its `mediaId` in `media_index`, else a URL + stored directly on the item (the defensive/constructed shape).""" + media_id = item.get("mediaId") or item.get("media_id") + if media_id is not None: + url = media_index.get(str(media_id)) + if url: + return url + return _find_url_by_key(item) + + +def _block_prefix(block_type: Any) -> str: + """The markdown prefix for a heading / list / quote block `type` (else "").""" + if isinstance(block_type, str): + return _BLOCK_PREFIXES.get(block_type, "") + return "" + + def _prepend_cover( blocks: list[ArticleBlock], container: dict[str, Any] | None ) -> list[ArticleBlock]: @@ -272,13 +358,49 @@ def _prepend_cover( return [cover, *blocks] -def _log_dropped_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> None: +def _warn_if_media_unresolved( + raw_blocks: list[Any], + container: dict[str, Any] | None, + media_index: dict[str, str], + blocks: list[ArticleBlock], +) -> None: + """WARN when the payload carried media but the body resolved ZERO images. + + The original #39 defect was exactly this: media present (atomic blocks + + `media_entities` + `cover_media`) yet every image silently dropped. A + genuinely text-only article (no atomic blocks, no media siblings) is NOT + flagged — so this fires only on a real media-resolution drift, not on every + prose-only piece. + """ + if any(isinstance(b, ArticleImageBlock) for b in blocks): + return + had_atomic = any(isinstance(r, dict) and r.get("type") == "atomic" for r in raw_blocks) + cover = container.get("cover_media") if isinstance(container, dict) else None + if had_atomic or media_index or _media_info_url(cover): + logger.warning( + "article: content_state has media indicators (atomic blocks / " + "media_entities / cover_media) but resolved 0 images — media " + "resolution may have drifted." + ) + + +def _log_partial_gallery(resolved: int, unresolved: int) -> None: + """WARN that a multi-image gallery only partially resolved (some items lost).""" + logger.warning( + "article: a MEDIA block resolved %d image(s) but %d gallery item(s) had " + "no URL — a media_entities key drift may be hiding images.", + resolved, + unresolved, + ) + + +def _log_dropped_block(raw: dict[str, Any], entity_by_key: dict[str, Any]) -> None: """WARN when a non-text block references an entity we could not render. A genuinely empty spacer block (no entity) is silent — only an entity-bearing block that produced no image is a real content drop worth surfacing. """ - entity = _first_entity(raw, entity_map) + entity = _first_entity(raw, entity_by_key) if entity is None: return data = entity.get("data") @@ -291,54 +413,7 @@ def _log_dropped_block(raw: dict[str, Any], entity_map: dict[str, Any]) -> None: ) -def _image_block( - raw: dict[str, Any], entity_by_key: dict[str, Any], media_index: dict[str, str] -) -> ArticleImageBlock | None: - """An `ArticleImageBlock` when `raw` references an inline image, else None. - - Resolves the CDN URL in two ways, in order: (1) the REAL X indirection — - `data.mediaItems[i].mediaId` looked up in `media_index` (built from the - sibling `media_entities[]`); (2) the defensive fallback — a URL stored - directly on the entity `data` (`media_url_https`, `mediaUrl`, … the old - CONSTRUCTED shape). A non-image entity (LINK/TWEET/…) is never an image. - """ - entity = _first_entity(raw, entity_by_key) - if entity is None: - return None - if str(entity.get("type", "")).upper() not in _IMAGE_ENTITY_TYPES: - return None - data = entity.get("data") - if not isinstance(data, dict): - return None - url = _resolve_media_url(data, media_index) or _find_url_by_key(data) - if not url: - return None - return ArticleImageBlock(media=MediaPhotoPending(url=url), alt=_alt_text(data)) - - -def _resolve_media_url(data: dict[str, Any], media_index: dict[str, str]) -> str | None: - """Resolve a `MEDIA` entity's CDN URL via `data.mediaItems[].mediaId`. - - Each `mediaItems[i]` carries a `mediaId` (never the URL itself); the URL is - looked up in `media_index` (`str(media_id)` → CDN URL, from the sibling - `media_entities[]`). Returns the first resolvable URL, or None when no item - resolves (the caller then tries the defensive URL-in-entity fallback). - """ - items = data.get("mediaItems") - if not isinstance(items, list): - return None - for item in items: - if not isinstance(item, dict): - continue - media_id = item.get("mediaId") or item.get("media_id") - if media_id is not None: - url = media_index.get(str(media_id)) - if url: - return url - return None - - -def _first_entity(raw: dict[str, Any], entity_map: dict[str, Any]) -> dict[str, Any] | None: +def _first_entity(raw: dict[str, Any], entity_by_key: dict[str, Any]) -> dict[str, Any] | None: """Resolve the first entity referenced by `raw`'s entityRanges, or None.""" ranges = raw.get("entityRanges") or raw.get("entity_ranges") if not isinstance(ranges, list) or not ranges: @@ -346,17 +421,17 @@ def _first_entity(raw: dict[str, Any], entity_map: dict[str, Any]) -> dict[str, first = ranges[0] if not isinstance(first, dict) or first.get("key") is None: return None - entity = entity_map.get(str(first["key"])) + entity = entity_by_key.get(str(first["key"])) return entity if isinstance(entity, dict) else None def _find_url_by_key(node: Any) -> str | None: """The image CDN URL, preferring the canonical key GLOBALLY. - Searches the whole entity `data` tree once per key in `_IMAGE_URL_KEYS` - priority order, so a deep `media_url_https` beats a shallow bare `url` - (a bare `url` may be a link/thumbnail; `media_url_https` is the canonical - full-size CDN photo PR4's size-cascade wants). + Searches the whole `node` tree once per key in `_IMAGE_URL_KEYS` priority + order, so a deep `media_url_https` beats a shallow bare `url` (a bare `url` + may be a link/thumbnail; `media_url_https` is the canonical full-size CDN + photo PR4's size-cascade wants). """ for key in _IMAGE_URL_KEYS: url = _first_http_value_for_key(node, key) @@ -384,6 +459,7 @@ def _first_http_value_for_key(node: Any, key: str) -> str | None: def _alt_text(data: dict[str, Any]) -> str | None: + """First non-empty alt-text string under `data`'s known alt keys, or None.""" for key in _ALT_KEYS: value = data.get(key) if isinstance(value, str) and value: diff --git a/src/xbrain/fetch_x.py b/src/xbrain/fetch_x.py index 614ce0e..1da8809 100644 --- a/src/xbrain/fetch_x.py +++ b/src/xbrain/fetch_x.py @@ -185,7 +185,13 @@ def _structured_article(captured: list[dict], url: str) -> ContentSourceSuccess never a silent empty success. The parse is wrapped so ANY parser exception (incl. a `RecursionError` on a pathological payload) degrades to the fallback instead of aborting the fetch — the "degrade, not crash" guarantee. + + When MORE THAN ONE captured payload parses to blocks (e.g. X emits a + preview/skeleton article response before the full-body one), the RICHEST body + (most blocks) is selected rather than the first — so a truncated preview never + masquerades as the complete article — and the ambiguity is logged. """ + parsed: list[tuple[str | None, list[ArticleBlock]]] = [] for payload in captured: try: title, blocks = parse_article_content_state(payload) @@ -198,23 +204,33 @@ def _structured_article(captured: list[dict], url: str) -> ContentSourceSuccess ) continue if blocks: - n_images = sum(1 for b in blocks if isinstance(b, ArticleImageBlock)) - logger.info( - "article: built structured body (%d blocks, %d images) for %s", - len(blocks), - n_images, - url, - ) - return ContentSourceSuccess( - kind="x_article", - url=url, - title=title, - text=_flatten_blocks(blocks), - blocks=blocks, - http_status=200, - attempts=1, - ) - return None + parsed.append((title, blocks)) + if not parsed: + return None + if len(parsed) > 1: + logger.warning( + "article: %d captured payloads parsed to blocks for %s; selecting the " + "richest — a preview/duplicate article response may be present", + len(parsed), + url, + ) + title, blocks = max(parsed, key=lambda item: len(item[1])) + n_images = sum(1 for b in blocks if isinstance(b, ArticleImageBlock)) + logger.info( + "article: built structured body (%d blocks, %d images) for %s", + len(blocks), + n_images, + url, + ) + return ContentSourceSuccess( + kind="x_article", + url=url, + title=title, + text=_flatten_blocks(blocks), + blocks=blocks, + http_status=200, + attempts=1, + ) def _log_article_fallback(captured: list[dict], url: str) -> None: From 6efa010597aa9e7565ff098e0c1a1e8843c9ba0e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 08:57:53 +0200 Subject: [PATCH 5/6] test(#39): cover galleries, coercion, dedup, tripwire, production path, titles Close the review-flagged test gaps: - Multi-image gallery (one block per item), partial-gallery WARN, int/str media_id coercion, list-shape MEDIA drop WARN, caption-atomic-drop, stray-URL guard, ordered-list/header-one/three/blockquote prefixes (constructed, test_article.py). - Real payloads (test_article_real.py): title extraction per fixture; cover==inline dedup; media-tripwire WARN vs no-warn on a text-only article; replaced the TAUTOLOGICAL invariant test with an independent text-run separator-structure check plus a model-validator check via fetch_x._flatten_blocks. - Production wrapper (test_fetch_x.py): _structured_article on the REAL OpenWiki payload (cover-first + inline, title, off the sibling media_entities/cover_media), and richest-of-multiple-payloads selection with the ambiguity WARN. Co-Authored-By: Claude Opus 4.8 --- tests/test_article.py | 157 +++++++++++++++++++++++++++++++++++++ tests/test_article_real.py | 83 +++++++++++++++++--- tests/test_fetch_x.py | 56 +++++++++++++ 3 files changed, 285 insertions(+), 11 deletions(-) diff --git a/tests/test_article.py b/tests/test_article.py index f88f2bd..cb60bf2 100644 --- a/tests/test_article.py +++ b/tests/test_article.py @@ -334,3 +334,160 @@ def test_dropped_media_block_is_logged(caplog): _title, blocks = parse_article_content_state(_payload(content_state)) assert blocks == [] assert any("dropped a non-text block" in r.message for r in caplog.records) + + +# --- REAL-shape (list entityMap + media_entities indirection) edge cases ------ +# +# These mirror the live X shape validated in tests/test_article_real.py, but +# CONSTRUCTED so they can exercise branches the three captured fixtures do not +# reach: multi-image galleries, int/str media_id coercion, list-shape drops, +# the stray-URL guard, and the extra heading/list/quote block types. + + +def _real_shape_payload( + content_state: dict, *, media_entities=None, cover_media=None, title: str = "Real Read" +) -> dict: + """A payload shaped like the LIVE X response: `content_state` sits beside + `media_entities` / `cover_media` on the `article_results.result` node.""" + result: dict = {"__typename": "Article", "title": title, "content_state": content_state} + if media_entities is not None: + result["media_entities"] = media_entities + if cover_media is not None: + result["cover_media"] = cover_media + return {"data": {"article": {"article_results": {"result": result}}}} + + +def _media_entity(media_id, url: str) -> dict: + return {"media_id": media_id, "media_info": {"original_img_url": url}} + + +def _atomic_media_block(entity_key: int, *, text: str = " ") -> dict: + return { + "key": "g", + "text": text, + "type": "atomic", + "entityRanges": [{"offset": 0, "length": 1, "key": entity_key}], + } + + +def test_media_gallery_yields_one_image_block_per_resolvable_item(): + url1 = "https://pbs.twimg.com/media/G1.jpg" + url2 = "https://pbs.twimg.com/media/G2.jpg" + content_state = { + "blocks": [_atomic_media_block(0)], + "entityMap": [ + { + "key": 0, + "value": { + "type": "MEDIA", + "data": {"mediaItems": [{"mediaId": "1"}, {"mediaId": "2"}]}, + }, + } + ], + } + payload = _real_shape_payload( + content_state, media_entities=[_media_entity("1", url1), _media_entity("2", url2)] + ) + _title, blocks = parse_article_content_state(payload) + assert [b.media.url for b in blocks if isinstance(b, ArticleImageBlock)] == [url1, url2] + + +def test_partial_gallery_drops_unresolved_item_and_warns(caplog): + url1 = "https://pbs.twimg.com/media/G1.jpg" + content_state = { + "blocks": [_atomic_media_block(0)], + "entityMap": [ + { + "key": 0, + "value": { + "type": "MEDIA", + "data": {"mediaItems": [{"mediaId": "1"}, {"mediaId": "missing"}]}, + }, + } + ], + } + payload = _real_shape_payload(content_state, media_entities=[_media_entity("1", url1)]) + with caplog.at_level("WARNING", logger="xbrain.extract.article"): + _title, blocks = parse_article_content_state(payload) + assert [b.media.url for b in blocks if isinstance(b, ArticleImageBlock)] == [url1] + assert any("gallery item(s) had no URL" in r.message for r in caplog.records) + + +def test_media_id_int_str_coercion_matches(): + # media_entities carries the id as an INT; the item's mediaId is a STR. + url = "https://pbs.twimg.com/media/COERCE.jpg" + content_state = { + "blocks": [_atomic_media_block(0)], + "entityMap": [ + {"key": 0, "value": {"type": "MEDIA", "data": {"mediaItems": [{"mediaId": "12345"}]}}} + ], + } + payload = _real_shape_payload(content_state, media_entities=[_media_entity(12345, url)]) + _title, blocks = parse_article_content_state(payload) + assert [b.media.url for b in blocks if isinstance(b, ArticleImageBlock)] == [url] + + +def test_list_shape_media_that_does_not_resolve_is_dropped_and_warned(caplog): + content_state = { + "blocks": [_atomic_media_block(0)], + "entityMap": [ + {"key": 0, "value": {"type": "MEDIA", "data": {"mediaItems": [{"mediaId": "nope"}]}}} + ], + } + payload = _real_shape_payload(content_state, media_entities=[]) # empty index -> unresolved + with caplog.at_level("WARNING", logger="xbrain.extract.article"): + _title, blocks = parse_article_content_state(payload) + assert [b for b in blocks if isinstance(b, ArticleImageBlock)] == [] + assert any("dropped a non-text block" in r.message for r in caplog.records) + + +def test_caption_bearing_atomic_image_that_fails_resolution_is_dropped_not_texted(): + # A MEDIA atomic block whose text is a caption (not blank) must NOT be demoted + # to a text run when its image fails to resolve — it is a dropped image. + content_state = { + "blocks": [_atomic_media_block(0, text="a caption")], + "entityMap": [ + {"key": 0, "value": {"type": "MEDIA", "data": {"mediaItems": [{"mediaId": "nope"}]}}} + ], + } + payload = _real_shape_payload(content_state, media_entities=[]) + _title, blocks = parse_article_content_state(payload) + assert blocks == [] # not [ArticleTextBlock("a caption")] + + +def test_failed_gallery_does_not_grab_a_stray_data_url(): + # mediaItems present but unresolvable: a stray click-through `url` on the + # entity data must NOT be emitted as the image (the real-shape guard for E). + content_state = { + "blocks": [_atomic_media_block(0)], + "entityMap": [ + { + "key": 0, + "value": { + "type": "MEDIA", + "data": { + "mediaItems": [{"mediaId": "nope"}], + "url": "https://example.com/click", + }, + }, + } + ], + } + payload = _real_shape_payload(content_state, media_entities=[]) + _title, blocks = parse_article_content_state(payload) + assert [b for b in blocks if isinstance(b, ArticleImageBlock)] == [] + + +def test_heading_list_and_quote_block_prefixes_are_baked_in(): + content_state = { + "blocks": [ + {"key": "1", "text": "H1", "type": "header-one"}, + {"key": "3", "text": "H3", "type": "header-three"}, + {"key": "o", "text": "step", "type": "ordered-list-item"}, + {"key": "q", "text": "quote", "type": "blockquote"}, + ], + "entityMap": [], + } + _title, blocks = parse_article_content_state(_payload(content_state)) + texts = [b.text for b in blocks if isinstance(b, ArticleTextBlock)] + assert texts == ["# H1", "\n\n### H3", "\n\n1. step", "\n\n> quote"] diff --git a/tests/test_article_real.py b/tests/test_article_real.py index 01b858e..8d6fd1b 100644 --- a/tests/test_article_real.py +++ b/tests/test_article_real.py @@ -21,7 +21,13 @@ import pytest from xbrain.extract.article import parse_article_content_state -from xbrain.models import ArticleImageBlock, ArticleTextBlock, ContentSourceSuccess +from xbrain.fetch_x import _flatten_blocks +from xbrain.models import ( + ARTICLE_PARAGRAPH_SEP, + ArticleImageBlock, + ArticleTextBlock, + ContentSourceSuccess, +) _FIXTURES = Path(__file__).parent / "fixtures" @@ -95,23 +101,34 @@ def test_link_entities_never_become_images() -> None: def test_headings_and_bullets_render_as_markdown() -> None: - """`header-two` → `## ` and `unordered-list-item` → `- `, baked into runs.""" + """`header-two` → `## ` and `unordered-list-item` → `- `, with the `\\n\\n` + separator applied BEFORE the prefix (an independent order check).""" _title, blocks = parse_article_content_state(_load("Wiki_Memory")) texts = [b.text for b in blocks if isinstance(b, ArticleTextBlock)] - assert any(t.lstrip("\n").startswith("## ") for t in texts) - assert any(t.lstrip("\n").startswith("- ") for t in texts) + # A non-first heading run is "\n\n## …"; a non-first bullet run is "\n\n- …". + assert any(t.startswith(ARTICLE_PARAGRAPH_SEP + "## ") for t in texts) + assert any(t.startswith(ARTICLE_PARAGRAPH_SEP + "- ") for t in texts) @pytest.mark.parametrize("name", ["OpenWiki", "Wiki_Memory", "Headcount_AI"]) -def test_flattened_text_invariant_holds(name: str) -> None: - """`text` == concat of the `ArticleTextBlock` texts — enforced by the model. - - Building a `ContentSourceSuccess` from the parsed blocks + flattened text - exercises the `_text_matches_blocks` validator, so a broken separator/prefix - that desynced the two would raise here. +def test_text_run_separator_structure(name: str) -> None: + """The FIRST text run never leads with the `\\n\\n` separator; every + subsequent run does. An INDEPENDENT structural check (not a tautology that + re-derives `text` from the same blocks): a broken separator would fail it. """ _title, blocks = parse_article_content_state(_load(name)) - flat = "".join(b.text for b in blocks if isinstance(b, ArticleTextBlock)) + runs = [b.text for b in blocks if isinstance(b, ArticleTextBlock)] + assert runs, "every real fixture carries prose" + assert not runs[0].startswith(ARTICLE_PARAGRAPH_SEP) + assert all(r.startswith(ARTICLE_PARAGRAPH_SEP) for r in runs[1:]) + + +@pytest.mark.parametrize("name", ["OpenWiki", "Wiki_Memory", "Headcount_AI"]) +def test_flattened_text_validates_against_model(name: str) -> None: + """The source built the way `fetch_x` builds it (`text = _flatten_blocks`) + satisfies the `_text_matches_blocks` model validator on real payloads.""" + _title, blocks = parse_article_content_state(_load(name)) + flat = _flatten_blocks(blocks) source = ContentSourceSuccess( kind="x_article", url="https://x.com/i/article/1", @@ -123,6 +140,50 @@ def test_flattened_text_invariant_holds(name: str) -> None: assert source.text == flat +@pytest.mark.parametrize( + "name, title", + [ + ("OpenWiki", "Introducing OpenWiki, an open source agent for repo documentation"), + ("Wiki_Memory", "Wiki Memory"), + ("Headcount_AI", "The case for headcount in the age of AI"), + ], +) +def test_real_payload_title_is_extracted(name: str, title: str) -> None: + """The article title is pulled from the real payload (flows to the source).""" + got, _blocks = parse_article_content_state(_load(name)) + assert got == title + + +def test_cover_equal_to_an_inline_image_is_deduped() -> None: + """When `cover_media` resolves to a URL already present inline, it is NOT + emitted twice — the lead image renders once, not doubled.""" + payload = _load("OpenWiki") + payload["cover_media"]["media_info"]["original_img_url"] = _OPENWIKI_INLINE + _title, blocks = parse_article_content_state(payload) + urls = _image_urls(blocks) + assert urls == [_OPENWIKI_INLINE] # single image, no doubled lead + + +def test_media_tripwire_warns_when_media_present_but_zero_images(caplog) -> None: + """Media indicators present (atomic block) but 0 images resolved → WARN — + the exact regression signal the original #39 defect lacked.""" + payload = _load("OpenWiki") + payload.pop("media_entities", None) + payload.pop("cover_media", None) + with caplog.at_level("WARNING", logger="xbrain.extract.article"): + _title, blocks = parse_article_content_state(payload) + assert _image_urls(blocks) == [] + assert any("resolved 0 images" in r.message for r in caplog.records) + + +def test_text_only_article_does_not_trip_the_media_warning(caplog) -> None: + """A genuinely text-only article (no atomic/media siblings) is NOT flagged.""" + with caplog.at_level("WARNING", logger="xbrain.extract.article"): + _title, blocks = parse_article_content_state(_load("Headcount_AI")) + assert _image_urls(blocks) == [] + assert not any("resolved 0 images" in r.message for r in caplog.records) + + def test_missing_media_entities_drops_image_but_keeps_text() -> None: """No `media_entities`/`cover_media` → images unresolved, text still complete. diff --git a/tests/test_fetch_x.py b/tests/test_fetch_x.py index 636f152..7b2603a 100644 --- a/tests/test_fetch_x.py +++ b/tests/test_fetch_x.py @@ -1,5 +1,7 @@ # tests/test_fetch_x.py +import json from datetime import datetime, timezone +from pathlib import Path import xbrain.fetch_x as fx from xbrain.fetch_x import ( @@ -562,3 +564,57 @@ def test_attach_x_sources_uses_injected_clock(): _attach_x_sources(item, [_blocks_source()], now=lambda: fixed) assert item.content is not None assert item.content.fetched_at == fixed + + +# --- _structured_article on the REAL captured payload shape (#66) --- +# +# Unlike the constructed `_article_payload()` above, these feed a real trimmed +# `article_results.result` (from tests/fixtures/art-*.json) through the PRODUCTION +# wrapper, so the BFS-locates-container-then-reads-`media_entities`/`cover_media` +# combination is exercised end-to-end, not just the pure parser. + +_FIXTURES = Path(__file__).parent / "fixtures" +_OPENWIKI_TITLE = "Introducing OpenWiki, an open source agent for repo documentation" +_OPENWIKI_COVER = "https://pbs.twimg.com/media/HMKNwxAbUAEMrOF.jpg" +_OPENWIKI_INLINE = "https://pbs.twimg.com/media/HMKNQeJbMAA9ljZ.jpg" + + +def _real_result(name: str) -> dict: + return json.loads((_FIXTURES / f"art-{name}.json").read_text(encoding="utf-8")) + + +def _wrap_result(result: dict) -> dict: + """Nest a real `article_results.result` under a full GraphQL response shape.""" + return {"data": {"article": {"article_results": {"result": result}}}} + + +def test_structured_article_on_real_openwiki_payload(): + source = fx._structured_article([_wrap_result(_real_result("OpenWiki"))], _ARTICLE_URL) + assert isinstance(source, ContentSourceSuccess) + assert source.kind == "x_article" + assert source.title == _OPENWIKI_TITLE + images = [b for b in source.blocks if isinstance(b, ArticleImageBlock)] + # cover first, then the inline MEDIA image — resolved off the sibling arrays. + assert [b.media.url for b in images] == [_OPENWIKI_COVER, _OPENWIKI_INLINE] + assert isinstance(source.blocks[0], ArticleImageBlock) + assert source.text == "".join(b.text for b in source.blocks if isinstance(b, ArticleTextBlock)) + + +def test_structured_article_selects_richest_of_multiple_payloads(caplog): + preview = _wrap_result( + { + "title": "Preview", + "content_state": { + "blocks": [{"key": "a", "text": "preview only", "type": "unstyled"}], + "entityMap": [], + }, + } + ) + full = _wrap_result(_real_result("OpenWiki")) + with caplog.at_level("WARNING", logger="xbrain.fetch_x"): + source = fx._structured_article([preview, full], _ARTICLE_URL) + assert isinstance(source, ContentSourceSuccess) + # The rich OpenWiki body wins over the 1-block preview, and the ambiguity logs. + assert source.title == _OPENWIKI_TITLE + assert len(source.blocks) > 5 + assert any("selecting the" in r.message for r in caplog.records) From a2fd7b17f4c9eba89ae9c4d951f0aa81468f7d7f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20Gonz=C3=A1lez-Pacheco?= Date: Sat, 4 Jul 2026 09:03:51 +0200 Subject: [PATCH 6/6] =?UTF-8?q?docs:=20add=20user=20manual=20=E2=80=94=20t?= =?UTF-8?q?utorial=20walkthrough,=20digest-video=20example,=20troubleshoot?= =?UTF-8?q?ing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README covered install/config/command-reference but had no narrated end-to-end walkthrough, no worked digest-video example, and no troubleshooting. - docs/tutorial.md — zero-to-wiki walkthrough (sync → vocab/enrich/topics → media/describe → digest-video → dashboard), with expected output per step. - docs/digest-video.md — worked example (prereqs, run, the "## Video digest" note shape, model selection, video selection). - docs/troubleshooting.md — FAQ: expired X session, the re-saw-0 tripwire, rate limits, parakeet/ffmpeg PATH (incl. cron/launchd minimal-PATH), digest-video slowness/timeouts, silent-video "sin voz", iCloud generate hang, API keys. - README: link the three from Documentation + a "New to XBrain?" pointer at Quick start. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 7 +++ docs/digest-video.md | 111 ++++++++++++++++++++++++++++++++++ docs/troubleshooting.md | 110 ++++++++++++++++++++++++++++++++++ docs/tutorial.md | 128 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 356 insertions(+) create mode 100644 docs/digest-video.md create mode 100644 docs/troubleshooting.md create mode 100644 docs/tutorial.md diff --git a/README.md b/README.md index 4a54fe3..9d83bd1 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,10 @@ uv run xbrain status # see the counts `sync` builds the mechanical layers. The LLM layers (`vocab`, `enrich`, `topics`) are run explicitly — see [The pipeline](#the-pipeline). +> **New to XBrain?** The [**Tutorial**](docs/tutorial.md) walks through the whole +> thing end-to-end — pull your posts, add topics, download + describe media, and +> digest a bookmarked talk — with the output you should see at each step. + --- ## Prerequisites @@ -1211,6 +1215,9 @@ client. Respect X's Terms of Service. | Document | Description | |----------|-------------| +| [docs/tutorial.md](docs/tutorial.md) | **Start here** — end-to-end walkthrough from install to a searchable wiki. | +| [docs/digest-video.md](docs/digest-video.md) | Worked example: turn a bookmarked talk into transcript + slide notes. | +| [docs/troubleshooting.md](docs/troubleshooting.md) | Common failures & fixes (auth, PATH, digest-video, iCloud). | | [ARCHITECTURE.md](ARCHITECTURE.md) | How XBrain is shaped: pipeline stages, artifacts, rubrics, executors, invariants. | | [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute — including PRs written with AI agents. | | [LICENSE](LICENSE) | MIT. | diff --git a/docs/digest-video.md b/docs/digest-video.md new file mode 100644 index 0000000..e2dbab6 --- /dev/null +++ b/docs/digest-video.md @@ -0,0 +1,111 @@ +# `digest-video` — turn bookmarked talks into readable notes + +`digest-video` manufactures **text** from a video so it flows through the normal +enrich → topics → generate pipeline like any other post. For each selected video +it does an **ephemeral** fetch, transcribes the audio with an external local +transcriber, attaches the transcript as an `x_video` content source, and +**discards the bytes** (the corpus never lands on disk). `--frames` adds a visual +layer: it extracts the slide key-frames and describes each with a vision model. + +## Prerequisites + +The heavy lifting is **external** — xbrain core carries no ML/ffmpeg dependency. +Install once (see [Local models for `digest-video`](../README.md#local-models-for-digest-video-apple-silicon)): + +```bash +brew install ffmpeg # frame extraction + audio probe +uv tool install parakeet-mlx # ASR (Apple Silicon) +uv tool install mlx-vlm # vision, only for --frames +``` + +and point `config.toml` at the wrappers: + +```toml +[transcribe] +command = "/abs/path/to/xbrain/scripts/xbrain-transcribe" # wraps parakeet-mlx + +[vision] +command = "/abs/path/to/xbrain/scripts/xbrain-vision" # local + cloud selector +model = "qwen-7b" +``` + +## Run it + +```bash +# Transcript only (no vision, no ffmpeg-frames) — fast: +uv run xbrain digest-video --all-pending + +# → Vídeos: transcritos 6, sin voz 2, ya digeridos 0, fallidos 0, sin vídeo 1, ... +# Dedup: 9 items ← 9 vídeos (6 transcritos este run). +``` + +Read the summary: **transcritos** = had speech, **sin voz** = silent (no audio +track — GIFs, muted clips; attached as `has_speech=false`, not a failure), +**fallidos** = a real transcribe failure, **sin vídeo** = the video couldn't be +fetched (deleted / unavailable). Videos are **deduped by identity** — N bookmarks +of the same clip are fetched + transcribed once. + +Add `--frames` for slide-heavy talks: + +```bash +uv run xbrain digest-video --all-pending --frames +# → ... Visual: 5 con slides, 4 talking-head (saltados). +``` + +`--frames` extracts key frames (ffmpeg scene-detection + interval sampling), +classifies the video as **slides** vs **talking-head** (talking-heads are skipped +— no vision calls wasted), and describes each slide of a slide video. The slide +images are embedded in the note like downloaded photos. + +Then render: + +```bash +uv run xbrain generate +``` + +## What you get + +The item's note gains a `## Video digest` section: + +```markdown +## Video digest: Elon Musk on the first thing to do when starting a company + +> Uh, the goal with Tesla was really to try to show what electric cars can do, +> because people had the wrong impression… (full transcript) + +![[_media/1874.../frames/0.png]] +> Slide: a line chart of Model S range vs. price, 2012–2015. +``` + +The transcript + slide descriptions are plain note text, so they feed `enrich` +(summary + topics) and are **searchable** in Obsidian. A silent video with no +slides degrades gracefully to a one-line "silent video" note. + +## Choosing the model, per run + +`config.toml` `[vision].model` is the default; `--vision-model` overrides it for +one run. The `scripts/xbrain-vision` selector routes the name: + +| `--vision-model` | Backend | Notes | +|------------------|---------|-------| +| `qwen-3b` / `qwen-7b` / `qwen-32b` / `` | local (mlx-vlm) | free, offline; `qwen-32b` needs ~20 GB RAM | +| `opus` / `sonnet` / `haiku` / `claude-` | cloud (Claude) | best quality; needs `ANTHROPIC_API_KEY`; frames leave the machine | + +```bash +uv run xbrain digest-video --ids --frames --vision-model opus +uv run xbrain digest-video --topic ai-coding --frames --vision-model qwen-7b +``` + +## Selecting which videos + +```bash +--ids a,b,c # specific item ids +--topic ai-coding # every video whose post is in that topic +--all-pending # every not-yet-digested video (idempotent; re-runs skip done ones) +--source bookmarks|tweets|all --limit N --language en +``` + +`digest-video` is destructive (rewrites `items.json`) → it auto-snapshots first. +Re-running skips videos already carrying an `x_video` source unless `--force`. + +Slow? See [Troubleshooting → digest-video](troubleshooting.md#digest-video-is-slow-or-times-out). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..b2283af --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,110 @@ +# Troubleshooting & FAQ + +Common failures and how to fix them. Most are environment issues (auth, PATH, +external tools), not bugs. + +## X session expired / auth fails + +Symptoms: `extract`/`sync` scrapes 0 posts, or `status` says it can't +authenticate. X sessions are short-lived. + +Fix — re-import cookies from a browser you're logged in to: + +```bash +# Chrome — log in to x.com in Chrome first, then: +.venv/bin/python scripts/import_chrome_session.py +# → "auth_token: OK" + +# Safari — log in in Safari, grant your terminal "Full Disk Access" +# (System Settings → Privacy & Security), then: +.venv/bin/python scripts/import_safari_session.py +``` + +`xbrain login` (in-app Playwright login) exists but is unreliable with +Google/SSO accounts — the automated browser gets blocked. Cookie import is the +recommended path. + +## "Re-saw 0 known items on a non-empty store" — the run aborts without saving + +A safety tripwire: extraction saw none of the items it already has, which almost +always means an **expired session** or an X GraphQL change, not that your +bookmarks vanished. It aborts rather than overwrite good data. Re-authenticate +(above) and re-run. If you're sure the store is stale, `--force` overrides it. + +## Getting rate-limited / the browser stalls + +`extract` runs **headful** (visible Chromium) by default to look human, paces +itself, and backs off on `429`. If you still hit limits, wait and re-run — the +store is incremental, so you lose nothing. Don't run many extracts back-to-back. + +## `parakeet-mlx` / `ffmpeg` not found (digest-video) + +``` +transcriber '.../xbrain-transcribe' exited 1: FileNotFoundError: 'parakeet-mlx' +``` + +The external tools aren't on `PATH`. Two cases: + +- **Interactive shell:** install them (`brew install ffmpeg`, + `uv tool install parakeet-mlx mlx-vlm`) and make sure `~/.local/bin` + + `/opt/homebrew/bin` are on your `PATH`. +- **cron / launchd / a scheduled job:** these run with a **minimal PATH** that + excludes `~/.local/bin` and `/opt/homebrew/bin`. Set the job's environment + explicitly — e.g. in a launchd plist: + + ```xml + EnvironmentVariables + + PATH/Users/you/.local/bin:/opt/homebrew/bin:/usr/bin:/bin + + ``` + + When testing a job, reproduce its env (`env -i HOME=$HOME PATH=... your-cmd`), + not your shell — your shell's full PATH hides the bug. + +## `digest-video` is slow or times out + +Local vision (`--frames`) is the bottleneck: a slide-heavy talk can have up to +40 key-frames, and a local VLM reloads the model per frame. On a 16 GB Mac, +`qwen-7b` is ~2 min/frame → a long talk takes over an hour. + +- **First run of a large model** can exceed the 300 s per-frame timeout while it + *downloads* — pre-pull once: `~/.local/share/uv/tools/mlx-vlm/bin/python -c + "from mlx_vlm import load; load('mlx-community/Qwen2.5-VL-7B-Instruct-4bit')"`. +- **Too slow overall?** Use a smaller model (`--vision-model qwen-3b`), or + transcript-only (drop `--frames`), or cloud (`--vision-model opus`, needs + `ANTHROPIC_API_KEY`). +- Frame extraction never hangs the run — ffmpeg is bounded by its own timeout. + +## Every video comes back `fallidos` / `sin voz` + +- `sin voz` (silent): the video has **no audio track** at the source (GIFs, + muted screencasts). This is expected — it attaches as `has_speech=false` + ("silent video"), not an error. Verify with `yt-dlp -f bestaudio ` + (errors = no audio exists). +- `fallidos` (real failures): usually `parakeet-mlx` not found (see the PATH + section above) — the fix is almost always the environment, not the video. + +## `generate` hangs or takes very long + +If your vault is on **iCloud** with "Optimize Mac Storage" on, files can be +evicted to the cloud (dataless), and reading/writing them blocks on +re-download — worst at night with no activity. Run `generate` while the machine +is active, or keep the vault folder materialized (turn off Optimize Storage for +it). `data/items.json` already holds every digest, so a slow `generate` never +loses data — just re-run it. + +## Do I need an API key? + +No. The default execution mode (`vocab`/`enrich`/`topics`/`describe`) uses a +**Claude Code session** — no key, no cost. `ANTHROPIC_API_KEY` is only for +`--executor api` (unattended LLM runs) and cloud vision (`--vision-model opus`). +`FIRECRAWL_API_KEY` is an optional fallback fetcher for JavaScript-heavy pages. + +## Where's the source of truth? Can I delete the vault notes? + +`data/items.json` is the hub — the markdown is **derived and disposable**. +Delete `items/`, `topics/`, `_index.md` and re-run `generate` any time. Every +destructive command auto-snapshots `items.json` first (see +[Snapshots & safety](../README.md#snapshots--safety)); restore from +`data/snapshots/` if needed. diff --git a/docs/tutorial.md b/docs/tutorial.md new file mode 100644 index 0000000..0c1909f --- /dev/null +++ b/docs/tutorial.md @@ -0,0 +1,128 @@ +# Tutorial — from zero to a searchable wiki + +A worked, end-to-end walkthrough: install XBrain, turn *your* X bookmarks into an +Obsidian knowledge base, and digest a bookmarked talk into readable notes. Every +command is copy-paste; the → lines show what you should see. + +New here? Do the [Quick start](../README.md#quick-start) first (install + +authenticate), then come back — this tutorial picks up from a logged-in install. + +--- + +## 1. Confirm you're set up + +```bash +uv run xbrain status +# → Items: 0 +# → con enlace: 0 +# → ... +``` + +An empty store with no error means config + auth are good. If `status` complains +about config, copy `config.toml.example` to `config.toml` and set your vault path ++ X handle. If it can't authenticate, re-run the cookie import (see +[Troubleshooting](troubleshooting.md#x-session-expired--auth-fails)). + +## 2. Pull your posts and build the mechanical wiki + +```bash +uv run xbrain sync # extract (scrape X) + fetch (article bodies) + generate +uv run xbrain status +# → Items: 812 +# → con enlace: 143 +# → última extracción bookmarks: 2026-07-04 ... +``` + +`sync` scrapes your bookmarks + own tweets into `data/items.json`, fetches the +linked article bodies, and writes one markdown note per post into your vault. +Open the vault in Obsidian — you already have `items/*.md` and `_index.md`. + +> `sync` runs **headful** by default (a visible Chromium) to look human; it +> paces itself and backs off on rate limits. First run scrolls your whole +> history, so it's the slow one. + +## 3. Add the topic layer (the LLM stages) + +The mechanical layers need no LLM. The *understanding* layers — a topic +vocabulary, per-post summaries + topics, and topic-page overviews — do: + +```bash +uv run xbrain vocab # induce ~45 topics from the corpus +uv run xbrain enrich # summary + topics for each post +uv run xbrain topics # write a topic page per cluster +uv run xbrain generate # re-render the vault with the new layers +``` + +By default these use the **claude-code execution mode** (no API key, no cost): +each stage exports a worksheet you fill in a Claude Code session, then +`--apply`. To run them unattended with the API instead, add `--executor api` +(needs `ANTHROPIC_API_KEY`). See [Execution modes](../README.md#execution-modes). + +Now your vault has three layers: `items/` (posts), `topics/` (thematic pages), +and `_index.md` (the map). Open `_index.md` in Obsidian and click into a topic. + +## 4. Download the media + +```bash +uv run xbrain media # download bookmarked photos +uv run xbrain download-videos --yes # download videos (prints a size gate first) +``` + +Photos embed under each post note. To make photos **searchable**, add vision +descriptions: + +```bash +uv run xbrain describe --executor claude-code # export a worksheet +# fill it in a Claude Code session, then: +uv run xbrain describe --apply data/describe-worksheet.json +uv run xbrain generate +``` + +Each photo now renders with a one-line caption under it — plain note text, so +Obsidian's search finds "that chart about pricing". + +## 5. Digest a bookmarked video + +This turns a saved talk into a readable, topic-linked note. It needs the local +tooling from [Local models for `digest-video`](../README.md#local-models-for-digest-video-apple-silicon) +(ffmpeg + parakeet-mlx, plus mlx-vlm for `--frames`). See the worked example in +[digest-video.md](digest-video.md). + +```bash +# Transcript only (fast): every bookmarked video → an x_video transcript source +uv run xbrain digest-video --all-pending + +# With the visual layer: also describe the slides of slide-heavy talks +uv run xbrain digest-video --all-pending --frames +uv run xbrain generate +# → the video's note now has a "## Video digest" section: transcript + slides +``` + +## 6. See the whole corpus at a glance + +`generate` also writes `dashboard.html` — a self-contained interactive dashboard +(counts, topics, authors, growth over time, photo thumbnails), with drill-down and +deep links back to each post + note. Open it from the **📊 Dashboard** link at the +top of `_index.md`, or directly in your browser: + +```bash +# //dashboard.html — from your config.toml [paths]: +open ~/Documents/Vault/vault/learnings/x-knowledge/dashboard.html +``` + +## Keeping it fresh + +Re-run periodically — everything is **incremental and idempotent**: + +```bash +uv run xbrain sync # pull new bookmarks/tweets, re-render +uv run xbrain enrich # enrich only the new posts +uv run xbrain topics # refresh topic pages +uv run xbrain generate +``` + +The markdown is **derived and disposable** — delete and regenerate any time. The +source of truth is `data/items.json` (snapshotted before every destructive op; +see [Snapshots & safety](../README.md#snapshots--safety)). + +Stuck? → [Troubleshooting](troubleshooting.md).