Skip to content

feat(capture): flag-gated #xray:capture navigation marker (27 K.4) #182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bryanmatthewsimonson wants to merge 4 commits into from
Closed

feat(capture): flag-gated #xray:capture navigation marker (27 K.4) #182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3120831
feat(capture): flag-gated #xray:capture navigation marker (27 K.4)
bryanmatthewsimonson Jul 17, 2026
04bab21
fix(phase-27): apply the S/F/K adversarial-review findings
bryanmatthewsimonson Jul 17, 2026
726019e
fix(capture): the marker-strip must never abort the capture (27 K.4)
bryanmatthewsimonson Jul 17, 2026
1b6c1cf
feat(llm): offer Fable 5 + Sonnet 5; disclose model refusals honestly
bryanmatthewsimonson Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 111 additions & 147 deletions .claude/skills/xray-capture/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: xray-capture
description: Capture articles into X-Ray by URL, enumerate an article's outbound-link frontier for corpus expansion, and run a scouting suggest pass. Use when the user asks to capture a URL into X-Ray, capture the links from an article, expand a case corpus, or scout what an article would yield. Requires the claude-in-chrome connector and the X-Ray extension loaded.
description: Capture articles into X-Ray by URL and enumerate an article's outbound-link frontier for corpus expansion, driving the loaded extension through the claude-in-chrome connector. Use when the user asks to capture a URL into X-Ray, capture the links from an article, or expand a case corpus. Requires the claude-in-chrome connector, the X-Ray extension loaded, and the Capture-automation flag ON.
---

# X-Ray capture — drive the extension, expand the corpus
Expand All @@ -12,165 +12,129 @@ never judge, and **nothing model-produced becomes durable X-Ray data
through you** — proposals are accepted by the human in the reader,
or not at all.

## Connector limits (verified 2026-07-16 — design around these, not against them)

- The connector **cannot navigate to `chrome-extension://` URLs** (it
forces an `https://` prefix) and **cannot attach the debugger to
extension pages** ("Cannot attach debugger to chrome-extension://
pages"). Everything that needed extension-page JS — IDB reads,
`chrome.tabs.sendMessage`, `xray:llm:suggest` — is unreachable.
- Synthetic keys do not fire extension command shortcuts, and OS-level
keystrokes into browsers are blocked by the computer-use tier.
- Therefore the ONLY verb you have against the extension is
**navigation of ordinary pages** — which is exactly what the
`#xray:capture` marker (Phase 27 K.4, flag-gated) exists for.

## Hard rules (the project constitution — these bind every step)

1. **No auto-accept, ever.** Never write entities, claims, links,
facts, findings, or baselines into `chrome.storage.local` or the
extension's IndexedDB. The ONLY writes you may cause are the ones
the extension itself performs in response to its own capture flow
(`xray:capture` → reader auto-archive). If a suggest pass returns
proposals, you REPORT them; the human accepts in the reader.
facts, findings, or baselines anywhere. The ONLY writes you may
cause are the extension's own capture flow (the marker → reader
auto-archive).
2. **Never touch publish paths.** No relay messages, no signing.
3. **The extension's API key stays in its service worker.** Talk to
the LLM only via the extension's own `xray:llm:*` messages (which
gate on the user's flags + key). Never read `xray:llm:key`.
4. **Pace yourself.** Captures run sequentially — one tab, one URL at
a time, waiting for each load. Suggest passes are paid API calls:
state the count and get the user's go-ahead in chat before running
more than one.
5. **Capture only where directed.** The user names the seed URL(s) or
3. **Pace yourself.** Captures run sequentially — one URL at a time,
waiting for each load and verification before the next.
4. **Capture only where directed.** The user names the seed URL(s) or
approves the frontier list before you fan out. Skip login-walled or
paywalled pages you cannot render; report them as uncapturable
rather than working around access controls.
6. **Forensic findings stay out of scope** for suggestion scouting
(maintainer decision 2026-07-16: not until the attribution fixes
are proven on real articles).
5. **Suggest-pass scouting is currently NOT runnable by this skill**
(it needs extension-context messaging the connector cannot reach).
The reader's own Suggest button is the human's path; recommend it
per-article instead of simulating it.

## Preflight (once per session)

1. Confirm the claude-in-chrome tools are available (load via
ToolSearch if deferred).
2. Find the X-Ray extension origin: call `tabs_context`. If no
`chrome-extension://` tab is open, ask the user to open the X-Ray
portal (extension icon → or `chrome://extensions` → X-Ray →
Details → open `src/portal/index.html`), then re-read
`tabs_context` and note the origin `chrome-extension://<ID>`.
Keep that portal tab open — it is your command surface.
3. Capability probe: run `javascript_tool` on the portal tab with
`typeof chrome.tabs.query` — expect `"function"`. If the tool
cannot execute JS on extension pages, STOP and tell the user this
skill needs that capability; do not improvise another write path.

## Capture one URL (K.1)

1. Open/navigate a NON-portal tab to the article URL. Wait for load
(screenshot or `get_page_text` sanity check — a real article body,
not a consent wall or error page).
2. From the **portal tab**, run `javascript_tool`:

```js
(async () => {
const tabs = await chrome.tabs.query({ url: '<ARTICLE_URL_PATTERN>' });
if (!tabs.length) return 'no-tab';
const resp = await chrome.tabs.sendMessage(tabs[0].id, { type: 'xray:capture' });
return resp && resp.ok ? 'capture-sent' : ('failed: ' + JSON.stringify(resp));
})()
```

(`url` patterns need a trailing `*` for query params; the message
lands in the content script — `src/content/index.js` — which opens
the reader, and the reader auto-archives on open.)
3. **PDFs:** skip steps 1–2 and instead open a tab at
1. Confirm the claude-in-chrome tools are loaded (ToolSearch if
deferred), then `tabs_context_mcp {createIfEmpty: true}`.
2. Ask the user to flip **Options → Advanced → Capture automation** ON
(the `#xray:capture` marker is default-off; it gates only the
marker). Remind them to turn it back off when the session ends.
3. Probe once: navigate a group tab to a harmless captureable page
with `#xray:capture` appended, wait ~3s, then `javascript_tool`:
`document.documentElement.dataset.xrayCaptured` —
- `"ok"` → capture fired; a reader tab opened (outside your group —
you won't see it; that's expected).
- `"flag-off"` → the flag isn't on; ask again.
- `undefined` → the extension may not be loaded, or the page blocked
the content script; try one reload, then report.

## Capture one URL

1. Navigate a group tab to `<URL>#xray:capture` (if the URL already
has a fragment, the marker replaces it — note that to the user).
2. Wait for load + ~5s (the marker waits 1.5s after init to let
dynamic pages settle, then opens the reader, which auto-archives).
**SPAs (Medium, some news sites) need ~10s** — a capture that fires
before the body renders archives an empty shell.
3. Verify on the SAME tab (ordinary page — attachable):
`document.documentElement.dataset.xrayCaptured` → expect `"ok"`.
`"error"`/`undefined` → report the URL as failed (paywall? consent
wall? script-blocked?); retry at most once.
4. **A stamp of `"ok"` means the capture FIRED, not that the content
was good.** Check `document.title` in the same probe: a title like
`"archive.is"`, `"Just a moment…"`, `"Medium"`, or a bare hostname
means an interstitial/loading state was live when the capture ran —
wait for the real title and re-capture (see the re-capture note
below). Batch the title into the verification expression:
`({stamp: document.documentElement.dataset.xrayCaptured, title: document.title, len: document.body.innerText.length})`
— `len` under ~2000 on a supposed article is another tell.
5. **Re-capturing:** navigate somewhere else first (`https://example.com`)
and back. Going from `<url>` to `<url>#xray:capture` used to be a
silent no-op (same-document navigation → no re-init); a `hashchange`
listener now covers it, but a full navigation is still the reliable
path, and re-capture is safe — the archive keys by URL and keeps the
prior version.
6. Report title (from the tab) and move to the next URL.
5. **Tab pileup is real:** every capture opens a reader tab the
connector cannot close (they're outside the group). Tell the user
at the end how many reader tabs they'll find and that each shows a
captured article ready for claim extraction.
6. **PDFs:** the reader's `?pdf=` path is an extension-page URL the
connector cannot open. Give the user the exact
`chrome-extension://<ID>/src/reader/index.html?pdf=<ENCODED_URL>`
— the reader's tabless PDF path captures directly.
4. Verify the archive landed — from the portal tab:

```js
(async () => {
const db = await new Promise((res, rej) => {
const r = indexedDB.open('xray-archive');
r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error);
});
const recs = await new Promise((res, rej) => {
const tx = db.transaction('articles').objectStore('articles').getAll();
tx.onsuccess = () => res(tx.result); tx.onerror = () => rej(tx.error);
});
db.close();
const hit = recs.find((r) => r.url && r.url.includes('<URL_DISTINCTIVE_PART>'));
return hit ? { ok: true, title: hit.article && hit.article.title,
links: (hit.article && hit.article.links || []).length }
: { ok: false, captured: recs.length };
})()
```

READ-ONLY: this is verification, never a write. If verification
fails, report it (paywall? consent wall? CSP?) and move on — do not
retry more than once.
5. Close the article tab if you opened it; report `title` + link
count to the user.

## Enumerate the frontier (K.1)

From the portal tab, read the seed article's outbound links and
subtract what's already captured:

```js
(async () => {
const db = await new Promise((res, rej) => {
const r = indexedDB.open('xray-archive');
r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error);
});
const recs = await new Promise((res, rej) => {
const tx = db.transaction('articles').objectStore('articles').getAll();
tx.onsuccess = () => res(tx.result); tx.onerror = () => rej(tx.error);
});
db.close();
const seed = recs.find((r) => r.url && r.url.includes('<SEED_URL_PART>'));
if (!seed) return 'seed-not-captured';
const have = new Set(recs.map((r) => r.url));
const links = ((seed.article && seed.article.links) || [])
.filter((l) => l && l.url && !l.internal)
.filter((l) => !have.has(l.url));
return { frontier: links.map((l) => ({ url: l.url, text: l.text, count: l.count })),
truncated: !!(seed.article && seed.article.links_truncated) };
})()
```

Present the frontier as a numbered list (url + anchor text + cite
count). **Wait for the user to pick** which to capture (or confirm
"all"), then loop the capture procedure over the picks, one at a
time, reporting progress. Notes: `internal` is same-host-approximate;
`links_truncated` means the capture capped at 100 links — say so.

## Scouting suggest pass (K.2 — optional, costs API tokens)

Purpose: tell the researcher what a captured article would yield
BEFORE they invest reader time. It does NOT feed the reader's review
panel — accepting still happens there (and the reader's own Suggest
run is a second paid pass; say so when recommending it).

1. The pass scopes itself to the USER'S stored suggestion kinds
(`xray:llm:suggest_kinds` — the worker reads it; the request cannot
override it, by design). Read it first (portal tab,
`chrome.storage.local.get`) so you can tell the user what the pass
will cover. If `findings` or `baselines` are enabled there, note
that per rule 6 you will not elaborate those proposals in your
summary — review them in the reader.
2. Get the article text from the archive record
(`record.article.markdown` — fall back to `textContent` if absent).
3. From the portal tab:

```js
(async () => new Promise((res) => chrome.runtime.sendMessage({
type: 'xray:llm:suggest',
request: { articleText: <TEXT>, articleUrl: <URL>, articleTitle: <TITLE> }
}, res)))()
```

Gates (llmAssist flag + key + at-least-one-kind) are enforced
worker-side; if the response is `{ok:false}`, relay its error
verbatim (it names the Options toggle to flip).
4. Report a scouting summary in chat: counts per kind + the two or
three most load-bearing claims verbatim (skip findings/baselines
per rule 6). Recommend which articles deserve a reader pass. Do
not store anything.
link to open by hand instead.

## Enumerate the frontier

Extension IDB is unreachable, so the frontier comes from a **workspace
backup export** the user hands you:

1. Ask the user: Options → Advanced → Workspace → download backup
(source bytes not needed), and give you the file path (usually
`~/Downloads/…json`).
2. Parse it in node (it is one JSON object):
`databases['xray-archive'].stores.articles` (explore the exact
nesting — `format` and `exportedAt` sit at the top level) → each
record has `url`, `articleHash`, and `article.links` (shape
`{url, text, count, internal}`, capped at 100 with
`article.links_truncated`).
3. Frontier = every non-`internal` link URL across the seed records,
minus every `url` already in the archive, deduped, with citation
counts and which seeds cite it. Disclose `links_truncated` seeds.
4. Present the frontier as a numbered pick list sorted by citation
count. **Wait for the user to pick** (or confirm "all"), then loop
the capture procedure sequentially over the picks.
5. The export is a snapshot — captures made after it won't be in it.
Re-request an export if you need a fresh frontier; don't guess.

## Failure modes

| Symptom | Meaning | Do |
|---|---|---|
| `javascript_tool` refuses extension pages | capability missing | Stop; tell the user (preflight step 3) |
| `no-tab` from capture | URL pattern mismatch | Add `*` for query strings; re-query |
| `sendMessage` throws "no receiving end" | content script not injected (chrome:// page, PDF viewer, race) | PDFs → the `?pdf=` reader path; else reload the tab once |
| Verification finds no record | paywall/consent wall/CSP ate the capture | Report as uncapturable; do not bypass |
| Suggest returns `ok:false` | flag off or no key | Relay the error; the user flips it in Options |
| `dataset.xrayCaptured === "flag-off"` | Capture automation off | Ask the user to enable it in Options → Advanced |
| `dataset.xrayCaptured` undefined after reload | content script absent (blocked page, chrome://, PDF viewer) | Report uncapturable; PDFs → hand the user the `?pdf=` reader link |
| `"error"` stamp | the marker branch threw | Read the console (`pattern: "X-Ray"`) — it logs the reason verbatim |
| Stamp `"ok"` but the title is `archive.is` / `Just a moment…` / `Medium` | captured an interstitial, not the article | Wait for the real title, then re-capture via a full navigation |
| Stamp never appears on one specific host | the content script may be throwing before the marker runs | Console first, guess never — this is how the archive.is `<base href>` bug was found (JOURNAL 2026-07-17) |
| Navigate mangles the URL | you passed a `chrome-extension://` URL | Don't — ordinary pages only (see Connector limits) |
| Frontier stale | export predates recent captures | Ask for a fresh export |

## Reporting back

Give the user a per-URL table (title + outcome), not just a count.
State plainly: how many reader tabs are now open (one per capture, all
outside the group — you cannot close them), which URLs failed and why,
and which captured content you are UNSURE about. An honest "this one
may have caught a loading page" is worth more than a green checkmark
they later discover was empty.
40 changes: 40 additions & 0 deletions docs/JOURNAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,46 @@ or files, and the "so-what" for future readers.

---

## 2026-07-17 — A cosmetic replaceState aborted the capture it preceded (27 K.4)

Tags: `bug`, `external`.

Found in the first live agent-driven capture run, on `archive.is`
(several COVID-corpus frontier links are archive.is snapshots of
paywalled reporting). The `#xray:capture` marker fired once, then
never again on that host:

```
SecurityError: Failed to execute 'replaceState' on 'History':
A history state object with URL 'https://d8w6fmknl1l6hl.archive.is/7gYpy'
cannot be created in a document with origin 'https://archive.is'
```

archive.is serves snapshot content under a **randomized subdomain** and
sets a `<base href>` to it. `history.replaceState(null, '', pathname +
search)` resolves its RELATIVE argument against the document's **base
URI**, not its origin — so the rewrite targeted another host and threw.
The marker-strip sat before `UI.openReader()` inside the same `try`, so
a cosmetic address-bar tidy killed the capture. It "worked" on the very
first hit only because the base tag hadn't been applied yet — which is
also why that capture archived the interstitial rather than the article.

Two fixes, both worth keeping in mind for any content-script URL work:
build the replacement **absolute from `location.origin`** (never
relative — `<base href>` is another site's to control), and give
best-effort cosmetics **their own try/catch** so they can never abort
the load-bearing work behind them. Also fixed alongside: a hash-only
navigation (`<url>` → `<url>#xray:capture`) is a same-document
navigation, so `init()` never re-runs and the marker silently no-oped;
a `hashchange` listener now covers it.

No unit test would have caught either — both need a real page with a
cross-origin base tag. The lesson generalizes past this slice: a
try-block that wraps a nice-to-have around a must-have inherits the
nice-to-have's failure modes.

---

## 2026-07-16 — The scope question never reached the LLM; proposals were never asked for (27 S.1/S.2)

Tags: `bug`, `design`.
Expand Down
Loading