Skip to content

feat: add shared dialog capture #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
konard merged 12 commits into from
Jun 25, 2026
Merged

feat: add shared dialog capture #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b09f2dc
Initial commit with task details
konard Jun 25, 2026
65d7dcc
feat: add shared dialog capture
konard Jun 25, 2026
fb5f5de
test(js): harden process e2e loopback URLs
konard Jun 25, 2026
a3f7f2b
test(js): isolate process e2e fixture server
konard Jun 25, 2026
40f5e09
fix(js): harden proxy response headers
konard Jun 25, 2026
b51fa92
fix(js): request identity encoding for html fetches
konard Jun 25, 2026
e35556b
fix(js): request identity encoding for gdocs exports
konard Jun 25, 2026
8b4e041
fix(js): request identity encoding for github api
konard Jun 25, 2026
20046e4
fix(js): harden stream proxy response handling
konard Jun 25, 2026
4e9f135
fix(js): handle upstream stream closes
konard Jun 25, 2026
d8cc305
fix(js): stream proxy responses with explicit writes
konard Jun 25, 2026
78225bc
fix(js): preserve stream content length
konard Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 24 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ A CLI and microservice to fetch URLs and render them as:
- **ZIP archive**: Markdown/HTML + locally downloaded images
- **PDF**: Print-quality document
- **DOCX**: Word document
- **Shared AI dialog transcript**: ChatGPT share captures with structured diagnostics for unsupported providers

## Language Implementations

Expand Down Expand Up @@ -60,6 +61,7 @@ Both implementations expose the same API:
| `GET /fetch?url=<URL>` | Proxy fetch content |
| `GET /stream?url=<URL>` | Stream content |
| `GET /search?q=<QUERY>` | Capture structured search-provider results |
| `GET /shared-dialog?url=<URL>` | Capture shared AI dialog transcripts as JSON, Markdown, text, or Links Notation |

FormalAI integration should use the stable HTTP/CLI contract documented in
[`docs/formalai-contract.md`](docs/formalai-contract.md).
Expand Down Expand Up @@ -88,6 +90,9 @@ web-capture https://example.com --format png -o screenshot.png
# Create a ZIP archive
web-capture https://example.com --archive

# Capture a shared AI dialog transcript as structured JSON
web-capture shared-dialog https://chatgpt.com/share/SHARE_ID

# Start as API server
web-capture --serve

Expand All @@ -97,24 +102,24 @@ web-capture --serve --port 8080

## CLI Options

| Option | Short | Description | Default |
| ------------------------ | ----- | ----------------------------------------------------------------------------------------------------- | --------------------- |
| `--serve` | `-s` | Start as HTTP API server | - |
| `--port` | `-p` | Port to listen on | 3000 |
| `--format` | `-f` | Output format: `markdown`/`md`, `html`, `txt`/`text`, `image`/`png`, `jpeg`, `pdf`, `docx`, `archive` | `markdown` |
| `--output` | `-o` | Output file path. Use `-o -` for stdout | auto-derived from URL |
| `--data-dir` | | Base directory for auto-derived output paths | `./data/web-capture` |
| `--engine` | `-e` | Browser engine (JS only): `puppeteer`, `playwright` | `puppeteer` |
| `--embed-images` | | Keep images inline as base64 data URIs (self-contained file) | `false` |
| `--no-extract-images` | | Alias for `--embed-images` | `false` |
| `--extract-images[=DIR]` | | Extract images to `DIR/images/` (or next to the output) and download remote images | - |
| `--keep-original-links` | | Keep remote image URLs as direct links (the default markdown behavior) | `false` |
| `--images-dir` | | Subdirectory name for extracted images | `images` |
| `--archive` | | Create archive: `zip` (default), `7z`, `tar.gz`, `tar` | - |
| `--extract-latex` | | Extract LaTeX formulas | `true` |
| `--extract-metadata` | | Extract article metadata | `true` |
| `--post-process` | | Apply post-processing | `true` |
| `--detect-code-language` | | Detect code block languages | `true` |
| Option | Short | Description | Default |
| ------------------------ | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| `--serve` | `-s` | Start as HTTP API server | - |
| `--port` | `-p` | Port to listen on | 3000 |
| `--format` | `-f` | Output format: `markdown`/`md`, `html`, `txt`/`text`, `image`/`png`, `jpeg`, `pdf`, `docx`, `archive`, `json`, `meta-language`, `demo-memory` | `markdown` |
| `--output` | `-o` | Output file path. Use `-o -` for stdout | auto-derived from URL |
| `--data-dir` | | Base directory for auto-derived output paths | `./data/web-capture` |
| `--engine` | `-e` | Browser engine (JS only): `puppeteer`, `playwright` | `puppeteer` |
| `--embed-images` | | Keep images inline as base64 data URIs (self-contained file) | `false` |
| `--no-extract-images` | | Alias for `--embed-images` | `false` |
| `--extract-images[=DIR]` | | Extract images to `DIR/images/` (or next to the output) and download remote images | - |
| `--keep-original-links` | | Keep remote image URLs as direct links (the default markdown behavior) | `false` |
| `--images-dir` | | Subdirectory name for extracted images | `images` |
| `--archive` | | Create archive: `zip` (default), `7z`, `tar.gz`, `tar` | - |
| `--extract-latex` | | Extract LaTeX formulas | `true` |
| `--extract-metadata` | | Extract article metadata | `true` |
| `--post-process` | | Apply post-processing | `true` |
| `--detect-code-language` | | Detect code block languages | `true` |

## Image Handling

Expand Down Expand Up @@ -168,6 +173,7 @@ Both implementations expose the same API:
| `GET /docx?url=<URL>` | DOCX with embedded images |
| `GET /fetch?url=<URL>` | Proxy fetch content |
| `GET /stream?url=<URL>` | Stream content |
| `GET /shared-dialog?url=<URL>` | Shared AI dialog transcript capture with structured diagnostics |

## Docker

Expand Down
29 changes: 29 additions & 0 deletions docs/case-studies/issue-141/raw-data/chatgpt-share-6a3825b9.demo_memory.lino
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
demo_memory
event "0c9f0151-b5a1-402f-afc3-6bd34a0d01d2"
role "user"
content "box@87ffc301f5eb:~$ sleep 30m && hive-cleanup -f\n\nmake a loop of that (infinite), answer with only single line"
demoLabel "issue-552-chatgpt-share"
conversationId "6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
conversationTitle "Infinite loop script"
evidence "https://chatgpt.com/share/6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
event "fc908946-280f-4e2b-adcc-f27a97538685"
role "assistant"
content "```bash\nwhile true; do sleep 30m && hive-cleanup -f; done\n```"
demoLabel "issue-552-chatgpt-share"
conversationId "6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
conversationTitle "Infinite loop script"
evidence "https://chatgpt.com/share/6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
event "a9f6893d-b2bf-43aa-a55c-9cea559f5190"
role "user"
content "Can we use `screen -R auto-cleanup` to execute that line inside, also making it all using single line?"
demoLabel "issue-552-chatgpt-share"
conversationId "6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
conversationTitle "Infinite loop script"
evidence "https://chatgpt.com/share/6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
event "a5fb0b5c-6831-4ccf-863e-bb0770e65d96"
role "assistant"
content "```bash\nscreen -dmS auto-cleanup bash -c 'while true; do sleep 30m && hive-cleanup -f; done'\n```"
demoLabel "issue-552-chatgpt-share"
conversationId "6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
conversationTitle "Infinite loop script"
evidence "https://chatgpt.com/share/6a3825b9-8de4-83ee-9c24-52fd1eb38d24"
Loading
Loading