feat: add web and local session ingestion

[jamespheffernan]

Summary

aicrawl now has functional ingestion for the main non-export sources this project needs: live/captured ChatGPT and Claude web payloads, native ChatGPT macOS app cache conversation IDs as live-web detail seeds, plus local OpenClaw, Codex, Gemini, Claude Code (including Claude desktop/local-agent transcript roots), Cursor, and Hermes session stores/files or roots. OpenClaw imports preserve Discord/Telegram-style sourceChannel and sender metadata when present, and directory imports tolerate control-only session files without leaking private paths. It also has safe source-audit dry-runs, official-export reconciliation reports, status-level web-sync and local-source freshness with durable cursor metadata, dedicated browser-profile launch/reuse for web sync, a macOS LaunchAgent generator for recurring bounded web sync, recurring local transcript-root imports, and one-command default local-root scheduling, and an aggregate-only live-provider smoke runner for final field validation.

sync web --provider chatgpt|claude --cdp-url <url> attaches to an already authenticated browser target, finds or opens a provider page, runs same-origin list/detail fetch() calls from page context, imports the fetched detail batch into SQLite under chatgpt_web or claude_web, preserves raw payloads, updates sync_state, and makes messages searchable. For ChatGPT, --chatgpt-app-cache <dir> can seed that live detail fetch from native macOS app conversations-v3-*/*.data filenames without reading opaque cache bodies. Without --cdp-url, live sync web launches or reuses a dedicated browser profile using --profile, binds CDP to 127.0.0.1, discovers Chrome's local endpoint from DevToolsActivePort, and reports login_required on first-run profiles without writing the archive. sync web --source <json-or-zip> still imports captured provider detail payloads through the same archive path. aicrawl import accepts --provider openclaw, --provider codex, --provider gemini, --provider claude-code, --provider cursor, and --provider hermes for characterized local files/stores, and recursively imports local provider roots for OpenClaw/Codex/Claude Code *.jsonl, Gemini *.json, Cursor store.db/state.vscdb, and Hermes state.db or sidecar session JSON/JSONL. Hermes root imports prefer state.db over sidecar files. Directory import and dry-run skip control-only local transcript files with redacted warnings, report skipped_sources, and cap warning arrays at 100 entries plus a truncation summary. aicrawl import --dry-run --json audits candidate counts without creating or writing the archive. aicrawl reconcile <official-export> --provider chatgpt|claude|auto --json compares official exports against the archive read-only and reports source/archived/missing coverage plus divergent normalized message projections. aicrawl schedule launchd ... writes CDP-attach, profile-launch, or local-import LaunchAgent plists without loading them or storing browser credentials. With --import-path, recurring local imports run aicrawl import <path> --provider <provider> --json for OpenClaw, Codex, Gemini, Claude Code/Claude desktop local-agent, Cursor, or Hermes roots and rely on import idempotency to avoid duplicate rows. With --local-defaults, setup discovers standard roots that already exist and writes the matching import LaunchAgents in one pass, with distinct labels for Claude Code CLI and Claude Desktop roots. Live web batches skip individual 403/404/410 conversation detail responses while preserving accessible conversations in the same batch, record those observations in conversation_sync_status as inaccessible, and avoid destructive deletion for archived/deleted/inaccessible items.

Dry-run web sync validates the browser-profile/CDP boundary, inspects optional redacted network captures for list/detail endpoint contracts, reports captured source candidate counts, performs best-effort list-only live candidate counting when --cdp-url is supplied, and reports freshness without writing. Non-dry-run sync with stale/partial/missing captures fails before archive writes with stable contract_<state> errors. status --json also exposes web_sync freshness for chatgpt_web and claude_web, including last_checked_at, candidate counts, and cursor metadata, so stale or never-run syncs are visible from the normal control surface. Live web sync records provider_updated_at watermarks when provider payloads expose timestamps; later runs pass the cursor back into the adapter, skip older list candidates, and return a successful no-change result without detail fetches or archive writes.

Design Notes

Concern	Decision
Auth boundary	Keep cookies, bearer tokens, session headers, and browser storage inside the browser profile or attached CDP target.
Browser profile launch	Launch Chrome/Chromium/Microsoft Edge with a dedicated --user-data-dir, CDP bound to 127.0.0.1, and an ephemeral remote-debugging port by default; reuse a live profile endpoint before starting a new browser process.
First-run login	If the dedicated profile is new, open the provider page and return login_required guidance without writing archive rows or scraping credentials.
Live browser fetch	Execute same-origin fetch() from the provider page context through a narrow CDP client; do not extract credentials.
Contract drift	Classify captures as matched, partial, stale, missing, empty, or not_checked; non-matched captures fail before writes with contract_<state> errors, and provider live fetch fails before archive writes if required response keys are absent.
Captured/live payload writes	Import ChatGPT/Claude detail payload batches through provider-specific adapters and existing archive idempotency. Live batches use private temp files that are removed after import, record inaccessible 403/404/410 details in conversation_sync_status, and do not delete existing archive rows.
Import audit	import --dry-run shares the same parsers as import, reports counts/warnings, caps warning arrays, reports skipped directory sources, and does not initialize or write the archive.
Web dry-run candidate counts	sync web --dry-run --cdp-url attaches only to an already open provider page, fetches list endpoints to count candidates, does not fetch details, and returns live_list_unavailable warnings instead of failing the whole preflight when candidate inspection cannot attach. With --chatgpt-app-cache, dry-run counts filename-derived ChatGPT app cache IDs locally as chatgpt_app_cache_ids without reading cache bodies or attaching to CDP.
Export reconciliation	reconcile shares official export parsers with import, opens the archive read-only, and reports missing coverage plus divergent normalized message projections. Backfill/refresh remains explicit through aicrawl import.
Freshness visibility	status --json.web_sync reports ChatGPT/Claude web source freshness, and status --json.local_sources reports OpenClaw, Codex, Gemini, Claude Code/Claude desktop local-agent, Cursor, and Hermes import freshness from sync_state, including never_imported sources.
Durable web cursors and statuses	Schema v3 records source-hash cursors for file/captured imports, provider_updated_at cursors for live web syncs, and per-conversation seen/inaccessible status rows; unchanged cursor runs update check freshness without re-importing rows.
Recurring sync and local imports	schedule launchd writes plists that run bounded sync web via --cdp-url or --profile, recurring aicrawl import <path> --provider <provider> --json via --import-path, or one-pass standard local-root import jobs via --local-defaults.
Local agent files/roots	Import OpenClaw JSONL, Codex rollout JSONL, Gemini CLI session JSON, Claude Code project JSONL (including Claude desktop/local-agent roots under ~/Library/Application Support/Claude), Cursor store.db, Hermes state.db/session JSON/JSONL, and provider-specific local roots with aggregate reports that avoid full private paths. OpenClaw preserves source-channel and sender metadata when present.
ChatGPT macOS app cache	Discover native ChatGPT app conversation IDs from conversations-v3-*/*.data filenames, sort them by newest file modification time, and fetch exact details through the authenticated ChatGPT web path; never read or decode opaque .data cache bodies.
Claude Code scope	Keep visible user/assistant text; skip control events, thinking blocks, tool calls, and tool results.
Cursor scope	Open one store.db or installed state.vscdb read-only as one conversation; use legacy meta identity/title or workspace-state directory identity, rowid ordering, and visible user/assistant/system text; skip non-message blobs, tool calls, and tool results.
Hermes scope	Open state.db read-only as the canonical Hermes session store when present; import exported session JSON/JSONL as fallback; index visible user/assistant/system/developer/tool content; skip session_meta and empty/internal records.

Residuals

The live ChatGPT/Claude path is deliberately bounded: it syncs a configurable number of recent conversations per run, or exact ChatGPT IDs seeded from the native macOS app cache, and depends on the current web app response shape. Logged-in ChatGPT and Claude CDP write smokes have now passed against Jimmy's Chrome profiles using isolated temp archives. Unattended provider login remains out of scope; the browser profile is launched, but the user logs in normally.

Cursor branch/path metadata is not exposed through the observed store.db or state.vscdb message blobs, so Cursor imports mark path membership unknown instead of fabricating edges.

Validation

• GOWORK=off go mod tidy
• git diff --exit-code -- go.mod go.sum
• GOWORK=off go vet -p 1 ./...
• GOWORK=off go test -p 1 -count=1 ./...
• git diff --check
• bash -n scripts/live-provider-smoke.sh
• scripts/live-provider-smoke.sh --help
• scripts/live-provider-smoke.sh --provider chatgpt --cdp-url http://127.0.0.1:1 --chatgpt-app-cache <synthetic-cache> --mode dry-run --max-conversations 1
• Public temp-home release smoke from docs/RELEASE_CHECKLIST.md: build CLI into a temp dir, run fixture imports, web source syncs, reconciliation, status, searches, read-only SQL rejection, LaunchAgent generation, Markdown export, and CrawlBar manifest generation
• Schema/cursor/status tests: v1 databases migrate through schema v3; imports record source-hash cursors and seen conversation statuses; live web sync records provider update watermarks plus inaccessible detail statuses; repeated unchanged live syncs return no-change stats and update last_checked_at
• CLI smoke: build ./cmd/aicrawl, initialize a temp-home archive, run reconcile ./testdata/redacted/chatgpt-export.fixture.zip --provider chatgpt --json before import and verify missing rows, import the fixture, then reconcile again and verify missing coverage and divergent messages are zero
• CLI smoke: build ./cmd/aicrawl and verify --help exposes --cdp-url, --max-conversations, --browser, --remote-debugging-port, claude-code, cursor, hermes, and reconcile
• CLI smoke: import ./testdata/redacted/chatgpt-export.fixture.json --dry-run --json reports chatgpt/chatgpt_export counts and leaves status --json as uninitialized
• CLI smoke: schedule launchd --provider chatgpt --cdp-url http://127.0.0.1:9222 --out <tmp> writes a plist with ProgramArguments, --cdp-url, and StartInterval
• CLI smoke: schedule launchd --provider chatgpt --profile <tmp-profile> --browser /tmp/fake-chrome --remote-debugging-port 0 --out <tmp> writes a profile-launch plist with --profile, --browser, and --remote-debugging-port and no credential material
• CLI smoke: built CLI schedule launchd --provider codex --import-path <tmp-codex-root> --out <tmp> reports mode: import, writes an aicrawl import <path> --provider codex --json plist, validates with plutil -lint, and includes no web-sync or credential material
• CLI smoke: built CLI schedule launchd --provider claude-code --import-path "~/Library/Application Support/Claude" --out <tmp> expands the real space-containing Claude desktop path, validates the plist with plutil -lint, and preserves the path as one ProgramArguments string
• CLI smoke: built CLI imports the Codex redacted fixture, then status --json reports local_sources with Codex as seen, source_kind: codex_jsonl, imported counts, cursor_kind: source_hash, and OpenClaw as never_imported
• CLI smoke: built CLI schedule launchd --local-defaults --interval-minutes 5 --json in a temp home with Codex, Claude Desktop, and Hermes roots writes three import plists, skips four missing roots, preserves the Claude Desktop path with spaces, and all generated plists pass plutil -lint
• CLI smoke: after importing testdata/redacted/chatgpt-web-conversation.fixture.json, status --json reports web_sync with chatgpt_web as seen and claude_web as never_synced
• CLI smoke: stale ChatGPT capture with a valid web source fails with contract_stale and status --json remains uninitialized
• Manual smoke: import testdata/redacted/claude-web-conversation.fixture.json, search for web sync claude fixture assistant phrase, then dry-run sync web and verify claude_web freshness is seen
• Manual smoke: import OpenClaw, Codex, and Gemini redacted fixtures, then search for gemini cli fixture assistant phrase through the CLI
• Real local ChatGPT macOS app cache dry-run smoke against ~/Library/Application Support/com.openai.chat: scanned filename-derived cache IDs without writing the archive or reading .data bodies, reported source_kind: chatgpt_app_cache_ids, candidate_conversations: 215, messages: 0, source_warnings: 0, and freshness_state: archive_missing
• Real local Codex dry-run smoke against ~/.codex/sessions: scanned 1,255 JSONL sources without writing the archive, found 1,235 conversations and 56,933 messages, skipped 20 metadata/control-only sources, and emitted 20 redacted ordinal-only warnings
• Real local Claude Code dry-run smoke against ~/.claude/projects: scanned 2,067 JSONL sources without writing the archive, found 2,067 conversations and 15,977 messages with no warnings
• Real local Claude desktop dry-run smoke against ~/Library/Application Support/Claude: scanned 285 JSONL sources without writing the archive, found 285 conversations and 4,077 messages with no warnings
• Real local Gemini dry-run smoke against ~/.gemini: shape-filtered session JSON discovery skipped config/auth JSON without printing paths, scanned 309 session JSON sources, found 309 conversations and 615 messages, and emitted 101 bounded parser warnings with truncated 1 additional warnings as the final summary
• Real local Cursor dry-run smoke against the installed app support directory: scanned 4 state.vscdb sources without writing the archive, found 1 importable conversation and 3,313 messages, skipped 3 non-chat/empty state stores, and emitted 3 redacted ordinal-only warnings
• Real local Hermes dry-run smoke against ~/.hermes: selected the root state.db canonical source without writing the archive, reported provider: hermes, source_kind: hermes_session, sources: 1, conversations: 15, messages: 617, attachments: 0, and warnings: 9 without printing transcript text
• Real local OpenClaw dry-run smoke against ~/.openclaw/agents/main/sessions: scanned 769 JSONL sources without writing the archive, found 416 conversations and 30,627 messages, skipped 353 control-only sources, and emitted 101 bounded warnings with truncated 537 additional warnings as the final summary
• CLI smoke: copy the Codex redacted fixture into a temp session root, run directory dry-run, verify status remains uninitialized, import the directory, search for codex jsonl fixture assistant phrase, and repeat import to verify already_imported_sources: 1
• Unit test: mutate one archived message projection after import and verify reconcile reports divergent_messages: 1 without message bodies
• Manual smoke: build the CLI, import testdata/redacted/claude-code-session.fixture.jsonl with --provider claude-code, and search for claude code jsonl fixture assistant phrase
• Generated SQLite fixture tests: Cursor store.db, Cursor state.vscdb, and Hermes parser/CLI import/search paths create synthetic SQLite stores at test time, avoiding checked-in binary/private DB fixtures
• Hermes parser tests cover state.db, exported session JSON, exported session JSONL, no-visible-text warning aggregation, root-directory state.db preference over sidecar sessions, and provider-filtered search
• Fake-browser tests: TestLaunchProfileStartsBrowserWithDedicatedProfile, TestLaunchProfileReusesExistingProfileEndpoint, and TestSyncWebLaunchesMissingProfileAndReportsLoginRequired exercise profile launch/reuse/login-required behavior without real credentials
• OpenClaw parser tests: preserve redacted Telegram source-channel/sender metadata, aggregate no-visible-text message warnings per file, skip control-only files during directory import/dry-run, and cap warning output/stored import warnings
• Live fetch unit tests: ChatGPT and Claude skip 403/404/410 conversation details while preserving accessible details in the same batch; repeated provider list pages stop pagination instead of looping or requesting missing offsets
• Live dry-run tests: ChatGPT/Claude adapter inspections count list candidates without detail fetches; CLI dry-run reports source.kind: live_list, preserves an uninitialized archive, and reports live_list_unavailable as a warning when no provider page target is attached
• Fake-CDP integration tests: TestSyncWebLiveCDPImportsSearchableChatGPTPayload exercises command path, CDP attach, page-context fetch responses, archive import, and search without real provider credentials; TestSyncWebLiveCDPRecordsInaccessibleDetailStatus verifies accessible details import while skipped 404 details are stored in conversation_sync_status with timestamps
• Real logged-in ChatGPT CDP diagnostic against Chrome at 127.0.0.1:9224: page network reload showed ChatGPT itself reading /backend-api/conversation/<id> with HTTP 200, while cookie-only synthetic fetches returned empty lists/404 detail responses; the fix hydrates the bearer token from same-origin /api/auth/session inside the page and never prints or persists the token
• Real logged-in ChatGPT CDP dry-run smoke against Chrome at 127.0.0.1:9224: normal authenticated history list reported source_kind: live_list, candidate_conversations: 3, and warnings: 0
• Real logged-in ChatGPT CDP write smoke against Chrome at 127.0.0.1:9224: normal list write imported 1 conversation and 11 messages into the isolated temp archive with warnings: 0
• Real logged-in ChatGPT CDP seeded write smoke against the visible conversation ID: synthetic cache-ID seed imported 1 conversation and 11 messages into the isolated temp archive with warnings: 0
• Real logged-in Claude CDP dry-run smoke against Chrome at 127.0.0.1:9225: authenticated profile reported source_kind: live_list, candidate_conversations: 3, and warnings: 0 after raising the CDP websocket read limit for larger Claude list responses
• Real logged-in Claude CDP write smoke against Chrome at 127.0.0.1:9225: imported 1 conversation and 6 messages into the isolated temp archive with warnings: 0
• Live provider smoke readiness: both ChatGPT and Claude logged-in CDP dry-run/write smoke have passed; first-run unattended provider login remains out of scope
• CE browser pipeline check: agent-browser is installed, but this repo has no browser app, dev server, HTML UI, or route files to exercise.

---