Export prompts and conversations

[tony]

Summary

Add a read-only export surface that moves matched prompts and conversations out of agentgrep into portable artifacts: a streaming NDJSON default for data interchange, a Markdown transcript for humans, ML-training profiles (OpenAI/Anthropic messages), and an optional Mermaid graph of thread/similarity structure. Export lands as a dedicated CLI verb, an MCP tool plus a parameterized resource, and a TUI action, all consuming the same normalized records and the same stable id from #80.

Problem & need

Once a prompt or conversation is found, agentgrep has no way to get it out in a shape another tool can consume. The user pains this blocks:

• Archival / sharing — save a session as a diffable file, paste a transcript into a PR or bug report.
• Offline reading — a self-contained Markdown/HTML copy that survives store rotation or a machine migration.
• Feeding fine-tune / eval datasets — emit conversations as messages JSONL for training or evaluation pipelines.
• Referencing — cite a specific record/thread by a stable id (bookmarks Bookmark prompts / conversations #79, an export manifest, a similarity seed Find similar prompts and conversations #82).

What is missing today:

• There is no export command and no document renderer. Output selection is a boolean mutually-exclusive group (--json / --ndjson / --ui) folded into OutputMode by add_output_mode_options / parse_output_mode in src/agentgrep/cli/parser.py. src/agentgrep/cli/renderers.py is ANSI-terminal only; src/agentgrep/cli/serializers.py emits SearchRecordPayload JSON/NDJSON. None of these is a portable document format.
• There is no conversation-assembly primitive. Conversation scope still yields flat per-turn SearchRecords (role, timestamp, session_id, conversation_id); the only grouping helper, group_by_session in src/agentgrep/ranking.py, buckets on session_id alone and collapses cross-backend Nones. Turning turns into an ordered role transcript is new work.
• There is no stable id to key an export on. search_record_fingerprint in src/agentgrep/mcp/refs.py is the closest prototype (a sha256 over canonical JSON including text_sha256 and the home-collapsed display path), but it is MCP-only and disagrees with the engine's weaker record_dedupe_key in src/agentgrep/_engine/orchestration.py (kind, agent, store, session_id|conversation_id|path, text). Those are the two identity schemes live on master; two more text-hash schemes exist only in unmerged workstreams (an unmerged on-disk DB cache workstream's source_id/record_id and Insights graph similarity engine + non-gated transformers LLM defaults #69's sha1(text)[:16] graph key). An export that cites an id must consume the unified Deterministic IDs for conversations / prompts #80 id rather than pick one of these or invent another.

The substrate is otherwise ready: normalized SearchRecord/FindRecord (src/agentgrep/records.py), a schema_version-stamped envelope (build_envelope), a streaming event generator (iter_search_events / aiter_search_events in src/agentgrep/_engine/search.py), and home-eliding paths via format_display_path. Export is therefore mostly a new writer set plus a thin verb/tool, not new engine work.

Where it fits

Export must reach parity across all four surfaces, driven by one frontend-neutral assembly + payload shape so the formats never drift:

• Headless / CLI — a dedicated agentgrep export verb (grep already owns -o/--only-matching and -m/--max-count in src/agentgrep/cli/parser.py, so overloading search is awkward). stdout by default, -o FILE to write, -o - for explicit stdout; pipeable for jq -c.
• TUI (pi-minimal) — an "export selection as markdown" action off src/agentgrep/ui/. Per ADR 0011, the rendering/serialization pass runs on an @offload worker; the pump-side callback only flips state, never performs the write.
• MCP — an export_records tool that consumes existing agref1: refs / agcur1: cursors and returns rendered content, plus a parameterized export resource for single-unit addressability.

Performance / opt-in posture. Export is a separate verb + tool + resource. Format writers (especially HTML templating) are lazy-imported and only reachable from the export path, so users who never export pay zero cold-start and zero per-search tax. NDJSON streams straight from the event generator rather than collecting the full result set. This mirrors how db and ranking keep heavy imports function-local.

Design directions

Tiered, opt-in-first. Tier 0 ships with no new dependencies; higher tiers are --format / --profile choices, not new engine work.

Tier 0 — data interchange (default, no new deps)

New agentgrep export verb with an ExportArgs dataclass (mirroring the db subparser as the template for a structured non-search subcommand) and a run_export_command dispatch branch beside the existing run_*_command handlers.

• --format ndjson (default) — one serialized record per line, flushed from iter_search_events; constant memory, appendable, pipeable. This is the existing --ndjson serializer reframed as an export target plus the Deterministic IDs for conversations / prompts #80 id per line.
• --format json — a single build_envelope document for small --limited selections; not streamable, so cap it.
• Introduce the ADR-0006-canonical --format on this verb rather than adding more boolean flags. Default sink is stdout; -o/--output writes, -o - forces stdout.
• Determinism: impose an export-owned total order (timestamp then Deterministic IDs for conversations / prompts #80 id) independent of scheduler nondeterminism, plus stable JSON key order, so reruns are byte-identical and diffable. The engine's newest-first + session-dedupe default is not a total order.
• Redaction is load-bearing: every payload path already routes record.path through format_display_path; export must keep that and never emit raw pathlib.Path. Add --redact to hash/drop text while keeping ids + shape, and default MCP exports to ids + metadata with bodies behind an explicit --bodies / bodies=true (mirroring ADR 0005's --include-text posture).

Tier 1 — human artifacts (--format markdown|html)

• Markdown transcript — front-matter carrying the Deterministic IDs for conversations / prompts #80 id + provenance (agent/store/session_id/model/timestamp), then per-turn ## role sections with fenced bodies (bump fence length when a body contains triple backticks). Round-trips via the front-matter id, not the prose. Doubles as the TUI "copy as markdown" output and the MCP text content.
• Conversation assembly is the real new primitive: a frontend-neutral helper (natural home: src/agentgrep/ranking.py or a new sibling) that groups by session_id-or-conversation_id-or-path (not bare session_id), orders turns by timestamp then discovery order, and labels role turns against USER_ROLES. Define it once so CLI, MCP, and TUI render from the same structure.
• Mermaid graph block (opt-in sub-block, gated on Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69) — a fenced flowchart/gitGraph where nodes are keyed by short Deterministic IDs for conversations / prompts #80 ids and edges are similarity/thread links, emitted in sorted order for deterministic diagrams. Renders inline on GitHub and in the docs.
• Self-contained HTML — lowest priority, deferrable. A single file with inlined assets; user text MUST be HTML-escaped (an exported prompt is an XSS surface). Keep the writer lazy-imported.

New writers live in a new src/agentgrep/cli/export.py sibling that consumes the plain-dict SearchRecordPayload (never imports pydantic) so it inherits the pydantic-free fallback automatically; add a test mirroring test_json_output_falls_back_without_pydantic.

Tier 2 — ML interchange (--profile, layered over NDJSON)

Factor a neutral internal messages schema with thin provider adapters (agentgrep is cross-agent; no vendor is privileged):

• --profile openai-messages (default) — {"messages":[{"role","content"},...]} per conversation, the de-facto fine-tune shape.
• --profile anthropic-messages — same idea with system hoisted and content-block support.
• --profile sharegpt — offered only as a labeled legacy option; document its deprecation toward role/content templates.
• Do not ship ChatML as an export target — it is a downstream template render of messages, not a data format. Ship structured messages and let the consumer apply a template.
• Carry agentgrep provenance (agent/store/Deterministic IDs for conversations / prompts #80 id) in a sibling metadata field or a separate manifest so the vendor messages shape stays clean.

Tier 3 — tabular (--format csv, prompt scope)

One row per record via stdlib csv.writer (never hand-rolled joins — embedded newlines/commas/quotes must go through the writer). Lossy for conversations; scope to prompts for spreadsheet/dedup-audit use.

MCP surface

• export_records tool — takes refs=[agref1:...] or a query, plus format/profile/bodies, returns a FastMCP ToolResult carrying content blocks (a ResourceLink to defer-fetch a large transcript, or an inline EmbeddedResource for a small markdown/mermaid artifact) alongside structured_content. Because of the 512KB response cap, large sets must paginate via the existing agcur1: cursors or return a link rather than inline bytes.
• Export resource — a parameterized agentgrep://export/{ref}?format=markdown template (mirroring the existing agentgrep://sources/{agent} pattern) so one conversation is addressable by id: the "referenceable" half to the tool's "bulk/streamed" half.
• Any user-text tool arg must be added to the audit middleware's redaction set so it is not logged in the clear.

Prior art & inspiration

• git/git@v2.54.0 Documentation/git-format-patch.adoc — stdout-default vs -o <dir>, and the "portable self-describing unit that can be re-applied" framing for an exported record.
• pallets/click@8.4.1 src/click/types.py — the File type where a lone - means stdin/stdout; the convention for -o - keeping every format pipeable.
• jupyter/nbformat@v5.10.4 nbformat/v4/nbformat.v4.schema.json — versioned JSON schema + stable per-cell id; the discipline that makes exports re-importable and diffable (schema_version is already present; add the Deterministic IDs for conversations / prompts #80 id).
• openai/openai-cookbook@857eb0b examples/Chat_finetuning_data_prep.ipynb — the {"messages":[{role,content}]} per-line JSONL schema for the default openai-messages profile.
• huggingface/trl@b98dd9f docs/source/dataset_formats.md — the "conversational" (messages) vs "standard" distinction that justifies a neutral internal messages schema with provider profiles.
• huggingface/transformers@effde20 docs/source/en/chat_templating.md — evidence that ChatML (<|im_start|>) is produced by applying a chat template to messages; ship messages, not ChatML.
• axolotl-ai-cloud/axolotl@bf19bff src/axolotl/utils/schemas/validation.py — ShareGPT (from/value) deprecation toward role/content; offer ShareGPT only as a labeled legacy profile.
• anthropics/anthropic-sdk-python@v0.116.0 src/anthropic/types/message_param.py — the Anthropic Messages request shape (system hoisted, content may be blocks) for the anthropic-messages profile.
• mermaid-js/mermaid@11.16.0 docs/syntax/flowchart.md and docs/syntax/gitgraph.md — node/edge grammar for a Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69 similarity graph and a single conversation's thread structure, keyed by Deterministic IDs for conversations / prompts #80 ids.
• jlowin/fastmcp@v3.4.2 fastmcp_slim/fastmcp/tools/base.py and fastmcp_slim/fastmcp/resources/template.py — ToolResult carrying content blocks + structured_content, and ResourceTemplate for the parameterized export resource.
• jqlang/jq@jq-1.8.2 docs/content/manual/manual.yml — the -c line-oriented streaming-consumer expectation that makes NDJSON the pipeable default.
• BurntSushi/ripgrep@15.1.0 GUIDE.md — the per-line begin/match/end JSON event model agentgrep's grep serializers already mirror; reuse for streamed NDJSON export.
• python/cpython@v3.15.0b1 Lib/csv.py — csv.writer quoting so the CSV profile never hand-rolls delimiters around multi-line prompt text.

Relationship to other issues

• Deterministic IDs for conversations / prompts #80 (deterministic IDs) — hard prerequisite. Export must carry ONE stable id per record/conversation, used as both the re-import upsert key and the citation handle. Deterministic IDs for conversations / prompts #80 must unify search_record_fingerprint (src/agentgrep/mcp/refs.py) with record_dedupe_key (src/agentgrep/_engine/orchestration.py) so the exported id and the dedupe key agree; Export prompts and conversations #81 consumes that id rather than inventing another of its own.
• Bookmark prompts / conversations #79 (bookmark). A bookmark references a record or a conversation by Deterministic IDs for conversations / prompts #80 id (the id is addressable at both levels); export cites the same id, so a bookmark set is directly exportable/shareable.
• Find similar prompts and conversations #82 (find-similar) / Insights graph similarity engine + non-gated transformers LLM defaults #69 (insights graph). The Mermaid graph block needs similarity edges from Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69; export is a downstream consumer that can also dump a neighbor set. Conversation-level export needs the stable conversation id that Find similar prompts and conversations #82/Insights graph similarity engine + non-gated transformers LLM defaults #69 also require.
• ADR 0006 (public CLI/MCP contract). Export introduces the canonical --format output selector the ADR reserves and is the privacy sharp edge — the operation that moves prompt text off-box — so bodies must be explicit/opt-in and machine-readable diagnostics/next-actions must stay free of prompt text and local paths.
• ADR 0004 (headless planning + streaming execution). Export reuses the streaming event model (iter_search_events) and scope narrowing rather than adding engine work.
• ADR 0005 (insights ladder). The Markdown renderer is shared with the insights report; heavy writers (HTML) follow the lazy dependency ladder.
• ADR 0011 (non-blocking TUI). A TUI export action must run its serialization on an @offload worker; the pump-side confirm callback only flips state.

Open questions

1. Verb vs flag. Ship a dedicated agentgrep export verb, or search --format markdown|jsonl? A verb isolates deterministic-sort/profile/redaction from search semantics and sidesteps grep's existing -o, at the cost of more surface.
2. Conversation assembly ownership. Where does the frontend-neutral transcript-assembly primitive live, and how does it degrade where conversation_id is nullable, a cwd path (Pi), or absent entirely (Cursor CLI prompt_history, VS Code inline history)? Falling back to display_path + ordinal keeps grouping stable, but the grouping key must be documented.
3. Re-import scope. Is round-trip import in scope for Export prompts and conversations #81 or a follow-up? agentgrep is read-only over the source stores, so "re-importable" likely means "another consumer can upsert by the Deterministic IDs for conversations / prompts #80 id," not that agentgrep ingests its own exports. Confirm whether an import verb exists at all.
4. Redaction defaults. Bodies-on for CLI but ids-only over MCP? Should --redact hash text (keeping text_sha256 stable) or drop it entirely, and does a redacted export stay round-trippable at the id level only?
5. Text-encoding stability. search_record_fingerprint (src/agentgrep/mcp/refs.py) encodes text with a plain .encode("utf-8"), which raises on lone surrogates from imperfectly-decoded stores. The exported id / text_sha256 needs errors="surrogatepass" to stay stable — is that fix owned by Deterministic IDs for conversations / prompts #80 or Export prompts and conversations #81?
6. ML profile set and provenance. Confirm shipping openai-messages + anthropic-messages now, sharegpt as labeled legacy, and ChatML out. How is agentgrep provenance (agent/store/Deterministic IDs for conversations / prompts #80 id) carried alongside the vendor messages shape — a sibling metadata field per line, or a separate manifest?
7. MCP response size. Given the 512KB response cap, does export_records default to ResourceLink (defer-fetch) for large transcripts and inline EmbeddedResource only under a size threshold, and how does that interact with agcur1: pagination?

Goal

Idempotent completion condition for /goal. It asserts an end state, not an action: if the state already holds, the first evaluation passes and nothing is edited.

/goal Issue #81 (Export) is decidable: its body separates a first contract from deferred tiers and answers each question in one sentence. Report FIRST in the shared master checkout: git rev-parse --abbrev-ref HEAD (prints master) and gh issue view 81. Today all seven numbered questions still end in a question mark and four format tiers read as one deliverable, so nothing judges branch issue-81-export; if contract, decisions and deferrals are written, stop with no edits. End state: the body names the first contract (a dedicated export verb, deterministic NDJSON and Markdown, a stdout-or-file sink, bodies policy, bounded selection, schema_version, one shared field allowlist, an MCP tool with an inline size ceiling, and a TUI action obeying ADR 0011: no open, scandir, sqlite, subprocess, network, json load/dump or unbounded CPU reachable from a pump callable even one hop down; new pump entrypoints @pump_only, workers @offload) and defers, not cancels: json envelope, html, csv, mermaid, vendor messages profiles, redaction and re-import. Two seams stay open: the MCP tool treats record refs as versioned opaque handles whose encoding may change, so agref1:/agcur1: bytes are not the contract; and the allowlist excludes local source paths but reserves conversation/thread identity as an additive future field, which #82 and #142 need. Export consumes the #80 id rather than minting one; format, ceiling and allowlist stay versioned. Do not assert export reads only live stores; allocate no ADR number. Only one mutation is permitted, gh issue edit 81 --body-file: no comment, label or state change, no branch touched, git status --porcelain empty. If gh writes are unavailable, print the final body plus that command and stop; that counts as met. Or stop after 8 turns and report what remains.

• Proof — gh issue view 81 --json body --jq '.body'
• Idempotence — With the first contract, seven decisions and deferral list present, the opening gh issue view transcript already satisfies the condition, so the one permitted edit is skipped and nothing is appended on re-run.
• Blocked by — nothing (it records that export consumes the Deterministic IDs for conversations / prompts #80 id, but does not require Deterministic IDs for conversations / prompts #80 to be resolved first)
• Blocks — nothing hard; Add portable record export across surfaces #122 reads it as acceptance criteria
• Preserves — Higher export tiers (json envelope, html, csv, mermaid, vendor messages profiles, redaction, re-import) are deferred rather than cancelled; refs are consumed as versioned opaque handles so agref1:/agcur1: evolution and retirement stay open; conversation/thread identity is reserved as an additive allowlist field for Find similar prompts and conversations #82's similarity citation and Durable prompt corpus and derived search indexes (ADR 0019) #142's upsert key; export asserts nothing about there being no index or provider.