A streaming NDJSON pipeline for composing agent tools without losing the record structure at each pipe boundary. Ships the apipe binary.
When the wintermute tools compose through shell + jq, every pipe flattens. A recall query returns hits with their evidence attached; pipe it through jq '.[].id' to take the top few and the evidence is gone — what comes out the other end is a list of ids, and getting the records back means re-querying. The structure that made the output useful does not survive being turned into bytes and reparsed at each stage.
apipe keeps it. The contract is one shared NDJSON record schema — every line has kind, source, id, ts — plus a thin streaming runtime whose stages enrich records rather than reduce them. A stage reads records and writes records; the envelope is invariant, so a ten-stage pipeline is as composable as a one-stage one. jq flattens because it has no schema to preserve; apipe has one, and that is the whole difference.
curl -fsSL https://raw.githubusercontent.com/j0yen/agent-pipe/main/install.sh | bashOr from a checkout:
git clone --depth 1 https://github.com/j0yen/agent-pipe.git
cd agent-pipe && ./install.shBoth run cargo install --path . --locked, which puts apipe in ~/.cargo/bin. Needs cargo / rustc 1.85+ and git. If apipe isn't found afterward, add ~/.cargo/bin to $PATH.
recall query 'foo' --format apipe | apipe top 5 | apipe prettyrecall emits records, apipe top 5 keeps the five highest-scoring ones — with their evidence intact, because the record is what flows — and apipe pretty renders a table for a human to read. Swap the last stage for apipe to-paths and the same pipeline feeds a shell script instead.
Each subcommand reads NDJSON on stdin and (except the sinks) writes NDJSON on stdout. Every stage validates each line against the required fields first; a malformed record is echoed unchanged with a 1-indexed diagnostic on stderr, never silently dropped.
| Stage | What it does |
|---|---|
apipe pass |
Identity filter. Validates every line; use it to type-check a pipeline. |
apipe top N [--by <field>] |
Keep the N highest by score (or --by). Stable on ties; bounded memory. |
apipe sort -k <field> [--desc] |
Sort all records by a field — numeric if every value is numeric, else lexicographic. Stable. |
apipe filter '<expr>' |
Keep records where a <field> <op> <literal> predicate holds. Ops: == != < <= > >= and ~ (substring). |
apipe schema [--kind <name>] |
List known kinds, or print the JSON Schema for one. |
apipe pretty |
Sink: a width-aware human-readable table. Output is not NDJSON. |
apipe to-paths |
Sink: bare path strings from file_event records or any record with payload.path. |
apipe from-paths |
Inverse of to-paths: wrap bare path strings back into file_event records. |
Every record carries the same envelope — kind, source, id, ts — plus kind-specific payload fields. Three kinds ship today: recall_hit, transcript_turn, file_event. apipe schema lists them; apipe schema --kind recall_hit prints the JSON Schema.
This is the runtime slice. apipe and the shared schema are complete and tested; the producer side — teaching the existing wintermute tools a --format apipe adapter mode — lands separately. Until a tool speaks the schema natively, feed apipe records you've shaped yourself.
apipe is the connective tissue for the wintermute tools and recall: the format they emit and the runtime that composes them. It was built from a PRD through the autobuilder pipeline.
Dual-licensed under MIT or Apache-2.0, at your option.