smartpipe

Semantic pipes and queries for your terminal.

Run PDFs, images, audio, video, and text through Unix verbs that understand their input.

▶ Watch the full 86-second demo - sound, controls, no download.

Install

macOS - Homebrew is the happy path:

brew install prabal-rje/tap/smartpipe

Linux - one line (uses Homebrew if you have it, bootstraps uv otherwise; works on macOS too):

curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh

Windows:

powershell -ExecutionPolicy Bypass -c "irm https://prabal-rje.github.io/smartpipe/install.ps1 | iex"

Prefer your own Python tooling? Any of these works on every platform:

uv tool install smartpipe-cli           # uv
pipx install smartpipe-cli              # pipx
pip install smartpipe-cli               # pip
uvx --from smartpipe-cli smartpipe      # zero-install trial

Stay current with smartpipe update - it detects how smartpipe was installed and runs that tool's own upgrade.

Point it at a model

# log in with a ChatGPT account - covers most people
smartpipe auth login

No ChatGPT plan? Bring a cloud API key, or run local Ollama - see Models & providers.

Examples

# summarize each filing, figures included
smartpipe map "summarize the key risk" 'filings/*.pdf'

# keep only the calls that sound frustrated - audio, understood
smartpipe filter "the caller sounds frustrated" 'calls/*.mp3'

# text on stdin works the same way
echo "hello world" \
| smartpipe map "translate to Spanish"
# → hola mundo

PDFs pass through with their figures. Scanned pages route to a vision model. Audio is sent natively where the model hears it, transcribed locally otherwise. Video is sent whole where the wire supports it, or split into frames plus a transcript. Each conversion is noted per row.

It composes with grep, jq, sort, and tail -f: stdin to stdout, one item at a time.

Verbs

A verb is one operation on your data - map, filter, cluster. Each reads stdin (or named FILES) and writes stdout, so verbs pipe into each other and into ordinary Unix tools. Every verb is documented at prabal-rje.github.io/smartpipe.

Semantic verbs call a model:

Verb	What it does	Feels like
map	transform each item - text or media - with a prompt	sed, but it understands
extend	add extracted fields; keep everything else	your record, plus columns
filter	keep items matching a plain-English condition	grep, but semantic
embed / top_k	vectors; rank by similarity	sort | head, by meaning
reduce	synthesize many items into one	awk END, but literate
join	match two inputs (--kind inner|leftouter|anti); --on alone is free	SQL join, but semantic
cluster	group by meaning, label each group	themes with sizes and quotes
distinct	fold near-duplicates; --exact is free	sort -u, by meaning
diff	what distinguishes two sets	the post-incident answer
outliers	the items least like the rest	novelty, surfaced
graph	corpus → entity/relationship graph; --fast is free	the case wall, with citations

Free verbs never call a model. Run them first to cut the corpus before any paid stage:

Verb	What it does	Feels like
where	filter on exact field predicates	SQL WHERE
summarize	count, average, percentiles, time buckets	SQL GROUP BY
sort	order items by a field	sort
sample	take a seeded random subset	shuf
getschema	list fields, types, and coverage	head, for structure
split	break items into pieces (pages, minutes)	split
chart	terminal bars, SVG, facets, time series	quick plots

Some semantic verbs have a conditionally free mode: join --on (key equality, no prompt), distinct --exact (hash-only folding), graph --fast (local NER, on-device), map/extend --dry-run (compose without sending), and smartpipe schema with a braces/DSL expression. Each makes no chat-model calls; graph --fast is $0 by default and spends only if you configure a cloud --embed-model or a remote --stt-model (both disclosed at run time).

A one-minute tour

# 1. point smartpipe at a model (ChatGPT login, a cloud key, or local Ollama)
smartpipe use

# 2. ask one question across a folder of mixed documents
smartpipe map "What does this say about pricing?" 'docs/*.pdf'

# 3. typed extraction - braces carry names, types, and guidance
cat tickets.jsonl \
| smartpipe extend "Add {label enum(bug, feature, praise), urgency number: 0 to 1}"

# 4. group feedback by meaning, label each theme, chart it
cat feedback.txt \
| smartpipe cluster --explode members \
| smartpipe chart cluster --top 8 --save themes.svg

# 5. cut for free with `where`, then let the model judge only what is left
cat app.log \
| smartpipe where 'text has "ERROR"' \
| smartpipe filter "an actual outage"

# 6. save a whole pipeline as a file; it runs as a command
smartpipe run triage.sem --dry-run   # prints the stage graph and cost, makes zero calls

# 7. month-end close: the vision model IS the OCR; the anti-join is the worklist
smartpipe map "Extract {vendor string, invoice_number string, total number}" 'invoices/2026-06/*.pdf' \
| tee june-invoices.ndjson \
| smartpipe join "the same payment" --right ledger.jsonl --kind anti > missing-from-ledger.jsonl

# 8. video RAG, no vector database: index a folder of recordings once, ask any day
smartpipe embed 'sessions/**/*.mp4' > sessions.embeddings
smartpipe top_k 3 --near "user gives up after the coupon fails" < sessions.embeddings

# 9. the corpus as a knowledge graph - $0 by default: local NER, on-device
smartpipe graph --fast 'case-files/*.md' --save case.html

A .txt on a pipe arrives one item per line; --as file treats the whole file as one item (feeding smartpipe has the full ingestion table).

Numbers 7 and 8 are full recipes - invoice reconciliation and video RAG - two of a dozen in the cookbook.

New to this? The Learn track starts at zero and assumes nothing, including what a "model" is.

Highlight: the knowledge graph

Number 9 is new: graph --fast turns a folder of mixed files into an interactive, cited knowledge graph entirely on-device by default - a local NER model finds the entities you name, co-occurrence weights the edges, and hovering any edge shows the files behind it. A focus prompt upgrades the strongest edges to model-read relations. Full story: the graph page and the cookbook recipe.

Try it on real files

No corpus handy? smartpipe-playground ships 26 MB of CC0 / public-domain practice files - invoices, reports, photos, recordings, screen sessions, and JSONL data:

curl -L https://github.com/prabal-rje/smartpipe-playground/releases/download/v1/smartpipe-playground-v1.tar.gz | tar xz
cd smartpipe-playground

smartpipe map "Extract {vendor, invoice_number, total number}" 'invoices/*.pdf'
smartpipe filter "the customer sounds frustrated" 'recordings/*.mp3'
smartpipe top_k 3 --near "supply chain risk" 'reports/*.pdf'

For AI agents

Driving smartpipe from an agent? SKILL.md is the operating manual - invocation patterns, the cost model, machine-output contracts, and quoting gotchas, written for programmatic use.

Where your data goes

Some steps run locally no matter which chat model you pick: embeddings (fastembed) and transcription (whisper) are built in.

For chat, Ollama runs models on your machine. Any cloud model sends that run's data to its provider - gpt-5.4-mini, claude-opus-4-8, gemini-3.1-flash-lite, mistral-large-latest, openrouter/….

API keys come from environment variables, or smartpipe auth login stores one securely for you. ChatGPT subscribers can log in instead of using a key. smartpipe usage keeps local run and token totals; see Privacy & security for where credentials live and how to remove them.

Unix behavior

• stdout is data, stderr is diagnostics. Progress and receipts never touch your pipe.
• Adapts to where it runs. Readable tables at a terminal; JSONL when piped into another command.
• Order-preserving. Output order matches input order, even with parallel calls.
• Failure-tolerant. One bad item is a warning, not a crash.

Learn more

Full docs: prabal-rje.github.io/smartpipe.

• Learn track - zero to first result, six short chapters
• Install - packages and platforms
• Working with files & media - PDFs, scans, images, audio, video
• CLI reference - every flag, format, and exit code
• Models & providers - ChatGPT login, cloud keys, local Ollama
• Privacy & security

How to cite

If smartpipe is useful in your research, cite it (or run smartpipe cite):

@software{gupta_smartpipe_2026,
  author = {Gupta, Prabal},
  title = {smartpipe: semantic pipes for your terminal},
  year = {2026},
  version = {1.6.0},
  license = {Apache-2.0},
  url = {https://github.com/prabal-rje/smartpipe}
}

GitHub's "Cite this repository" button (from CITATION.cff) gives APA too.

Development

Built in the open, under Apache-2.0. Contributor setup and the quality gates are in CONTRIBUTING.md; the manual release pass lives in qa/. The CLI surface is a SemVer contract.