Skip to content
zsltgPublic

About

Run jq filters across NoSQL databases and their dump files to query, copy, diff and restore data, from one static binary. A filter that names keys reads only those keys.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

1 watching

Forks

Latest commit

 

History

666 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

iq

CI Coverage
OpenSSF Scorecard OpenSSF Best Practices CodeScene Code Health
Release Go Reference Go version License: MIT

Documentation · Quick start · Install · AI agents · Usage · Drivers · Dumps · Contributing

iq

iq runs jq filters to query, dump, copy, diff and write data across NoSQL databases, and their dump files, from a single static binary. See Drivers for supported databases.

iq registers a MongoDB source, reads one document by key, filters a scan with a pushed-down predicate, explains the plan, and prints the result as gron

Why iq

  • Query any supported store with one language. The same jq filter runs on Redis, MongoDB, Cassandra, DynamoDB and the other drivers, and gives the same result on each. One filter can also join data from several sources, with source() or iq combine.
  • A filter that names keys fetches only those keys. A .[] filter scans the source in pages.
  • Query and compare dumps without a database server. Restore them to a live store.
  • Compare two sources, or a source and a dump, by data, stats, or inferred schema. iq diff returns diff(1) exit codes, so a CI job can fail on drift.
  • Infer a schema from a sample of any source with iq schema, as JSON Schema draft 2020-12 or an Open Data Contract Standard v3.2.0 contract.
  • Copy typed data across stores in one command, with no export script and no conversion script.
  • Give an AI agent access to every store with one Agent Skill and one MCP server. The skill works with any agent that reads the Agent Skills format. The server is read-only by default.

iq is inspired by sq, whose command set it deliberately follows to make the tool feel familiar. Read Comparison for how iq relates to other tools, and How it works for the read routes and pushdown.

Note

iq is built with AI assistance. Every change passes the full test suite, container-backed integration tests for every backend, and a mutation gate before it lands (see CONTRIBUTING.md).

Feedback and bug reports are very welcome. Report security problems privately, see the security policy.

Quick start

You need Docker, and iq on your PATH. To install iq on Linux or macOS:

curl -fsSL https://raw.githubusercontent.com/zsltg/iq/main/install.sh | sh

For other platforms, see Install. If the installer prints a note about PATH, add the directory that the note names.

Run each command on its own. If a command fails, stop and read its error.

Start a disposable MongoDB that only this machine can reach:

docker run -d --name iq-quickstart -p 127.0.0.1:27018:27017 mongo:8.2

Register the database as the active source. iq add connects to make sure that the source works, and --timeout 30s gives MongoDB time to start:

iq add -a --timeout 30s 'mongodb://localhost:27018/iq?collection=books'

Write two sample books. iq reads piped stdin as a source and takes each key from the id field:

printf '%s\n' '{"id":1,"title":"Dune","year":1965,"price":9.99}' \
              '{"id":2,"title":"Project Hail Mary","year":2021,"price":14.5}' |
  iq --insert books --key-field id

Read one book by its key. iq fetches only that document:

iq '.["2"]'

The output is similar to the following:

{
  "_id": "2",
  "id": 2,
  "price": 14.5,
  "title": "Project Hail Mary",
  "year": 2021
}

Filter a scan, and show the query plan:

iq '.[] | select(.year > 2015) | {title, price}' --explain

The output is similar to the following, where server filter shows the part of the filter that MongoDB runs. The full filter then runs again on the client, so the result is the same with or without pushdown:

query plan
source: books (mongo)
...
mongo calls:
  find(<filter>): server-side pre-filter, then the full jq re-runs client-side
  cursor streamed in batches of 100

server filter (mongo find):
  {                                 — .year > 2015
    "$or": [
      {"year": {"$gt": 2015}},
      {"year": {"$type": ["string", "array", "object"]}}
    ]
  }
...

Without --explain, the same command prints {"price": 14.5, "title": "Project Hail Mary"}.

When you are done, remove the container and the source:

docker rm -f iq-quickstart
iq rm books

Install

iq ships as a single static binary (no runtime dependencies, no CGO), prebuilt for Linux, macOS, and Windows on amd64 and arm64.

Linux

curl -fsSL https://raw.githubusercontent.com/zsltg/iq/main/install.sh | sh

The script downloads the release for your OS/arch, verifies its SHA-256 against the release checksums, and installs the binary. IQ_VERSION pins a version and IQ_INSTALL_DIR picks the target directory. Or grab a .deb, .rpm, .apk, or Arch .pkg.tar.zst from the releases. Each release carries a cosign signature and SLSA build provenance, see Verify a release.

macOS

brew install zsltg/tap/iq

The Linux curl … | sh one-liner works on macOS too.

Windows

scoop bucket add zsltg https://github.com/zsltg/scoop-bucket
scoop install iq

Go

go install github.com/zsltg/iq@latest

From source

git clone https://github.com/zsltg/iq
cd iq && make build

Shell completions and man page

The .deb, .rpm, .apk and .pkg.tar.zst packages install bash, zsh and fish completions and the iq(1) man page. For a brew, scoop, go-install or source build, iq completion <shell> prints a completion script and iq man prints the man page. Read Shell completions for the setup of each shell.

Use with AI agents

Agent Skill

iq ships an Agent Skill, a single Markdown file (skills/iq/SKILL.md) that any agent reading the Agent Skills format can load. It covers these topics:

  • Finding a source
  • --explain before every scan
  • --dry-run before every write
  • The machine-readable output flags and the JSON error shape
  • The rules around destructive commands.
npx skills add zsltg/iq

MCP server

For an agent that speaks MCP, iq mcp serves the same core over stdio.

It is read-only by default. --allow writes|exec|destructive opens the rest. A tool that is not allowed is never registered. Every result is capped, every error is redacted, and every destructive call is confirmed.

Claude Code:

claude mcp add iq -- iq mcp --timeout 30s

Codex CLI:

codex mcp add iq -- iq mcp --timeout 30s

Antigravity CLI:

agy mcp add iq -- iq mcp --timeout 30s

OpenCode, Cursor, Cline, the Antigravity IDE and Copilot in VS Code take a JSON block. Read AI agents for the block and the configuration file of each client.

Usage

Sources

Driver list:

iq driver ls

Add a collection named "books" from a MongoDB source and make it active:

iq add -a 'mongodb://localhost:27017/iq?collection=books'

Inspect a source:

iq inspect books

List sources:

iq ls

Check Sources for more details.

Query data

Fetch an item with the ID "2":

iq '.["2"]'

Filter a scan and reshape each item:

iq '.[] | select(.year > 2015) | {title, price}'

Print a formatted query plan:

iq '.[] | select(.year > 2015) | {title, price}' --explain -v

Check Query data for more details.

Diff

Diff schema of two sources:

iq diff --schema dev qa

Diff the items with the same ID from two sources:

iq diff 'dev=.["1"]' 'qa=.["1"]'

Diff items key by key:

iq diff dev qa

Check Diff for more details.

Write data

Important

Queries are read-only. --insert, --replace, iq data clear, iq data drop and iq data delete write to the target. iq exec forwards a native command to the database, so it can write too. Use --explain to see the query plan, or --dry-run to report the effect of a write without changing anything. iq exec has no dry run.

Copy one source into another, keys preserved (a cross-driver copy carries values only):

iq --src books --insert books2

Create a lossless (typed) dump:

iq --src cache --typed -o dump.jsonl

Add a dump as a source and restore it to a live source:

iq add file:///dump.jsonl -n snap
iq --src snap --insert cache

Copy only the items a filter selects:

iq '.[] | select(.year > 2015)' --src books --insert recent

Check Write data for more details.

Cross-source query

Compose across sources:

iq 'INDEX(source("users"; ".[]"); .id) as $u
    | source("orders"; ".[] | select(.total > 99)")
    | {name: $u[.userId].name, total}'

Combine across sources:

iq combine 'users=.[] | {id, name}' \
           'orders=.[] | select(.total > 99)' \
   --with '($users | INDEX(.id)) as $u | $orders[] | . + {name: $u[.userId].name}'

Check Cross-source queries for more details.

Keyspace commands

Delete two items:

iq data delete cache book:1 book:2

Empty a source called "cache":

iq data clear cache

Drop a collection called "orders":

iq data drop shop.orders

Check Delete, Clear and Drop for more details.

UNIX pipes

Implicit stdin source:

cat dump.jsonl | iq '.[]'

Check Insert for more details on piped stdin.

Drivers

iq picks the backend from a source's URI scheme. The query core is driver-agnostic, so further backends slot in behind the same port. The Drivers page documents these topics for each driver:

  • Keyspace mapping
  • Value encoding
  • Predicate pushdown
  • Raw commands.
Name Database Versions
cassandra Apache Cassandra 3.11+
couchbase Couchbase 7.x, 8.x (Community or Enterprise)
couchdb Apache CouchDB 2.x, 3.x
dynamodb Amazon DynamoDB AWS (managed)
elasticsearch Elasticsearch 8.x, 9.x
file Local dump file, read-only (see Backups and dumps) —
hbase Apache HBase 1.0+
mongo MongoDB 4.2+
neo4j Neo4j 5.x
opensearch OpenSearch 2.x, 3.x
redis Redis 7.0+

The Versions column lists the range of backend server versions the bundled client library supports.

Guarantees

Every backend honors the same contract. The Drivers page gives the details for each driver.

  • One URI, native nouns. The URI scheme picks the driver. The keyspace is the backend's own noun, such as ?collection=, ?table= or ?index=.
  • One jq interface. A filter that names keys fetches only those keys, and a missing key reads as null. A .[] filter streams the keyspace in pages. A filter that needs the whole keyspace at once runs only with --unbounded.
  • Pushdown never changes results. A pushed predicate only pre-filters. The full jq filter always runs again on the client, and --explain shows where each condition runs.
  • Capabilities are explicit. If a backend cannot support a command, the command fails with a clear message instead of emulating it. For example, Redis has no drop.
  • Values round-trip. Every value normalizes to JSON under a fixed encoding for each backend. A --typed dump restores losslessly through --insert.
  • Bounded and redacted. --timeout bounds every backend call. iq removes a URI's password from every listing, log line and error.
  • Native commands. iq exec sends a command in the backend's own language, such as CQL, Cypher or a MongoDB command document, to the backend unchanged.

Backups and dumps

A file:// source reads a database dump straight from disk. As a result, you can do these operations on a snapshot through the same jq interface, with no running server:

  • Query it
  • Inspect it for shape
  • Diff it against a live source
  • Restore it.

A file:// source is read-only. A file:// endpoint is never a copy destination. iq exec and iq inspect, which need a live server, do not apply.

Register a dump like any other source:

iq add -n snap file:///backups/prod.rdb

Query, diff and restore it with no server running:

iq --src snap '.["session:42"]'
iq diff snap cache --data
iq --src snap --insert cache
Type Description
jsonl iq typed JSON Lines / array
yaml iq typed YAML
mongoexport mongoexport Extended JSON
bson mongodump BSON
rdb Redis RDB snapshot
?format=dynamodb-json DynamoDB S3 export / scan JSON
?format=cassandra-csv cqlsh COPY TO CSV
?format=neo4j-json Neo4j APOC JSON export

A type with a bare name auto-detects (file:///<file_path>). For a type in the ?format= rows, you must pass the ?format= form (file:///<file_path>?format=<source_format>). The Drivers page documents what produces each format, the options some of them need, and the round-trip caveats.

See also

Where iq sits among other tools, and the jq ecosystem its filter language carries over.

  • jq manual: the language reference for the filters iq runs.
  • awesome-jq: a curated list of jq tools, guides, and resources.
  • sq: jq-style queries over SQL databases and document files.
  • gojq: the pure-Go jq implementation iq embeds.
  • jaq: a Rust jq clone focused on speed and stricter semantics.
  • yq: jq-style filters for YAML, TOML, and XML.
  • fq: jq for binary formats.
  • jc: converts classic CLI output to JSON.
  • jqp: a TUI playground that live-previews a filter as you type.
  • ijq: interactive jq with a side-by-side input and output view.

Contributing

Read CONTRIBUTING.md to start, and DEVELOPMENT.md for the build, test and gate commands. Report security problems privately through the security policy.

License

MIT. See LICENSE.

About

Run jq filters across NoSQL databases and their dump files to query, copy, diff and restore data, from one static binary. A filter that names keys reads only those keys.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages