feat(pipeline): add Docker path for the netflow pipeline toolchain

[flamboh]

Problem

Building a database today requires rustup, a C compiler, the full autotools stack (autoconf, automake, libtool, flex, bison, pkg-config), initialized git submodules, and a local build of the pinned nfdump fork. These are the most error-prone, platform-sensitive steps in new-user setup.

Solution

An optional Docker path for the pipeline only — the dashboard stays native (Bun + Node), since Cloudflare is the deploy target and a web container would have no production role.

• Dockerfile — multi-stage on ubuntu:24.04 (matching CI). The builder clones the nfdump fork at the commit pinned by the superproject gitlink, passed as a build arg; a copied submodule checkout cannot work in a build context because its .git is a pointer into the host repo. Runs the existing vendor/scripts/compile-nfdump.sh unmodified, builds netflow-db with cargo build --locked --release, and ships a slim runtime with just the two binaries. The runtime stage re-runs smoke checks (netflow-db contract-version, nfdump -o atlantis no-match stream) so a broken image fails at build time.
• scripts/netflow-db-docker.sh — wrapper mirroring scripts/netflow-db.sh. Reads the pinned commit from git ls-tree, builds the image when missing (--build to force), mounts data/ read-write and datasets.json read-only at /workspace, mounts each --capture-root read-only at its identical host path so root_path in datasets.json works unchanged, and runs as the invoking uid:gid so outputs stay user-owned.
• shell.nix — adds gcc and git, making the Nix shell self-sufficient for the native nfdump build (verified below).

Documentation

docs/user/setup-pipeline.md is reorganized by phase rather than by path. It previously described Docker and native as parallel tracks, which duplicated the pipeline command and left shared knowledge — inclusive date semantics, MAAD defaults, flow selection, config mode, verification — sitting under headings marked "natively", where a Docker reader had no reason to keep reading.

It now reads: Choose a path (a table plus one substitution rule) → Docker setup / Native setup (one-time work) → shared usage sections that apply to both. --nfdump no longer appears in any example; it lives in the native setup prose and the options table, which is what removed the duplicate command and the per-section Docker footnotes.

Separately, concrete example dates are replaced with <YYYY-MM-DD> placeholders across the docs (root README.md, setup-pipeline.md, operations.md, querying.md, pipeline-contract.md) so readers substitute a date their captures actually cover.

Flows to exercise

./scripts/netflow-db-docker.sh --capture-root /abs/path/to/captures \
  pipeline --dataset <id> --start-date <YYYY-MM-DD> --end-date <YYYY-MM-DD>
./scripts/netflow-db-docker.sh verify data/<id>/netflow.sqlite --dataset-id <id> --require-data

Expected: first run builds atlantis-netflow-db:local (in-image smoke checks pass), the database lands at data/<id>/netflow.sqlite owned by you, and the dashboard discovers it as before. No --nfdump flag needed — the fork is the default nfdump on PATH inside the image.

Also worth a read-through of docs/user/setup-pipeline.md as a newcomer would: the Docker path should be followable end to end without bouncing into the native sections.

Edge cases and decisions

• Capture roots must be absolute, comma-free directories (docker --mount syntax); the wrapper validates and errors clearly.
• Output paths must stay under data/ — it is the only writable mount; a custom db_path outside it is not reachable in the container (documented).
• --capture-root is only needed for commands that read captures (pipeline), not for verify or compare. Documented as one rule rather than a per-command note.
• macOS bind-mount I/O over large capture trees is slower than native; noted in docs.

Verification

Automated, on this branch:

• bun run format, bun run lint, bun run typecheck all pass.
• Both risky Dockerfile stages were replicated exactly and pass:
  • nfdump stage: fresh clone of the fork at pinned 16348e7 + vendor/scripts/compile-nfdump.sh in a bare tree — full runatlantis.sh suite green, both smoke checks pass, binary staged at the exact path the Dockerfile copies. Ran inside the updated nix-shell, which also proves the gcc/git additions complete that environment.
  • Rust stage: cargo build --locked --release from exactly the Dockerfile's COPY file set; the binary answers contract-version → 4.
• nix-instantiate shell.nix evaluates with the additions.
• Inbound doc anchors (#verify-the-output, #select-flows, #use-a-pipeline-configuration from troubleshooting.md and datasets.md) were checked before renaming headings; all still resolve.

Remaining manual: the actual docker build could not run on the authoring host (no docker-group access). On any Docker-enabled machine, run ./scripts/netflow-db-docker.sh --build contract-version — expected output is 4 after the image builds (~15–30 min first time).

---

Authored with Claude Code (Fable 5) as orchestrator; container implementation drafted by GPT-5.6-sol (xhigh reasoning) via codex, then reviewed and verified by the orchestrator.