Take any tabular dataset and run it through a complete modern data stack — ingestion → warehouse → transformation → testing → docs → visualization — with AI layered at the build, orchestration, and consumption seams, and deliberately kept out of the deterministic transform path.
One command, no cloud accounts, no API keys for the core data flow, runs offline:
python run.py up # download → load → profile → dbt run → dbt test
python run.py dashboard # open the BI dashboard
# (with make installed: `make up`, `make dashboard`)
In the dashboard, the Ask tab takes a plain-English question, grounds it in the live schema, generates DuckDB SQL, and runs it read-only — but only after it clears a layered validation pipeline (L1–L7) that statically checks every column, dry-runs the query before a single row is read, and sanity-checks the result. Each layer lights up pass / retry / fail in a live trace, and every number in the answer traces back to the SQL that produced it. A query that can't be validated fails transparently instead of guessing. From there, a good answer doesn't have to evaporate: promote it into a reviewable dbt model or pin it to the dashboard as a chart — so an ad-hoc question becomes durable, tested infrastructure in a couple of clicks.
The reference dataset is AnAge (the Animal Ageing & Longevity Database) — ~4,600 species. The marts answer questions like "which animals live far longer than their body size predicts?" (answer: deep-sea rockfish, the olm, tortoises, naked mole-rats — and yes, the data backs it up).
Deterministic core, AI at the edges. The
raw → staging → martsflow is hand-written, version-controlled, tested dbt SQL. It produces the same result on every run, on every clone. AI assists in building it, operating it, and querying it — but never executes the transforms. No LLM regenerates SQL at runtime. Reproducibility is the product.
This boundary is a deliberate engineering stance, not a limitation. "Automate what's safe; curate what matters."
┌─────────────────────────────────────────────┐
AI (build-time) ───▶ │ scaffold: profile.json → proposed dbt code │
└─────────────────────────────────────────────┘
│ (human reviews & commits)
▼
┌──────────┐ ┌──────────┐ ┌───────────────── DETERMINISTIC CORE ─────────────────┐
│ any │ │ ingest │ │ raw.* ──▶ staging.stg_* ──▶ marts.mart_* │
│ csv/tsv/ │──▶│ (DuckDB │──▶│ (1:1, typed) (curated semantics) + dbt tests │
│ parquet/ │ │ autotype)│ │ one local warehouse.duckdb file │
│ json │ └──────────┘ └───────────────────────────────────────────────────────┘
└──────────┘ │ │ │
▼ ▼ ▼
profile.json Streamlit dashboard NL→SQL query agent
┌─────────────────────────┐
AI (run-time) ───▶ orchestrate: │ AI (consumption) │
run / monitor / explain test failures / │ L1 intent → L2 gen → │
flag row-count drift │ L3 static → L4 dry-run →│
│ L5 exec → L6 plausible →│
│ L7 explain + trust badge│
└─────────────────────────┘
Prereqs: Python 3.10+ (only hard requirement). Optional: an Anthropic API key
for the three AI layers; make if you want the make aliases.
pip install -r requirements.txt
python run.py up # build the whole pipeline, green
python run.py dashboard # http://localhost:8501
python run.py docs # dbt lineage graph (raw → staging → marts)
# AI layers (need a key: ANTHROPIC_API_KEY env, or a one-line anthropic.txt):
python run.py scaffold <name> --write # generate staging+tests, wire them into dbt
python run.py orchestrate anage --break # run + monitor + explain an injected failure
python run.py ask "which animals live longest for their size?"Every target takes an optional dataset name (default anage); make users pass
DATASET=…. The core (up, dashboard, docs) needs no key; the AI layers
(scaffold's LLM tier, orchestrate's explanations, ask, and the dashboard's Ask /
Build / chart features) each need an Anthropic key — set once, used by all.
The python run.py … form always works from a clone. You can also install sediment
as a real command-line tool and run it against a project directory anywhere:
pip install -e . # from a clone — installs the `sediment` command
# or straight from GitHub, no clone:
pip install "git+https://github.com/camharris93/sediment.git"
sediment up # same as `python run.py up`
sediment ask "which animals live longest for their size?"sediment <target> is exactly python run.py <target> — every target above works
either way. To start a fresh project of your own (separate from this repo):
sediment init ~/my-analytics # scaffold a new workspace (dbt project + datasets/)
cd ~/my-analytics
# drop a file in datasets/example/data/ and edit datasets/example/config.yml
sediment up examplesediment locates the active workspace (where datasets/, dbt_project/, and the
warehouse live) by walking up from the current directory, or from $SEDIMENT_HOME if
set — so one installed tool can drive many separate projects.
On PyPI: the published name will be
sediment-stack(the bare namesedimentis taken by an unrelated project), so the eventual public install ispip install sediment-stack— the command stayssediment. Not published yet; the git/clone installs above are the current distribution path.
engine/ the reusable, dataset-AGNOSTIC framework (written once)
config.py paths, dataset config, key resolution
ingest.py any file → raw.<table> (generic)
profile.py raw table → profile.json (generic)
scaffold.py profile → proposed dbt code (AI, build-time)
orchestrate.py run / monitor / explain the core (AI, run-time)
modeling.py promote a chat answer -> dbt model (AI, build, gated)
charting.py suggest a chart spec for the report (AI, build, gated)
report_config.py persist custom report blocks (build, gated)
query/ NL→SQL agent over the marts (AI, consumption)
grounding.py intent.py generation.py static_validation.py
dry_run.py execution.py plausibility.py translation.py orchestrator.py cli.py
datasets/
anage/
config.yml the entire human-authored contract for the dataset
data/ the source file
scaffold/ generated proposals for review
report_blocks.json AI-built charts pinned to the report (committable)
dbt_project/ dbt Core project (DuckDB)
models/staging/ stg_anage + _sources.yml + tests
models/marts/ mart_longevity_by_class, mart_aging_outliers (curated)
dashboard/app.py Streamlit BI — Report / Ask (chat) / Build tabs
run.py / Makefile the one-command wrapper (run.py = the `sediment` command)
pyproject.toml packaging (`pip install` → `sediment` CLI) + ruff/pytest config
tests/ pytest suite — incl. adversarial SQL-guard tests (offline, no key)
evals/ NL→SQL golden-question harness (replay = no key; --live = full L1→L7)
warehouse.duckdb the entire warehouse — one file (git-ignored)
The core is dataset-agnostic for everything that's safely automatable; mart semantics stay human-curated (that's the whole design — see §"The dataset-agnostic contract"). Onboarding a new single-table dataset:
# 1. Drop your file in and declare the contract.
mkdir -p datasets/sales/data && cp ~/orders.csv datasets/sales/data/
cat > datasets/sales/config.yml <<'YAML'
name: sales
source: data/orders.csv # path (relative to the dataset dir) or a URL
table: orders # lands as raw.orders
# delimiter: "\t" # optional; auto-sniffed for csv/tsv
YAML
# 2. Land it, profile it, and WIRE a typed/tested staging layer into dbt.
python run.py load sales
python run.py profile sales # -> datasets/sales/profile.json
python run.py scaffold sales --write # installs stg_orders + tests, adds the
# raw.orders entry to _sources.yml
# 3. Curate the marts (the human-judgment step). With a key, the scaffolder also
# drops proposed mart stubs in datasets/sales/scaffold/*.proposed.sql — review,
# move keepers into dbt_project/models/marts/, and edit to taste. (Without a key,
# you write the marts yourself against the freshly-built staging layer.)
# 4. Build + test, then consume.
python run.py up sales
python run.py dashboard # Ask tab + AI chart builder work on any data
python run.py ask "top 10 orders by value"scaffold --write collapses the old manual wiring (copy the staging model in, hand-add
a sources.yml entry) into one idempotent command. It never auto-installs marts —
their semantics are yours to decide. Re-running is safe; it won't overwrite a curated
staging model unless you pass --force.
Multi-table datasets are supported — declare a tables: list instead of a single
source/table, and the engine loops ingest/profile over them, derives the
relationships between them, installs a staging model per table, and (with a key)
proposes join marts across the inferred relationships:
name: shop
tables:
- {source: data/customers.csv, table: customers}
- {source: data/orders.csv, table: orders}python run.py load shop && python run.py profile shop # derives customers→orders [1:many]
python run.py scaffold shop --write # installs both staging models + join-mart proposalsMultiple datasets coexist in the one warehouse file, each in its own schemas —
<dataset>_raw / <dataset>_staging / <dataset>_marts. So AnAge lives in anage_marts
etc. and a second dataset never touches it. The dashboard's dataset selector scopes
the Report, the chat's grounding, and model-building to one dataset at a time (AnAge gets
its curated charts; every other dataset gets an auto-report + the AI chart builder).
Even identical table names across datasets are fine. dbt requires globally-unique
model names, so each model's node name is dataset-prefixed (anage__stg_anage) while a
dbt alias keeps the warehouse relation clean (anage_staging.stg_anage). Two datasets can
both have an orders table — they build as north__stg_orders / south__stg_orders into
north_staging.stg_orders / south_staging.stg_orders with no collision. Refs use the
prefixed node names; everything querying the warehouse sees the clean relation.
| Layer | Status |
|---|---|
| ingestion, profiling, staging generation, test inference, orchestration, NL querying | fully generic & automatic |
| mart semantics — what business questions the marts answer | human-curated per dataset |
No tool can know what matters in arbitrary data; pretending otherwise breaks the trust model. So the engine automates everything up to the marts, and the marts stay a human judgment call.
-
Build-time copilot (
engine/scaffold.py) — profiles the source and proposes a staging model,schema.ymlwith conservative inferred tests, and proposed mart stubs with explanatory comments. A deterministic baseline runs with no key (mechanical snake_case staging + key tests); the LLM enhances it. Output is written todatasets/<name>/scaffold/for human review — never auto-committed. Add--writeto install the staging model + tests into the dbt project and merge theraw.<table>sources entry (idempotent; marts are never auto-installed). -
Orchestration brain (
engine/orchestrate.py) — drivesload → dbt run → dbt test, reads dbt's structured artifacts, explains test failures in plain English with the offending rows, and flags row-count drift vs. the last run. The LLM only acts at the decision points; the pipeline runs (and fails loudly) without a key. Trypython run.py orchestrate anage --breakto watch it catch and explain a deliberately-injected failing test. -
Consumption layer (
engine/query/) — a trust-first, layered NL→SQL agent (ported from the siblingsql-engineproject and adapted to DuckDB). Schema is pulled dynamically from the warehouse, so it adapts to any dataset. The layers:Layer Role L1 intent restate the question, surface assumptions (auditable) L2 generation constrained DuckDB SQL over the grounded schema only L3 static check sqlglot validates columns/tables without touching the db L4 dry-run DuckDB EXPLAIN— authoritative bind/semantic check, no executionL5 execution read-only guard + row cap, full provenance L6 plausibility join fan-out + result-value sanity checks L7 translation plain-English answer + trust badge; every number traces to the SQL L3/L4 failures feed structured violations back to L2 in a bounded self-correction loop. A query that can't be validated is a transparent failure, not a confident guess. Available as a CLI (
python run.py ask "…") and built into the dashboard's 💬 Ask tab, which renders a live L1→L7 trace (each layer lights up pass/retry/fail with its intent, SQL, validation detail, and trust badge) as the answer is computed. -
Build a model from chat (
engine/modeling.py+ dashboard 🛠 Build tab) — promote a validated chat answer into a dbt model. The chat SQL is rewritten frommarts.x/staging.xinto{{ ref('x') }}, then you can preview it in an isolated_sandboxschema, save it as a reviewablemodels/marts/<name>.sql, ordbt runit straight intomarts. Even here the unit of work is reviewable dbt SQL — the chat never silently mutates the curated marts. -
AI chart builder + editable report (
engine/charting.py,engine/report_config.py) — the dashboard's 📊 Report tab is editable in build mode: point it at a mart (or a chat answer's SQL), and the agent proposes a chart spec (revived from sql-engine's L7VizHint), which you preview and pin to the report. Build a model from chat, then chart it in two clicks — the Build tab seeds the Report customizer with the new mart. Saved charts persist todatasets/<name>/report_blocks.jsonand render for everyone (a heuristic fallback picks a sensible chart when no key is set); only editing the report is gated to build mode.
The capability tension — "we can't open the ability to build models to everyone we share a report with" — is resolved by making build a property of the deployment, not a per-user toggle:
| view (default) | build | |
|---|---|---|
| NL→SQL Ask | ✅ (read-only at the engine: L5 guard + read-only DB) | ✅ |
| Build a model | ⛔ refused server-side, not just hidden | ✅ |
| Set by | a bare/deployed streamlit run (or SEDIMENT_MODE=view) |
the local authoring app python run.py dashboard |
So you share a report in view mode: viewers get the charts and the chat, never model
creation. You author in build mode locally (or on a private instance), review the
generated dbt, and commit — the published report only ever ships the built, tested
marts. One shared app with per-user build rights is a deliberate non-goal (PRD §2); the
upgrade path is a password (st.secrets) / SSO in front of the Build tab.
Two curated marts (dbt_project/models/marts/):
-
mart_longevity_by_class— lifespan stats rolled up by taxonomic class. -
mart_aging_outliers— every species ranked by how far its actual longevity beats the body-size→lifespan power law (fit in log-log space with DuckDB'sregr_slope/regr_intercept— deterministic SQL, no library, no AI). The top over-performers:rank species class weight longevity size-predicted ratio 1 Rougheye rockfish Teleostei 495 g 205 yr 15.6 yr 13.1× 2 Olm Amphibia 17 g 102 yr 9.8 yr 10.4× 3 Eastern box turtle Reptilia 372 g 138 yr 15.0 yr 9.2×
The framework's pitch is trust, so the framework itself is tested — not just the data. Everything here runs offline, no API key:
pip install -r requirements.txt pytest ruff
pytest -m "not live" # the deterministic engine + the SQL trust-boundary guards
ruff check . # lint
python evals/harness.py # replay the NL→SQL golden questions through L3–L6tests/— the deterministic engine under test: ingest auto-typing, profiling, the scaffold baseline, config/grounding, and — most importantly — the read-only execution guard.test_read_only_guard.pyis adversarial: every known way to turn "a SELECT" into a write, a;-chained second statement, a local-file read (read_csv('/etc/passwd')), or an extension load must be refused, and ordinary analytical SQL must pass. The guard is the security boundary of the NL→SQL agent; if you extend it, add the attack it defends against here first.evals/— golden questions over AnAge with result invariants (the rockfish is rank 1). Replay mode (default, no key) runs known-good SQL through the deterministic layers L3–L6 and checks the values — CI-safe.--liveruns the full L1→L7 pipeline from the natural-language question (needs a key).- CI (
.github/workflows/ci.yml) runs lint + the test suite on Linux and Windows × Python 3.10 and 3.13, then proves a fresh clone builds a green, tested pipeline with one command — fully offline.
| Layer | Choice | Account needed? |
|---|---|---|
| Warehouse | DuckDB (one file) | No |
| Transform | dbt Core + dbt-duckdb |
No |
| Visualization | Streamlit (pure-Python) | No |
| Ingestion | Python + DuckDB auto-typing | No |
| AI layers | Claude API (Anthropic SDK) | Key — AI layers only |
| Orchestration | Python runner / Makefile | No |
- ✅ 1. Generic ingest + profiler → DuckDB
- ✅ 2. AnAge worked example through dbt (staging, curated marts, tests) + dashboard;
upruns green - ✅ 3. AI build-time scaffolding (deterministic baseline + LLM enhancement)
- ✅ 4. Orchestration agent (run / monitor / explain, with
--breakdemo) - ✅ 5. NL→SQL query agent over the marts (dynamic schema, self-correcting)
- ✅ 6. In-app chat (dashboard Ask tab) + build-a-model-from-chat with view/build governance
- ✅ 7. Multi-table datasets (looped ingest/profile, derived relationships, join-mart scaffolding) + a dataset selector that scopes the Report/Ask/Build to one dataset
- ✅ 8. Per-dataset schema namespacing (
<dataset>_raw/_staging/_marts) + dataset-prefixed model names withalias— multiple datasets coexist in one warehouse, even sharing table names - ✅ 9. Hardening: pytest suite (incl. adversarial SQL-guard tests) + NL→SQL eval harness,
cross-platform CI matrix, hardened read-only guard (blocks local-file reads / multi-statement /
file writes), and
pip installpackaging (thesedimentcommand +sediment init)
Where it goes from here: ROADMAP.md. Want to help: CONTRIBUTING.md.
- Visualization → Streamlit for v1: pure-Python, guaranteed-green offline, no Node install. Evidence.dev is the documented upgrade path when a slicker static site is wanted.
- Orchestration → a Python runner (
run.py), not Dagster/Prefect. It keeps the one-command promise cross-platform (Windows has nomake); a DAG engine is a clean later swap behind the same targets. - Test inference is deliberately conservative — the scaffolder only proposes a test the profile justifies (a test that would fail on real data is worse than no test), and everything it proposes is for human review.
- Live hosting: clone-and-run only for now; the dashboard builds to a shareable static-ish Streamlit app if/when desired.