Skip to content

ROADMAP.md

Latest commit

 

History

History
201 lines (178 loc) · 12 KB

ROADMAP.md

File metadata and controls

201 lines (178 loc) · 12 KB
Raw

Stelis — Roadmap

Planning horizons, so any proposed feature has an obvious home. Horizons are ordered by dependency and risk, not by date. Everything past Horizon 0 is provisional and expected to be reshaped by what Horizon 0 teaches. The test for "which horizon does this belong in?" is: what must be true and working before this feature can be built well?

See DESIGN.md for the rationale behind the commitments and deferrals referenced here.

Horizon 0 — Now (Phase 1)

The coarse batch build. One working thing, end to end, on the simplest real case.

  • Racket build system over the beeatlas batch pipeline.
  • Datalog as a metadata language about the build (deps, versions, staleness); transformations stay external.
  • Target selection + up-to-date skipping.
  • Input-addressed task caching; derived-vs-authoritative node types.
  • Per-task hermetic runtime invocation (dual-interpreter case).
  • Gates as first-class nodes; partial success; explicit output destinations.
  • Streaming per-task logs.
  • Determinism-testing harness (build twice, compare hashes).

Exit condition: run.py's linear sequence retired for the occurrences.db build; minimal-upstream rebuild proven reproducible.

Horizon 1 — Delivered (2026-07-16)

Make it explain itself, remember, and rebuild incrementally. Still batch, still one repo, still linear time.

  • ✓ Provenance, first-class: --why / --explain; staleness and blame as Datalog (provenance-datalog.rkt). The build reasoning is inspectable.
  • ✓ Trace/graph persistence: an append-only observation history under .stelis/ (history.rkt) — per-build records with per-key ('dir) and per-column ('db-relation) granularity, once-per-topology graph snapshots, a --history browser, and a Datalog projection. Freshness stays content-hash + graph, never the sequence (ADR 0005).
  • ✓ Incremental rebuild via early cutoff: content-addressed outputs; stop propagation when a rebuild produces identical content (ADR 0003).
  • ✓ Data-quality rules as build nodes: in-process rule nodes (exec.rkt's rule-check), with the record-count integrity gate as the first rule (data-quality.rkt, ADR 0006). The editorial half — flags for end users — is published derived data, so it moved to Horizon 2 (see below).
  • ✓ Broaden target coverage: every terminal deliverable — not just occurrences.db — now plans, builds, and verifies byte-identical.

Exit condition (met): the build explains itself, remembers its observations across builds, and cuts off propagation on unchanged content — proven across all beeatlas targets. Horizon 2's substrate (the observation history) is built; the engine is host-portable (BEEATLAS_DIR / NOTES_DB_PATH).

Horizon 2 — Now

Fold in change-over-time and cross the file/value boundary. This is where the batch→streaming arc completes and the browser gets fed.

Re-ordered 2026-08-03 — most of this now waits. This horizon was written as a list of things to build next. The build detour has paid out (DESIGN.md, "The detour has paid out") and the pull is data modeling, so it is now a list of things that wait for a pull, with two exceptions:

  • st-hdm's per-page provenance closes out the build work. Half-delivered, and the last item here still claimed on build-infrastructure grounds. When the notes/ → pages hop lands, the build arc is done.
  • The taxon reasoning arc is the active line — it is the one entry in this horizon that is data modeling, and the ADR 0008 argument for it was always a value argument rather than a substrate one.

Everything else below stays named and reasoned, because a good idea with nobody waiting on it should be recorded, not built — which is exactly what the slotting rule at the bottom already said, now applied to Stelis's own build features. The entries are unchanged; what changed is that they need a pull to start.

  • Serve from the build host (ADR 0007) — delivered. beeatlas.net serves from an Apache vhost on maderas over a directory Stelis owns (st-bgy); S3/CloudFront left the serving path and the /api/notes kludge is gone (st-vjd); a note write commits, then synchronously publishes before responding — reload-sees-it as a build property (st-nee). The render itself LEFT the graph in the Model Y amendment (st-5em): Stelis narrowed to the data engine and the site build consumes the export via npm run fetch-data.
  • The render returns, as two nodes (ADR 0007's per-page-provenance amendment, st-hdm) — the active arc. Not for speed: beeatlas already won that externally (its ADR 0016/0017/0019 took a note publish 23s → 8.5s). For per-page provenance — --history <pages>:<path> reaching the note that produced the page, which a renderer structurally cannot do for itself. app-bundle and site-content have disjoint inputs and are already separate npm scripts. The enabling half is delivered: per-key blame (st-nbu) walks the observation history backward and is verified on a real notes build; the missing hop is notes/ → the pages. Retiring beeatlas's three hand-rolled re-implementations of Stelis machinery (its build receipt, its render-key env var, its bundle gate) is the test that it landed.
  • Streaming / CRUD ingestion — salishsea's model: small frequent content-addressed snapshots at the ingestion boundary; near-real-time incorporation of API data into artifacts.
  • Delta-based propagation (Z-sets / DBSP-shaped) where coarse over-rebuilding hurts; retraction-clean incremental maintenance. The H1 observation history (content + basis per output, plus per-key/per-column granularity) is the substrate this folds over — the natural entry point into this horizon (st-066).
  • Editorial data-quality flags (moved from H1): rules that flag records for end users (dup collector-day, out-of-state, bee-vs-flower) and travel with the data into published outputs — they annotate, never block (ADR 0006). The dbt-vs-Stelis fork is settled toward dbt (ADR 0008): these are per-record / per-cluster leaf predicates, no more natural in Datalog than SQL — so they stay external and earn no substrate.
  • Taxon reasoning with learner explanations (ADR 0008, st-650 verdict): the value prop that earns the substrate's reasoning half. Characterization by taxonomic-rank inheritance (unbounded-depth closure, most-specific-wins) → typing the existing bee_specialist_hosts / bee_parasite_hosts edges over those traits → an ecological at-risk closure (necessity through obligate edges only), so each derived trait carries its proof — published, baked into pages as learner-facing "why" (the operator --why engine, audience flipped). Render-by-query is not pulled forward — explanations are precomputed and baked, so the H3 substrate below keeps its deferral. Inheritance (st-ozp) is delivered: a new derivation invoke variant runs a transform inside the engine (taxon-inherit.rkt rules + taxon-derive.rkt seam), and taxon-reasoning publishes species_reasoning.json from curated assertions in data/taxon-traits.rktd. Seven assertions — at subfamily, tribe, genus, and subgenus rank, across three families — characterize 106 species; Bee-Gap independently agrees on 95 and has no value for the other 11 (the coverage win, measured). The cross-check runs BOTH ways: it also names species Bee-Gap calls parasitic that no assertion reaches, which is how a bad source record surfaces. Most-specific-wins is still deferred (no real lineage conflicts yet; trait-conflicts blocks publication the day one appears). Next: type the existing edges over these characterizations, then the at-risk closure.
  • Demand-directed evaluation via magic sets, if/when goal-directedness is needed.
  • Compile-to-TS emission: specialized projections compiled to small self-contained frontend modules — each a derived, content-addressed node in the build graph.
  • Differential-testing harness for emitted artifacts: run the general interpreter and the specialized emission on the same inputs; compare.

Horizon 3 — Speculative / deferred

Named so features have a home, not scheduled. Each entry has a "why deferred" in DESIGN.md.

  • Sentential-Datalog render substrate (the server pivot's deferred half, ADR 0007): pages rendered by querying a fact database at request time, replacing 11ty. Why deferred: required by nothing shipped so far. st-650's exploration resolved (ADR 0008) — flags don't earn it, and taxon reasoning earns only the reasoning half (now H2 above), whose explanations are precomputed and baked. Render-by-query itself is still unclaimed and needs its own concrete value prop: a query parameterized at request time, over a space too large or too live to bake. And it is squeezed from both sides — fixed projections bake (H2), while open/interactive queries over a small fact base (bee taxa + edges is shippable) go client-side via compile-to-TS emission (also H2, above). That leaves H3 only the cases a small atlas doesn't have: a fact base too large to ship, or a query needing live/secret server-only state. Likely empty for this use case — a home, not a plan.
  • Non-linear time: git-like branching and grafting of database-programs; distribution. The hard problem; everything else assumes it away.
  • Review / staleness workflow layer: human and LLM review nodes; three-valued staleness (clean / dirty-rebuildable / suspect); doc-depends-on-source and similar semantic edges; LLM verification with human reconstruction. The ambition to make GitHub/issue-tracker-shaped coordination fall out of the dependency graph — while owning the cross-tool edges that currently live nowhere, not reimplementing those tools.
  • External engine integration: reach for Feldera/DBSP for fine-grained incremental maintenance at scale.
  • Compile-to-Rust for a specific hot path that matters and doesn't fit Feldera.
  • ASP for genuine search/repair problems (e.g. minimal dataset corrections).
  • WASM in-browser engine execution (platform-choice tripwire).
  • Rhombus migration of selected modules.
  • Layer 2 breadth (community collaboration features) and Layer 3 (tooling/patterns library) generalized across beeatlas, salishsea, and beyond.

Slotting rule of thumb

  • Is it build-system machinery? → name the data question it answers. As of 2026-08-03 this is the first test, not the last: Stelis has enough build system to run the pipelines it was built for, so a new build feature is premature unless something in the data work is waiting on it. st-hdm passes (a page's provenance is a question about the data, and no renderer can answer it); "delta propagation would be elegant" does not.
  • But a defect in a property Stelis already claims is not a new feature, and needs no data question. Added 2026-08-06, because the rule above reads as though it defers these and it must not. Closing the gap between what the engine promises and what it enforces is maintenance of the existing system, and it is never premature: st-zb9 (an artifact with no producer is indistinguishable from a forgotten one) and st-5e6 (build-graph never checks that edge names refer to declared artifacts) are both cases where the graph is complete today only by authoring discipline. The test that separates the two: would a user of the current system be surprised to learn it doesn't do this? If yes, it is a defect — fix it. If it is capability the system never claimed, the data question applies.
  • Does it require change-over-time or streaming? → Horizon 2 at the earliest.
  • Does it require branching/merging or non-linear time? → Horizon 3.
  • Does it require provenance or persistence? → Horizon 1 at the earliest.
  • Is it needed to retire run.py for one target, reproducibly? → Horizon 0.
  • Is it a general-substrate feature with no user waiting on it? → it is probably premature; find the user first, or move it to Horizon 3 with a reason.