Proposal: Reposition the Tycoon CLI as a control plane — rewrite ingestion first

[JesuFemi-O]

Proposal: Reposition the Tycoon CLI as a control plane — rewrite ingestion first

What this is

Tycoon CLI today is a Python command-line tool that helps a user run dbt, ingest data via dlt, and stitch a handful of adjacent pieces (Fivetran metadata, dashboards, project scaffolding) into a coherent workflow. It works. People can install it from PyPI, point it at a project, and get value.

This proposal argues that the shape of the tool — the abstractions it's built around — limits how far it can go. It proposes a rewrite of the ingestion layer (with foundations for the rest of the stack) to reposition Tycoon CLI as something more ambitious: a stack-aware control plane that humans and analytics agents can act through, regardless of which underlying tools a user already has.

The proposal is captured in three documents in the accompanying draft PR. This issue summarizes the thinking and asks for sign-off to begin implementation.

The problem

The current ingestion code conflates "source type" with "execution runtime." SourceConfig.type means a specific dlt verified source; the runner has no concept of who executes a given source, only what it is. The two ideas — what is this data and who runs the ingestion — are tangled together.

Three consequences fall out of that:

• Adding a non-dlt runtime (Fivetran, Airbyte, sling, a user's existing dlt project) means patching the dispatch core, not slotting in an adapter.
• A user who already has Fivetran or Airbyte is asked to think in dlt-shaped terms even when their stack isn't dlt-shaped.
• The codebase grows surface area that looks like platform features (bundled FastAPI server, Dagster scaffolding, ai integration) when the differentiating value is coordination over whatever stack the user already runs.

In short: the current shape is fine for "a CLI that wraps dlt," but it constrains the larger thing we want to be.

The repositioning

A platform is a thing users live inside. It owns the runtimes, the storage, the auth, the UI. Adjacent tools in the analytics-engineering category lean platform-shaped — they absorb the user's stack to deliver value.

A control plane is a thing users act through. It owns coordination, observation, and addressability. It does not own the runtimes underneath; it makes them legible and actionable as a single surface.

The proposal commits to the second positioning. Concretely, the design follows a Bring Your Own Stack (BYOS) principle: if you already have ingestion in Fivetran, dbty leverages it. If you already have a dlt project, dbty leverages it. If you have nothing yet, dbty runs dlt natively to get you moving. The user isn't asked to port, rewrite, or replatform anything.

The differentiation matters. A platform with our current resources is a fight we won't win on bundled features. A control plane is a different category — one where the value compounds the more existing tools a user already has.

The design philosophy (in five positions)

The full thinking is in MANIFESTO.md. The load-bearing positions:

1. A Source is an abstraction, not a runtime. Every Source — whether it's a dlt verified source, a user's custom dlt resource, an existing dlt project, or a Fivetran connector — answers the same questions through the same interface (identity, runtime binding, handle, capabilities, metadata, data, run). Execution becomes a capability gated by clearance, not a method every source implements.
2. Metadata is the substrate, not a downstream concern. Every action emits structured events into a metadata store the control plane owns. Every read resolves through that store or a uniform adapter. The CLI, the JSON output, and any future surface (TUI, MCP) are renderings of the same canonical model.
3. Agent-latchability is a design constraint, not a feature. Every command is machine-introspectable; structured output is not optional. An analytics agent can latch onto the control plane without bespoke per-tool integration.
4. The cockpit principle. A beginner sees a small intelligible surface — one Runtime, one Destination, a focused catalog. Surface area expands only when the user takes actions that require it. The basic user never has to learn the word "Runtime." The sophisticated user is never constrained by the basic user's affordances.
5. No private doors. Future adapters (sling, Airbyte, dbt-cloud, MotherDuck) reach what they need through the same interfaces the first adapters used. A change to the core is a signal the abstraction is wrong, not a feature.

Where we start: ingestion

Ingestion first, for three reasons:

• It's the load-bearing test of the abstraction. If the Source/Runtime/Catalog model holds against real adapters, the rest of the architecture earns its weight. If it doesn't, we'd rather find out before committing to Transformation or Presentation.
• The Phase 1 scope — dlt as the in-process runtime, Fivetran as the delegated runtime — exercises both ends of the spectrum (local-runnable vs. delegated execution). An abstraction designed against one example is a fiction.
• Ingestion is where users feel pain today. Adjacent tools either replatform you to their warehouse or lock you into their connector set. A control plane that leverages whichever you already have is the most direct expression of BYOS.

Explicitly out of scope for the rewrite POC: Transformation rewrite, Semantics, Presentation (Rill comes later), additional surfaces (MCP, TUI, web), and the existing FastAPI server / Dagster scaffolding / ai integration.

The plan in one breath

The full plan is in PLAN.md. The shape:

A new package lives at src/_tycoon/ during the rewrite window — the leading underscore signals "in-progress, will-be-renamed." A new entrypoint command tycli exposes it. The existing src/tycoon/ package and the tycoon command stay untouched in the same repo as a working fallback. When the rewrite reaches its architectural acceptance test (the Fivetran Runtime adapter lands with zero changes to core/), a single cutover PR deletes the old package and renames the new one. Estimated calendar effort: 4–6 weeks.

Four sequenced steps, each with explicit acceptance criteria:

1. Rails up (half a day) — empty package, working CI, pre-commit, CLAUDE.md.
2. Protocols + first dlt adapter, no CLI (~1 week) — proves the abstractions hold against a real adapter before the CLI shape locks in.
3. CLI surface, designed top-down (~1 week) — the Phase 1 vertical slice (attach, source add, run, status, all with --json).
4. Port four named patterns from tycoon/ (~3–5 days) — metadata hook pattern, Pydantic source-config shape, command-surface separation, soft-fail capture pattern. Explicit don't-port list keeps scope honest.

After the four steps: Phase 2 adds the Fivetran Runtime adapter as the architectural acceptance test. If it requires changes to core/, the Runtime Protocol is wrong and we fix the Protocol first. This is the moment of truth for the design.

The ask

Approving this proposal means: green light to begin Step 1 of the plan, with the understanding that Step 2 ships the abstractions that make or break the whole thing. If the Runtime Protocol designed against dlt turns out wrong for Fivetran in Phase 2, we revise it before any more adapters compound the mistake — not after.

React in-thread for direction or concerns at any altitude. Line-level questions on the docs go on the PR (#78).

References

• Draft PR with the three documents: docs(proposal): rewrite foundations — manifesto, architecture, plan #78
• MANIFESTO.md — philosophy and shape
• ARCHITECTURE.md — C4-structured technical design
• PLAN.md — execution plan, decisions, risks