dmfdeploy
Replies: 4 comments
On PostgreSQL and Apache AGEThe source concept document proposes PostgreSQL for indexed/queryable application state, with Apache AGE as a later extension if graph traversal becomes operationally important. Worth engaging with directly rather than folding into "a derived index". Postgres already exists here — three timesThe cluster already runs three Postgres instances: Authentik, AWX and NetBox, each app-owned with its own volume. So "use Postgres" is not a question of introducing a database; it is a question of whose.
So the plugin route and the "use Postgres" route are the same route, not alternatives. If we want queryable, relational, indexed design state, the plugin is how this platform gets Postgres tables without adding a fourth database. That is a stronger argument for the plugin than the tenancy one it currently leads with — and unlike the tenancy argument, it does not evaporate if cluster-per-tenant holds. It does not change the recommendation in the discussion body, because it answers a different question. Forgejo is proposed as the place designs are authored and versioned; Postgres-via-plugin would be where designs are indexed for query. Those can coexist, provided the index stays strictly derived — regenerable from the artifacts, never edited directly, never a second source of truth. Apache AGE: defer, and probably indefinitelyAGE adds property-graph storage and openCypher to Postgres. Getting it here means either installing an extension into NetBox's database that NetBox does not know about — a custom Postgres image, unsupported, and a hazard at every NetBox upgrade — or standing up a separate Postgres purely to hold the extension. Neither is justified by the workload. A media workload graph is small: a production is on the order of tens of nodes, not millions. The traversal questions people reach for a graph database to answer — "what feeds this encoder, transitively", "which workloads touch this facility" — are a recursive CTE in plain Postgres, or a loop in application code over a graph that fits comfortably in memory. Graph databases earn their keep at a scale and query complexity this problem does not have and is unlikely to acquire. The source document already says AGE is "not required initially" and should be evaluated "only after the semantic model proves useful". Agreed, with one sharpening: the trigger for revisiting should be a specific query that is provably too slow or too awkward in plain SQL, named and measured — not the general feeling that a graph deserves a graph database. Without that trigger, AGE is a specialised stateful dependency acquired on aesthetics. The point that survives from the source documentIts actual insight here is not the technology choice but the separation: canonical designs in a versioned artifact store, indexed state somewhere queryable, and layout kept apart from semantics. That separation is right regardless of which database ever appears underneath the index. |
Can dmf-cms stay glue, or must it eventually own a database?The concern behind this is worth stating plainly: nobody wants a large custom system where mature ones already exist. The preference is for dmf-cms to be a UI and glue layer between apps that already do their jobs well. That is achievable, and it is the stronger architecture rather than a compromise. Every requirement of authoring is already met by something deployed:
Nothing there needs a console-owned database. There is none today: no ORM or SQL dependency anywhere in The one thing that pressures itThe index — questions like "which workloads use function X". Be concrete about scale: a facility holds tens to low hundreds of designs, not millions. Reading and filtering them is entirely adequate, and Forgejo has code search besides. An index is a performance optimisation, not a correctness requirement, and it can be added later — derived — without moving the source of truth. Authoring facilities later reinforces this rather than straining itFacilities are NetBox: sites, devices, racks, with a full API. Authoring facilities means writing to NetBox, not storing them somewhere new. The console would be a better interface onto data NetBox already owns. The rule that keeps this glue rather than a monster
The moment the console stores something authoritative that is not in Forgejo or NetBox, it becomes a third source of truth and the thing we are trying to avoid exists. Every "just one small table" is that moment. Worth adopting as an explicit constraint so it is defended deliberately rather than eroded incidentally. Two honest costs
How does a half-finished design get saved?Save is a commit.
Incompleteness is not a storage problem — it is a validation verdict, derived on read rather than stored. A design with three unconnected ports saves perfectly happily. What it cannot do is Plan or Provision, and the surface says exactly why: "3 connections incomplete". That is the same model #557 lands for the rail: structural preconditions gate actions, not navigation, and a stage with nothing actionable states plainly why and reads as healthy rather than as an error. A half-finished design is precisely that state, so this needs no new mechanism — it reuses one already being built. Properties worth taking deliberately
If unsaved-work loss becomes a real complaintEscalate to autosave onto a draft branch, promoted to the main line by Save. Still git, still no new store. That is the important property: the next step is available without breaking the cache-never-own rule. The tempting alternative — a drafts table in the console — is exactly the moment to hold the line. |
Status: a parked plan, and nine open decisions This comment describes a design plan that exists on a branch. It does not start any work. What the branch is. The branch What the plan covers. It follows the proposal in this discussion: design intent persisted as declarative artifacts in the in-cluster Forgejo, with save as a commit. It corrects several premises in the discussion against source. It records which questions it settles as proposals and which it leaves open. Read it here: https://github.com/dmfdeploy/dmfdeploy/blob/design/media-workload-canvas/docs/plans/DMF%20Media%20Workload%20Design%20Canvas%20Plan%202026-09-11.md The nine open decisions. Each one is a separate request for comment.
Disagree here. If you think a proposal is wrong, say so in an issue or in this discussion. Disagreement is the purpose of this request for comment. A short note that a premise does not hold is as useful as a full alternative. Much of the platform is built by AI agents under operator orchestration — by design, and documented as method in JOURNEY.md. This is a request for comment. Nothing in the parked plan is authorised to start. |
|
Update, 2026-10-20: the plan moved to main The plan described in the comment above is no longer only on the Sections were renumbered. Old to new:
The old sections 11, 11a, 12 and 13 (provenance, review history and the readiness notes) are not in the new plan. Where the nine open decisions went:
The links inside those questions now point to the plan on |
Uh oh!
There was an error while loading. Please reload this page.
Why this exists
Creating a media workload is being reshaped: create makes a blank container, Design selects the media functions and how they relate, Plan assigns them to facility resources, Provision deploys. Investigating that exposed a keystone gap.
There is nowhere to put a Design or Plan decision. Today "what media functions a workload has" and "where they run" are derived on the fly from the member services that are already deployed — the answer to "what is in this workload" is literally "whatever is currently running". There is no durable record of a selection, no record of a relationship, and no write seam for either. Selection currently lives in a browser store and is lost on refresh; the page says so itself.
Everything downstream of that gap already has a home. This one does not.
What is already settled — please don't re-litigate
The concept worth adopting
From an independent concept document, sharpened against the above. Model a workload as a first-class directed property graph, where the graph is the canonical design intent and any visual editor is merely one interface onto it.
The parts that earn their place:
The sharpest line in that document, and the one that should shape sequencing: the valuable abstraction is not the boxes-and-lines editor, it is the schema and the compiler. The expensive UI is therefore not on the critical path.
Where design intent persists — the leading answer
Forgejo, as declarative artifacts, with local designs never leaving the cluster.
This is not a fourth store bolted on. The platform already runs Forgejo in-cluster, and both consumers already read from it:
forgejo-netbox-data, enabled today, 24h interval, credential from OpenBao.So a design written to Forgejo is already reachable by the automation engine that deploys it and by the source of truth that records what got deployed. That is a materially different proposition from adding a new database.
Local designs stay local; public designs are mirrored out. A facility's own workload designs live only in its in-cluster Forgejo — private by construction, no external dependency, and it works air-gapped. Designs intended to be shared (templates, reference designs) mirror outward. That also refines ADR-0014's mirror model rather than contradicting it: inbound mirroring for upstream source repos, origin for local designs, outbound mirroring for the designs a facility chooses to publish.
The honest limitation. A NetBox Data Source syncs files, not queryable objects. Design files become visible to NetBox; they do not become NetBox models you can filter, relate or report on. So a question like "which workloads use function X" is not answered by the sync alone — that needs an index derived from the designs. An index is a legitimate role for the console or a plugin, but it must be derived, never a second source of truth.
The alternatives, and why they are weaker
NetBox plugin-owned models. A tenanted MediaWorkload with rows for designed functions and links. Inherits Postgres, API, permissions, change logging and tenancy for free. Costs: a new repo, a custom NetBox image plus PLUGINS config, a version pin coupling us to a NetBox major, and migrations on upgrade. It also contradicts ADR-0046 decision 1 and ADR-0037 decision 2, so both would need amendments distinguishing declared design links (config, persisted) from observed flows (live, never persisted).
Its strongest argument was that only a real object can carry a tenant, so an empty container can be scoped. That argument may be hollow: ADR-0039 gives every env a single shared Tenant
DMF, and ADR-0020 assigns the tenant axis to cluster-per-tenant. If each tenant gets their own cluster and their own NetBox, a tenant foreign key solves a problem the intended architecture does not have. Settling that is question 1 below.A console-owned database. Ships fastest, needs no plugin. But dmf-cms is deliberately stateless — no ORM, no PVC, single replica, and its one in-process store documents that durability would need Postgres or Redis. It forks the source of truth and relocates the tenancy problem rather than solving it.
Things any answer must reconcile with
topology-parent:/topology-source:tags, which the console already reads. Any new relationship model is a second store for an edge NetBox already records — say which wins.topology-params.schema.yamlexpresses N sources feeding one viewer, deliberately ("N copies, not N distinct"). It cannot expresscamera → mixer → encoder. A chain needs it to become a directed graph.What to build first
The schema, not the editor: functions, typed ports, connections, constraints — plus compatibility validation, which is what makes typed ports worth having. A handful of reference functions to prove it. Representation before authoring: drawing what a topology already declares is cheap and demo-useful; letting an operator author a graph needs the persistence question answered first.
Questions for discussion
topology-parent:/topology-source:tags versus a new relationship model — which is authoritative, and is there a migration?All reactions