Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 43 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,13 @@ The inspector above shows one entity carrying an enumerated domain (`asset_class
a named primary key and a unique key, user-defined properties, and its relationships,
all first-class in the model.

Relationship lines anchor to the exact columns that join, not to the entity headers, so
with composite keys or several relationships between the same pair the diagram still says
which columns join which; hovering an attribute highlights the relationships it takes part
in and dims the rest.

![The ER canvas with a relationship line running from one entity's foreign-key attribute row to the referenced entity's primary-key row, with crow's-foot cardinality](docs/assets/canvas.png)

## In VS Code

Install the extension from `vscode/modelith-vscode-0.1.0.vsix`:
Expand Down Expand Up @@ -207,8 +214,8 @@ Modelith represents the core data-modeling taxonomy as first-class, git-tracked
|---|---|
| Conceptual / logical / physical layers | Separate object kinds, referenced by immutable ULID |
| Entities and attributes | Name, domain, role (business key / surrogate / attribute / measure), nullability |
| Relationships | Four cardinalities, identifying vs non-identifying, optionality, crow's-foot rendering |
| Named keys | Primary, alternate, unique, and index key groups with ordered, composite members |
| Relationships | Four cardinalities, identifying vs non-identifying, optionality, crow's-foot rendering; lines anchor to the concrete attribute rows on both sides, not the entity box |
| Named keys | Primary, alternate, unique, and index key groups with ordered, composite members; validated (one primary key per entity, a warning on a nullable PK column, and an error when a foreign key's domains don't match) |
| Domains and reference data | Reusable domains, inline enumerations, and shared code sets that emit dbt `accepted_values` tests |
| Subtypes and supertypes | Category clusters with a discriminator and a physical materialization strategy (single-table or table-per-subtype) |
| User-defined properties | Extensible metadata on any object, flowing through to dbt `meta` |
Expand All @@ -235,6 +242,10 @@ deterministically from the one definition:
| Semantic layer | `mdl emit semantic` | MetricFlow semantic models and metrics, or OSI |
| Ontology | `mdl export rdf` / `shacl` | RDF/OWL with SKOS alignments, and SHACL shapes |
| Knowledge graph | `mdl export r2rml` | A W3C R2RML mapping: the deterministic term-map from warehouse rows to typed, ontology-aligned graph nodes; fails loud on any unmapped entity unless `--allow-unmapped` |
| SQL DDL | `mdl export sql --dialect …` | `CREATE TABLE` with primary keys, foreign-key `REFERENCES`, `UNIQUE`, and `NOT NULL`, in a chosen dialect (postgres/snowflake/duckdb/…) — the universal interchange target |
| Mermaid | `mdl export mermaid` | A Mermaid `erDiagram` with PK/FK/UK markers and crow's-foot cardinality, rendering natively in GitHub, GitLab, and most markdown |
| DBML | `mdl export dbml` | Database Markup Language that opens directly in dbdiagram.io, dbdocs, and ChartDB |
| CSV | `mdl export csv` | A flat attributes sheet (entity, attribute, type, PK/FK, nullable) for spreadsheets and BI tools |

`mdl generate --emit-contract` (also `--emit-pydantic`, `--emit-graph`, `--emit-r2rml`)
turns Modelith into a contract factory: on every regeneration it drops a fresh, valid
Expand Down Expand Up @@ -268,6 +279,14 @@ misclassification on non-standard naming is visible immediately, not discovered
time; the conventions it keys off are overridable via `mdl reverse --naming <file.yaml>`
(medallion `gold_`, `f_`/`d_`, non-English). See [Reverse engineering a real warehouse](#reverse-engineering-a-real-warehouse) below.

**Import & interchange.** Bring a model in from the wider toolset and hand it back out.
Import **SQL DDL** (parsed with a real SQL AST, so dialects, quoting, composite keys, and
inline vs table-level constraints all work), a **Mermaid `erDiagram`**, or a **JSON
Schema** — each parsed into entities, attributes, keys, and relationships that stage as a
normal proposal. Export **SQL DDL, Mermaid, DBML, CSV** (alongside the compile targets
above). All of it is a menu in the Model tab's toolbar, and on the CLI as `mdl import
sql|mermaid|json-schema` and `mdl export sql|mermaid|dbml|csv`.

**Drift detection.** `mdl drift` compares the committed model to a compiled warehouse and
classifies each difference as breaking, additive, or cosmetic, with a CI gate mode and a
reconcile mode. A 400-model breaking change classifies in under thirty seconds.
Expand All @@ -282,6 +301,19 @@ below.
customer-owned Jinja profile. A Collibra adapter ships, along with OpenLineage emission
and a conformance kit that validates a bespoke mapping in CI.

### Import and export, from the Model tab

The Model tab's toolbar has **Export** and **Import** menus. Export downloads the model
in any interchange format; import parses SQL DDL, a Mermaid diagram, or a JSON Schema and
stages it as a proposal you review before it lands.

![The Export menu open in the Model-tab toolbar: a SQL dialect selector and download items for SQL DDL, Mermaid erDiagram, DBML, CSV, an ODCS data contract, and Neo4j Cypher](docs/assets/export-menu.png)

![The Import panel on its SQL DDL tab, with a dialect selector, a paste area, and a file picker; tabs switch to Mermaid erDiagram and JSON Schema](docs/assets/import-panel.png)

An imported model is visible on the canvas immediately as a staged preview — nothing
touches disk or git until you submit the proposal.

## Reverse engineering a real warehouse

`mdl reverse` lifts a compiled dbt project into a logical model. On a real, organically
Expand Down Expand Up @@ -796,6 +828,15 @@ lists your open proposal branches. Stage an edit and the tray takes you to
by severity, everything else dimmed to context. Erwin's Complete Compare is a
tree and structurally cannot do this.

![The Review screen: a Counterparty entity marked modified, a before/after definition diff with the added words highlighted, and a footer showing the review route (Governance), the reviewers, the CI gates, and that it merges cleanly onto main](docs/assets/review-screen.png)

Submitting opens one dialog that carries the whole git handoff — a word-level diff of
each change, who the proposal is attributed to (the resolved identity, not a
self-asserted name), an editable branch name, the route-advice callout when a change
spans review routes, and the branch and reviewers it will land on.

![The Submit-for-review dialog: the definition diff, "Signed in as Debasish Bose", an editable title and branch name, a callout explaining the change spans routes A and E so it takes the stricter Governance review, and the target branch sme/debasish-bose/update-definition with its reviewers](docs/assets/propose-dialog.png)

A banner states the git reality plainly. There is no lock file and no "claim this
area" button: the branch **is** the lock, so the banner says which branch you are
on, and greys out Submit with the reason when the working tree is dirty — rather
Expand Down
20 changes: 16 additions & 4 deletions canvas/src/ModelCanvas.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -182,22 +182,34 @@ export function ModelCanvas({
// flags below rather than a parallel highlight system.
const relsForAttr = new Set<string>(); // relationship ids touching hoveredAttr
const entsForAttr = new Set<string>(); // entities on those relationships
// The attribute rows to emphasise at BOTH ends, keyed by entity. Populated for a
// hovered edge (both its paired columns) and for a hovered attribute (that column
// plus the columns it joins to at the other end), so hovering one data element
// lights up the data element it links to on the far side.
const hitAttrsByEntity = new Map<string, Set<string>>();
const mark = (entity: string, attrs: string[]) => {
if (!attrs.length) return;
const s = hitAttrsByEntity.get(entity) ?? new Set<string>();
attrs.forEach((a) => s.add(a));
hitAttrsByEntity.set(entity, s);
};
if (hoveredAttr) {
for (const r of doc.relationships) {
if (r.from.attributes.includes(hoveredAttr) || r.to.attributes.includes(hoveredAttr)) {
relsForAttr.add(r.id);
entsForAttr.add(r.from.entity);
entsForAttr.add(r.to.entity);
// light up the joined columns on BOTH sides of this relationship
mark(r.from.entity, r.from.attributes);
mark(r.to.entity, r.to.attributes);
}
}
}
// attribute rows to emphasise, keyed by entity, when an edge is hovered
const hitAttrsByEntity = new Map<string, Set<string>>();
if (hoveredEdge) {
const r = doc.relationships.find((x) => x.id === hoveredEdge);
if (r) {
hitAttrsByEntity.set(r.from.entity, new Set(r.from.attributes));
hitAttrsByEntity.set(r.to.entity, new Set(r.to.attributes));
mark(r.from.entity, r.from.attributes);
mark(r.to.entity, r.to.attributes);
}
}

Expand Down
33 changes: 33 additions & 0 deletions canvas/src/sme/ModelWorkspace.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,25 @@ export function ModelWorkspace({

{creatingArea && (
<NewSubjectAreaModal
objects={doc.entities.flatMap((e) =>
e.conceptual
? [
{
id: e.conceptual.id,
kind: "conceptual_entity" as const,
name: e.conceptual.name,
definition: e.conceptual.definition ?? null,
synonyms: [],
subject_area: e.conceptual.subject_area ?? null,
// the picker only reads id/name/definition; the rest of the
// GlossaryTerm shape is filled with empties to satisfy the type.
stewardship: null,
ontology: null,
where_used: [],
},
]
: [],
)}
onClose={() => setCreatingArea(false)}
onCreate={(name, definition, members) => {
// Client-minted so the area keeps its identity from preview to PR, and
Expand Down Expand Up @@ -176,6 +195,20 @@ export function ModelWorkspace({
)}
<span className="erd-hint">drag between entities to relate them</span>
<span className="erd-panels">
<button
className="erd-action ghost"
onClick={() => canvasRef.current?.relayout()}
title="Auto-arrange the diagram"
>
Re-layout
</button>
<button
className="erd-action ghost"
onClick={() => canvasRef.current?.fitView()}
title="Fit the whole model in view"
>
Fit
</button>
<button
className={"erd-action ghost" + (panel === "ontology" ? " on" : "")}
onClick={() => setPanel((p) => (p === "ontology" ? null : "ontology"))}
Expand Down
39 changes: 26 additions & 13 deletions canvas/src/sme/NewSubjectAreaModal.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { useEffect, useMemo, useState } from "react";
import { expandSubjectArea, fetchSubjectArea, fetchSubjectAreas } from "../api";
import { expandSubjectArea, fetchSubjectAreas } from "../api";
import type { ClosureHop, GlossaryTerm } from "../types";
import { Pane } from "./SubjectAreaEditor";

Expand All @@ -10,16 +10,21 @@ import { Pane } from "./SubjectAreaEditor";
* intention. This is the same Available/Included picker the Subject areas tab
* uses, with the expansion preview, wrapped around the create. */
export function NewSubjectAreaModal({
objects,
onCreate,
onClose,
}: {
/** every conceptual object in the model — the pool the two panes partition. Passed
* in from the projected model so the picker is populated even when the model has
* NO subject areas yet (creating the first one must still list its objects). */
objects: GlossaryTerm[];
/** name plus the members to seed it with; both staged as one proposal */
onCreate: (name: string, definition: string, members: string[]) => void;
onClose: () => void;
}) {
const [name, setName] = useState("");
const [definition, setDefinition] = useState("");
const [all, setAll] = useState<GlossaryTerm[]>([]);
const [all] = useState<GlossaryTerm[]>(objects);
const [members, setMembers] = useState<string[]>([]);
const [picked, setPicked] = useState<Set<string>>(new Set());
const [filter, setFilter] = useState("");
Expand All @@ -30,20 +35,19 @@ export function NewSubjectAreaModal({
const [seedAreaId, setSeedAreaId] = useState<string>("");
const [error, setError] = useState<string | null>(null);

// Every conceptual object in the model, from any existing area's detail — the
// two panes are a partition of that set. Expansion needs a real area id to POST
// against, so borrow the first one; it only reads the graph.
// The object pool comes from `objects` (the projected model), so the picker is
// populated with no network call and works when the model has zero areas. The only
// thing that needs an existing area is the "add related objects" EXPANSION, which
// POSTs against a saved area's graph — borrow any one for that; if there are none,
// expansion is simply unavailable (a brand-new, unsaved area has no graph to walk).
useEffect(() => {
fetchSubjectAreas()
.then((d) => {
const first = d.subject_areas[0];
if (!first) return;
setSeedAreaId(first.id);
return fetchSubjectArea(first.id).then((detail) =>
setAll([...detail.included, ...detail.available]),
);
if (d.subject_areas[0]) setSeedAreaId(d.subject_areas[0].id);
})
.catch((e) => setError(String(e)));
.catch(() => {
/* no area to seed expansion from; the picker still works from `objects` */
});
}, []);

const memberSet = useMemo(() => new Set(members), [members]);
Expand Down Expand Up @@ -201,7 +205,16 @@ export function NewSubjectAreaModal({
onChange={(e) => setLevels(Math.max(1, Math.min(10, Number(e.target.value))))}
/>
</label>
<button className="sme-secondary" onClick={runExpand}>
<button
className="sme-secondary"
onClick={runExpand}
disabled={!seedAreaId}
title={
seedAreaId
? "Preview related objects to add"
: "Available once the model has at least one saved subject area to walk the graph from"
}
>
Preview
</button>
</div>
Expand Down
5 changes: 4 additions & 1 deletion canvas/src/styles.css
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,10 @@ body { font-family: var(--font); background: var(--bg); color: var(--text); }
.attr-row.empty { color: var(--text-dim); font-style: italic; }
/* attribute-level relationship highlighting (issue #5): the rows an edge/attribute
hover touches brighten; the rest recede. */
.attr-row.rel-hit { background: var(--accent-soft); }
/* a joined column at either end of the hovered relationship: accent wash + a left
marker + brighter name, so the linked data elements read as a pair across cards. */
.attr-row.rel-hit { background: var(--accent-soft); box-shadow: inset 2px 0 0 var(--accent); }
.attr-row.rel-hit .attr-name { color: var(--accent); font-weight: 600; }
.attr-row.rel-dim { opacity: 0.35; }
.attr-row.endpoint .attr-name { font-style: normal; }
/* per-attribute connection ports: sit on the row's own edge, revealed on hover so
Expand Down
Binary file added docs/assets/export-menu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/import-panel.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/propose-dialog.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/review-screen.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading