feat(workflow-tengo): make ColumnUniversalId a first-class column reference

.changeset/cool-pandas-listen.md

@@ -0,0 +1,38 @@
+---
+"@platforma-sdk/workflow-tengo": minor
+"@milaboratories/pl-model-common": patch
+---
+
+Make `ColumnUniversalId` a first-class column reference in the workflow.
+
+A block can now pass any id the model mints — bare leaf, `ColumnFilteredId`,
+`ColumnOverriddenId`, `ColumnDiscoveredId` — straight into
+`createPBundleBuilder().addSingle(...)`, `pt.frameFromColumnBundle` and
+`tableBuilder`, and the workflow resolves it, reconstructs its effective spec,
+and builds the linker join a discovered id describes.
+
+New `:pframes.column-id` decodes and applies the layers (pure functions over
+JSON, unit-tested offline against the TypeScript source of truth). New
+`bundle.getQueryEntry(id)` compiles an id into a `SpecQueryJoinEntry` for
+`pt.p._rawQueryEntry`, folding every `path` hop into a nested `linkerJoin`; new
+`bundle.getAxesSpecOf(id)` reports the axes a column actually contributes,
+projecting out each linker's one-side axes via the new `pSpec.linkerSides` /
+`pSpec.splitAxes` / `pSpec.linkerJoinAxesSpec`.
+
+`EnrichmentRef` becomes a derived special case: `columnId.fromEnrichmentRef`
+normalizes the v1 wire form into a `ColumnDiscoveredKey`, and `tableBuilder`
+resolves both spellings through one path. The type is deprecated; existing
+callers keep working.
+
+Fixes two latent defects found on the way:
+
+- `bundle.getAxesSpec` read raw pool specs while `getSpec` applied axis filters,
+  so a filtered id could contribute axes that its own spec no longer had.
+  `pt.frameFromColumnBundle` calls the former before the latter, so the
+  disagreement was reachable.
+- `pSpec.A_IS_LINKER_COLUMN` was read by `pt.p.linkerJoin` but never exported,
+  which would have panicked on the builder's first use.
+
+No behaviour change for existing id shapes: the registration and read paths now
+share one key derivation that reproduces all four previous derivations, and the
+legacy `FilteredPColumnId` keeps keying on its `source`.

docs/column-access-api.md

@@ -733,3 +733,78 @@ label }` wire shape with `refsWithEnrichments`. Its `@deprecated` note points at
 since that changes the stored value from a `PlRef` to a `ColumnUniversalId`.
 
 Removed names and their one-to-one replacements are in the migration doc.
+
+## Workflow Side (Tengo)
+
+A `ColumnUniversalId` is a first-class column reference in
+`@platforma-sdk/workflow-tengo`, not just something the host understands. The
+model mints an id; the workflow resolves it, reconstructs its spec, and — for a
+discovered id — builds the join. Blocks pass ids straight through their args.
+
+`:pframes.column-id` is the decoder: `decode`, `kindOf`, `isUniversalId`,
+`poolKey`, `unwrap`, `extractLeafId`, `referencedIds`, `applyAxisFilters`,
+`applySpecOverrides`, `effectiveSpec`, `fromEnrichmentRef`. Every function is
+pure over decoded JSON, so it is unit-tested offline against the TypeScript
+source of truth (`spec/ids.ts`, `spec/filtered_column.ts`, `spec/overridden.ts`).
+
+### Where each layer is realised
+
+| Layer        | Realised                                                          | Mechanism                                                                          |
+| ------------ | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
+| leaf         | resolution time                                                   | `bquery.resolve` on the terminal `{__isRef, blockId, name}`                        |
+| `Filtered`   | read time (bundle) / execution time (frame)                       | spec: axes dropped; data: `:pframes.slice-data`, or `pt.p.slicedColumn` in a frame |
+| `Overridden` | read time                                                         | spec math only — the engine's query language has no `specOverride` node            |
+| `Discovered` | execution time                                                    | `bundle.getQueryEntry` → `bquery.buildQuery` → `pt.p._rawQueryEntry`               |
+
+### Pool keys
+
+A rich id registers and reads under **the id itself**, never under its leaf. Two
+projections of one physical column, or one column reached by two linker routes,
+are different columns to a consumer and must not collapse into one pool entry —
+this is the workflow's counterpart to `rebrandLeafId`. It also makes the
+registration key and the read key identical by construction, so the collector
+and the unmarshaller cannot drift.
+
+The one exception is the legacy `FilteredPColumnId` (`{source, axisFilters}` with
+a *selector map* source), which still keys on its source. Marker fields are
+therefore tested before that duck-test.
+
+### Axes of a discovered column
+
+`getSpec` on a discovered id passes through to the hit — the linker chain enables
+co-indexing, it does not remap the hit's own axes, and the engine needs the hit's
+real axes to resolve the join.
+
+The axes the column *contributes to a frame* are different, and that is what
+`getAxesSpecOf` reports: each hop inner-joins its linker and projects out that
+linker's one-side axes, so the result lands on the many side of the outermost
+hop. `pSpec.linkerSides` derives the two sides from the linker's own spec by
+splitting its `axesSpec` into parent-connected components (exactly two are
+required; the component holding axis 0 is the one side). A hit axis the route
+never touches survives, which a blanket empty `axesSpec` would lose.
+
+### Deliberate divergences from the model side
+
+- An `axesSpec` patch whose index is past the end of the spec **panics** instead
+  of appending. The workflow registers specs against data that already exists,
+  and an axis with no data behind it is a spec the engine cannot satisfy.
+- Axis filters on a discovered column **panic**: pinning them would have to
+  become a `sliceAxes` node inside the join, and the query builder emits only
+  column / linkerJoin / innerJoin. Filter after the join instead.
+- `queriesQualifications` is **not** carried. Its keys are the `PObjectId`s of
+  external primary columns, and the workflow addresses its primaries by `PlRef`
+  and frame key — the key space does not exist here. `columnQualifications` does
+  reach `tableBuilder`'s join entry.
+- `tableBuilder` accepts a bare or discovered id but refuses one carrying spec
+  overrides or axis filters: it resolves specs as futures, so there is nothing to
+  patch or slice at that point. Use a column bundle for those.
+
+### `EnrichmentRef` is derived
+
+`EnrichmentRef` (`{__isEnrichment: "v1", hit, path[].linker, qualifications}`) is
+the strictly narrower spelling of the same idea: its hit and every hop must be
+bare global ids and its only step type is `linker`. `fromEnrichmentRef`
+normalizes it into a `ColumnDiscoveredKey`, and `tableBuilder` funnels both
+spellings through one resolution path. The reverse conversion is lossy and
+deliberately absent — a discovered key whose hit or hop carries a projection has
+no v1 representation. The type is deprecated, with no removal date.

lib/model/common/src/ref.ts

@@ -142,6 +142,12 @@ export type EnrichmentStep = EnrichmentLinkerStep;
  * (`canonicalize({ __isRef: true, blockId, name })`) which the workflow decodes
  * to a PlRef before resolving. For a prerun-sourced hop, supply a resolved
  * `{spec, data}` map in place of the id — `tableBuilder` accepts both.
+ *
+ * @deprecated Pass a `ColumnDiscoveredId` instead. It says the same thing —
+ * a hit plus the linker route to it — without restricting the hit and the hops
+ * to bare global ids, and the workflow now normalizes this form into it
+ * (`pframes.column-id.fromEnrichmentRef`). Kept working for existing callers;
+ * no removal date.
  */
 export type EnrichmentRef = {
   readonly __isEnrichment: "v1";

sdk/workflow-tengo/src/pframes/axes-split.test.tengo

@@ -0,0 +1,138 @@
+// Tests for the parent-connected-component math in `:pframes.spec`
+// (`splitAxes`, `linkerSides`, `linkerJoinAxesSpec`).
+//
+// Ported from pframes-rs: `AxesSpec::split` (packages/spec/src/axes/axes_spec.rs),
+// the linker one-side/many-side derivation (packages/spec/src/linker_index.rs),
+// and the linkerJoin axis projection rule (the joined axes minus the linker's
+// one-side axes).
+
+test := import(":test")
+pSpec := import(":pframes.spec")
+
+axis := func(name, ...parents) {
+	a := { name: name, type: "String" }
+	if len(parents) > 0 {
+		a.parentAxes = parents[0]
+	}
+	return a
+}
+
+// A linker between `clusterId` (one side) and `clonotypeKey` (many side).
+// Axis 0 lands in the one-side component, matching the engine's convention.
+linkerSpec := func() {
+	return {
+		kind: "PColumn",
+		name: "linker",
+		valueType: "Int",
+		annotations: { "pl7.app/isLinkerColumn": "true" },
+		axesSpec: [
+			axis("clusterId"),
+			axis("clonotypeKey")
+		]
+	}
+}
+
+//
+// splitAxes
+//
+
+TestSplitAxesSingleComponent := func() {
+	// b and c both hang off a, so all three are one component.
+	axesSpec := [axis("a"), axis("b", [0]), axis("c", [0])]
+	test.isEqual([[0, 1, 2]], pSpec.splitAxes(axesSpec))
+}
+
+TestSplitAxesIndependentAxesAreSeparateComponents := func() {
+	test.isEqual([[0], [1]], pSpec.splitAxes([axis("a"), axis("b")]))
+}
+
+TestSplitAxesOrdersComponentsByLowestMember := func() {
+	// a(0) — c(2) joined; b(1) alone. Component containing 0 comes first.
+	axesSpec := [axis("a"), axis("b"), axis("c", [0])]
+	test.isEqual([[0, 2], [1]], pSpec.splitAxes(axesSpec))
+}
+
+TestSplitAxesTransitiveChainCollapses := func() {
+	// a <- b <- c is a single component even though c never names a.
+	axesSpec := [axis("a"), axis("b", [0]), axis("c", [1])]
+	test.isEqual([[0, 1, 2]], pSpec.splitAxes(axesSpec))
+}
+
+TestSplitAxesThreeComponents := func() {
+	axesSpec := [axis("a"), axis("b"), axis("c"), axis("d", [1])]
+	test.isEqual([[0], [1, 3], [2]], pSpec.splitAxes(axesSpec))
+}
+
+TestSplitAxesEmpty := func() {
+	test.isEqual([], pSpec.splitAxes([]))
+}
+
+//
+// linkerSides
+//
+
+TestLinkerSidesSplitsOneAndMany := func() {
+	sides := pSpec.linkerSides(linkerSpec())
+	test.isEqual([0], sides.oneSide)
+	test.isEqual([1], sides.manySide)
+}
+
+TestLinkerSidesWithCompositeManySide := func() {
+	// Many side is a two-axis group: clonotypeKey plus a child axis.
+	spec := {
+		kind: "PColumn",
+		name: "linker",
+		valueType: "Int",
+		axesSpec: [
+			axis("clusterId"),
+			axis("clonotypeKey"),
+			axis("chain", [1])
+		]
+	}
+	sides := pSpec.linkerSides(spec)
+	test.isEqual([0], sides.oneSide)
+	test.isEqual([1, 2], sides.manySide)
+}
+
+//
+// linkerJoinAxesSpec — linker axes lead, one-side axes projected out
+//
+
+TestLinkerJoinAxesSpecProjectsOutOneSide := func() {
+	// Hit lives on the one side; after the join only the many side survives.
+	hitAxes := [axis("clusterId")]
+	got := pSpec.linkerJoinAxesSpec(linkerSpec(), hitAxes)
+	test.isEqual([axis("clonotypeKey")], got)
+}
+
+TestLinkerJoinAxesSpecKeepsHitAxesBeyondTheRoute := func() {
+	// A hit carrying an axis the route does not touch keeps it — this is the
+	// case a blanket `axesSpec: []` would silently lose.
+	hitAxes := [axis("clusterId"), axis("sampleId")]
+	got := pSpec.linkerJoinAxesSpec(linkerSpec(), hitAxes)
+	test.isEqual([axis("clonotypeKey"), axis("sampleId")], got)
+}
+
+TestLinkerJoinAxesSpecDeduplicatesSharedAxes := func() {
+	// The hit already sits on the many side: no duplicate column.
+	hitAxes := [axis("clonotypeKey")]
+	got := pSpec.linkerJoinAxesSpec(linkerSpec(), hitAxes)
+	test.isEqual([axis("clonotypeKey")], got)
+}
+
+TestLinkerJoinAxesSpecDistinguishesAxesByDomain := func() {
+	// Same name, different domain — distinct axes, so nothing is projected out
+	// or deduplicated by name alone.
+	linker := {
+		kind: "PColumn",
+		name: "linker",
+		valueType: "Int",
+		axesSpec: [
+			{ name: "group", type: "String", domain: { side: "primary" } },
+			{ name: "group", type: "String", domain: { side: "secondary" } }
+		]
+	}
+	hitAxes := [{ name: "group", type: "String", domain: { side: "primary" } }]
+	got := pSpec.linkerJoinAxesSpec(linker, hitAxes)
+	test.isEqual([{ name: "group", type: "String", domain: { side: "secondary" } }], got)
+}