feat(080): row-type discovery probe + template-seeded generation (cxs2 spec 080)#65
Open
acmeguy wants to merge 4 commits into
Open
feat(080): row-type discovery probe + template-seeded generation (cxs2 spec 080)#65acmeguy wants to merge 4 commits into
acmeguy wants to merge 4 commits into
Conversation
…2 spec 080) - NEW POST /v1/discover-row-types: exact two-pass row-type discovery (novelty = full snapshot every run; stats via per-table cursors: semantic_events (timestamp,event_gid), data_points (timestamp,signature), entities _version + uniqExact(gid) snapshot counts); monotone map-key/ JSON-path inventories; series head-label enrichment (headSchema). - NEW GET /v1/published-template: the published global template by name (the FR-020 maturation package's seed). - smart-generate: optional template_name (seed via buildCubesFromTemplate, probe-pruned by the request filters) + cube_meta (family-#2 provenance, replaces the 013 default_model/template_checksum stamp); template path forces skip_llm, disables deep-profile sampling (registry pruning must not flap), forces the target table into internalTables (partition scoping), and merges regenerations via mergeTemplateModel so ai_generated/team content survives. - buildCubesFromTemplate: additive filters/cubeName/cubeMeta options. - FraiOS JIT provisioning: the row-type-pipeline provider provisions as team ADMIN (dataschemas select/update perms are owner/admin-only) with an upgrade path for already-provisioned identities. Verified live from cxs2 e2e (somi.is, 13.9M-row semantic_events + 2.4M-row data_points): cxs2 suites US2 6/6, US3 10/10, US5 2/2. cubejs npm test: 620/622 (2 pre-existing partition.test.js failures on clean main). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ies head labels - discover-row-types (semantic_events): identity = event value, but rows with an empty event register under synthetic 'type:<type>' keys (owner amendment — page/screen/identify/log/custom carry no event name). - discover-row-types (data_points): optional headSchema — series display labels resolved from <headSchema>.timeseries (non-fatal when absent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner refinement: an event type spans multiple sources across rows (source.label/type are scalar Nullable(String) per row). Add a sourceEvidence config on semantic_events and groupUniqArray(64) over `source.label` in both the novelty + stats passes, surfaced as observedSources alongside observedTypes. groupUniqArray drops NULLs, so sourceless platform events fall out cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner: count() + max(timestamp) are the PRIMARY active-indicator and were missing/zero for most types because they came from the cursor-bounded delta. - Novelty (full-partition) pass now computes total_count + min/max per key — always correct, independent of the delta cursor. count/min/max are cheap columnar aggregates. - The delta pass carries ONLY the expensive union-mergeable evidence (distinct Segment types, source labels, map keys, JSON paths). - Cursor bound is derived from the delta slice (its max IS the global max), removing a second full-partition scan. - Truncation: raise caps (types 100, sources 256) + uniqExact totals + evidenceTruncated flag so capped arrays are no longer silently lossy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to smartdataHQ/cxs2#807 (internal SL row-type models). All changes additive.
New endpoints
POST /v1/discover-row-types— exact two-pass row-type discovery: novelty is a FULL snapshot every run (presence never rides on cursor semantics); stats ride per-table cursors (semantic_events(timestamp,event_gid)tuple-compare,data_points(timestamp,signature),entities_version+uniqExact(gid)current-state counts). Returns monotone map-key/JSON-path inventories (capped) and the next cursor from scan bounds. Series head-label enrichment viaheadSchema(non-fatal).GET /v1/published-template?name=— the published global template (seed for the FR-020 maturation context package).smart-generate (backward compatible)
template_name: seed viabuildCubesFromTemplate+ probe-prune against the requestfilters(published-template fetch replicated in cubejs —templateResolver.js, same GraphQL surface the reconciler's feed reads).cube_meta: caller provenance (marker family Fix: resolve datasource access issues by simplifying user query logic #2,managed_by: row-type-pipeline) stamped INSTEAD of the 013default_model/template_checksumpair.skip_llm; disables deep-profile sampling (registry pruning must not flap on sampled key inventories — the pre-existingpartition.test.jsfailures document the adjacent gap); forces the target table intointernalTablesso the generated cube bakes partition scoping even when team settings don't enumerate the table; regenerations merge viamergeTemplateModel(ai_generated/team content survives; auto_generated refreshes; template converges).Auth/JIT
FraiOS JIT provisioning: tokens with
provider: row-type-pipelineprovision as team admin (dataschemas select/update permissions are owner/admin-only — a plain member reads an EMPTY dataschemas list, silently), with an upgrade path for already-provisioned identities.Verification
npm test: 620/622 — the 2 failures are pre-existing on clean main (verified via stash).🤖 Generated with Claude Code