Add SEMARCHY dialect and scaffold TypeScript OSI<>Semarchy converter - #1
Open
amergey wants to merge 7 commits into
Open
Add SEMARCHY dialect and scaffold TypeScript OSI<>Semarchy converter#1amergey wants to merge 7 commits into
amergey wants to merge 7 commits into
Conversation
Register Semarchy as a new spoke in the hub-and-spoke converter model. Spec change (SEMARCHY dialect), synced across all artifacts: - core-spec/osi-schema.json: add SEMARCHY to the Dialect enum - core-spec/spec.md, core-spec/spec.yaml: document the dialect - validation/validate.py: map SEMARCHY (skip sqlglot SQL validation) - python/src/osi/models.py: add SEMARCHY to OSIDialect - converters/index.md: add SEMARCHY to Supported Vendors Both examples still validate against the schema. New converter converters/semarchy (TypeScript, Node >=18, ESM) — the repo's first non-Python/Java converter, mirroring osi-dbt's CLI shape: - commander CLI: osi-to-semarchy / semarchy-to-osi subcommands - OSI types mirroring python/src/osi/models.py - dialect fallback chain (SEMARCHY -> ANSI_SQL -> warn), fully implemented - ajv-based schema validation for OSI and Semarchy output (best-effort) - vitest tests (dialect fallback green; roundtrip todo pending schema) The per-construct import/export mapping is stubbed and marked TODO(schema): it needs the Semarchy semantic-model JSON Schema, which must be dropped into converters/semarchy/schemas/. typecheck, tests, and build all pass on Node 20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flesh out the scaffolded converter into a working bidirectional mapping between OSI and the Semarchy xDM model, based on the Semarchy model JSON Schemas (Entity, Reference, UniqueKey). Model shape: a Semarchy model is a directory of _type-discriminated YAML objects; the OSI side is a single YAML document. Added io.ts to read/write that directory form. Mapping (per project decision, core ER only): - Entity <-> dataset (source = <_package>.<physicalTableName>) - EntityAttribute <-> field (name = _name, ANSI_SQL expr = physicalName, Date/Timestamp -> dimension.is_time) - Entity.primaryKey <-> dataset.primary_key - UniqueKey <-> dataset.unique_keys (matched by owning entity) - Reference <-> relationship (fromEntity=many, toEntity=one) Lossy by design (documented in README): metrics, SemQL enrichers, type definitions, views, and UI/process objects are dropped with a warning rather than preserved in custom_extensions. Export fills the Semarchy required fields OSI lacks with sensible defaults; composite keys flatten to their first column. Validation: Ajv2020 for OSI output (draft 2020-12), draft-07 Ajv per Semarchy object type. Bundled schemas under schemas/. Tests (vitest): dialect fallback, TPC-DS export (schema-valid), fixture import (schema-valid, drop-warning, time dimension), and a Semarchy->OSI->Semarchy roundtrip preserving the core ER structure. typecheck, tests (8), and build all pass on Node 20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Validated against a real Semarchy model (dxp-demo ProductRetail); adjust the importer to the conventions that differ from the bare schema: - Directory is a nested tree of .seml (YAML) files, not a flat dir of .yaml: walk recursively and accept .seml/.yaml/.yml. - Model name comes from the dedicated `Model` object (_type: Model), not the first _package; `Model` is consumed as metadata (not warned as unsupported). - References are fully qualified: primaryKey (`Pkg.entities.X.X.attr`), fromEntity/toEntity, and unique-key attributes are FQNs -> resolve to the local (last-segment) name; entity lookup for relationships is by local name. - dataType casing varies (STRING vs String, TIMESTAMP/DATE vs Timestamp/Date): is_time detection is now case-insensitive (isTimeDataType helper). - source is qualified with the model's root package (<pkg>.<physicalTableName>). - Foreign key column read from reference.foreignAttribute._name. - writeSemarchyDir emits <_name>.<_type>.seml. Fixtures rewritten to mirror the real format (nested dirs, .seml, a Model object, FQN references, a UniqueKey). Roundtrip test normalizes FQN entity names. typecheck + 8 tests pass on Node 20; importing the real ProductRetail model produces OSI that passes validation/validate.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make OSI -> Semarchy export produce a re-importable native model instead of flat bare-name files: - Generate a root `Model` object (_type: Model) carrying the model name. - Regenerate Semarchy fully-qualified names: entity packages (<pkg>.entities.<Entity>), primaryKey (<pkg>.entities.<E>.<E>.<attr>), reference fromEntity/toEntity and foreignAttribute.entity, and unique-key entity/attribute references. - Emit a reference foreignAttribute (the FK column) so imports resolve it. - writeSemarchyDir writes the native tree: <pkg>.Model.seml at the root, entities/<Name>/<Name>.Entity.seml, entities/<Name>/unique_keys/, and references/<Name>.Reference.seml. Added a file-based export->reimport test. Verified a full roundtrip on the real ProductRetail demo model: the generated tree mirrors the native layout (same file paths and FQNs), re-imports, and the re-imported OSI passes validation/validate.py with all 8 datasets and 9 relationships preserved. typecheck + 9 tests pass on Node 20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement the SemQLEnricher <-> OSI computed-field mapping, the intended use of
the SEMARCHY dialect (SemQL is not SQL and is carried verbatim, not translated).
Import (Semarchy -> OSI): each SemQLEnricher expression {attributeName,
expression} adds a SEMARCHY dialect entry to the matching field, keeping its
ANSI_SQL column reference (multi-dialect, lossless). Unmatched attributes get a
SEMARCHY-only field (warned). The enricher-level `condition` has no OSI
equivalent and is dropped with a warning.
Export (OSI -> Semarchy): fields carrying a SEMARCHY expression are grouped back
into one SemQLEnricher per entity. Crucially, the entity attribute's physicalName
now comes from the ANSI_SQL expression, never the SemQL, so a computed field no
longer mangles the SemQL into a column name. Enrichers are written under
entities/<Name>/enrichers/.
SemQLEnricher schema bundled and validated. Fixture gains a Customer enricher
(with a dropped condition). typecheck + 11 tests pass; CLI roundtrip on the
fixture shows fullName carrying both ANSI_SQL (FULL_NAME) and SEMARCHY
(UPPER(fullName)), and export regenerating the enricher while keeping the
physical column.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An enricher is a data-preparation rule: once it runs, the MDM column holds the materialized value, so a consumer of the model sees a plain column. Turning it into a query-time OSI expression on import over-states it. - Import: enrichers are dropped with a warning; the enriched field keeps only its ANSI_SQL column reference (no SEMARCHY expression attached). - Export: unchanged — a field carrying a SEMARCHY dialect expression is still grouped into a SemQLEnricher per entity, with physicalName taken from ANSI_SQL. Tests updated: import asserts the field stays a plain column and the enricher is dropped; export is now exercised from a hand-authored OSI model that carries a SEMARCHY expression. README and memory updated. typecheck + 11 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Register Semarchy as a new spoke in the hub-and-spoke converter model.
Spec change (SEMARCHY dialect), synced across all artifacts:
New converter converters/semarchy (TypeScript, Node >=18, ESM) — the repo's first non-Python/Java converter, mirroring osi-dbt's CLI shape:
The per-construct import/export mapping is stubbed and marked TODO(schema): it needs the Semarchy semantic-model JSON Schema, which must be dropped into converters/semarchy/schemas/. typecheck, tests, and build all pass on Node 20.