diff --git a/README.md b/README.md index 62aa081..03017cd 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ SkyData Studio is the post-ingestion data engineering application in the Sky ecosystem. It begins where SkyCommand's ingestion responsibility ends and prepares trusted data for analytical consumption by SkyWeb Analytics, Power BI, and future client-facing applications. -**Current status:** Phases 0 through 5 are complete, and **Phase 6.1 is complete** with a Docker-isolated dbt/Postgres runtime, three tested staging → intermediate → mart models, 14 dbt data tests, and 3/3 model/layer readiness over the 26,335-row Federal Funds Rate proof. **Phase 6.2 is now in progress:** SkyData Studio reads dbt's generated manifest and run-result artifacts to expose a live Data Models catalogue with build, test, column, and direct dependency evidence. +**Current status:** Phases 0 through 5 are complete, **Phase 6.1 is complete** with a Docker-isolated dbt/Postgres runtime and the proven three-layer Federal Funds Rate model chain, and **Phase 6.2 is complete** with an artifact-backed Data Models catalogue reporting 3/3 models ready, 1 source, and 14 data tests. **Phase 6.3 is now in progress:** the governed mart becomes SkyData Studio's first dbt semantic model, with a stable entity, reusable dimensions, and four governed metrics projected from dbt artifacts. --- @@ -155,7 +155,7 @@ Unblock-File .\scripts\dbt.ps1 # only when Windows marks the trusted local help .\scripts\dbt.ps1 build ``` -Phase 6.1 materializes the proven `stg_fed_funds_rate → int_fed_funds_rate_changes → fct_fed_funds_rate_daily` chain. Open `/workspace/transformations` for physical relation readiness and row-count proof. Phase 6.2 reads the generated `target/manifest.json` and `target/run_results.json` artifacts through `GET /api/v1/transformations/dbt/models`; open `/workspace/models` to inspect model layers, materializations, tests, columns, tags, and direct dependencies. Generated dbt artifacts remain ignored by Git and are refreshed by the next dbt build. +Phase 6.1 materializes the proven `stg_fed_funds_rate → int_fed_funds_rate_changes → fct_fed_funds_rate_daily` chain. Open `/workspace/transformations` for physical relation readiness and row-count proof. Phase 6.2 reads the generated `target/manifest.json` and `target/run_results.json` artifacts through `GET /api/v1/transformations/dbt/models`; open `/workspace/models` to inspect model layers, materializations, tests, columns, tags, and direct dependencies. Phase 6.3 adds model-embedded semantic definitions to the governed mart, a dedicated DAY-grain MetricFlow time spine, and projects semantic evidence through `GET /api/v1/transformations/dbt/semantic`; open `/delivery/semantic` to inspect the semantic model, primary entity, dimensions, and governed metrics. Generated dbt artifacts remain ignored by Git and are refreshed by the next dbt build. --- @@ -406,7 +406,8 @@ The repository currently includes: - a Phase 5.3 daily DFF timetable plus bounded Airflow REST API v2 backfill controls; - a completed Phase 5.4 native Airflow asset-event bridge from terminal successful SkyCommand FRED/DFF ingestion evidence, including duplicate-signal reuse; - a completed Phase 6.1 Dockerized dbt/Postgres runtime plus the first governed staging → intermediate → mart model chain with 14 green data tests; -- a Phase 6.2 artifact-backed dbt model catalogue surface for logical model, column, test, build, and dependency evidence; +- a completed Phase 6.2 artifact-backed dbt model catalogue surface for logical model, column, test, build, and dependency evidence; +- a Phase 6.3 model-embedded dbt semantic definition surface with one governed semantic model, reusable dimensions, and four governed metrics; - backend/frontend validation, repository map, and compact handoff tooling. -The active implementation target is **Phase 6.2 — dbt Model Catalogue and Artifact Evidence**. Phase 6.1 is complete: the isolated runtime, PostgreSQL connection, three-layer DFF model chain, 26,335-row grain, 14 data tests, Transformations API, and workbench readiness proof are green. Phase 6.2 now turns dbt's generated artifacts into the logical Data Models surface without duplicating dbt-owned metadata in Studio PostgreSQL. +The active implementation target is **Phase 6.3 — dbt Semantic Model and Governed Metric Foundation**. Phase 6.1 is complete: the isolated runtime, PostgreSQL connection, three-layer DFF model chain, 26,335-row grain, 14 data tests, Transformations API, and workbench readiness proof are green. Phase 6.2 is also complete: dbt's generated artifacts now drive a 3/3-ready logical Data Models catalogue without duplicating dbt-owned metadata in Studio PostgreSQL. Phase 6.3 extends that same single-authority pattern into semantic metadata: one mart-owned semantic model, one stable primary entity, reusable dimensions, four governed metrics, and one dedicated DAY-grain MetricFlow time spine surfaced through artifact evidence. diff --git a/apps/api/skydata_studio/api/routes/dbt.py b/apps/api/skydata_studio/api/routes/dbt.py index e3fd839..9347217 100644 --- a/apps/api/skydata_studio/api/routes/dbt.py +++ b/apps/api/skydata_studio/api/routes/dbt.py @@ -2,9 +2,14 @@ from sqlalchemy.exc import SQLAlchemyError from skydata_studio.db.session import SessionDependency -from skydata_studio.schemas.dbt import DbtModelCatalogueSummary, DbtTransformationSummary +from skydata_studio.schemas.dbt import ( + DbtModelCatalogueSummary, + DbtSemanticLayerSummary, + DbtTransformationSummary, +) from skydata_studio.services.dbt_transformations import ( dbt_model_catalogue, + dbt_semantic_layer, dbt_transformation_summary, ) @@ -37,3 +42,17 @@ def dbt_models() -> DbtModelCatalogueSummary: "Run .\\scripts\\dbt.ps1 build and refresh the model catalogue." ), ) from error + + +@router.get("/semantic", response_model=DbtSemanticLayerSummary) +def dbt_semantic() -> DbtSemanticLayerSummary: + try: + return dbt_semantic_layer() + except (OSError, ValueError, TypeError) as error: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail=( + "SkyData Studio could not read the dbt semantic artifacts. " + "Run .\\scripts\\dbt.ps1 build and refresh the semantic layer." + ), + ) from error diff --git a/apps/api/skydata_studio/api/routes/platform.py b/apps/api/skydata_studio/api/routes/platform.py index b54d57f..8786090 100644 --- a/apps/api/skydata_studio/api/routes/platform.py +++ b/apps/api/skydata_studio/api/routes/platform.py @@ -83,7 +83,7 @@ def platform_summary() -> PlatformSummary: product="SkyData Studio", subtitle="Data Engineering Workbench", theme="Aurora Foundry", - current_phase="Phase 6.2 — dbt Model Catalogue and Artifact Evidence", + current_phase="Phase 6.3 — dbt Semantic Model and Governed Metric Foundation", boundary=( "SkyData Studio starts after SkyCommand ingestion and publishes governed " "analytical products for SkyWeb Analytics, Power BI, and future consumers." diff --git a/apps/api/skydata_studio/schemas/dbt.py b/apps/api/skydata_studio/schemas/dbt.py index f4fa014..377de28 100644 --- a/apps/api/skydata_studio/schemas/dbt.py +++ b/apps/api/skydata_studio/schemas/dbt.py @@ -67,3 +67,55 @@ class DbtModelCatalogueSummary(BaseModel): source_count: int = Field(ge=0) test_count: int = Field(ge=0) models: list[DbtModelCatalogueItem] + + +class DbtSemanticEntitySummary(BaseModel): + name: str + entity_type: Literal["PRIMARY", "UNIQUE", "FOREIGN", "NATURAL"] + expression: str | None = None + description: str | None = None + + +class DbtSemanticDimensionSummary(BaseModel): + name: str + dimension_type: Literal["TIME", "CATEGORICAL"] + expression: str | None = None + granularity: str | None = None + description: str | None = None + + +class DbtSemanticMetricSummary(BaseModel): + unique_id: str + name: str + label: str + metric_type: Literal["SIMPLE", "RATIO", "CUMULATIVE", "DERIVED", "CONVERSION"] + description: str | None = None + aggregation: str | None = None + expression: str | None = None + time_dimension: str | None = None + semantic_model: str | None = None + + +class DbtSemanticModelSummary(BaseModel): + unique_id: str + name: str + description: str | None = None + relation: str | None = None + default_time_dimension: str | None = None + entities: list[DbtSemanticEntitySummary] + dimensions: list[DbtSemanticDimensionSummary] + metric_names: list[str] + + +class DbtSemanticLayerSummary(BaseModel): + project_name: str = "skydata_studio" + phase: str = "Phase 6.3 — dbt Semantic Model and Governed Metric Foundation" + artifact_status: Literal["READY", "PENDING", "MISSING"] + generated_at: str | None = None + dbt_version: str | None = None + semantic_model_count: int = Field(ge=0) + metric_count: int = Field(ge=0) + entity_count: int = Field(ge=0) + dimension_count: int = Field(ge=0) + semantic_models: list[DbtSemanticModelSummary] + metrics: list[DbtSemanticMetricSummary] diff --git a/apps/api/skydata_studio/services/dbt_transformations.py b/apps/api/skydata_studio/services/dbt_transformations.py index 9ff6c2f..d1d36bc 100644 --- a/apps/api/skydata_studio/services/dbt_transformations.py +++ b/apps/api/skydata_studio/services/dbt_transformations.py @@ -12,6 +12,11 @@ DbtModelColumnSummary, DbtModelDependencySummary, DbtRelationSummary, + DbtSemanticDimensionSummary, + DbtSemanticEntitySummary, + DbtSemanticLayerSummary, + DbtSemanticMetricSummary, + DbtSemanticModelSummary, DbtTransformationSummary, ) @@ -211,6 +216,7 @@ def dbt_model_catalogue(target_dir: Path | None = None) -> DbtModelCatalogueSumm and node.get("resource_type") == "model" and node.get("package_name") == "skydata_studio" and bool((node.get("config") or {}).get("enabled", True)) + and "semantic_utility" not in (node.get("tags") or []) } test_nodes = [ node @@ -309,3 +315,302 @@ def dbt_model_catalogue(target_dir: Path | None = None) -> DbtModelCatalogueSumm test_count=len(test_nodes), models=models, ) + + +def _semantic_relation(semantic_model: dict[str, Any]) -> str | None: + node_relation = semantic_model.get("node_relation") + if isinstance(node_relation, dict): + schema = node_relation.get("schema_name") + alias = node_relation.get("alias") + if isinstance(schema, str) and schema and isinstance(alias, str) and alias: + return f"{schema}.{alias}" + relation_name = node_relation.get("relation_name") + if isinstance(relation_name, str) and relation_name: + return relation_name + model = semantic_model.get("model") + return str(model) if isinstance(model, str) and model else None + + +def _semantic_default_time_dimension(semantic_model: dict[str, Any]) -> str | None: + direct_value = semantic_model.get("agg_time_dimension") + if isinstance(direct_value, str) and direct_value: + return direct_value + defaults = semantic_model.get("defaults") + if not isinstance(defaults, dict): + return None + value = defaults.get("agg_time_dimension") + return str(value) if isinstance(value, str) and value else None + + +def _semantic_entities(semantic_model: dict[str, Any]) -> list[DbtSemanticEntitySummary]: + raw_entities = semantic_model.get("entities") + if not isinstance(raw_entities, list): + return [] + entities: list[DbtSemanticEntitySummary] = [] + for entity in raw_entities: + if not isinstance(entity, dict): + continue + raw_type = str(entity.get("type") or "").upper() + if raw_type not in {"PRIMARY", "UNIQUE", "FOREIGN", "NATURAL"}: + continue + entities.append( + DbtSemanticEntitySummary( + name=str(entity.get("name") or "unnamed_entity"), + entity_type=cast(Any, raw_type), + expression=(str(entity.get("expr")) if entity.get("expr") else None), + description=( + str(entity.get("description")) if entity.get("description") else None + ), + ) + ) + return entities + + +def _semantic_dimensions( + semantic_model: dict[str, Any], +) -> list[DbtSemanticDimensionSummary]: + raw_dimensions = semantic_model.get("dimensions") + if not isinstance(raw_dimensions, list): + return [] + dimensions: list[DbtSemanticDimensionSummary] = [] + for dimension in raw_dimensions: + if not isinstance(dimension, dict): + continue + raw_type = str(dimension.get("type") or "").upper() + if raw_type not in {"TIME", "CATEGORICAL"}: + continue + type_params = dimension.get("type_params") + granularity = None + if isinstance(type_params, dict): + value = type_params.get("time_granularity") + if isinstance(value, str) and value: + granularity = value.upper() + dimensions.append( + DbtSemanticDimensionSummary( + name=str(dimension.get("name") or "unnamed_dimension"), + dimension_type=cast(Any, raw_type), + expression=(str(dimension.get("expr")) if dimension.get("expr") else None), + granularity=granularity, + description=( + str(dimension.get("description")) + if dimension.get("description") + else None + ), + ) + ) + return dimensions + + +def _semantic_measure_index( + semantic_models: dict[str, Any], +) -> dict[str, tuple[str, dict[str, Any], str | None]]: + index: dict[str, tuple[str, dict[str, Any], str | None]] = {} + for semantic_model in semantic_models.values(): + if not isinstance(semantic_model, dict): + continue + semantic_name = str(semantic_model.get("name") or "") + default_time_dimension = _semantic_default_time_dimension(semantic_model) + raw_measures = semantic_model.get("measures") + if not isinstance(raw_measures, list): + continue + for measure in raw_measures: + if not isinstance(measure, dict): + continue + name = measure.get("name") + if isinstance(name, str) and name: + index[name] = (semantic_name, measure, default_time_dimension) + return index + + +def _semantic_metric_measure_name(metric: dict[str, Any]) -> str | None: + type_params = metric.get("type_params") + if not isinstance(type_params, dict): + return None + measure = type_params.get("measure") + if isinstance(measure, dict): + name = measure.get("name") + return str(name) if isinstance(name, str) and name else None + return None + + +def _semantic_metric_model_name( + metric: dict[str, Any], + semantic_models: dict[str, Any], +) -> str | None: + for key in ("semantic_model", "semantic_model_name"): + value = metric.get(key) + if isinstance(value, str) and value: + return value.split(".")[-1] + + depends_on = metric.get("depends_on") + dependency_ids = depends_on.get("nodes", []) if isinstance(depends_on, dict) else [] + for dependency_id in dependency_ids: + if not isinstance(dependency_id, str): + continue + semantic_model = semantic_models.get(dependency_id) + if isinstance(semantic_model, dict): + return str(semantic_model.get("name") or dependency_id) + + metric_path = metric.get("original_file_path") + if isinstance(metric_path, str) and metric_path: + matches = [ + semantic_model + for semantic_model in semantic_models.values() + if isinstance(semantic_model, dict) + and semantic_model.get("original_file_path") == metric_path + ] + if len(matches) == 1: + return str(matches[0].get("name") or "") or None + return None + + +def dbt_semantic_layer(target_dir: Path | None = None) -> DbtSemanticLayerSummary: + target = target_dir or _default_dbt_target_dir() + manifest_path = target / "manifest.json" + if not manifest_path.exists(): + return DbtSemanticLayerSummary( + artifact_status="MISSING", + semantic_model_count=0, + metric_count=0, + entity_count=0, + dimension_count=0, + semantic_models=[], + metrics=[], + ) + + manifest = _read_json(manifest_path) + metadata = manifest.get("metadata") + metadata = metadata if isinstance(metadata, dict) else {} + + raw_semantic_models = manifest.get("semantic_models") + raw_metrics = manifest.get("metrics") + semantic_models_payload = ( + cast(dict[str, Any], raw_semantic_models) + if isinstance(raw_semantic_models, dict) + else {} + ) + metrics_payload = ( + cast(dict[str, Any], raw_metrics) if isinstance(raw_metrics, dict) else {} + ) + + project_semantic_models = { + unique_id: semantic_model + for unique_id, semantic_model in semantic_models_payload.items() + if isinstance(semantic_model, dict) + and semantic_model.get("package_name") == "skydata_studio" + } + project_metrics = { + unique_id: metric + for unique_id, metric in metrics_payload.items() + if isinstance(metric, dict) and metric.get("package_name") == "skydata_studio" + } + + measure_index = _semantic_measure_index(project_semantic_models) + metric_names_by_semantic_model: dict[str, list[str]] = {} + metrics: list[DbtSemanticMetricSummary] = [] + for unique_id, metric in sorted( + project_metrics.items(), key=lambda item: str(item[1].get("name") or "") + ): + measure_name = _semantic_metric_measure_name(metric) + semantic_name = _semantic_metric_model_name(metric, project_semantic_models) + measure: dict[str, Any] = {} + default_time_dimension = None + if measure_name and measure_name in measure_index: + semantic_name, measure, default_time_dimension = measure_index[measure_name] + elif semantic_name: + semantic_model = next( + ( + candidate + for candidate in project_semantic_models.values() + if isinstance(candidate, dict) + and candidate.get("name") == semantic_name + ), + None, + ) + if isinstance(semantic_model, dict): + default_time_dimension = _semantic_default_time_dimension(semantic_model) + + if semantic_name: + metric_names_by_semantic_model.setdefault(semantic_name, []).append( + str(metric.get("name") or unique_id) + ) + + metric_type = str(metric.get("type") or "simple").upper() + if metric_type not in {"SIMPLE", "RATIO", "CUMULATIVE", "DERIVED", "CONVERSION"}: + metric_type = "SIMPLE" + aggregation_value = metric.get("agg", measure.get("agg")) + expression_value = metric.get("expr", measure.get("expr")) + time_dimension_value = metric.get( + "agg_time_dimension", + measure.get("agg_time_dimension", default_time_dimension), + ) + aggregation = ( + str(aggregation_value) if isinstance(aggregation_value, str) else None + ) + expression = str(expression_value) if isinstance(expression_value, str) else None + time_dimension = ( + str(time_dimension_value) + if isinstance(time_dimension_value, str) + else default_time_dimension + ) + metrics.append( + DbtSemanticMetricSummary( + unique_id=unique_id, + name=str(metric.get("name") or unique_id), + label=str(metric.get("label") or metric.get("name") or unique_id), + metric_type=cast(Any, metric_type), + description=( + str(metric.get("description")) if metric.get("description") else None + ), + aggregation=aggregation.upper() if aggregation else None, + expression=expression, + time_dimension=time_dimension, + semantic_model=semantic_name, + ) + ) + + semantic_models: list[DbtSemanticModelSummary] = [] + for unique_id, semantic_model in sorted( + project_semantic_models.items(), key=lambda item: str(item[1].get("name") or "") + ): + entities = _semantic_entities(semantic_model) + dimensions = _semantic_dimensions(semantic_model) + name = str(semantic_model.get("name") or unique_id) + semantic_models.append( + DbtSemanticModelSummary( + unique_id=unique_id, + name=name, + description=( + str(semantic_model.get("description")) + if semantic_model.get("description") + else None + ), + relation=_semantic_relation(semantic_model), + default_time_dimension=_semantic_default_time_dimension(semantic_model), + entities=entities, + dimensions=dimensions, + metric_names=sorted(metric_names_by_semantic_model.get(name, [])), + ) + ) + + entity_count = sum(len(model.entities) for model in semantic_models) + dimension_count = sum(len(model.dimensions) for model in semantic_models) + artifact_status: Literal["READY", "PENDING", "MISSING"] = ( + "READY" if semantic_models and metrics else "PENDING" + ) + + return DbtSemanticLayerSummary( + artifact_status=artifact_status, + generated_at=( + str(metadata.get("generated_at")) if metadata.get("generated_at") else None + ), + dbt_version=( + str(metadata.get("dbt_version")) if metadata.get("dbt_version") else None + ), + semantic_model_count=len(semantic_models), + metric_count=len(metrics), + entity_count=entity_count, + dimension_count=dimension_count, + semantic_models=semantic_models, + metrics=metrics, + ) diff --git a/apps/web/src/App.jsx b/apps/web/src/App.jsx index c955c19..8f47062 100644 --- a/apps/web/src/App.jsx +++ b/apps/web/src/App.jsx @@ -12,6 +12,7 @@ import Airflow from './pages/Airflow.jsx'; import SchedulesBackfills from './pages/SchedulesBackfills.jsx'; import Transformations from './pages/Transformations.jsx'; import DataModels from './pages/DataModels.jsx'; +import SemanticLayer from './pages/SemanticLayer.jsx'; import PlaceholderPage from './pages/PlaceholderPage.jsx'; const PAGE_META = { @@ -67,6 +68,7 @@ function App() { } /> } /> } /> + } /> } /> } /> } /> @@ -82,6 +84,7 @@ function App() { '/orchestration/airflow', '/orchestration/runs', '/orchestration/backfills', + '/delivery/semantic', ].includes(path)) .map(([path, [title, subtitle]]) => ( Development - Phase 6.2 dbt model catalogue + Phase 6.3 semantic model + metrics diff --git a/apps/web/src/pages/SemanticLayer.jsx b/apps/web/src/pages/SemanticLayer.jsx new file mode 100644 index 0000000..a0f3270 --- /dev/null +++ b/apps/web/src/pages/SemanticLayer.jsx @@ -0,0 +1,240 @@ +import { useCallback, useEffect, useMemo, useState } from 'react'; +import StatusPill from '../components/StatusPill.jsx'; +import { getJson } from '../services/api.js'; + +const EMPTY_SEMANTIC_LAYER = { + artifact_status: 'MISSING', + generated_at: null, + dbt_version: null, + semantic_model_count: 0, + metric_count: 0, + entity_count: 0, + dimension_count: 0, + semantic_models: [], + metrics: [], +}; + +function generatedLabel(value) { + if (!value) return 'Awaiting semantic artifacts'; + const date = new Date(value); + if (Number.isNaN(date.getTime())) return value; + return date.toLocaleString(); +} + +function titleCase(value) { + if (!value) return '—'; + return value.toLowerCase().replaceAll('_', ' ').replace(/\b\w/g, (letter) => letter.toUpperCase()); +} + +function SemanticLayer() { + const [summary, setSummary] = useState(EMPTY_SEMANTIC_LAYER); + const [loading, setLoading] = useState(true); + const [error, setError] = useState(''); + + const loadSemanticLayer = useCallback(async () => { + setLoading(true); + setError(''); + try { + setSummary(await getJson('/api/v1/transformations/dbt/semantic')); + } catch (requestError) { + setError(requestError.message); + } finally { + setLoading(false); + } + }, []); + + useEffect(() => { + const controller = new AbortController(); + getJson('/api/v1/transformations/dbt/semantic', { signal: controller.signal }) + .then(setSummary) + .catch((requestError) => { + if (requestError.name !== 'AbortError') setError(requestError.message); + }) + .finally(() => { + if (!controller.signal.aborted) setLoading(false); + }); + return () => controller.abort(); + }, []); + + const semanticReady = summary.artifact_status === 'READY'; + const metricNames = useMemo( + () => new Set(summary.metrics.map((metric) => metric.name)), + [summary.metrics], + ); + + return ( +
+
+
+ PHASE 6.3 · DBT SEMANTIC MODEL + GOVERNED METRICS +

Semantic Layer

+

+ Keep business definitions beside the dbt mart that owns them, then project semantic + entities, dimensions, and governed metrics from dbt artifacts without creating a + second metadata authority inside SkyData Studio. +

+
+ +
+ + {error ?
!{error}
: null} + +
+
+ Artifacts + {summary.artifact_status} + {generatedLabel(summary.generated_at)} +
+
+ Semantic Models + {summary.semantic_model_count} + {semanticReady ? 'Governed model graph discovered' : 'Run dbt build to refresh semantics'} +
+
+ Governed Metrics + {summary.metric_count} + Simple metrics owned by dbt +
+
+ Dimensions + {summary.dimension_count} + {summary.entity_count} semantic entity +
+
+ dbt Core + {summary.dbt_version || '—'} + Artifact-producing runtime +
+
+ +
+
+
+ SEMANTIC MODEL GRAPH +

Governed mart → entity + dimensions → reusable metrics

+
+ +
+ + {summary.semantic_models.length ? ( +
+ {summary.semantic_models.map((model) => ( +
+
+
+ SEMANTIC MODEL + {model.name} + {model.relation || 'relation pending'} +
+ +
+

{model.description || 'Semantic annotations owned by the dbt mart model.'}

+
+
+ PRIMARY ENTITY + {model.entities.find((entity) => entity.entity_type === 'PRIMARY')?.name || '—'} + {model.entities.find((entity) => entity.entity_type === 'PRIMARY')?.expression || 'model column'} +
+
+ DEFAULT TIME + {model.default_time_dimension || '—'} + Metric aggregation seam +
+
+ METRICS + {model.metric_names.filter((name) => metricNames.has(name)).length} + Governed definitions +
+
+
+ Dimensions +
+ {model.dimensions.map((dimension) => ( + + {dimension.name} + {titleCase(dimension.dimension_type)}{dimension.granularity ? ` · ${dimension.granularity.toLowerCase()}` : ''} + + ))} +
+
+
+ ))} +
+ ) : ( +
+ No semantic model artifacts are available yet. + Run .\scripts\dbt.ps1 build, then refresh this page. +
+ )} +
+ +
+
+
+ METRIC CATALOGUE +

Business measures defined once, beside the model

+
+ {summary.metrics.length} metrics +
+ {summary.metrics.length ? ( +
+ {summary.metrics.map((metric) => ( +
+
+
+ {titleCase(metric.metric_type)} +

{metric.label}

+ {metric.name} +
+ +
+

{metric.description || 'Governed dbt metric.'}

+
+
Aggregation
{metric.aggregation || '—'}
+
Expression
{metric.expression || '—'}
+
Time
{metric.time_dimension || '—'}
+
Semantic model
{metric.semantic_model || '—'}
+
+
+ ))} +
+ ) : ( +
+ No governed metrics discovered. + The dbt manifest will become the runtime evidence after the next semantic build. +
+ )} +
+ +
+
+
+
ACCEPTANCE CONTRACT

What Phase 6.3 proves

+ 6.3 +
+
    +
  1. 01
    One primary entityDaily observations have a stable semantic key.
    2. +
  2. 02
    Reusable dimensionsTime and direction cuts are governed beside the mart.
    3. +
  3. 03
    Four metricsAverage, minimum, maximum, and observation count share one definition seam.
    4. +
+
+
+
+
BOUNDARY

Definitions, not a hosted metric service

+ ∑ +
+

+ Phase 6.3 proves portable dbt semantic definitions and artifact evidence. SkyData Studio + does not pretend to be a hosted dbt Semantic Layer query service; downstream execution + stays a later integration boundary. +

+
One authorityPortable metricsArtifact proof
+
+
+
+ ); +} + +export default SemanticLayer; diff --git a/apps/web/src/pages/StudioOverview.jsx b/apps/web/src/pages/StudioOverview.jsx index 46fd161..60c6b0f 100644 --- a/apps/web/src/pages/StudioOverview.jsx +++ b/apps/web/src/pages/StudioOverview.jsx @@ -5,7 +5,7 @@ const FALLBACK = { product: 'SkyData Studio', subtitle: 'Data Engineering Workbench', theme: 'Aurora Foundry', - current_phase: 'Phase 6.2 — dbt Model Catalogue and Artifact Evidence', + current_phase: 'Phase 6.3 — dbt Semantic Model and Governed Metric Foundation', boundary: 'SkyData Studio starts after SkyCommand ingestion and publishes governed analytical products for downstream consumers.', capabilities: [], }; @@ -41,7 +41,7 @@ function StudioOverview() {
- AURORA FOUNDRY · PHASE 6.2 + AURORA FOUNDRY · PHASE 6.3

Shape trusted data into analytical products.

{summary.boundary}

@@ -61,7 +61,7 @@ function StudioOverview() {
Platform API{apiState}FastAPI contract service
-
Current Phase6.2dbt artifact-backed model catalogue
+
Current Phase6.3governed semantic model + metrics
Pipeline ProofREADYCurated mart materialization proven
OrchestratorAirflow 3Batch and asset workflows
@@ -110,11 +110,11 @@ function StudioOverview() {
-
CURRENT IMPLEMENTATION

dbt model catalogue and artifact evidence

PHASE 6.2
+
CURRENT IMPLEMENTATION

dbt semantic model and governed metric foundation

PHASE 6.3
    -
  1. 01
    Artifact-backed catalogueRead dbt manifest and run-result artifacts instead of duplicating model metadata in Studio storage.
    2. -
  2. 02
    Model dependency evidenceProject direct source/model dependencies, layers, materializations, tags, columns, and tests.
    3. -
  3. 03
    Live Data Models workbenchInspect the latest dbt build graph and model contracts from one portfolio-ready surface.
    4. +
  4. 01
    Semantic model contractAnnotate the governed mart with one primary entity and reusable time/categorical dimensions.
    5. +
  5. 02
    Governed metricsDefine average, minimum, maximum, and observation-count metrics beside the dbt model that owns them.
    6. +
  6. 03
    Artifact-backed semantic workbenchProject parsed semantic-model and metric evidence without copying dbt-owned definitions into Studio storage.
diff --git a/apps/web/src/styles.css b/apps/web/src/styles.css index 81ccaca..6bc84d5 100644 --- a/apps/web/src/styles.css +++ b/apps/web/src/styles.css @@ -722,6 +722,7 @@ a { color: inherit; } .transformation-metric-grid { grid-template-columns: 1fr; } } .status-missing { color: var(--amber); background: rgba(243,189,85,0.08); } +.status-pending { color: var(--amber); background: rgba(243,189,85,0.08); } /* Phase 6.2 — dbt model catalogue and artifact evidence */ @@ -777,3 +778,201 @@ a { color: inherit; } .data-model-metric-grid, .data-model-column-list article { grid-template-columns: 1fr; } } + + +/* Phase 6.3 semantic layer */ +.semantic-intro { + align-items: end; +} + +.semantic-metric-grid { + grid-template-columns: repeat(5, minmax(0, 1fr)); +} + +.semantic-model-grid { + display: grid; + grid-template-columns: 1fr; + gap: 14px; +} + +.semantic-model-card, +.semantic-metric-card { + border: 1px solid rgba(80, 231, 207, 0.18); + border-radius: 14px; + background: linear-gradient(145deg, rgba(7, 27, 37, 0.92), rgba(6, 18, 29, 0.96)); +} + +.semantic-model-card { + padding: 18px; +} + +.semantic-model-heading, +.semantic-metric-card header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 16px; +} + +.semantic-model-heading > div, +.semantic-metric-card header > div { + display: grid; + gap: 5px; +} + +.semantic-model-heading span, +.semantic-definition-block > span, +.semantic-contract-grid span, +.semantic-metric-card header span, +.semantic-metric-card dt { + color: var(--muted); + font-size: 0.66rem; + font-weight: 800; + letter-spacing: 0.1em; + text-transform: uppercase; +} + +.semantic-model-heading strong { + font-size: 1.08rem; +} + +.semantic-model-heading code, +.semantic-metric-card code { + color: var(--cyan); + font-size: 0.75rem; +} + +.semantic-model-card > p, +.semantic-metric-card > p { + margin: 14px 0; + color: var(--soft); + line-height: 1.55; +} + +.semantic-contract-grid { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: 10px; + margin: 16px 0; +} + +.semantic-contract-grid > div { + display: grid; + gap: 5px; + min-width: 0; + padding: 12px; + border: 1px solid rgba(80, 231, 207, 0.12); + border-radius: 10px; + background: rgba(3, 14, 22, 0.58); +} + +.semantic-contract-grid strong { + overflow: hidden; + color: var(--text); + text-overflow: ellipsis; + white-space: nowrap; +} + +.semantic-contract-grid small { + color: var(--muted); +} + +.semantic-definition-block { + display: grid; + gap: 9px; +} + +.semantic-chip-list { + display: flex; + flex-wrap: wrap; + gap: 8px; +} + +.semantic-chip-list > span { + display: grid; + gap: 2px; + min-width: 150px; + padding: 9px 11px; + border: 1px solid rgba(139, 114, 255, 0.2); + border-radius: 9px; + background: rgba(139, 114, 255, 0.06); +} + +.semantic-chip-list strong { + font-size: 0.76rem; +} + +.semantic-chip-list small { + color: var(--muted); + font-size: 0.67rem; +} + +.semantic-metric-list { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 12px; +} + +.semantic-metric-card { + padding: 16px; +} + +.semantic-metric-card h3 { + margin: 2px 0 0; + font-size: 1rem; +} + +.semantic-metric-card dl { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 8px; + margin: 0; +} + +.semantic-metric-card dl > div { + display: grid; + gap: 4px; + padding: 9px 10px; + border-radius: 8px; + background: rgba(2, 13, 21, 0.6); +} + +.semantic-metric-card dd { + margin: 0; + overflow: hidden; + color: var(--text); + font-size: 0.76rem; + text-overflow: ellipsis; + white-space: nowrap; +} + +.semantic-empty { + display: grid; + gap: 6px; + min-height: 110px; + place-content: center; + text-align: center; + color: var(--muted); +} + +.semantic-empty strong { + color: var(--text); +} + +.semantic-empty code { + color: var(--cyan); +} + +@media (max-width: 1180px) { + .semantic-metric-grid { + grid-template-columns: repeat(3, minmax(0, 1fr)); + } +} + +@media (max-width: 820px) { + .semantic-metric-grid, + .semantic-contract-grid, + .semantic-metric-list { + grid-template-columns: 1fr; + } +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index bc7a99c..9796211 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -58,6 +58,10 @@ metadata_domain ─ metadata_system ─ metadata_connection SkyCommand assets enter the registry as `RAW` assets. Studio-created products advance through `STAGING`, `INTERMEDIATE`, `MART`, `SEMANTIC`, and `REPORT` layers. The registry becomes the stable metadata boundary used by the pipeline workbench, dbt artifact ingestion, lineage, and analytics delivery. +### dbt semantic metadata boundary + +Phase 6.3 keeps semantic authorship beside the dbt model that owns the analytical grain and supplies MetricFlow with a dedicated DAY-grain calendar time spine. The first contract flows `dbt_mart.fct_fed_funds_rate_daily → fed_funds_rate_daily semantic model → governed metrics → dbt artifacts → Studio Semantic Layer workbench`. SkyData Studio reads generated evidence; it does not copy dbt semantic definitions into Studio PostgreSQL. This phase establishes portable definitions and observability only, leaving hosted semantic-query execution and downstream BI integrations as later delivery boundaries. + ## Product blueprint and mapping boundary Phase 3.2 adds a design-time contract between registered source assets and intended targets. A mapping is metadata, not execution: it records how a pipeline *should* move and shape data before Phase 4 introduces a runnable pipeline engine. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7703ac5..c989a2d 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -407,7 +407,7 @@ Closure evidence: ## Phase 6.2 — dbt Model Catalogue and Artifact Evidence -**Status:** In progress. +**Status:** Complete. Scope: @@ -426,3 +426,36 @@ Acceptance proof: - the Data Models workbench displays the staging → intermediate → mart dependency chain and exposes model detail for tags, columns, tests, and direct lineage; - removing/cleaning the dbt target produces an explicit MISSING artifact state rather than stale or fabricated metadata; - local validation and GitHub checks remain green. + +Closure evidence: + +- `GET /api/v1/transformations/dbt/models` reports `artifact_status=READY`, dbt Core 1.12.0, 3 discovered models, 3 ready models, 1 source, and 14 data tests; +- the Data Models workbench displays the complete staging → intermediate → mart graph as READY and exposes the same three models in the runtime inventory; +- model-level test evidence resolves to 3 staging tests, 3 intermediate tests, and 5 mart tests, with the remaining 3 tests attached directly to the curated source; +- direct lineage resolves `fed_funds_rate → stg_fed_funds_rate → int_fed_funds_rate_changes → fct_fed_funds_rate_daily`; +- local validation completes with 52 pytest tests, clean Ruff/mypy, zero npm vulnerabilities, clean ESLint, and a successful Vite production build. + +## Phase 6.3 — dbt Semantic Model and Governed Metric Foundation + +**Status:** In progress. + +Scope: + +- adopt dbt Core 1.12's model-embedded semantic YAML specification on the governed Federal Funds Rate mart; +- add a dedicated daily MetricFlow time spine so time-based semantic parsing and aggregation have a continuous DAY-grain calendar; +- define `fed_funds_rate_daily` as the first Studio semantic model without creating a second semantic metadata store; +- add a stable `observation_key` primary entity while preserving the proven one-row-per-day mart grain; +- expose `observation_date`, `observation_month`, `observation_year`, and `rate_direction` as governed semantic dimensions; +- define four simple governed metrics: average rate, minimum rate, maximum rate, and distinct observation count; +- project parsed semantic-model and metric evidence from dbt artifacts through `GET /api/v1/transformations/dbt/semantic`; +- replace the Semantic Layer placeholder with an artifact-backed workbench for semantic models, entities, dimensions, and metrics; +- keep Phase 6.3 limited to portable definitions and artifact evidence rather than claiming a hosted dbt Semantic Layer query service. + +Acceptance proof: + +- `./scripts/dbt.ps1 build` completes 4 physical dbt models (3 business models plus 1 semantic utility time spine) and 14 data tests with `PASS=18`, `WARN=0`, `ERROR=0`, and `SKIP=0`; +- the generated dbt manifest contains exactly 1 Studio semantic model and 4 Studio metrics; +- `GET /api/v1/transformations/dbt/semantic` reports `artifact_status=READY`, 1 semantic model, 4 metrics, 1 entity, and the expected dimensions; +- the Semantic Layer workbench displays the governed model, primary entity, dimensions, and all four metrics from the generated artifact evidence; +- `dbt_mart.fct_fed_funds_rate_daily` retains 26,335 rows, `dbt_mart.time_spine_daily` exists at DAY granularity, and the business-model Data Models workbench remains 3/3 ready because semantic utility models are excluded from that catalogue; +- local validation and GitHub checks remain green. diff --git a/docs/SkyData_RepoMap.md b/docs/SkyData_RepoMap.md index f5167d1..7eeef1a 100644 --- a/docs/SkyData_RepoMap.md +++ b/docs/SkyData_RepoMap.md @@ -89,6 +89,7 @@ SkyDataStudio/ │ │ ├── Pipelines.jsx │ │ ├── PlaceholderPage.jsx │ │ ├── SchedulesBackfills.jsx +│ │ ├── SemanticLayer.jsx │ │ ├── SourceMappings.jsx │ │ ├── StudioOverview.jsx │ │ └── Transformations.jsx @@ -165,9 +166,9 @@ SkyDataStudio/ │ │ ├── int_fed_funds_rate_changes.sql │ │ └── schema.yml │ ├── marts/ - │ │ ├── .gitkeep │ │ ├── fct_fed_funds_rate_daily.sql - │ │ └── schema.yml + │ │ ├── schema.yml + │ │ └── time_spine_daily.sql │ └── staging/ │ ├── schema.yml │ ├── sources.yml @@ -196,6 +197,7 @@ SkyDataStudio/ │ │ │ │ └── unique_int_fed_funds_rate_changes_observation_date.sql │ │ │ ├── marts/ │ │ │ │ ├── fct_fed_funds_rate_daily.sql + │ │ │ │ ├── time_spine_daily.sql │ │ │ │ └── schema.yml/ │ │ │ │ ├── accepted_values_fct_fed_funds__ce738edf7a8c271de47c966afcab8ce8.sql │ │ │ │ ├── not_null_fct_fed_funds_rate_daily_observation_date.sql @@ -224,6 +226,7 @@ SkyDataStudio/ │ │ │ └── unique_int_fed_funds_rate_changes_observation_date.sql │ │ ├── marts/ │ │ │ ├── fct_fed_funds_rate_daily.sql + │ │ │ ├── time_spine_daily.sql │ │ │ └── schema.yml/ │ │ │ ├── accepted_values_fct_fed_funds__ce738edf7a8c271de47c966afcab8ce8.sql │ │ │ ├── not_null_fct_fed_funds_rate_daily_observation_date.sql diff --git a/docs/repository-map.txt b/docs/repository-map.txt index 554e237..a82bba2 100644 --- a/docs/repository-map.txt +++ b/docs/repository-map.txt @@ -70,11 +70,13 @@ SkyDataStudio │ │ ├── pages │ │ │ ├── Airflow.jsx │ │ │ ├── DataAssets.jsx +│ │ │ ├── DataModels.jsx │ │ │ ├── MetadataRegistry.jsx │ │ │ ├── PipelineRuns.jsx │ │ │ ├── Pipelines.jsx │ │ │ ├── PlaceholderPage.jsx │ │ │ ├── SchedulesBackfills.jsx +│ │ │ ├── SemanticLayer.jsx │ │ │ ├── SourceMappings.jsx │ │ │ ├── StudioOverview.jsx │ │ │ └── Transformations.jsx @@ -157,7 +159,8 @@ SkyDataStudio │ │ ├── marts │ │ │ ├── .gitkeep │ │ │ ├── fct_fed_funds_rate_daily.sql -│ │ │ └── schema.yml +│ │ │ ├── schema.yml +│ │ │ └── time_spine_daily.sql │ │ └── staging │ │ ├── schema.yml │ │ ├── sources.yml @@ -170,6 +173,7 @@ SkyDataStudio │ ├── tests │ │ ├── .gitkeep │ │ └── assert_fed_funds_rate_reasonable.sql +│ ├── .user.yml │ ├── dbt_project.yml │ ├── profiles.yml │ └── profiles.yml.example diff --git a/tests/test_dbt_transformations.py b/tests/test_dbt_transformations.py index b78cd05..89e85c3 100644 --- a/tests/test_dbt_transformations.py +++ b/tests/test_dbt_transformations.py @@ -164,9 +164,313 @@ def test_dbt_model_catalogue_projects_manifest_and_run_results(tmp_path: Path) - assert summary.models[1].test_count == 1 +def test_dbt_model_catalogue_excludes_semantic_utility_models(tmp_path: Path) -> None: + target_dir = tmp_path / "target" + target_dir.mkdir() + manifest = { + "metadata": {"generated_at": "2026-08-11T18:20:00Z", "dbt_version": "1.12.0"}, + "sources": {}, + "nodes": { + "model.skydata_studio.fct_fed_funds_rate_daily": { + "resource_type": "model", + "package_name": "skydata_studio", + "name": "fct_fed_funds_rate_daily", + "schema": "dbt_mart", + "alias": "fct_fed_funds_rate_daily", + "description": "Business mart", + "original_file_path": "models/marts/fct_fed_funds_rate_daily.sql", + "tags": ["phase6", "mart"], + "columns": {}, + "config": {"enabled": True, "materialized": "table"}, + "depends_on": {"nodes": []}, + }, + "model.skydata_studio.time_spine_daily": { + "resource_type": "model", + "package_name": "skydata_studio", + "name": "time_spine_daily", + "schema": "dbt_mart", + "alias": "time_spine_daily", + "description": "MetricFlow utility", + "original_file_path": "models/marts/time_spine_daily.sql", + "tags": ["phase6", "mart", "semantic_utility"], + "columns": {"date_day": {"description": "Calendar day"}}, + "config": {"enabled": True, "materialized": "table"}, + "depends_on": {"nodes": []}, + }, + }, + } + run_results = { + "results": [ + { + "unique_id": "model.skydata_studio.fct_fed_funds_rate_daily", + "status": "success", + }, + { + "unique_id": "model.skydata_studio.time_spine_daily", + "status": "success", + }, + ] + } + (target_dir / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + (target_dir / "run_results.json").write_text( + json.dumps(run_results), encoding="utf-8" + ) + + summary = dbt_model_catalogue(target_dir) + + assert summary.model_count == 1 + assert summary.ready_model_count == 1 + assert [model.name for model in summary.models] == ["fct_fed_funds_rate_daily"] + + def test_dbt_model_catalogue_returns_missing_when_manifest_is_absent(tmp_path: Path) -> None: summary = dbt_model_catalogue(tmp_path / "target") assert summary.artifact_status == "MISSING" assert summary.model_count == 0 assert summary.models == [] + + +def test_dbt_semantic_layer_projects_manifest_semantic_evidence(tmp_path: Path) -> None: + from skydata_studio.services.dbt_transformations import dbt_semantic_layer + + target_dir = tmp_path / "target" + target_dir.mkdir() + manifest = { + "metadata": {"generated_at": "2026-08-11T16:20:00Z", "dbt_version": "1.12.0"}, + "semantic_models": { + "semantic_model.skydata_studio.fed_funds_rate_daily": { + "package_name": "skydata_studio", + "name": "fed_funds_rate_daily", + "description": "Federal Funds Rate semantic model", + "original_file_path": "models/marts/schema.yml", + "node_relation": { + "schema_name": "dbt_mart", + "alias": "fct_fed_funds_rate_daily", + }, + "defaults": {"agg_time_dimension": "observation_date"}, + "entities": [ + { + "name": "fed_funds_observation", + "type": "primary", + "expr": "observation_key", + } + ], + "dimensions": [ + { + "name": "observation_date", + "type": "time", + "type_params": {"time_granularity": "day"}, + }, + { + "name": "rate_direction", + "type": "categorical", + "type_params": None, + }, + ], + "measures": [ + { + "name": "average_federal_funds_rate", + "agg": "average", + "expr": "rate", + "agg_time_dimension": "observation_date", + }, + { + "name": "federal_funds_observation_count", + "agg": "count_distinct", + "expr": "observation_key", + "agg_time_dimension": "observation_date", + }, + ], + } + }, + "metrics": { + "metric.skydata_studio.average_federal_funds_rate": { + "package_name": "skydata_studio", + "name": "average_federal_funds_rate", + "label": "Average Federal Funds Rate", + "description": "Average rate", + "type": "simple", + "type_params": {"measure": {"name": "average_federal_funds_rate"}}, + }, + "metric.skydata_studio.federal_funds_observation_count": { + "package_name": "skydata_studio", + "name": "federal_funds_observation_count", + "label": "Federal Funds Observation Count", + "description": "Observation count", + "type": "simple", + "type_params": { + "measure": {"name": "federal_funds_observation_count"} + }, + }, + }, + } + (target_dir / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + + summary = dbt_semantic_layer(target_dir) + + assert summary.artifact_status == "READY" + assert summary.semantic_model_count == 1 + assert summary.metric_count == 2 + assert summary.entity_count == 1 + assert summary.dimension_count == 2 + assert summary.semantic_models[0].relation == "dbt_mart.fct_fed_funds_rate_daily" + assert summary.semantic_models[0].default_time_dimension == "observation_date" + assert summary.semantic_models[0].entities[0].entity_type == "PRIMARY" + assert summary.metrics[0].aggregation == "AVERAGE" + assert summary.metrics[0].expression == "rate" + assert summary.metrics[0].semantic_model == "fed_funds_rate_daily" + + +def test_dbt_semantic_layer_returns_pending_before_semantic_artifacts(tmp_path: Path) -> None: + from skydata_studio.services.dbt_transformations import dbt_semantic_layer + + target_dir = tmp_path / "target" + target_dir.mkdir() + (target_dir / "manifest.json").write_text( + json.dumps( + { + "metadata": { + "generated_at": "2026-08-11T16:20:00Z", + "dbt_version": "1.12.0", + }, + "semantic_models": {}, + "metrics": {}, + } + ), + encoding="utf-8", + ) + + summary = dbt_semantic_layer(target_dir) + + assert summary.artifact_status == "PENDING" + assert summary.semantic_model_count == 0 + assert summary.metric_count == 0 + + +def test_dbt_semantic_endpoint_projects_semantic_catalogue( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from skydata_studio.schemas.dbt import ( + DbtSemanticDimensionSummary, + DbtSemanticEntitySummary, + DbtSemanticLayerSummary, + DbtSemanticMetricSummary, + DbtSemanticModelSummary, + ) + + expected = DbtSemanticLayerSummary( + artifact_status="READY", + generated_at="2026-08-11T16:20:00Z", + dbt_version="1.12.0", + semantic_model_count=1, + metric_count=1, + entity_count=1, + dimension_count=1, + semantic_models=[ + DbtSemanticModelSummary( + unique_id="semantic_model.skydata_studio.fed_funds_rate_daily", + name="fed_funds_rate_daily", + relation="dbt_mart.fct_fed_funds_rate_daily", + default_time_dimension="observation_date", + entities=[ + DbtSemanticEntitySummary( + name="fed_funds_observation", + entity_type="PRIMARY", + expression="observation_key", + ) + ], + dimensions=[ + DbtSemanticDimensionSummary( + name="observation_date", + dimension_type="TIME", + granularity="DAY", + ) + ], + metric_names=["average_federal_funds_rate"], + ) + ], + metrics=[ + DbtSemanticMetricSummary( + unique_id="metric.skydata_studio.average_federal_funds_rate", + name="average_federal_funds_rate", + label="Average Federal Funds Rate", + metric_type="SIMPLE", + aggregation="AVERAGE", + expression="rate", + time_dimension="observation_date", + semantic_model="fed_funds_rate_daily", + ) + ], + ) + + monkeypatch.setattr(dbt_route, "dbt_semantic_layer", lambda: expected) + response = client.get("/api/v1/transformations/dbt/semantic") + + assert response.status_code == 200 + payload = response.json() + assert payload["artifact_status"] == "READY" + assert payload["semantic_model_count"] == 1 + assert payload["metric_count"] == 1 + assert payload["semantic_models"][0]["entities"][0]["entity_type"] == "PRIMARY" + + +def test_dbt_semantic_layer_supports_v112_inline_metric_artifact_shape( + tmp_path: Path, +) -> None: + from skydata_studio.services.dbt_transformations import dbt_semantic_layer + + target_dir = tmp_path / "target" + target_dir.mkdir() + semantic_unique_id = "semantic_model.skydata_studio.fed_funds_rate_daily" + manifest = { + "metadata": {"generated_at": "2026-08-11T16:30:00Z", "dbt_version": "1.12.0"}, + "semantic_models": { + semantic_unique_id: { + "package_name": "skydata_studio", + "name": "fed_funds_rate_daily", + "original_file_path": "models/marts/schema.yml", + "node_relation": { + "schema_name": "dbt_mart", + "alias": "fct_fed_funds_rate_daily", + }, + "agg_time_dimension": "observation_date", + "entities": [ + { + "name": "fed_funds_observation", + "type": "primary", + "expr": "observation_key", + } + ], + "dimensions": [ + { + "name": "observation_date", + "type": "time", + "type_params": {"time_granularity": "day"}, + } + ], + } + }, + "metrics": { + "metric.skydata_studio.average_federal_funds_rate": { + "package_name": "skydata_studio", + "name": "average_federal_funds_rate", + "label": "Average Federal Funds Rate", + "type": "simple", + "agg": "average", + "expr": "rate", + "agg_time_dimension": "observation_date", + "depends_on": {"nodes": [semantic_unique_id]}, + } + }, + } + (target_dir / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + + summary = dbt_semantic_layer(target_dir) + + assert summary.artifact_status == "READY" + assert summary.semantic_models[0].default_time_dimension == "observation_date" + assert summary.semantic_models[0].metric_names == ["average_federal_funds_rate"] + assert summary.metrics[0].aggregation == "AVERAGE" + assert summary.metrics[0].expression == "rate" + assert summary.metrics[0].semantic_model == "fed_funds_rate_daily" diff --git a/tests/test_platform.py b/tests/test_platform.py index 83ca6ea..68eab5a 100644 --- a/tests/test_platform.py +++ b/tests/test_platform.py @@ -12,7 +12,9 @@ def test_platform_summary_exposes_product_boundary() -> None: assert payload["product"] == "SkyData Studio" assert payload["theme"] == "Aurora Foundry" assert "SkyCommand" in payload["boundary"] - assert payload["current_phase"] == "Phase 6.2 — dbt Model Catalogue and Artifact Evidence" + assert payload["current_phase"] == ( + "Phase 6.3 — dbt Semantic Model and Governed Metric Foundation" + ) assert {item["code"] for item in payload["capabilities"]} >= { "AIRFLOW", "DBT", diff --git a/transformations/dbt/skydata_studio/models/marts/.gitkeep b/transformations/dbt/skydata_studio/models/marts/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/transformations/dbt/skydata_studio/models/marts/fct_fed_funds_rate_daily.sql b/transformations/dbt/skydata_studio/models/marts/fct_fed_funds_rate_daily.sql index f1fe7d0..bf6afdc 100644 --- a/transformations/dbt/skydata_studio/models/marts/fct_fed_funds_rate_daily.sql +++ b/transformations/dbt/skydata_studio/models/marts/fct_fed_funds_rate_daily.sql @@ -1,4 +1,5 @@ select + observation_date::text as observation_key, observation_date, observation_month, observation_year, diff --git a/transformations/dbt/skydata_studio/models/marts/schema.yml b/transformations/dbt/skydata_studio/models/marts/schema.yml index 375ac1a..e5da61f 100644 --- a/transformations/dbt/skydata_studio/models/marts/schema.yml +++ b/transformations/dbt/skydata_studio/models/marts/schema.yml @@ -1,19 +1,78 @@ version: 2 models: + - name: time_spine_daily + description: Daily MetricFlow time spine used for governed time joins and aggregations. + time_spine: + standard_granularity_column: date_day + columns: + - name: date_day + description: Continuous daily calendar grain for MetricFlow semantic queries. + granularity: day + - name: fct_fed_funds_rate_daily description: Consumer-ready daily Federal Funds Rate fact model produced by dbt. + semantic_model: + enabled: true + name: fed_funds_rate_daily + agg_time_dimension: observation_date columns: + - name: observation_key + description: Stable one-row-per-day semantic entity key derived from the observation date. + entity: + name: fed_funds_observation + type: primary - name: observation_date description: "Grain and business key: one row per observation date." + granularity: day + dimension: + type: time data_tests: - not_null - unique + - name: observation_month + description: Calendar month containing the daily observation. + granularity: month + dimension: + type: time + - name: observation_year + description: Calendar year containing the daily observation. + dimension: + type: categorical - name: rate + description: Federal Funds Effective Rate for the observation date. data_tests: - not_null - name: rate_direction + description: Direction of movement compared with the prior available observation. + dimension: + type: categorical data_tests: - accepted_values: arguments: values: ['BASELINE', 'UP', 'DOWN', 'UNCHANGED'] + metrics: + - name: average_federal_funds_rate + label: Average Federal Funds Rate + description: Average effective Federal Funds Rate over the selected semantic time grain. + type: simple + agg: average + expr: rate + - name: minimum_federal_funds_rate + label: Minimum Federal Funds Rate + description: Minimum effective Federal Funds Rate over the selected semantic time grain. + type: simple + agg: min + expr: rate + - name: maximum_federal_funds_rate + label: Maximum Federal Funds Rate + description: Maximum effective Federal Funds Rate over the selected semantic time grain. + type: simple + agg: max + expr: rate + - name: federal_funds_observation_count + label: Federal Funds Observation Count + description: Distinct daily Federal Funds Rate observations in the selected semantic slice. + type: simple + agg: count_distinct + expr: observation_key diff --git a/transformations/dbt/skydata_studio/models/marts/time_spine_daily.sql b/transformations/dbt/skydata_studio/models/marts/time_spine_daily.sql new file mode 100644 index 0000000..1216658 --- /dev/null +++ b/transformations/dbt/skydata_studio/models/marts/time_spine_daily.sql @@ -0,0 +1,9 @@ +{{ config(tags=['semantic_utility']) }} + +select + generated_date::date as date_day +from generate_series( + date '1954-01-01', + current_date + interval '366 days', + interval '1 day' +) as generated_dates(generated_date)