From 184fb25b60f807a6facff19c47a663dacd1a74d5 Mon Sep 17 00:00:00 2001 From: Iliia_Veselov Date: Wed, 1 Jul 2026 11:40:28 +0300 Subject: [PATCH 1/2] feat: Add sdmx data models --- architecture/README.md | 1 + architecture/sdmx-compatibility.md | 13 ++++ architecture/sdmx-data-models.md | 111 +++++++++++++++++++++++++++++ 3 files changed, 125 insertions(+) create mode 100644 architecture/sdmx-data-models.md diff --git a/architecture/README.md b/architecture/README.md index 82321c7..db47a99 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -15,6 +15,7 @@ design, services, tools, and integration requirements. | **[🔎 Data Query & Hybrid Search](./data-query-hybrid-search.md)** | How NL queries become grounded SDMX data, and how hybrid indicator search works (conceptual) | • Composite indicators
• Keyword + semantic + LLM
• Two-phase indexing
• Availability grounding | | **[🔌 Application MCP](./mcp.md)** | Agentic access via Model Context Protocol | • Channel tools surfaced to AI agents
• DIAL Application registration
• MCP-spec discovery + IDP OAuth flow | | **[📊 SDMX Compatibility](./sdmx-compatibility.md)** | SDMX standards and requirements guide | • Version support (2.1/3.0)
• Metadata requirements
• Performance standards
• Quality checklist | +| **[📐 SDMX Data Models](./sdmx-data-models.md)** | SDMX data-model patterns and which StatGPT supports | • Conventional (supported)
• Shared DSD
• Indicator-per-dataflow
• Why indicator search fits one model | ### Design Documentation/res diff --git a/architecture/sdmx-compatibility.md b/architecture/sdmx-compatibility.md index c2cf409..5bdb737 100644 --- a/architecture/sdmx-compatibility.md +++ b/architecture/sdmx-compatibility.md @@ -34,6 +34,18 @@ effectively with statistical data sources. - ✅ Available Constraint endpoint - ✅ Data endpoint +## 📐 Supported Data Models + +StatGPT is an indicator-search-based system and supports only the **conventional +(multi-indicator dataflow)** data model. Providers whose indicator is encoded by *which +dataflow (or table)* you pick — the **shared DSD** (e.g. OECD) and **indicator-per-dataflow / +dedicated-DSD-per-dataset** (e.g. Eurostat, ISTAT, ABS, ILO) models — are **not supported yet**. + +The three data models, their example providers, and why StatGPT supports one of them are +described in a dedicated document: + +➡️ **[SDMX Data Models & StatGPT Support](./sdmx-data-models.md)** + ## 🌐 Connectivity Requirements ### API Availability @@ -179,6 +191,7 @@ Use this checklist to ensure your SDMX metadata meets StatGPT requirements: 5. **Missing Concept Definitions**: Reduces context understanding 6. **Slow API Response Times**: Degrades user experience 7. **Authentication Complexity**: Currently not supported by StatGPT +8. **Unsupported Data Model**: Only the conventional multi-indicator dataflow model is supported — see [Supported Data Models](#-supported-data-models) and [SDMX Data Models & StatGPT Support](./sdmx-data-models.md) ## 💡 Recommendations for Data Providers diff --git a/architecture/sdmx-data-models.md b/architecture/sdmx-data-models.md new file mode 100644 index 0000000..5b43b73 --- /dev/null +++ b/architecture/sdmx-data-models.md @@ -0,0 +1,111 @@ +# SDMX Data Models & StatGPT Support + +This document describes the SDMX data-model patterns StatGPT encounters across providers and +which of them StatGPT supports. It complements the [SDMX Compatibility](./sdmx-compatibility.md) +requirements. + +## Overview + +Across SDMX providers there are three data models. StatGPT supports only the **conventional +(multi-indicator dataflow)** model — where the indicator is a dimension inside one broad +dataflow, which is what StatGPT's indicator search resolves and filters within (see +[Data Query & Hybrid Search](./data-query-hybrid-search.md)). The **shared DSD** and +**indicator-per-dataflow** models — where the indicator is instead encoded by *which dataflow +(or table)* you pick — are **not supported yet**. + +| # | Data model | Indicator is selected by… | Example providers | Supported | +|---|------------|---------------------------|-------------------|:---------:| +| 1 | **Conventional (multi-indicator dataflow)** | a dimension *inside* one broad dataflow | IMF, BIS | ✅ Yes | +| 2 | **Shared DSD** | *which dataflow* (one DSD split across many per-country / per-table dataflows) | OECD | ❌ Not yet | +| 3 | **Indicator-per-dataflow (dedicated DSD per dataset)** | *which dataflow* (one narrow table = one DSD) | Eurostat, ISTAT, ABS, ILO | ❌ Not yet | + +--- + +## 1. Conventional (multi-indicator dataflow) — ✅ Supported + +A dataset is a **single broad dataflow that holds many indicators**. + +**Shape:** + +- A relatively small number of dataflows. +- Each dataflow exposes a broad indicator/measure dimension whose code list holds the + individual indicators. +- One DSD may back one or several such broad dataflows. + +**Examples:** IMF, BIS. + +**Why it works:** the indicator index is populated from the codes on the broad indicator +dimension, and query resolution is a filter *within* a dataflow — exactly the flow StatGPT +implements. + +--- + +## 2. Shared DSD — ❌ Not supported yet + +One rich DSD is **deliberately partitioned into many semantically-distinct dataflows** — +typically one dataflow per country or per sub-table. The DSD is shared; the dataflows are +narrow slices of it. + +**Shape:** + +- One DSD backs many dataflows (mean dataflows-per-DSD well above 1); a large share of + dataflows ride a shared DSD. +- Each dataflow is a narrow slice of the shared structure — typically one country or one + sub-table — not a broad indicator dataflow. + +**Example: OECD** + +- ~1,536 dataflows over ~413 distinct DSDs (mean ≈ 3.65 dataflows per DSD); ~82% of dataflows + ride a shared DSD. +- Naming encodes the DSD in every dataflow id: `DSD_@DF_` (~96% of dataflows). +- The sharing is almost always one structure split by country or sub-table: + +| DSD | # dataflows | Example dataflows | +|-----|-------------|-------------------| +| `OECD.TAD.ADM:DSD_FFS` (fossil-fuel support) | 53 | `DSD_FFS@DF_FFS_ARG`, `…_ARM`, `…_AUS` (per country) | +| `OECD.CTP.TPS:DSD_REV_OECD` (tax revenue) | 39 | `DSD_REV_OECD@DF_REVAUS`, `…_AUT`, … (per country) | +| `OECD.SDD.NAD:DSD_NAMAIN10` (national accounts) | 35 | `DSD_NAMAIN10@DF_TABLE1`, `…_EXPENDITURE`, … (per table) | + +--- + +## 3. Indicator-per-dataflow (dedicated DSD per dataset) — ❌ Not supported yet + +Each dataset is **one narrow table — effectively one indicator — published as its own +dataflow with its own dedicated DSD**, producing thousands of dataflows. The indicator is +determined by *which* dataflow is chosen, not by a dimension within one. + +**Shape:** + +- Thousands of dataflows, each with its own dedicated DSD — a near-1:1 dataflow-to-DSD ratio, + often with `dataflow-id == DSD-id`. +- Each dataflow is a narrow table; there is no broad indicator dataflow to select and filter. +- The dataset-specific axis is usually a domain indicator dimension that is not consistent + across datasets. + +**Example: Eurostat** + +- ~8,245 dataflows over ~7,698 distinct DSDs, in near-perfect 1:1 — ~100% of base datasets + have `dataflow-id == DSD-id` (e.g. dataflow `NAMA_10_GDP` → DSD `NAMA_10_GDP`). +- The one genuinely dataset-specific axis is a domain indicator dimension (`na_item`, + `indic_is`, `indic_bt`, …), which is not consistent across datasets. +- Plus ~547 `$DV_` "data-view" dataflows (pre-saved filtered views) that reference their + parent dataset's DSD (e.g. `ISOC_CI_AC_I$DV_1043`). + +**Other providers with this model:** + +| Provider | Shape | +|----------|-------| +| **Eurostat** | ~8,245 dataflows, one dedicated DSD per dataset (`dataflow-id == DSD-id`), + `$DV_` views | +| **ISTAT** | ~810 real datasets, one DSD each, but ~78% of its ~4,876 dataflows are `_DF_..._N` saved-view variants of a parent dataset | +| **ABS** | ~1,223 dataflows, dedicated DSD ~1:1 (`dataflow-id == DSD-id`) | +| **ILO** | ~1,208 dataflows ↔ 1,208 DSDs, strict 1:1 (`DF_` → DSD ``) | + +--- + +## Summary + +| Data model | Example providers | Supported | +|------------|-------------------|:---------:| +| Conventional (multi-indicator dataflow) | IMF, BIS | ✅ | +| Shared DSD | OECD | ❌ | +| Indicator-per-dataflow (dedicated DSD per dataset) | Eurostat, ISTAT, ABS, ILO | ❌ | From 51e3a716fa086ec3ee4adf6d4197f464dd4da76b Mon Sep 17 00:00:00 2001 From: Iliia_Veselov Date: Wed, 22 Jul 2026 19:53:03 +0300 Subject: [PATCH 2/2] fix: Rephrase support levels --- architecture/README.md | 2 +- architecture/sdmx-compatibility.md | 16 ++++++----- architecture/sdmx-data-models.md | 46 +++++++++++++++++++----------- 3 files changed, 40 insertions(+), 24 deletions(-) diff --git a/architecture/README.md b/architecture/README.md index db47a99..0cfe3a8 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -15,7 +15,7 @@ design, services, tools, and integration requirements. | **[🔎 Data Query & Hybrid Search](./data-query-hybrid-search.md)** | How NL queries become grounded SDMX data, and how hybrid indicator search works (conceptual) | • Composite indicators
• Keyword + semantic + LLM
• Two-phase indexing
• Availability grounding | | **[🔌 Application MCP](./mcp.md)** | Agentic access via Model Context Protocol | • Channel tools surfaced to AI agents
• DIAL Application registration
• MCP-spec discovery + IDP OAuth flow | | **[📊 SDMX Compatibility](./sdmx-compatibility.md)** | SDMX standards and requirements guide | • Version support (2.1/3.0)
• Metadata requirements
• Performance standards
• Quality checklist | -| **[📐 SDMX Data Models](./sdmx-data-models.md)** | SDMX data-model patterns and which StatGPT supports | • Conventional (supported)
• Shared DSD
• Indicator-per-dataflow
• Why indicator search fits one model | +| **[📐 SDMX Data Models](./sdmx-data-models.md)** | SDMX data-model patterns and which StatGPT supports | • Conventional (supported)
• Shared DSD (supported, not optimal)
• Indicator-per-dataflow (not yet)
• Why indicator search fits one model | ### Design Documentation/res diff --git a/architecture/sdmx-compatibility.md b/architecture/sdmx-compatibility.md index 5bdb737..70da7a7 100644 --- a/architecture/sdmx-compatibility.md +++ b/architecture/sdmx-compatibility.md @@ -36,12 +36,14 @@ effectively with statistical data sources. ## 📐 Supported Data Models -StatGPT is an indicator-search-based system and supports only the **conventional -(multi-indicator dataflow)** data model. Providers whose indicator is encoded by *which -dataflow (or table)* you pick — the **shared DSD** (e.g. OECD) and **indicator-per-dataflow / -dedicated-DSD-per-dataset** (e.g. Eurostat, ISTAT, ABS, ILO) models — are **not supported yet**. - -The three data models, their example providers, and why StatGPT supports one of them are +StatGPT is an indicator-search-based system and gives **first-class support** to the +**conventional (multi-indicator dataflow)** data model. The **shared DSD** model (e.g. OECD) is +**supported, but not optimal** — it can be onboarded, but only one dataflow at a time, which makes +onboarding slow and can affect quality. The **indicator-per-dataflow / dedicated-DSD-per-dataset** +model (e.g. Eurostat, ISTAT, ABS, ILO), where the indicator is encoded by *which dataflow (or +table)* you pick, is **not supported yet**. + +The three data models, their example providers, and StatGPT's support for each are described in a dedicated document: ➡️ **[SDMX Data Models & StatGPT Support](./sdmx-data-models.md)** @@ -191,7 +193,7 @@ Use this checklist to ensure your SDMX metadata meets StatGPT requirements: 5. **Missing Concept Definitions**: Reduces context understanding 6. **Slow API Response Times**: Degrades user experience 7. **Authentication Complexity**: Currently not supported by StatGPT -8. **Unsupported Data Model**: Only the conventional multi-indicator dataflow model is supported — see [Supported Data Models](#-supported-data-models) and [SDMX Data Models & StatGPT Support](./sdmx-data-models.md) +8. **Non-conventional Data Model**: The conventional multi-indicator dataflow model has first-class support; the shared DSD model is workable but not optimal, and the indicator-per-dataflow model is not supported yet — see [Supported Data Models](#-supported-data-models) and [SDMX Data Models & StatGPT Support](./sdmx-data-models.md) ## 💡 Recommendations for Data Providers diff --git a/architecture/sdmx-data-models.md b/architecture/sdmx-data-models.md index 5b43b73..5a91099 100644 --- a/architecture/sdmx-data-models.md +++ b/architecture/sdmx-data-models.md @@ -6,22 +6,23 @@ requirements. ## Overview -Across SDMX providers there are three data models. StatGPT supports only the **conventional -(multi-indicator dataflow)** model — where the indicator is a dimension inside one broad -dataflow, which is what StatGPT's indicator search resolves and filters within (see -[Data Query & Hybrid Search](./data-query-hybrid-search.md)). The **shared DSD** and -**indicator-per-dataflow** models — where the indicator is instead encoded by *which dataflow -(or table)* you pick — are **not supported yet**. - -| # | Data model | Indicator is selected by… | Example providers | Supported | +Across SDMX providers there are three data models. StatGPT gives **first-class support** to the +**conventional (multi-indicator dataflow)** model — where the indicator is a dimension inside one +broad dataflow, which is what StatGPT's indicator search resolves and filters within (see +[Data Query & Hybrid Search](./data-query-hybrid-search.md)). The **shared DSD** model (e.g. OECD) +is **supported, but not optimal**: it can be onboarded, but only one dataflow at a time rather than +a whole DSD and all its dataflows at once. The **indicator-per-dataflow** model — where the +indicator is encoded by *which dataflow (or table)* you pick — is **not supported yet**. + +| # | Data model | Indicator is selected by… | Example providers | Support | |---|------------|---------------------------|-------------------|:---------:| -| 1 | **Conventional (multi-indicator dataflow)** | a dimension *inside* one broad dataflow | IMF, BIS | ✅ Yes | -| 2 | **Shared DSD** | *which dataflow* (one DSD split across many per-country / per-table dataflows) | OECD | ❌ Not yet | +| 1 | **Conventional (multi-indicator dataflow)** | a dimension *inside* one broad dataflow | IMF, BIS | ✅ Supported | +| 2 | **Shared DSD** | *which dataflow* (one DSD split across many per-country / per-table dataflows) | OECD | ⚠️ Supported, but not optimal | | 3 | **Indicator-per-dataflow (dedicated DSD per dataset)** | *which dataflow* (one narrow table = one DSD) | Eurostat, ISTAT, ABS, ILO | ❌ Not yet | --- -## 1. Conventional (multi-indicator dataflow) — ✅ Supported +## 1. Conventional (multi-indicator dataflow) — ✅ First-class support A dataset is a **single broad dataflow that holds many indicators**. @@ -40,7 +41,7 @@ implements. --- -## 2. Shared DSD — ❌ Not supported yet +## 2. Shared DSD — ⚠️ Supported, but not optimal One rich DSD is **deliberately partitioned into many semantically-distinct dataflows** — typically one dataflow per country or per sub-table. The DSD is shared; the dataflows are @@ -53,6 +54,19 @@ narrow slices of it. - Each dataflow is a narrow slice of the shared structure — typically one country or one sub-table — not a broad indicator dataflow. +**Support status:** this model *can* be onboarded, so it is not unsupported — but it is not +optimal. Because StatGPT onboards at the dataflow level, each of a shared DSD's dataflows has +to be onboarded **individually**, rather than onboarding the DSD once and picking up all of its +dataflows automatically. The practical consequences: + +- **Slow, laborious onboarding** — a DSD split into dozens or hundreds of per-country / per-table + dataflows means dozens or hundreds of separate onboarding steps. +- **Possible quality trade-offs** — because the indicator context is spread across many narrow + dataflows instead of one broad indicator dimension, search and grounding quality can suffer. + +So a shared-DSD provider is workable today; it is simply neither efficient nor optimal, and it is +not the model StatGPT is designed around. + **Example: OECD** - ~1,536 dataflows over ~413 distinct DSDs (mean ≈ 3.65 dataflows per DSD); ~82% of dataflows @@ -104,8 +118,8 @@ determined by *which* dataflow is chosen, not by a dimension within one. ## Summary -| Data model | Example providers | Supported | +| Data model | Example providers | Support | |------------|-------------------|:---------:| -| Conventional (multi-indicator dataflow) | IMF, BIS | ✅ | -| Shared DSD | OECD | ❌ | -| Indicator-per-dataflow (dedicated DSD per dataset) | Eurostat, ISTAT, ABS, ILO | ❌ | +| Conventional (multi-indicator dataflow) | IMF, BIS | ✅ Supported | +| Shared DSD | OECD | ⚠️ Supported, but not optimal | +| Indicator-per-dataflow (dedicated DSD per dataset) | Eurostat, ISTAT, ABS, ILO | ❌ Not yet |