feat: AI-add optional YAML workflow configuration

PLANS.md

@@ -200,3 +200,31 @@ print a direct local URL to `charts/index.html` before the artifact directory.
    root, and preserve relative offline loading.
 3. Run an overwrite smoke test to verify obsolete root files are removed, then
    browser-test the dashboard and run the full validation gate.
+
+## Optional YAML Workflow Configuration
+
+### Summary
+
+Add an optional `--config` YAML file for custom dataset paths, separators,
+canonical feature names, chart feature groups, unit-interval validation,
+timestamp parsing, workflow selection, calendar overrides, and TSAM mean
+preservation. Preserve the current no-config CLI behavior and let explicitly
+provided CLI flags override YAML values.
+
+### Milestones
+
+1. Add failing configuration tests for strict YAML schema validation, paths
+   relative to the YAML file, `countries: ALL`, subset countries, and defaults;
+   then add PyYAML as a direct dependency and implement the loader.
+2. Add failing CLI tests for optional `--config` loading and precedence
+   `built-in defaults < YAML < explicit CLI flags`; keep output publication
+   settings CLI-only.
+3. Add failing data tests for configurable timestamp columns/formats, inferred
+   common sampling frequency, complete-year validation, and a fixed 24-hour
+   TSAM period whose timestep count follows the inferred resolution.
+4. Add failing aggregation tests proving `preserve_column_means: true` exports
+   TSAM's rescaled representatives while retaining medoid-date provenance and
+   recording the effective setting and inferred frequency in the manifest.
+5. Add an example YAML file and README usage documentation, then run lint,
+   typecheck, all tests, notebook execution, CLI help, package build, diff
+   checks, and real default/configured CLI smoke runs.

README.md

@@ -65,6 +65,36 @@ uv run tsam-workflows grouped \
 Open `charts/index.html`, whose direct local URL is printed when the command
 finishes. It provides a responsive sidebar for every chart in one workspace.
 
+### Custom YAML Configuration
+
+Use the optional YAML configuration for custom input files, CSV separators,
+timestamp parsing, feature labels, chart feature groups, calendar rules, and
+mean-preserved representatives:
+
+```bash
+uv run tsam-workflows grouped \
+  --config examples/grouped-workflow.yaml \
+  --output-dir outputs/custom \
+  --overwrite
+```
+
+Relative dataset paths resolve from the YAML file's directory. Omitting
+`countries` or setting `countries: ALL` selects every available country; use a
+list such as `countries: [DE, FR]` for a subset. Explicit CLI options such as
+`--year`, `--countries`, and cluster counts override YAML values.
+
+Each dataset's `feature` becomes the canonical output token, for example raw
+column `DE` with `feature: solar` becomes `DE_solar_2025`. `feature_group`
+organizes related features in chart selectors and does not change clustering.
+`unit_interval: true` validates that every value is between zero and one; it
+does not normalize the data.
+
+The grouped workflow always creates representative 24-hour days. Sampling
+frequency is inferred from timestamps, must be regular and identical across
+datasets, and is recorded in `manifest.json`. Enabling
+`preserve_column_means` exports TSAM's rescaled synthetic representatives while
+retaining each selected medoid date as provenance.
+
 ## Data
 
 Input CSV files live in `data/`. The notebooks use the checked-in sample

examples/grouped-workflow.yaml

@@ -0,0 +1,44 @@
+input:
+  snapshot_column: snapshot
+  timestamp_format: "%d.%m.%Y %H:%M"
+
+datasets:
+  demand:
+    path: ../data/Demand_ENTSO_E.csv
+    separator: ";"
+    feature: demand
+    feature_group: demand
+    unit_interval: false
+  solar_cf:
+    path: ../data/solar_capacity_factors.csv
+    separator: ";"
+    feature: solar
+    feature_group: capacity_factors
+    unit_interval: true
+  onwind_cf:
+    path: ../data/onwind_capacity_factors.csv
+    separator: ","
+    feature: onwind
+    feature_group: capacity_factors
+    unit_interval: true
+  ror_cf:
+    path: ../data/ror_capacity_factors.csv
+    separator: ","
+    feature: ror
+    feature_group: capacity_factors
+    unit_interval: true
+  hydro_inflow:
+    path: ../data/hydro_inflow_scaled_deduped_2025.csv
+    separator: ","
+    feature: hydro
+    feature_group: hydro
+    unit_interval: false
+
+workflow:
+  year: 2025
+  countries: ALL
+  working_clusters: 5
+  non_working_clusters: 2
+  cluster_method: hierarchical
+  non_working_weekdays: [Saturday, Sunday]
+  preserve_column_means: false

pyproject.toml

@@ -16,6 +16,7 @@ dependencies = [
     "notebook>=7.0.0",
     "pandas>=3.0.1",
     "plotly>=6.7.0",
+    "pyyaml>=6.0.3",
     "tsam>=3.3.0",
 ]
 

src/approach_1_ALL.ipynb

@@ -10,6 +10,27 @@
     "This notebook is the interactive client for the grouped Option 1 workflow. The reusable loading, validation, calendar grouping, TSAM aggregation, CSV schema construction, and chart builders live in `tsam_workflows`.\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "table-of-contents",
+   "metadata": {},
+   "source": [
+    "## Table of Contents\n",
+    "\n",
+    "- [Method Overview](#Method-Overview)\n",
+    "- [Normalization](#Normalization)\n",
+    "- [Feature Preservation](#Feature-Preservation)\n",
+    "  - [Preservation Objectives](#Preservation-Objectives)\n",
+    "  - [Clustering Influence](#Clustering-Influence)\n",
+    "  - [Current Baseline](#Current-Baseline)\n",
+    "- [Imports And Configuration](#Imports-And-Configuration)\n",
+    "- [Run Workflow](#Run-Workflow)\n",
+    "- [Output Tables](#Output-Tables)\n",
+    "- [Summary Charts](#Summary-Charts)\n",
+    "- [Group-Level TSAM Diagnostic Drilldowns](#Group-Level-TSAM-Diagnostic-Drilldowns)\n",
+    "- [Optional CSV Export](#Optional-CSV-Export)\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "2352af61",
@@ -20,6 +41,215 @@
     "The workflow clusters days separately by calendar month and by working/non-working status. With the default 5 working-day and 2 non-working-day representative configuration, the year is split into 24 groups and produces 84 representative days.\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8e0202af",
+   "metadata": {},
+   "source": [
+    "# Normalization\n",
+    "\n",
+    "TSAM handles basic per-column normalization internally. If normalized columns or feature groups should contribute differently to clustering, their distance contributions can be normalized in a second step. TSAM implements this additional normalization through the top-level `weights=` argument.\n",
+    "\n",
+    "**Current grouped-workflow support**\n",
+    "\n",
+    "The grouped workflow and CLI use TSAM's default per-column normalization with equal column weights. Additional normalization through `normalize_column_means` or `weights` is not supported by the grouped workflow or CLI. The methods below are retained as technical reference and require custom code outside the current interface.\n",
+    "\n",
+    "The clustering pipeline is:\n",
+    "\n",
+    "`raw columns -> per-column min-max scaling -> optional mean scaling -> optional contribution normalization through weights -> clustering`\n",
+    "\n",
+    "**Built-in normalization**\n",
+    "\n",
+    "- TSAM independently maps each input column's observed range to `0..1` before clustering.\n",
+    "- This transformation is internal; representative values are returned in their original units.\n",
+    "- `ClusterConfig(normalize_column_means=True)` optionally divides each normalized column by its mean. It is disabled in the baseline.\n",
+    "- External affine scaling, such as min-max or z-score normalization, is generally redundant because TSAM subsequently applies its own min-max scaling.\n",
+    "\n",
+    "**Why additional contribution normalization may be needed**\n",
+    "\n",
+    "Equal column ranges do not guarantee equal influence. Distance still sums squared differences across every column and timestep, so a feature group's contribution depends on both its column count and normalized variability. `tsam.aggregate(weights=...)` is the implementation mechanism for normalizing these contributions after min-max scaling.\n",
+    "\n",
+    "**Candidate additional normalization strategies**\n",
+    "\n",
+    "The baseline uses no additional contribution normalization. The alternatives below are implemented through column weights:\n",
+    "\n",
+    "| Strategy | Purpose |\n",
+    "|---|---|\n",
+    "| No additional normalization | Keep TSAM's default equal column weights |\n",
+    "| Feature-group contribution normalization | Use `1 / sqrt(group_column_count)` to diagnose and correct column-count imbalance |\n",
+    "| Physical-scale normalization | Where defensible conversions exist, scale normalized differences to common physical units using demand range for load and installed capacity multiplied by observed capacity-factor range for renewable availability |\n",
+    "\n",
+    "**Illustrative pseudocode**\n",
+    "\n",
+    "The TSAM-level pseudocode below illustrates how these strategies can be expressed without modifying the raw input data. It is intentionally non-executable and cannot be selected through the current CLI.\n",
+    "\n",
+    "```python\n",
+    "NORMALIZATION_MODE = \"none\"\n",
+    "# Alternatives: \"equal_feature_groups\", \"physical_scale\"\n",
+    "\n",
+    "\n",
+    "def build_normalization_weights(data, mode):\n",
+    "    if mode == \"none\":\n",
+    "        return None\n",
+    "\n",
+    "    columns_by_group = group_columns_by_feature(data.columns)\n",
+    "\n",
+    "    if mode == \"equal_feature_groups\":\n",
+    "        return {\n",
+    "            column: 1 / sqrt(len(group_columns))\n",
+    "            for group_columns in columns_by_group.values()\n",
+    "            for column in group_columns\n",
+    "        }\n",
+    "\n",
+    "    if mode == \"physical_scale\":\n",
+    "        return {\n",
+    "            column: physical_range_in_common_power_units(column)\n",
+    "            for column in data.columns\n",
+    "        }\n",
+    "\n",
+    "    raise ValueError(\"Unknown normalization mode\")\n",
+    "\n",
+    "\n",
+    "normalization_weights = build_normalization_weights(\n",
+    "    group_features, NORMALIZATION_MODE\n",
+    ")\n",
+    "\n",
+    "aggregation_result = tsam.aggregate(\n",
+    "    data=group_features,\n",
+    "    n_clusters=n_clusters,\n",
+    "    weights=normalization_weights,\n",
+    "    cluster=tsam.ClusterConfig(\n",
+    "        method=\"hierarchical\",\n",
+    "        representation=\"medoid\",\n",
+    "        normalize_column_means=False,\n",
+    "    ),\n",
+    "    preserve_column_means=False,  # Feature preservation, not normalization.\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "For equal feature-group contributions, `group_column_count * (1 / sqrt(group_column_count)) ** 2 = 1`. The `physical_range_in_common_power_units` helper remains abstract until the required installed-capacity and unit-conversion metadata is available.\n",
+    "\n",
+    "No additional normalization strategy is selected in the baseline. Alternatives should be compared using group-level normalized errors, peak and extreme preservation, distance contributions, and downstream model results.\n",
+    "\n",
+    "**Mean preservation is separate**\n",
+    "\n",
+    "`preserve_column_means` is post-clustering behavior, not normalization. The baseline keeps it `False` so medoid profiles remain untouched observed days. Setting `workflow.preserve_column_means: true` in the optional YAML configuration rescales TSAM's representative outputs to preserve weighted means, so they are no longer exact medoids. The shared workflow exports TSAM's rescaled representative values while retaining the selected medoid date as provenance.\n",
+    "\n",
+    "[TSAM API reference](https://tsam.readthedocs.io/en/latest/api/tsam/api/)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "81b963fe",
+   "metadata": {},
+   "source": [
+    "# Feature Preservation\n",
+    "\n",
+    "Temporal aggregation cannot retain every original value. Feature preservation therefore starts by choosing which property should be retained in the representative periods or reconstructed series.\n",
+    "\n",
+    "**Current grouped-workflow support**\n",
+    "\n",
+    "The grouped workflow and CLI currently use hierarchical clustering with medoid representation. All-column mean preservation is supported through `workflow.preserve_column_means: true` in the optional YAML configuration; it exports TSAM's rescaled representative values and therefore produces synthetic profiles. Selective mean preservation, extreme-period preservation, and distribution representations are not supported by the grouped workflow or CLI. The remaining alternatives below are retained as technical reference and require custom code outside the current interface.\n",
+    "\n",
+    "## Preservation Objectives\n",
+    "\n",
+    "| Objective | TSAM setting | What it preserves |\n",
+    "|---|---|---|\n",
+    "| Exact observed profiles | `ClusterConfig(representation=\"medoid\")` | Complete observed periods, including relationships between columns |\n",
+    "| Annual means and totals | `preserve_column_means=True` | Occurrence-weighted column means; totals follow when the represented duration is unchanged |\n",
+    "| Extremes | `ExtremeConfig(...)` | Configured minimum or maximum values or periods for selected columns |\n",
+    "| Duration-curve shape | `ClusterConfig(representation=tsam.Distribution(...))` | An approximation of each column's value distribution, optionally including its minimum and maximum |\n",
+    "\n",
+    "Medoid and distribution representations apply to every input column. To preserve means only for selected columns, enable `preserve_column_means` and place every non-selected column in `rescale_exclude_columns`. `ExtremeConfig` can target selected columns directly.\n",
+    "\n",
+    "**Illustrative pseudocode**\n",
+    "\n",
+    "Each preservation objective changes different TSAM arguments. The TSAM-level example below is intentionally non-executable; only all-column mean preservation is currently exposed through grouped-workflow YAML.\n",
+    "\n",
+    "```python\n",
+    "PRESERVATION_MODE = \"exact_profiles\"\n",
+    "# Alternatives: \"selected_means\", \"selected_extremes\", \"all_distributions\"\n",
+    "\n",
+    "PRESERVED_FEATURE_GROUPS = {\"demand\"}\n",
+    "\n",
+    "\n",
+    "def build_preservation_options(columns, mode, feature_groups):\n",
+    "    selected_columns = columns_for_feature_groups(columns, feature_groups)\n",
+    "    non_selected_columns = [\n",
+    "        column for column in columns if column not in selected_columns\n",
+    "    ]\n",
+    "\n",
+    "    options = {\n",
+    "        \"cluster\": tsam.ClusterConfig(\n",
+    "            method=\"hierarchical\",\n",
+    "            representation=\"medoid\",\n",
+    "        ),\n",
+    "        \"preserve_column_means\": False,\n",
+    "        \"rescale_exclude_columns\": None,\n",
+    "        \"extremes\": None,\n",
+    "    }\n",
+    "\n",
+    "    if mode == \"exact_profiles\":\n",
+    "        return options\n",
+    "\n",
+    "    if mode == \"selected_means\":\n",
+    "        options[\"preserve_column_means\"] = True\n",
+    "        options[\"rescale_exclude_columns\"] = non_selected_columns\n",
+    "        return options\n",
+    "\n",
+    "    if mode == \"selected_extremes\":\n",
+    "        options[\"extremes\"] = tsam.ExtremeConfig(\n",
+    "            method=\"append\",\n",
+    "            max_value=selected_columns,\n",
+    "        )\n",
+    "        return options\n",
+    "\n",
+    "    if mode == \"all_distributions\":\n",
+    "        # Distribution representation applies to every input column.\n",
+    "        options[\"cluster\"] = tsam.ClusterConfig(\n",
+    "            method=\"hierarchical\",\n",
+    "            representation=tsam.Distribution(\n",
+    "                scope=\"global\",\n",
+    "                preserve_minmax=True,\n",
+    "            ),\n",
+    "        )\n",
+    "        return options\n",
+    "\n",
+    "    raise ValueError(\"Unknown preservation mode\")\n",
+    "\n",
+    "\n",
+    "preservation_options = build_preservation_options(\n",
+    "    columns=group_features.columns,\n",
+    "    mode=PRESERVATION_MODE,\n",
+    "    feature_groups=PRESERVED_FEATURE_GROUPS,\n",
+    ")\n",
+    "\n",
+    "aggregation_result = tsam.aggregate(\n",
+    "    data=group_features,\n",
+    "    n_clusters=n_clusters,\n",
+    "    **preservation_options,\n",
+    ")\n",
+    "\n",
+    "# Use TSAM's final values for modes that create synthetic representatives.\n",
+    "representative_values = aggregation_result.cluster_representatives\n",
+    "```\n",
+    "\n",
+    "- `selected_means` rescales only the selected feature-family columns.\n",
+    "- `selected_extremes` demonstrates maximum-value preservation. Minimum values or extreme period totals use the corresponding `ExtremeConfig` fields.\n",
+    "- `all_distributions` cannot honor `PRESERVED_FEATURE_GROUPS`; it affects every input column.\n",
+    "- Mean preservation must export `cluster_representatives`; otherwise the current original-row slicing restores the unscaled medoid values. Distribution representatives are synthetic and have no original row to slice, so that mode also requires a different export path.\n",
+    "\n",
+    "## Clustering Influence\n",
+    "\n",
+    "The top-level `weights=` argument changes how strongly columns influence clustering distances. It can improve representation of important features, but it does not guarantee preservation of their means, extremes, or distributions.\n",
+    "\n",
+    "## Current Baseline\n",
+    "\n",
+    "The baseline uses hierarchical clustering with medoid representation, no mean rescaling, no extreme-period configuration, and default equal column weights. It preserves the exact profiles of selected medoid days for every column, but it does not guarantee annual means, extremes, or duration curves.\n",
+    "\n",
+    "**API references:** [ClusterConfig](https://tsam.readthedocs.io/en/latest/api/tsam/api/#tsam.config.ClusterConfig) | [ExtremeConfig](https://tsam.readthedocs.io/en/latest/api/tsam/api/#tsam.config.ExtremeConfig)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "183b3ce2",