diff --git a/docs/migrations/NEXT.md b/docs/migrations/NEXT.md index 43a725d25..a6beb4663 100644 --- a/docs/migrations/NEXT.md +++ b/docs/migrations/NEXT.md @@ -1,6 +1,6 @@ # Migration: Next release -**Current pom version:** 1.12.10 +**Current pom version:** TBD **Target release version:** TBD *(renamed to `released/v.md` at release cut)* **Status:** under development @@ -21,152 +21,6 @@ For the migration documentation structure, see [`README.md`](README.md). | # | Script | Description | Risk | |---|---|---|---| -| 1 | [`sql/migrations/create-associated-dataset.sql`](../sql/migrations/create-associated-dataset.sql) | Create `associated_dataset` table for connecting InvenioRDM datasets to projects (FEAT-DATSET-01) | low — new empty table, no data loss on rollback | - -Each row links to its incremental script. The sections below expand each entry -with apply / verify / rollback detail. - ---- - -## Migration #1: `associated_dataset` table - -| Field | Value | -|---|---| -| **Story** | [#1467 — FEAT-DATSET-01: Connecting open, published datasets](https://github.com/qbicsoftware/data-manager-app/issues/1467) | -| **Feature** | [#1466 — FEAT-DATASET-CONNECTION](https://github.com/qbicsoftware/data-manager-app/issues/1466) | -| **ADRs** | [0001](../adr/0001-associated-datasets-domain-model.md), [0002](../adr/0002-invenio-rdm-api-client-credentials.md), [0003](../adr/0003-connection-lifecycle-stewardship.md) | -| **Scope** | Database schema only (new table) | -| **Script** | [`sql/migrations/create-associated-dataset.sql`](../sql/migrations/create-associated-dataset.sql) | -| **Target datasource** | `data_management` | - -### What it does - -Creates an empty `associated_dataset` table. Per [ADR-0001](../adr/0001-associated-datasets-domain-model.md): - -- Lives in the `project-management` bounded context (table name starts with `associated_dataset`, not under the existing `dataset_*` namespace, to keep the InvenioRDM connection concept distinct from the legacy OpenBIS raw-data tables). -- Has four **universal columns** (`title`, `pid`, `version`, `publication_date`) duplicated from the JSON blob for SQL sort/filter. -- Stores **source-specific metadata** (InvenioRDM creators, community, access details) in a MariaDB `JSON` column (`resource_metadata`). At the expected scale of <100 rows/project, this is efficient; the escape hatch for future scale is a generated virtual column + index — no entity change required. -- **Soft-delete for removal** (ADR-0001, ADR-0003): deleting a connection sets `connection_state = 'REMOVED'`, leaving the row as an audit tombstone. Active query paths filter `REMOVED` out. There is no `DELETE` statement in v1. -- **No SQL FK constraint on `experiment_id`** — the application enforces referential integrity in memory so that project-ACL changes and experiment deletions don't cascade-delete dataset connections. - -### What is NOT in this migration - -A second table, `user_invenio_rdm_credential` (for per-user Personal Access -Tokens to access restricted datasets), is planned for Stories 14/15 -(FEAT-DATSET-14, FEAT-DATSET-15). It is intentionally **not** created by this -migration because FEAT-DATSET-01 covers only public, open datasets. A follow-up -entry in this file — or in the next release's `NEXT.md` — will cover the -credentials table when those stories land. - -### Pre-flight - -Run these checks against the target database **before** applying: - -```sql --- 1. Confirm the table does NOT already exist -SELECT COUNT(*) AS table_exists -FROM information_schema.tables -WHERE table_schema = DATABASE() - AND table_name = 'associated_dataset'; --- Expected: 0 - --- 2. Confirm target datasource charset/collation matches the rest of the schema -SELECT @@character_set_database AS charset, - @@collation_database AS collation; --- Expected: utf8mb4, utf8mb4_unicode_ci - --- 3. Confirm you are connected to the data-management datasource -SELECT DATABASE(); --- Expected: data_management (or whatever your DM schema is named) -``` - -If the table already exists (return value = 1), this migration was already -applied — no further action required (the DDL is idempotent). - -### Apply - -```bash -mysql -u -h -P data_management \ - < sql/migrations/create-associated-dataset.sql -``` - -Or inline: - -```sql --- The full DDL is inlined in the script for review; see --- sql/migrations/create-associated-dataset.sql -``` - -### Verify - -```sql --- 1. Confirm 15 columns -SELECT COUNT(*) AS column_count -FROM information_schema.columns -WHERE table_schema = DATABASE() - AND table_name = 'associated_dataset'; --- Expected: 15 - --- 2. Confirm indexes are in place (4 keys + PK = 5 rows) -SELECT index_name, GROUP_CONCAT(column_name ORDER BY seq_in_index) AS columns -FROM information_schema.statistics -WHERE table_schema = DATABASE() - AND table_name = 'associated_dataset' -GROUP BY index_name -ORDER BY index_name; --- Expected rows: --- PRIMARY id --- idx_assoc_ds_project project_id --- idx_assoc_ds_project_state project_id,connection_state --- idx_assoc_ds_source_type source_type --- idx_assoc_ds_state connection_state - --- 3. Confirm the application can query the table -SELECT COUNT(*) AS active_connected_count -FROM associated_dataset -WHERE connection_state <> 'REMOVED'; --- Expected: 0 (empty table immediately after migration) - --- 4. Confirm no unintended table was created on the finance datasource -SELECT table_schema, table_name -FROM information_schema.tables -WHERE table_name = 'associated_dataset'; --- Expected: exactly one row, for the data-management schema -``` - -### Rollback - -Rollback is destructive — any rows inserted by the application after the -migration will be lost. Because FEAT-DATSET-01 is the first release introducing -this table, rollback is only meaningful before any datasets have been connected -by users. - -```sql --- DANGER: drops all connected-dataset connections. --- Safe only when the table is empty (see rollback plan below). -DROP TABLE IF EXISTS `associated_dataset`; -``` - -If the table contains connected-dataset rows, rollback requires one of: - -1. **Reverse the feature deployment** to a version that does not reference the - table, then drop the table. -2. **Truncate the table**, then drop it. Both cases lose all connection data. - -In either case, notify users that previously connected datasets will no longer -appear in their projects. - -### Operator notes - -- **No data is moved.** This migration only creates an empty table. -- **No downtime required.** `CREATE TABLE IF NOT EXISTS` on MariaDB takes - milliseconds and does not lock other tables. -- **No foreign key constraint is created** between - `associated_dataset.experiment_id` and `experiments_datamanager.id`. The - application enforces referential integrity in memory. -- **Future migration (same feature):** Stories 14/15 will add the - `user_invenio_rdm_credential` table. Expect a follow-up entry in this file - (or in a subsequent release's `NEXT.md`). --- @@ -230,4 +84,4 @@ When this release is ready to ship: 5. **Reset `NEXT.md`:** create a fresh empty copy from the template above for the *subsequent* release. -See [`README.md`](README.md) for the full migration documentation structure. +See [`README.md`](README.md) for the full migration documentation structure. \ No newline at end of file diff --git a/docs/migrations/released/README.md b/docs/migrations/released/README.md index ab1090199..108130c42 100644 --- a/docs/migrations/released/README.md +++ b/docs/migrations/released/README.md @@ -9,9 +9,9 @@ guide. ## Current state -**Empty.** The three-tier migration documentation structure was established in -2026-07 for FEAT-DATSET-01. Until the first release that uses the new -structure is cut, no files will appear here. +| Release | Guide | +|---|---| +| **1.14.0** | [`v1.14.0.md`](v1.14.0.md) | For migrations targeting the upcoming (unreleased) version, see [`../NEXT.md`](../NEXT.md). diff --git a/docs/migrations/released/v1.14.0.md b/docs/migrations/released/v1.14.0.md new file mode 100644 index 000000000..3957d4c9a --- /dev/null +++ b/docs/migrations/released/v1.14.0.md @@ -0,0 +1,343 @@ +# Migration: v1.14.0 + +**Previous version:** 1.13.0 +**Target release version:** 1.14.0 +**Status:** released + +> **To operators:** This document describes the schema changes for **Data Manager v1.14.0**. +> Apply these migrations when upgrading from v1.13.0 to v1.14.0. + +For the migration documentation structure, see [`README.md`](README.md). + +--- + +## Release-wide pre-flight + +Before applying any migrations: + +1. **Backup the `data_management` database.** + ```bash + mysqldump -u -h -P data_management > backup-data-manager-v1.13.0.sql + ``` +2. **Confirm connectivity** to the `data_management` datasource. + ```sql + SELECT DATABASE(); + -- Expected: data_management (or whatever your DM schema is named) + ``` +3. **Confirm charset / collation** matches the rest of the schema. + ```sql + SELECT @@character_set_database AS charset, + @@collation_database AS collation; + -- Expected: utf8mb4, utf8mb4_unicode_ci + ``` +4. **Stop the application** (recommended, not strictly required — see individual migration notes). + +--- + +## Summary of schema changes in this release + +| # | Script | Description | Risk | +|---|---|---|---| +| 1 | [`sql/migrations/create-associated-dataset.sql`](../sql/migrations/create-associated-dataset.sql) | Create `associated_dataset` table for connecting InvenioRDM datasets to projects (FEAT-DATSET-01) | low — new empty table, no data loss on rollback | +| 2 | [`sql/migrations/extend-dataset-visibility-in-project-overview.sql`](../sql/migrations/extend-dataset-visibility-in-project-overview.sql) | Add connected-dataset aggregates (count, open/restricted breakdown, last-connected) to `project_overview` view (FEAT-DATSET-09) | low — replaces an existing view; application fallback handles brief inconsistency | + +Apply migrations in the order shown above. Migration #1 must complete before #2 (the view depends on the table). + +--- + +## Migration #1: `associated_dataset` table + +| Field | Value | +|---|---| +| **Story** | [#1467 — FEAT-DATSET-01: Connecting open, published datasets](https://github.com/qbicsoftware/data-manager-app/issues/1467) | +| **Feature** | [#1466 — FEAT-DATASET-CONNECTION](https://github.com/qbicsoftware/data-manager-app/issues/1466) | +| **ADRs** | [0001](../../adr/0001-associated-datasets-domain-model.md), [0002](../../adr/0002-invenio-rdm-api-client-credentials.md), [0003](../../adr/0003-connection-lifecycle-stewardship.md) | +| **Scope** | Database schema only (new table) | +| **Script** | [`sql/migrations/create-associated-dataset.sql`](../sql/migrations/create-associated-dataset.sql) | +| **Target datasource** | `data_management` | + +### What it does + +Creates an empty `associated_dataset` table. Per [ADR-0001](../../adr/0001-associated-datasets-domain-model.md): + +- Lives in the `project-management` bounded context (table name starts with `associated_dataset`, not under the existing `dataset_*` namespace, to keep the InvenioRDM connection concept distinct from the legacy OpenBIS raw-data tables). +- Has four **universal columns** (`title`, `pid`, `version`, `publication_date`) duplicated from the JSON blob for SQL sort/filter. +- Stores **source-specific metadata** (InvenioRDM creators, community, access details) in a MariaDB `JSON` column (`resource_metadata`). At the expected scale of <100 rows/project, this is efficient; the escape hatch for future scale is a generated virtual column + index — no entity change required. +- **Soft-delete for removal** (ADR-0001, ADR-0003): deleting a connection sets `connection_state = 'REMOVED'`, leaving the row as an audit tombstone. Active query paths filter `REMOVED` out. There is no `DELETE` statement in v1. +- **No SQL FK constraint on `experiment_id`** — the application enforces referential integrity in memory so that project-ACL changes and experiment deletions don't cascade-delete dataset connections. + +### What is NOT in this migration + +A second table, `user_invenio_rdm_credential` (for per-user Personal Access +Tokens to access restricted datasets), is planned for later stories. It is intentionally +**not** created by this migration because FEAT-DATSET-01 covers only public, open datasets. + +### Pre-flight + +Run these checks against the target database **before** applying: + +```sql +-- 1. Confirm the table does NOT already exist +SELECT COUNT(*) AS table_exists +FROM information_schema.tables +WHERE table_schema = DATABASE() + AND table_name = 'associated_dataset'; +-- Expected: 0 + +-- 2. Confirm target datasource charset/collation matches the rest of the schema +SELECT @@character_set_database AS charset, + @@collation_database AS collation; +-- Expected: utf8mb4, utf8mb4_unicode_ci + +-- 3. Confirm you are connected to the data-management datasource +SELECT DATABASE(); +-- Expected: data_management (or whatever your DM schema is named) +``` + +If the table already exists (return value = 1), this migration was already +applied — no further action required (the DDL is idempotent). + +### Apply + +```bash +mysql -u -h -P data_management \ + < sql/migrations/create-associated-dataset.sql +``` + +Or inline: + +```sql +-- The full DDL is inlined in the script for review; see +-- sql/migrations/create-associated-dataset.sql +``` + +### Verify + +```sql +-- 1. Confirm 15 columns +SELECT COUNT(*) AS column_count +FROM information_schema.columns +WHERE table_schema = DATABASE() + AND table_name = 'associated_dataset'; +-- Expected: 15 + +-- 2. Confirm indexes are in place (4 keys + PK = 5 rows) +SELECT index_name, GROUP_CONCAT(column_name ORDER BY seq_in_index) AS columns +FROM information_schema.statistics +WHERE table_schema = DATABASE() + AND table_name = 'associated_dataset' +GROUP BY index_name +ORDER BY index_name; +-- Expected rows: +-- PRIMARY id +-- idx_assoc_ds_project project_id +-- idx_assoc_ds_project_state project_id,connection_state +-- idx_assoc_ds_source_type source_type +-- idx_assoc_ds_state connection_state + +-- 3. Confirm the application can query the table +SELECT COUNT(*) AS active_connected_count +FROM associated_dataset +WHERE connection_state <> 'REMOVED'; +-- Expected: 0 (empty table immediately after migration) + +-- 4. Confirm no unintended table was created on the finance datasource +SELECT table_schema, table_name +FROM information_schema.tables +WHERE table_name = 'associated_dataset'; +-- Expected: exactly one row, for the data-management schema +``` + +### Rollback + +Rollback is destructive — any rows inserted by the application after the +migration will be lost. Because FEAT-DATSET-01 is the first release introducing +this table, rollback is only meaningful before any datasets have been connected +by users. + +```sql +-- DANGER: drops all connected-dataset connections. +-- Safe only when the table is empty (see rollback plan below). +DROP TABLE IF EXISTS `associated_dataset`; +``` + +If the table contains connected-dataset rows, rollback requires one of: + +1. **Reverse the feature deployment** to a version that does not reference the + table, then drop the table. +2. **Truncate the table**, then drop it. Both cases lose all connection data. + +In either case, notify users that previously connected datasets will no longer +appear in their projects. + +### Operator notes + +- **No data is moved.** This migration only creates an empty table. +- **No downtime required.** `CREATE TABLE IF NOT EXISTS` on MariaDB takes + milliseconds and does not lock other tables. +- **No foreign key constraint is created** between + `associated_dataset.experiment_id` and `experiments_datamanager.id`. The + application enforces referential integrity in memory. + +--- + +## Migration #2: Extend `project_overview` view + +| Field | Value | +|---|---| +| **Story** | [#1475 — FEAT-DATSET-09: Show connected-dataset summary in project listing](https://github.com/qbicsoftware/data-manager-app/issues/1475) | +| **Feature** | [#1466 — FEAT-DATASET-CONNECTION](https://github.com/qbicsoftware/data-manager-app/issues/1466) | +| **ADRs** | [0001](../../adr/0001-associated-datasets-domain-model.md) | +| **Scope** | Database schema (view replacement) | +| **Script** | [`sql/migrations/extend-dataset-visibility-in-project-overview.sql`](../sql/migrations/extend-dataset-visibility-in-project-overview.sql) | +| **Target datasource** | `data_management` | + +### What it does + +Extends the `project_overview` view with four aggregate columns sourced from +the `associated_dataset` table: + +- `connectedDatasetCount` — total datasets connected to the project +- `openDatasetCount` — PUBLIC access_level count +- `restrictedDatasetCount` — RESTRICTED access_level count +- `lastConnectedOn` — most recent `connected_on` timestamp + +The aggregate uses a LEFT JOIN against a derived table so that projects with +no connected datasets still appear in the view (with zero counts and a NULL +`lastConnectedOn`). + +**Self-containment:** This migration inlines the measurement-aggregation +subqueries directly into the `project_overview` view definition rather than +joining through `project_measurements`. That avoids a cross-view dependency on +the `project_measurements` view definition being up-to-date at deployment time — +in environments where `project_measurements` has the older pre-IP-measurements +definition, a join-through approach would fail with *"Unknown column +'m.amountIpMeasurements' in SELECT"*. + +> **Fresh install note:** For fresh installs, run `sql/complete-schema.sql` +> instead of this migration script. It contains the canonical view definition +> (with the same inlined measurement subqueries and dataset aggregates) and a +> separate `project_measurements` view. + +### Pre-flight + +Run these checks **before** applying: + +```sql +-- 1. Confirm the `associated_dataset` table exists (required by the new view) +SELECT COUNT(*) AS table_exists +FROM information_schema.tables +WHERE table_schema = DATABASE() + AND table_name = 'associated_dataset'; +-- Expected: 1 + +-- 2. Confirm `project_overview` view exists +SELECT COUNT(*) AS view_exists +FROM information_schema.views +WHERE table_schema = DATABASE() + AND table_name = 'project_overview'; +-- Expected: 1 + +-- 3. Confirm you are connected to the data-management datasource +SELECT DATABASE(); +-- Expected: data_management (or whatever your DM schema is named) + +-- 4. Check current row count (for post-migration comparison) +SELECT COUNT(*) AS project_count FROM project_overview; +-- Record this value for verification after migration +``` + +### Apply + +```bash +mysql -u -h -P data_management \ + < sql/migrations/extend-dataset-visibility-in-project-overview.sql +``` + +The script drops and recreates the `project_overview` view. The full DDL is in +the script — review it before running. + +### Verify + +```sql +-- 1. Confirm the new columns exist +SELECT column_name +FROM information_schema.columns +WHERE table_schema = DATABASE() + AND table_name = 'project_overview' + AND column_name IN ('connectedDatasetCount', 'openDatasetCount', + 'restrictedDatasetCount', 'lastConnectedOn') +ORDER BY column_name; +-- Expected 4 rows + +-- 2. Confirm the view returns rows with new columns populated +SELECT projectId, projectTitle, connectedDatasetCount, openDatasetCount, + restrictedDatasetCount, lastConnectedOn +FROM project_overview +LIMIT 5; +-- Expected: same row count as pre-flight; new columns should be 0/NULL for projects +-- with no connected datasets + +-- 3. Confirm no unintended view was created on the finance datasource +SELECT table_schema, table_name +FROM information_schema.views +WHERE table_name = 'project_overview'; +-- Expected: exactly one row, for the data-management schema + +-- 4. Compare row count to pre-flight value +SELECT COUNT(*) AS project_count FROM project_overview; +-- Should match the pre-flight value +``` + +### Rollback + +Drop the new `project_overview` and re-create it from the previous definition. +The previous definition is available in git history: + +```bash +# Show the diff of the previous project_overview definition +git show 1.13.0:sql/complete-schema.sql | grep -A 200 "CREATE VIEW \`project_overview\`" + +# Or check out the previous version and extract it +git show 1.13.0:sql/complete-schema.sql > /tmp/complete-schema-1.13.0.sql +mysql -u -h -P data_management < /tmp/complete-schema-1.13.0.sql +``` + +> **Caution:** Restoring `complete-schema.sql` from 1.13.0 will also restore the +> full schema to that point, potentially overwriting other changes. For a +> targeted rollback, extract only the `CREATE VIEW \`project_overview\`` block +> from the 1.13.0 schema file and run it directly. + +### Operator notes + +- **Safe to run while the application is live** — views are dropped and + recreated atomically; in-flight queries return the old definition. +- **The `associated_dataset` table must exist** before running this script + (created by Migration #1 above). +- **COALESCE handles the NULL case** where a project has no connected datasets; + `COUNT` and `SUM` over an empty group would otherwise produce `NULL` and trip + the JPA entity's column nullability constraints. +- **`connection_state = 'CONNECTED'`** excludes soft-deleted (`REMOVED`) rows. + +--- + +## Post-migration checklist + +After applying all migrations: + +1. **Restart the application.** +2. **Verify the application starts without errors** — check logs for schema + mismatch warnings. +3. **Run a smoke test** against a project with no connected datasets to confirm + the `project_overview` view returns correct zero counts. +4. **Verify the connected-datasets feature** (if test data exists) — connect a + dataset and confirm counts update correctly. + +--- + +## Release cut metadata + +- **Frozen from:** `docs/migrations/NEXT.md` (commit at release cut) +- **Copied to:** `docs/migrations/released/v1.14.0.md` +- **NEXT.md reset:** Fresh template for the subsequent release +- **Migration scripts:** All 2 scripts in `sql/migrations/` \ No newline at end of file