DataTiles revision 8: FAIR semantics, Store PWA, payments and versioning

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.19.0 — 2026-08-29
+
+- advances the DataTiles schema to revision 8 with CF-first semantic variables, FAIR publication metadata, provenance, rights, optional integrity signatures, optional protected distribution, and immutable release identity;
+- adds deterministic NetCDF, GRIB2, and Zarr scientific import utilities that preserve numeric DNT1 values and record source and conversion provenance;
+- adds the optional Flask/SQLAlchemy DataTiles Store PWA, API, authentication, agreement enforcement, provider-neutral commerce, user libraries, and update discovery;
+- expands the normative specification, migration addenda, source-citation register, white paper, tutorials, and regression coverage for revisions 4 through 8.
+
 ## 0.10.0 — 2026-08-27
 
 - adds a command-line static-demo exporter, a Safari-compatible client-side numeric/vector portrayal, the full scientific playground toolset (relief, contours, smart labels, profile, compound query, and interactive 3D mesh), embedded checksum-identified data for direct `file://` use, an offline OpenLayers bundle, and an optional loopback-only launcher;

CITATION.cff

@@ -5,7 +5,7 @@ type: software
 authors:
   - family-names: Montella
     given-names: Raffaele
-version: 0.10.0
+version: 0.19.0
 date-released: 2026-08-27
 license: Apache-2.0
 keywords:

README.md

@@ -38,6 +38,10 @@ The spatial row follows MBTiles/TMS convention. Use `--xyz` on `put` and `get` t
 
 Version 0.10 adds conservative physical-table MBTiles fallback, a self-sufficient implementation specification, and an onboard edge-intelligence manifesto/white paper. It also includes mixed raster/vector content profiles, a tested five-lesson zero-to-hero curriculum, FAIR-by-design publication profile, OpenLayers scientific playground, comprehensive quality suite, protected CI/CD release path, interval axes, PROV-inspired provenance, scientific CRS records, bounded numeric-array decoding, OpenAPI description, and read-only OGC-style access.
 
+## NetCDF, GRIB, and Zarr import utilities
+
+Optional import utilities under `utils/` convert local or URL-identified NetCDF, GRIB2, and Zarr rectilinear grids into semantic DNT1 DataTiles. They preserve CF Standard Names when present, add GRIB2 crosswalk identifiers when available, retain source identity plus format-appropriate checksum provenance, and never pre-render scientific arrays as imagery. Install with `python -m pip install -e '.[utils]'`; see `docs/import-utilities.md`.
+
 ## Raster and vector content
 
 - Numeric raster matrices use the dependency-free DNT1 encoding and retain dtype, shape, byte order, nodata, scale, offset, unit, and compression.
@@ -69,6 +73,18 @@ The release workspace may also contain `dist/from-gaeta-to-maratea.datatiles` an
 
 Repository guidance is in [`AGENTS.md`](AGENTS.md), and the Markdown license notice is in [`LICENSE.md`](LICENSE.md). The complete Apache-2.0 legal text remains in `LICENSE`.
 
+## Zarr source ingestion
+
+DataTiles 0.13 adds `utils/zarr2datatiles.py` for Zarr v2/v3 stores. Local directory stores use the canonical `zarr-tree-sha256-v1` store digest; remote stores must be bound to an immutable snapshot with an authoritative checksum. Groups, consolidated metadata policy, fsspec storage backends, CF semantic mapping, explicit source/output rights, and credential-safe provenance are documented in `docs/zarr-source-profile.md`.
+
+## Optional cryptographic integrity and signatures
+
+DataTiles schema revision 6 can cryptographically bind the final logical scientific object to a signer without making signatures mandatory. `datatiles-integrity` creates deterministic SHA-256 logical manifests and optional Ed25519 signatures, either embedded or detached. Verification explicitly distinguishes mathematical validity from trust in the signer: an embedded public key is not, by itself, an authenticated publisher identity. The dependency-free core remains unchanged; install `.[integrity]` only when signing/verification is required. See `docs/digital-signatures.md`.
+
+## FAIR scientific publication
+
+DataTiles 0.12 treats FAIR, provenance, and licensing as enforceable publication evidence. Published objects separate persistent identity, DataCite 4.7 citation metadata, W3C PROV-aligned lineage, SPDX rights for the generated dataset/metadata and each source, cryptographic source identity, and external repository evidence. `fair_report(strict_publication=True)` is a release gate; it does not claim that FAIRness proves scientific validity or navigational fitness. See `docs/fair-publication-profile.md`, `docs/provenance.md`, and `docs/data-licensing.md`.
+
 ## Quality assurance and releases
 
 Pull requests run the complete suite on Python 3.10–3.13, validate playground JavaScript, build and install the distributions, and repeat the deterministic scientific-fixture build. Version tags additionally produce checksums, a GitHub build-provenance attestation, a GitHub Release, and—after the protected environment gate—a PyPI Trusted Publishing deployment. See [`docs/testing-and-release.md`](docs/testing-and-release.md) for the local protocol and repository configuration.
@@ -104,6 +120,24 @@ The [onboard intelligence white paper](docs/white-paper.md) presents DataTiles a
 
 New users can follow the [five-lesson DataTiles zero-to-hero tutorial](docs/tutorial/README.md), which combines formal discussion with a fully offline mixed raster/vector laboratory dataset.
 
+## Data acknowledgement and citation — required reading
+
+Data Tiles demo derivatives are multi-source scientific products. Every map, MBTiles file, figure, service, paper, and release must visibly credit every dataset that actually contributed cells or features and must ship its machine-readable provenance. The MIT software licence does not relicense any input data.
+The authoritative, source-by-source register is docs/data_sources_and_citation.md. It gives full bibliographic citations, mandatory map acknowledgements, licences, persistent identifiers, coverage/resolution limitations, frozen-manifest locations, and a FAIR release checklist.
+
+For the current From Gaeta to Maratea production build, acknowledge:
+
+JammeGaia22/MGDS bathymetry: Foglini, F., Tonielli, R., & Rovere, M. (2024), Multi-Resolution bathymetry grids of the Naples and Pozzuoli Gulf and Amalfi Coastal Area, Jamme_Gaia22 (2022), https://doi.org/10.60521/331667, CC BY 4.0.
+EMODnet fallback bathymetry: EMODnet Bathymetry Consortium (2024), EMODnet Digital Bathymetry (DTM 2024), https://doi.org/10.12770/cf51df64-56f9-4a99-b1aa-36b8d7b743a1, CC BY 4.0. EMODnet is used only where JammeGaia22 contains no finite measurement.
+Land/coastline: GSHHG 2.3.7 (Wessel & Smith, 1996, https://doi.org/10.1029/96JB00104) and, when enabled, S2Coast-2023 (Duan et al., https://doi.org/10.5281/zenodo.17092775, CC BY 4.0). These remain separate topology and high-water-line facts.
+Navigation and land context: © OpenStreetMap contributors, ODbL 1.0, https://www.openstreetmap.org/copyright.
+The required short-form map credit is:
+
+Bathymetry: Foglini, Tonielli & Rovere (2024), JammeGaia22/MGDS, doi:10.60521/331667; EMODnet Bathymetry Consortium DTM 2024, doi:10.12770/cf51df64-56f9-4a99-b1aa-36b8d7b743a1. Land/coastline: GSHHG 2.3.7 and S2Coast-2023. Context: © OpenStreetMap contributors, ODbL 1.0. Not for navigation.
+Do not list GMRT, GEBCO, EMODnet thematic products, or ISPRA as contributors unless the specific run manifest proves that they were used. Citation is source-specific evidence, not a generic project boilerplate.
+
+> **Software-licence consistency check.** The current public DataTiles repository declares Apache-2.0 in `LICENSE`. The preceding requested wording says MIT. Before publication, the README wording and `LICENSE` MUST agree; regardless of which software licence is adopted, it does not relicense input data.
+
 ## How to cite DataTiles
 
 Research using the software SHOULD cite the released software described by [`CITATION.cff`](CITATION.cff). Also cite the paper(s) that support the part of the scientific lineage or application you use:
@@ -121,3 +155,17 @@ This repository is a working reference implementation and a draft format specifi
 ## License
 
 Apache-2.0.
+
+## Optional commercial DRM
+
+DataTiles can optionally package a finalized lawful commercial product as an encrypted `.dtpkg` with recipient-specific issuer-signed licence grants. The portable profile uses AES-256-GCM, X25519/HKDF key wrapping, Ed25519 issuer signatures, and W3C ODRL 2.2 policy metadata. DRM is a technical access-control mechanism only: it does not relicense source data, remove attribution obligations, establish ownership, prove scientific validity, or make a product suitable for navigation. See `docs/drm-and-commercial-licensing.md`.
+
+## DataTiles Store PWA
+
+The optional `/store` application is a Python/Flask progressive web application backed by SQLAlchemy. It indexes machine-readable metadata from stored DataTiles files into a separate application database, supports full catalog search, authenticated browsing, role/group based authorization, an interactive chart-explorer-style preview of declared portrayal slices, and authorized downloads of the original DataTiles files. The bootstrap `admin` user belongs to the `administrators` group with the `admin` role; its initial password is explicitly configured in `store/config.py` and must be changed/secret-managed for production. The PWA never treats numeric scientific matrices as imagery and its service worker does not cache protected APIs, map payloads, or downloads. See `docs/store-pwa.md`. Managed authentication, verified-email self-registration, Google OIDC, Microsoft Entra ID tenants, generic OAuth2/OIDC, SMTP, public callback URL, and agreement settings are administrator-configurable in the PWA and API. Complete Store help is maintained under `store/docs/` and rendered by the PWA Help section.
+
+## Release versioning and optional Store payments
+
+Schema revision 8 defines `DataTiles-Release-Versioning-1`: a release can carry stable product identity, a human version label, a monotonically increasing release sequence, timestamp, predecessor, release-notes URI, and update-discovery URI. Published versioned releases are immutable; corrections and updates are new DataTiles objects with new checksums and signatures where used.
+
+The optional Store 0.4 adds a provider-neutral payment interface, with PayPal Orders v2 as the reference adapter, plus release-specific purchase entitlements, exact download history, a user library, and update notifications for users who previously acquired older sequences. Payment is disabled by default and is legally independent from data-licence acceptance: paying for a product never creates rights that the publisher does not possess. See `store/docs/payments.md` and `store/docs/versioning-and-updates.md`.

demo/from-gaeta-to-maratea/CITATION.cff

@@ -5,6 +5,6 @@ type: software
 authors:
   - family-names: Montella
     given-names: Raffaele
-version: 0.10.0
+version: 0.19.0
 date-released: 2026-08-27
 license: Apache-2.0

demo/from-gaeta-to-maratea/README.md

@@ -89,3 +89,19 @@ After `make all`, run `datatiles-serve work/gaeta-to-maratea.datatiles --port 80
 EMODnet is acquired at 2,754 × 1,167 regional cells; the compositing grid is 5,508 × 2,334 (nominal 50 m target spacing) so measured Jamme detail is not first collapsed to the coarser fallback grid. Storage extends through Web Mercator zoom 12. The client requests a bounded 128 × 128 map portrayal and a separate 96 × 72 analytic surface. The source-coverage layer makes every Jamme/EMODnet decision inspectable. Contour density is scale-dependent and remains limited by the selected source resolution; no new soundings are invented.
 
 The shelter field is a finite north-west ray test over the bathymetry water/land mask. It is a reproducible exposure proxy, not a wind-wave or atmospheric model, and must not support navigational or safety decisions.
+
+## FAIR, provenance, and rights gate
+
+The demo is also a publication-evidence exercise. Every upstream dataset must be represented by a checksum-identified provenance entity and an explicit source-rights record; the generated DataTiles dataset and its metadata have separate rights. The build must never infer permission from successful HTTP access. Before a scholarly release, replace illustrative identifiers with a registered PID, export DataCite 4.7 metadata and the W3C PROV-aligned graph, and archive the strict FAIR report with catalogue, landing-page, and metadata-retention evidence. `fair-publication-profile.json` lists the required evidence. The navigation warning remains independent of FAIR status: FAIRness does not establish hydrographic authority or safety fitness.
+
+## Source-specific citation gate
+
+`docs/data_sources_and_citation.md` is the authoritative citation register. A production run MUST record whether each acquired source actually contributed cells/features. Visible map/release acknowledgements MUST be checked against that frozen manifest: JammeGaia22 is primary bathymetry; EMODnet DTM 2024 is credited only when fallback cells are used; GSHHG topology and optional S2Coast-2023 high-water-line facts remain distinct; OpenStreetMap context retains ODbL attribution. Do not credit GMRT, GEBCO, EMODnet thematic products, ISPRA, or other candidates unless the run evidence proves contribution.
+
+## Optional signed release
+
+After the final source-contribution manifest, citations, rights, provenance, and FAIR checks are frozen, the demo MAY be signed with `datatiles-integrity`. Do not auto-sign intermediate NetCDF/GRIB/Zarr imports. The release should publish the detached signature and independently authenticated public key. A valid signature does not change the `Not for navigation` status.
+
+## Optional commercial edition
+
+The reference demo is not automatically converted into a proprietary product. A commercial edition may be created only if the frozen contribution manifest and every upstream licence permit that use. If enabled, register product/ODRL metadata in the inner DataTiles object, complete all acknowledgements, sign the frozen plaintext release if desired, and only then create `.dtpkg` packages and recipient licences. `Not for navigation` remains mandatory unless an independent competent authority establishes otherwise.

demo/from-gaeta-to-maratea/fair-publication-profile.json

@@ -0,0 +1,25 @@
+{
+  "profile": "DataTiles FAIR scholarly demonstration",
+  "purpose": "Template only; replace illustrative identifiers with registered publication evidence",
+  "required_evidence": [
+    "primary PID",
+    "dataset rights",
+    "metadata rights",
+    "source rights per provenance entity",
+    "source SHA-256 locks",
+    "W3C PROV-aligned graph",
+    "DataCite 4.7 metadata export",
+    "catalogue registration",
+    "landing page",
+    "metadata retention policy",
+    "scientific QA and limitations"
+  ],
+  "warning": "Demo outputs are research/visualization artifacts and are not for navigation.",
+  "optional_integrity_evidence": [
+    "DataTiles-Integrity-Manifest-1",
+    "Ed25519 detached/embedded signature",
+    "independently authenticated public key or certificate",
+    "optional Sigstore/timestamp/transparency evidence"
+  ],
+  "integrity_warning": "Signature validity does not establish scientific validity, legal compliance, hydrographic authority, or navigation fitness."
+}

demo/from-gaeta-to-maratea/runtime-lock.json

@@ -4,5 +4,5 @@
   "zlib": "1.3.2",
   "numpy": "2.3.5",
   "pillow": "12.3.0",
-  "datatiles": "0.10.0"
+  "datatiles": "0.19.0"
 }

docs/README.md

@@ -10,16 +10,27 @@ The [figure provenance register](figures/README.md) identifies the source docume
 | [MBTiles fallback](mbtiles-fallback.md) | selected-slice compatibility and physical-table export for conservative OpenLayers adapters |
 | [Onboard intelligence white paper](white-paper.md) | manifesto, AI evidence contract, practical marine/automotive architecture, and safety boundaries |
 | [Design evaluation](design-evaluation.md) | concise definition, audit findings, corrections, and compatibility conclusion |
-| [Zero-to-hero tutorial](tutorial/README.md) | five theory-led lessons with executable raster/vector laboratories |
+| [Zero-to-hero tutorial](tutorial/README.md) | seven theory-led lessons with executable raster/vector laboratories |
 | [FAIR by design](fair-by-design.md) | operational mapping from each FAIR principle to DataTiles evidence and publication gates |
+| [FAIR scientific publication profile](fair-publication-profile.md) | normative-supporting FAIR gates, DataCite 4.7 alignment, and release evidence |
+| [Scientific provenance](provenance.md) | W3C PROV-aligned entities, activities, agents, lineage, and audit rules |
+| [Data licensing and rights](data-licensing.md) | SPDX expressions, source/output/metadata rights separation, attribution, and access conditions |
+| [Cryptographic integrity and digital signatures](digital-signatures.md) | optional SHA-256 logical manifests, Ed25519 signatures, trust policy, offline verification, and Sigstore/in-toto interoperability |
+| [Data sources and citation](data_sources_and_citation.md) | authoritative source-by-source citations, acknowledgements, licences, frozen-manifest rules, and FAIR release checklist |
 | [Architecture](architecture.md) | component boundaries and design rationale |
 | [Numeric tiles](numeric-tiles.md) | DNT1 encoding and physical-value semantics |
 | [Vector tiles](vector-tiles.md) | multidimensional MVT/GeoJSON profiles and MBTiles projection |
 | [HTTP/OGC API](ogc-api.md) | resource model and interoperability scope |
 | [Playground](playground.md) | OpenLayers scientific operations and limitations |
+| [DataTiles Store PWA](store-pwa.md) | Flask/SQLAlchemy institutional catalog, metadata search, role-based access, interactive preview, licence/safety acceptance, API-first CRUD, and authorized downloads |
+| [DataTiles Store API v1](store-api.md) | Bearer authentication, agreement workflow, catalog/search, preview/download, managers CRUD, identity administration, and audit endpoints |
 | [Testing and release](testing-and-release.md) | local suite, CI gates, protected delivery, and release checklist |
 | [Profile demo](profile-demo.md) | two-point transect algorithm and evidence |
 | [Reproducibility](reproducibility.md) | exact reconstruction of the Naples reference object |
 | [Replicability](replicability.md) | protocol for independent regions, sources, variables, and implementations |
 | [Getting started](getting-started.md) | concise installation and API tutorial |
+| [Interoperability conventions](conventions.md) | CF-first variable semantics, units, CRS/datum, source identity, dimensions, and portrayal boundaries |
+| [Import utilities](import-utilities.md) | NetCDF/GRIB/Zarr local-or-URL ingestion, semantic mapping, numeric tiling, and provenance |
+| [Zarr source profile](zarr-source-profile.md) | normative Zarr v2/v3 source identity, checksums, groups, remote stores, FAIR provenance and rights |
 | [References](references.md) | research papers, standards, vocabularies, and source records |
+| [Commercial DRM](drm-and-commercial-licensing.md) | optional protected distribution, ODRL policy, recipient licences |