[raffmont]

DataTiles revision 8: FAIR semantics, integrity, DRM, Store PWA, payments, and release versioning

Summary

This PR evolves DataTiles from the 0.10-era schema into a FAIR-by-design, provenance-rich, semantically interoperable scientific data publication platform while preserving MBTiles compatibility and the core distinction between scientific evidence and portrayal.

It adds CF-first semantic variable identity, scientific import utilities for NetCDF/GRIB2/Zarr, FAIR/provenance/licensing metadata, source-specific citation requirements, optional digital signatures, optional commercial DRM, a Flask/SQLAlchemy DataTiles Store PWA, role-based authentication, licence/safety acceptance, API-first catalog management, optional external identity providers and email registration, provider-neutral payments with a PayPal reference adapter, user purchase/download history, update notifications, and normative DataTiles release versioning.

Major changes

DataTiles schema and semantics

• Advances the proposed schema through revision 8.
• Adds a semantic variable registry using CF Standard Names as the canonical vocabulary.
• Supports external crosswalks such as WMO GRIB2 and SeaDataNet/NERC identifiers.
• Preserves typed multidimensional coordinate addressing and MBTiles compatibility.
• Adds stable product identity and monotonic release sequencing for immutable versioned releases.

Scientific import utilities

• Adds utils/netcdf2datatiles.py.
• Adds utils/grib2datatiles.py.
• Adds utils/zarr2datatiles.py.
• Supports local and URL-identified sources.
• Preserves CF semantics, source provenance, checksums, rights and conversion parameters.
• Keeps numeric scientific arrays as DNT1 rather than silently pre-rendering imagery.

FAIR, provenance, licensing and citation

• Adds FAIR publication metadata and principle-level validation.
• Adds W3C PROV-aligned provenance and tile-level lineage.
• Adds structured rights/licensing metadata with SPDX-style expressions.
• Adds DataCite-shaped publication metadata without claiming DOI registration when none exists.
• Adds docs/data_sources_and_citation.md as the authoritative source-by-source citation register.
• Extends the white paper and tutorial with FAIR, provenance, licensing and citation guidance.

Cryptographic integrity

• Adds optional canonical integrity manifests over logical DataTiles content.
• Adds optional Ed25519 signatures.
• Separates cryptographic validity, content equality and trusted signer identity.
• Supports detached signatures and multiple independent signers.

Optional DRM / commercial distribution

• Adds an outer protected-distribution package instead of encrypting SQLite rows.
• Uses AES-256-GCM content encryption, X25519/HKDF recipient key wrapping, and Ed25519-signed licences.
• Uses W3C ODRL as the machine-readable usage-policy model.
• Keeps DRM legally and technically separate from source licensing and FAIR metadata.

DataTiles Store PWA

• Adds a Python/Flask PWA under /store.
• Uses SQLAlchemy 2.x for application data and DataTiles inspection.
• Adds searchable metadata catalog, preview/exploration, downloads, and admin/manager CRUD.
• Adds user/group/role/permission authorization.
• Adds managed password authentication plus optional Google, Microsoft Entra ID and generic OIDC/OAuth2 integration.
• Adds optional email self-registration with SMTP verification.
• Adds administrator configuration UI/API.
• Adds integrated Help generated from store/docs/.
• Adds mandatory licence acceptance plus not-for-navigation / AS-IS / no-liability acknowledgement before protected access.
• Exposes all material Store interactions through /api/v1 for third-party clients.

Payments and user library

• Adds a provider-neutral payment abstraction.
• Adds PayPal Orders v2 as the reference/demo adapter with sandbox mode by default.
• Records purchases separately from downloads.
• Adds a per-user library of purchased/downloaded DataTiles releases.
• Adds update notifications when a higher release sequence of an acquired product becomes available.
• Does not assume that a newer paid release is automatically included in an older entitlement.

Store governance

• Adds store/README.md.
• Adds store/LICENSE.md with Apache-2.0.
• Adds store/AGENTS.md with implementation and security constraints.
• Adds comprehensive store/docs/ documentation for installation, configuration, authentication, SMTP, management, API, agreements, payments, versioning, security and troubleshooting.

Compatibility

The design is additive and keeps the existing MBTiles compatibility model. Versioned DataTiles releases are immutable scientific objects; the Store maintains its own operational database and does not rewrite released DataTiles files during indexing, purchase tracking or user-management operations.

Safety and legal boundaries

• DataTiles is not an official nautical chart, ECDIS/ENC replacement or certified navigation system.
• Store users must explicitly accept current data rights/licence terms and the not-for-navigation / AS-IS agreement before protected preview/download.
• Software licensing does not relicense input datasets.
• DRM does not create commercial redistribution rights where upstream licences do not permit them.
• Payment, entitlement, licence acceptance, data provenance, cryptographic authenticity and scientific validity remain distinct concepts.

Testing

The extension bundle includes regression coverage for semantic variables, import utilities, FAIR metadata, provenance, integrity signatures, DRM, Store catalog behavior, agreement enforcement, authentication/configuration, release versioning, payments, user library and update notifications.

Before merge, apply the extension to a clean current main checkout and run:

python apply_semantic_extension.py /path/to/data-tiles
python -m pip install -e '/path/to/data-tiles[demo,utils,integrity]'
pytest -q

For the Store:

cd store
python -m pip install -e '.[prod]'
pytest -q

Migration notes

The bundle includes migrations through:

• revision 3 -> 4: semantic variable registry
• revision 4 -> 5: FAIR/publication/rights metadata
• revision 5 -> 6: integrity manifests/signatures
• revision 6 -> 7: commercial/ODRL metadata
• revision 7 -> 8: stable product identity and release versioning

Existing revision-3 datasets can migrate while preserving legacy behavior; semantic registration can be curated progressively for migrated content.

Documentation

The PR updates the main specification, revision addenda, white paper, FAIR/provenance/licensing documentation, source citation register, import-utility documentation, Store documentation and tutorial lessons through the Store/payment/versioning workflow.