Vectorprep2026

.github/workflows/metadata-catalog.yml

@@ -0,0 +1,115 @@
+name: Publish Metadata Catalog
+# TWO PARTS - validate (always run) and build and publish (only on Master branch)
+# The Python CODE EXECUTED comes from riverscapes-metadata which, as defined in pyproject.toml comes from RiverscapesXML repository
+# Validate searches for the layer_definitions.json and makes sure they are compliant with the current standard
+# Build and publish makes parquet files and uploads them to S3 path for consumption by Athena
+# Only triggered on changes with specific paths. Once other tools are on board we can loosen up or remove the `paths` limitation below  
+on:
+  push:
+    branches:
+      - master
+      - dev
+    paths:
+      - "packages/vector_prep/**"
+      - ".github/workflows/metadata-catalog.yml"
+  pull_request:
+    branches:
+      - master
+      - dev
+    paths:
+      - "packages/vector_prep/**"
+      - ".github/workflows/metadata-catalog.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  id-token: write  # needed for OIDC role assumption
+
+concurrency:
+  group: metadata-catalog-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  TABLE_NAME: layer_definitions
+  S3_BASE_PATH: s3://riverscapes-athena/riverscapes_metadata/layer_definitions_raw/0.8/
+  ATHENA_DATABASE: default
+  ATHENA_RESULT_BUCKET: s3://riverscapes-athena-output/query-results/metadata  # <-- ensure this exists
+  AWS_REGION: us-west-2                        
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install uv
+        run: |
+          python -m pip install --upgrade pip
+          pip install uv
+
+      - name: Sync dependencies
+        run: |
+          uv sync --no-dev --group catalog --upgrade-package riverscapes-metadata
+
+      - name: Generate partitioned Parquet metadata
+        id: generate-metadata
+        run: |
+          uv run export-layer-definitions-for-s3 --root .
+          echo "Index manifest:"
+          cat dist/index.json | jq .
+          echo "Partition files:"
+          jq -r '.partitions[].path' dist/index.json
+
+      - name: Upload metadata artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: metadata-dist
+          path: dist/
+          retention-days: 1
+
+  build-and-publish:
+    needs: validate
+    if: github.ref == 'refs/heads/master'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Configure AWS credentials (OIDC)
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: ${{ secrets.ATHENA_S3_ACCESS_ROLE_ARN }}
+          # aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          # aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: ${{ env.AWS_REGION }}
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install uv
+        run: |
+          python -m pip install --upgrade pip
+          pip install uv
+
+      - name: Sync dependencies
+        run: |
+          uv sync --no-dev --group catalog --upgrade-package riverscapes-metadata
+
+      - name: Download metadata artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: metadata-dist
+          path: dist/
+
+      - name: Upload and process metadata
+        run: |
+          uv run publish-metadata-to-s3 --root .
+

.vscode/extensions.json

@@ -1,7 +1,6 @@
 {
   "recommendations": [
     "GitHub.copilot-chat",
-    "GitHub.copilot",
     "mhutchie.git-graph",
     "ms-azuretools.vscode-docker",
     "ms-python.autopep8",

README.md

@@ -22,7 +22,7 @@
       - [Method 2: Using QGIS Python Environment](#method-2-using-qgis-python-environment)
 
 
-This is a monorepo housing the python open-source GIS tools for Riverscapes, and the documentation site tools.riverscapes.net. Packages include:  
+This is a monorepo housing the python open-source GIS tools for Riverscapes, and the documentation site [`tools.riverscapes.net`](https://tools.riverscapes.net). Packages include:  
 
 * [Riverscapes Context](./packages/rscontext)
 * [BRAT](./packages/brat)
@@ -98,7 +98,7 @@ Make sure you use the folder name that matches what this tool expects for the pr
 
 ## Documentation Site
 
-The documentation site is built with [Docusaurus](https://docusaurus.io/) and published  from the **`docs` branch** to [tools.riverscapes.net](https://tools.riverscapes.net).
+The documentation site is built with [Docusaurus](https://docusaurus.io/) and published from the **`docs` branch** to [tools.riverscapes.net](https://tools.riverscapes.net).
 
 ## Editing & preview changes using VS Code
 
@@ -110,16 +110,20 @@ yarn install
 yarn start
 ```
 
-
------------------------------------
+--------------------------------------
 
 ## Using UV for Environment Management
 
 This project uses [uv](https://github.com/astral-sh/uv) to manage Python virtual environments and dependencies. `uv` is an alternative to tools like `pipenv` and `poetry`.
 
-
 ## Environment Setup
 
+The UV *dependency groups* are for packages required only for local/CI use. These are not part of the package, so when users (or Cybercastor) run `pip install riverscapes-tools`, they won't get these extra libraries.  
+
+When you run `uv sync` for development, you get the development group by default. Any other groups will only be installed if explicitly called for, e.g. with `uv sync --group catalog`.
+
+Confusingly, in addition to these 'development dependencies' called groups, UV has 'optional dependencies', called extras ([reference](https://docs.astral.sh/uv/concepts/projects/dependencies/)). We aren't currently using optional dependencies in this repo.
+
 ### Prerequisites
 
 1. Install `uv` by following the [installation instructions](https://github.com/astral-sh/uv#installation) for your operating system.

Workspaces/Vector-Prep.code-workspace

@@ -22,6 +22,14 @@
     }
   ],
   "settings": {
+    "json.schemas": [
+      {
+        "fileMatch": [
+          "**/layer_definitions.json"
+        ],
+        "url": "https://xml.riverscapes.net/riverscapes_metadata/schema/layer_definitions.schema.json"
+      }
+    ],
     "python.envFile": "${workspaceFolder}/.env",
     "terminal.integrated.defaultProfile.linux": "zsh",
     "[python]": {
@@ -61,7 +69,7 @@
       "**/.venv/**": true,
       "**/__pycache__/**": true
     },
-  "python.defaultInterpreterPath": "${workspaceFolder:🏠 Git Root}/.venv/bin/python",
+    "python.defaultInterpreterPath": "${workspaceFolder:🏠 Git Root}/.venv/bin/python",
     "python.testing.autoTestDiscoverOnSaveEnabled": false
   }
-}
+}

packages/vector_prep/README.md

@@ -0,0 +1,18 @@
+# Vector Prep Tool
+
+The purpose of the tool is to provide consistent and reproducible preparation of (usually public Government data) vector data for use in the Riverscapes ecosystem.
+
+Example preparation includes:
+
+* checking and fixing for geometry errors, null or invalid etc (TO BE BETTER DOCUMENTED)
+* simplify as needed
+* Documentation and Metadata enrichment - ingesting any existing metadata (eg. in an ISO xml format), source urls etc., making sure it is referenced/included and also translating it into the riverscapes metadata `.json` format, published at [xml.riverscapes.net](https://xml.riverscapes.net/riverscapes_metadata/schema/layer_definitions.schema.json) and documented on [docs.riverscapes.net](https://docs.riverscapes.net/standards/metadata) so it can be published to the central S3/Athena repository.
+
+Store run manifests (json files documenting what was done/changed) in processed_data
+If we move to more automated & frequent data processing we'll revisit this - the logs will probably in s3, AWS glue will help etc.
+
+Planned, not yet implemented:
+
+* Identify/Verify a primary key field that can be used for joins. Could be composite of existing fields.
+* Identify/Verify a primary name field that can be used for labels. could be formulaic composite of multiple existing fields -- implemented in a view perhaps rather than materialized.
+* Export to parquet in WGS84 for use in reporting

packages/vector_prep/__version__.py

@@ -0,0 +1 @@
+__version__ = "0.1.1"

packages/vector_prep/changelog.md

@@ -0,0 +1,5 @@
+# Vector Prep Changelog
+
+## 0.1.1
+
+Beginning to make this a module that can help prepare data for use in reporting data platform (Athena), including enriching it with metadata.

packages/vector_prep/plan-addUniqueIdAndNameColumns.prompt.md

@@ -0,0 +1,20 @@
+## Plan: Add Unique ID and Name Columns to New Vector Layer
+
+The script will add `rs_reports_rowid` (UUID) and `rs_reports_rowname` (user-supplied/calculated) columns to a new layer in a GeoPackage. It will abort and report if any `rs_reports_rowname` values are not unique or exceed 150 characters.
+
+### Steps
+1. Accept input: GeoPackage path, source layer, new layer name, rowname expression/columns.
+2. Open the source layer using `geopandas`.
+3. Prepare new schema with added columns.
+4. For each feature:
+    - Generate UUID for `rs_reports_rowid`.
+    - Compute `rs_reports_rowname` using the provided expression/columns.
+5. Collect and check all `rs_reports_rowname` values for uniqueness and length.
+6. If violations exist, report details and abort.
+7. If valid, write features to the new layer in the GeoPackage.
+
+### Further Considerations
+1. Rowname expression option:
+    - Python string templates (e.g., "{STATE_NAME} {NAMELSAD}").
+2. Keep first version minimal. Report errors immediately rather than building in fallbacks. 
+