diff --git a/docs/info_for_nationaldataplatform/00-what-is-an-endpoint.md b/docs/info_for_nationaldataplatform/00-what-is-an-endpoint.md new file mode 100644 index 0000000..c5b00c7 --- /dev/null +++ b/docs/info_for_nationaldataplatform/00-what-is-an-endpoint.md @@ -0,0 +1,84 @@ +# What is an NDP Endpoint? + +An **NDP Endpoint** (often shortened to **NDP-EP**, or just **EP**) is an +institution's own data-publishing service that plugs into the National Data +Platform. The platform is **federated by design**: instead of every institution +uploading its data to one central place, each institution can run its own +Endpoint that holds its own catalog and storage. An Endpoint's writers can +selectively **publish** datasets and services upstream (through Pre-CKAN) so +they also appear in the central NDP catalog — but the Endpoint itself remains +independent. + +## What a single Endpoint provides + +An Endpoint is a small, self-contained service that offers: + +- **A web app** at `…/ep-api/ui/` where you search, browse, and (with the right + role) publish datasets, services, and resources. +- **A REST API** at `…/ep-api/` (interactive docs at `…/ep-api/docs`) so the + same operations can be automated from code or from the + [`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library. +- **A local catalog** — backed by **CKAN** or **MongoDB** — that stores the + metadata for the datasets, services, and resources the institution publishes. +- **S3-compatible object storage** (optional) for large files, with a built-in + **S3 Management** UI for writers and admins. +- **Identity, roles, and access requests** that reuse the user's NDP login + (CILogon-backed), so users do not create yet another account. +- **Operational link to NDP**: the Endpoint periodically reports its presence + and operational metrics (status, version, host load, catalog counts) to the + platform. **Publishing data upstream** to the central catalog is a separate + per-item action initiated by writers (through Pre-CKAN). + +## Where the Endpoint fits in the platform + +``` + ┌──────────────────────────────┐ + │ National Data Platform │ nationaldataplatform.org + │ (catalog · workspaces · │ (CILogon login, central + │ CollabStudio · …) │ catalog, JupyterHub …) + └──────────────┬───────────────┘ + │ federates · discovers + ┌──────────────────────┼──────────────────────┐ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ EP @ │ │ EP @ │ │ EP @ │ + │ Inst. A │ │ Inst. B │ │ Inst. C │ + └─────────┘ └─────────┘ └─────────┘ +``` + +Each Endpoint keeps full control of **its own data and policies**. The central +NDP catalog holds the items that EPs have chosen to publish upstream; the +platform's operational registry simply knows which EPs exist (through their +periodic status reports). + +## Who needs an Endpoint? + +| If you are… | You typically need… | +|---|---| +| A **researcher**, **educator**, or **student** consuming data on NDP | **No Endpoint.** Use the central NDP at [nationaldataplatform.org](https://nationaldataplatform.org/) to search the catalog, launch workspaces, and download data. | +| A researcher **publishing your own data** through your institution's Endpoint | **No new Endpoint.** Use the EP that your institution already runs — see [Using an Endpoint](01-using-an-endpoint.md). | +| An **institution, lab, or project** that wants to publish its own data into NDP under its own brand and policies | **An Endpoint of your own.** See [For institutional admins](05-for-institutional-admins.md). | + +## What the Endpoint is *not* + +- Not a place where you create users — identities come from NDP's central + authentication (CILogon → institutional login). +- Not a copy of the central NDP catalog — the EP holds **your institution's** + catalog; items only reach the central catalog when a writer chooses to + publish them upstream. +- Not a replacement for the workspaces / JupyterHub experience — those are + central NDP services and the EP integrates with them rather than duplicating + them. + +## Related pages + +- [Using an Endpoint](01-using-an-endpoint.md) — for researchers and educators. +- [Requesting access and the role tiers](02-requesting-access-and-roles.md) — + how to be allowed to publish. +- [Publishing data](03-publishing-data.md) — the `+ New` flows. +- [Automating with Python](04-automating-with-python.md) — the `ndp-ep` library. +- [For institutional admins](05-for-institutional-admins.md) — getting an + Endpoint for your organization. + +The source code for the Endpoint API and web app lives at +[github.com/national-data-platform/ep-api](https://github.com/national-data-platform/ep-api). diff --git a/docs/info_for_nationaldataplatform/01-using-an-endpoint.md b/docs/info_for_nationaldataplatform/01-using-an-endpoint.md new file mode 100644 index 0000000..f1bf917 --- /dev/null +++ b/docs/info_for_nationaldataplatform/01-using-an-endpoint.md @@ -0,0 +1,72 @@ +# Using an Endpoint + +If you are a researcher, educator or student on NDP, you do not need to install +anything to use an Endpoint: you reach it through your normal NDP login and use +its web interface or its API. + +## Where you reach an Endpoint + +An Endpoint is **its own destination**: you reach it directly at its URL — +typically `https:///ep-api/ui/`, given to you by your +institution or by whoever shared a dataset hosted on it. + +Endpoints are **independent of the central NDP catalog**. The catalog at +[nationaldataplatform.org](https://nationaldataplatform.org/) only contains +items that an Endpoint's writers have **explicitly published upstream**, going +through **Pre-CKAN** (staging) and into the central catalog. Datasets that +live only on an Endpoint are not visible from the central catalog unless +someone publishes them there. + +## Signing in + +The Endpoint reuses **your existing NDP identity**. You log in through +**CI Logon** with your **institutional account** — the same flow as the central +NDP. You do not create a separate Endpoint account, and commercial emails +(Gmail, Outlook, etc.) are not eligible. + +Once you are logged in, the Endpoint reads your identity and your role (more on +roles in [Requesting access and the role tiers](02-requesting-access-and-roles.md)). +A user without a role can still **see and search public data**; write actions +require an explicit role. + +## The Search experience + +The Endpoint's home page is **Search**. You can: + +- Type **free text** to match across **name, description and keywords**. +- Switch the **category** between **All · Datasets · Services · Organizations**. +- Choose the **catalog scope**: + - **Local** — only what this Endpoint publishes. + - **Global** — the federated NDP catalog (results from many Endpoints). +- Filter by **organization** or toggle **Yours** to see only items you own. + +Results expand to show their description, tags, license, version, geographic +extent (when present, drawn on a map) and the **resources** attached to each +dataset. + +## Using a resource + +A dataset can carry several kinds of resources, and each kind opens in the way +you expect: + +- **URL resources** — a link to a file or service (CSV, JSON, NetCDF, …). Click + to open or download. +- **S3 resources** — a pointer to an object in S3-compatible storage. The + Endpoint can generate a temporary **presigned URL** so you can fetch the + object without having direct S3 credentials. +- **Kafka topics** — a streaming data flow. The dataset page lists the broker + host, port and topic so you can connect with a Kafka client. + +## What if you want to publish? + +If your role is **read-only** ("viewer", or no role at all), the Endpoint will +show your search results but will not show the **`+ New`** menu or the S3 +Management entry. To publish, you need to **request access** — see +[Requesting access and the role tiers](02-requesting-access-and-roles.md). + +## Want to automate? + +Anything you do through the web app is also available through the Endpoint's +HTTP API (`/ep-api/docs` for interactive documentation) and through the +[`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library. See +[Automating with Python](04-automating-with-python.md). diff --git a/docs/info_for_nationaldataplatform/02-requesting-access-and-roles.md b/docs/info_for_nationaldataplatform/02-requesting-access-and-roles.md new file mode 100644 index 0000000..d2e2aaa --- /dev/null +++ b/docs/info_for_nationaldataplatform/02-requesting-access-and-roles.md @@ -0,0 +1,57 @@ +# Requesting access and the role tiers + +To **publish** or **modify** data on an Endpoint you need a role. Roles are +**per-Endpoint** (you may be a writer on one Endpoint and a viewer on another) +and they live in the central identity service, so they travel inside your NDP +login. You do not maintain a separate password per Endpoint. + +## The three role tiers + +| Role | Can do | +|---|---| +| 👁️ **Viewer** | View and search the catalog (including public data anyone can see). Read-only. | +| ✏️ **Writer** | Everything a viewer does, **plus** create / edit / delete organizations, datasets, services and resources; use the **S3 Management** tool. | +| 🛠️ **Admin** | Everything a writer does, **plus** review access requests and reach admin-only pages (dashboard, settings). | + +The tiers are **hierarchical**: an admin does not also need a writer role +assigned; a writer does not also need a viewer role assigned. You only need the +**highest tier you want**. + +**Without a role**, an authenticated user can still search and read public data +on the Endpoint, but cannot create or modify anything — this is the secure +default. + +## Requesting access + +If you sign in and the Endpoint denies a write action — or if you simply want to +contribute — request access from the Endpoint itself: + +1. Sign in with your NDP (CI Logon) account at your institution's Endpoint URL. +2. If you have no role yet, the Endpoint shows a **Request access** form with a + short **justification** field — describe what you want to do and which group + or project you belong to. +3. Submit. The request now appears as **pending** to the Endpoint's + administrators. + +## What the administrator does + +An Endpoint administrator opens the **Access Requests** page, reviews pending +requests and either **approves** the request with a tier — **Viewer**, **Writer** +or **Admin** — or **rejects** it with a brief reason. + +Approving grants the role to your account on this Endpoint. + +## After approval + +The new role takes effect when your **next token is issued** — typically on +your next sign-in. If the Endpoint still acts as if you had no role, sign out +and back in to refresh. + +If your role is removed or changed later, the same applies: the change is +visible after the next sign-in. + +## Once you have a writer role + +You can publish through the **`+ New`** menu and manage your data on the +**Search** page. See [Publishing data](03-publishing-data.md) for the available +flows and the fields each one requires. diff --git a/docs/info_for_nationaldataplatform/03-publishing-data.md b/docs/info_for_nationaldataplatform/03-publishing-data.md new file mode 100644 index 0000000..5332305 --- /dev/null +++ b/docs/info_for_nationaldataplatform/03-publishing-data.md @@ -0,0 +1,87 @@ +# Publishing data + +Publishing on an Endpoint requires the **writer** (or admin) role — see +[Requesting access and the role tiers](02-requesting-access-and-roles.md). + +Once you have a role, the navigation bar shows a **`+ New`** menu. The menu +groups every creation flow the Endpoint offers — six in total, split into +**catalog containers** and **data resources**. + +## At a glance + +| Kind | What it is | +|---|---| +| **Organization** | A top-level group that owns datasets and services. | +| **Dataset** | A logical container of related resources, owned by an organization. | +| **Service** | A network-accessible service (REST API, web app, etc.) owned by an organization. | +| **URL resource** | A link to a file or service (CSV, JSON, NetCDF, stream, …). | +| **S3 resource** | An object in S3-compatible storage. | +| **Kafka topic** | A streaming data flow registered as a system dataset. | + +Catalog containers (Organization, Dataset, Service) are the things users +**search and browse**. Data resources (URL, S3, Kafka) live **inside a dataset** +and point at the actual data. + +## A typical publishing flow + +1. **Create an Organization** (once per group/lab/project). +2. **Create a Dataset** under that Organization — give it a title, description + and tags. +3. **Add one or more resources** to the dataset: a URL to a file, an object in + S3, or a Kafka topic — whichever describes where the data lives. + +You can also register **Services** (an API, a dashboard, a webhook…) under the +special `services` organization — these appear in the catalog so other users +can discover and call them. + +## Field cheat-sheet + +Required fields in **bold**. Slugs (`name`, `service_name`, `resource_name`, +`dataset_name`) accept lowercase letters, digits, `_` and `-`. + +### Organization +**`name`**, **`title`**, `description`. + +### Dataset +**`name`**, **`title`**, **`owner_org`**, `notes`, `tags`, `groups`, +`license_id`, `version`, `extras`, `resources`, `private`. + +### Service +**`service_name`**, **`service_title`**, **`owner_org`** (must be `services`), +**`service_url`**, `service_type` (one of **API**, **UI**, **Trigger** or +free-text), `notes`, `health_check_url`, `documentation_url`, `extras`. + +### URL resource +**`resource_name`**, **`resource_title`**, **`owner_org`**, **`resource_url`**, +`file_type` (one of `stream`, `CSV`, `TXT`, `JSON`, `NetCDF` or custom), +`processing` (type-specific: CSV delimiter and header line, JSON `data_key`, +NetCDF `group`, …), `notes`, `mapping`, `extras`. + +### S3 resource +**`resource_name`**, **`resource_title`**, **`owner_org`**, **`resource_s3`** +(`s3://bucket/path` or `http(s)://…`), **`notes`** (may be empty), `extras`. + +### Kafka topic +**`dataset_name`**, **`dataset_title`**, **`owner_org`**, **`kafka_topic`**, +**`kafka_host`**, **`kafka_port`** (1–65535), **`dataset_description`**, +`mapping`, `processing`, `extras`. + +## Managing what you publish + +On the **Search** page, your own items expose **Publish** and **Delete** +actions. Use them to push a dataset upstream (when the Endpoint is connected +to a staging Pre-CKAN or to the federation) or to retire it. + +## When the data is large + +For large object datasets, the writer can also use the **S3 Management** tool +to create buckets, upload objects (drag-and-drop or file picker), generate +**presigned URLs** for time-limited sharing, view metadata, and delete objects. +S3 Management is restricted to writers and admins. + +## When you have many things to publish + +If you have many datasets or files to register, do it from code. The +[`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library performs every +operation in this page from a script — see +[Automating with Python](04-automating-with-python.md). diff --git a/docs/info_for_nationaldataplatform/04-automating-with-python.md b/docs/info_for_nationaldataplatform/04-automating-with-python.md new file mode 100644 index 0000000..187b1fe --- /dev/null +++ b/docs/info_for_nationaldataplatform/04-automating-with-python.md @@ -0,0 +1,122 @@ +# Automating with Python + +Every operation an Endpoint exposes through its web app is also available from +code, either by calling its **HTTP API** directly (interactive docs at +`…/ep-api/docs`) or — more conveniently — through the +[`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library. + +This is the path to take when you have **many** items to register, you want to +**schedule** ingestion, or you want to wire publishing into a CI pipeline or an +ETL job. + +## Install + +```bash +pip install ndp-ep +``` + +The library targets Python 3.10+ and has no native dependencies beyond +`requests`. + +## Connect + +You authenticate once and reuse the client for all calls. You can either pass a +bearer token you already have, or sign in with username/password: + +```python +from ndp_ep import APIClient + +# Option A — pass a bearer token (e.g. from your NDP session) +client = APIClient( + base_url="https://my-endpoint.example.org/ep-api", + token="", +) + +# Option B — sign in with username/password +client = APIClient(base_url="https://my-endpoint.example.org/ep-api") +client.get_token(username="me@my-institution.edu", password="…") +``` + +`base_url` is the same root the web app uses, **without** the `/ui/`. The +client honours the Endpoint's role checks — what your token can do from code is +exactly what your account can do from the web. + +## Common operations + +```python +# Browse organizations and search the catalog +client.list_organizations() +client.search_datasets("storm radar") + +# Create a dataset +client.register_dataset( + name="storm-radar-2025", + title="Storm radar reflectivity 2025", + owner_org="atmospheric-research", + notes="Hourly NEXRAD Level-II composites over CONUS.", + tags=["radar", "nexrad", "2025"], +) + +# Register a URL resource for that dataset +client.register_url( + resource_name="radar-jan-2025", + resource_title="Radar reflectivity — January 2025", + owner_org="atmospheric-research", + resource_url="https://data.example.org/radar/2025-01.nc", + file_type="NetCDF", +) + +# Register an S3 resource +client.register_s3( + resource_name="radar-archive-2025", + resource_title="NEXRAD radar archive, 2025", + owner_org="atmospheric-research", + resource_s3="s3://nexrad-archive/2025/", + notes="Annual archive, partitioned by month.", +) + +# Register a Kafka topic +client.register_kafka( + dataset_name="nexrad-live", + dataset_title="NEXRAD radar — live stream", + owner_org="atmospheric-research", + kafka_topic="nexrad.live", + kafka_host="kafka.example.org", + kafka_port=9092, + dataset_description="Live JSON feed of NEXRAD volume scans.", +) +``` + +## S3 management from code + +```python +# Buckets +client.list_buckets() +client.create_bucket("radar-archive-2025") +client.get_bucket_info("radar-archive-2025") +client.delete_bucket("radar-archive-2025") # the bucket must be empty + +# Objects +client.download_object("radar-archive-2025", "2025-01.nc") +``` + +The library mirrors the Endpoint's S3 Management UI: uploads, presigned URLs, +metadata reads, and deletes all have method equivalents. Like the UI, they +require the **writer** (or admin) role. + +## Typical use cases + +- **Bulk import.** Walk a directory tree, create one dataset per top-level + folder, and register each file inside as a URL or S3 resource. +- **Recurring ingestion.** Run a script on a schedule to register new outputs + of an instrument or pipeline. +- **CI integration.** Have a CI job publish a versioned dataset after a build. +- **ETL bridges.** Use the catalog as the source of truth for downstream + processing — read the dataset list from the Endpoint, then process each + resource. + +## Where to find more + +- The library on PyPI: . +- The library source: . +- The Endpoint's interactive HTTP API: `/ep-api/docs`. diff --git a/docs/info_for_nationaldataplatform/05-for-institutional-admins.md b/docs/info_for_nationaldataplatform/05-for-institutional-admins.md new file mode 100644 index 0000000..3a8f8ad --- /dev/null +++ b/docs/info_for_nationaldataplatform/05-for-institutional-admins.md @@ -0,0 +1,103 @@ +# For institutional admins + +An **NDP Endpoint** lets an institution publish its own data into the National +Data Platform under its own brand and policies, while reusing NDP's shared +identity and federation. This page is for the IT, dev-ops or data team that +runs the Endpoint on behalf of the institution. + +## Should we run an Endpoint? + +Run one if **any** of these applies: + +- Your institution wants its datasets and services to appear in NDP under its + own name, with its own organizations and curators. +- You need a place where institution members can publish data through a + familiar web UI and a Python library, without writing to a shared central + catalog. +- You want object-storage management (S3-compatible) integrated with the + catalog so large data products live next to their metadata. +- You want a Kafka-based stream of data to be discoverable from NDP. + +If none of the above applies and your users only **consume** data, you do not +need an Endpoint — they can use the central NDP directly. + +## The platform / institution split + +| The platform provides | You provide | +|---|---| +| **Identity** (AAI, CI Logon, role tiers) | The host (single Linux machine with Docker) | +| **Affinities** registration (your Endpoint's `EP_UUID`) | A **catalog database** (MongoDB or an existing CKAN) | +| **Federation** registration & periodic discovery | **Object storage** (MinIO or an existing S3) — optional | +| The `nationaldataplatform.org` shared experience | The institution's **branding** and content policies | + +In practice, the NDP team gives you what you need from their side as part of +**onboarding**; you do not have to provision identity or federation from +scratch. + +## How to obtain an Endpoint + +1. **Get an institutional NDP account** (CI Logon, institutional email — see + the central [Registration and Login](https://nationaldataplatform.org/documentation/registration/) + docs). +2. **Contact the NDP team** to start the onboarding for your institution. + The team registers your Endpoint in the federation and gives you back: + - the Endpoint's **`EP_UUID`** in Affinities, + - access credentials for the shared catalogs your Endpoint will need to + talk to (CKAN and / or Pre-CKAN admin tokens), + - the AAI configuration values (`AUTH_API_URL`, etc.) that the Endpoint + should call. +3. **Install the Endpoint on your host**. The source and the install + instructions live at + [github.com/national-data-platform/ep-api](https://github.com/national-data-platform/ep-api). + The Endpoint ships as a single Docker image (API + web UI). Depending on + your needs, you also bring up a catalog database (MongoDB or CKAN) and + optionally S3-compatible storage; the project's Docker Compose file uses + **profiles** so you can opt-in piece by piece. +4. **Provide `.env` values.** Every configuration variable is documented in the + project's `docs/configuration.md`, including which ones are required, what + each one is for, and where to get the value. +5. **Bootstrap the first administrator.** Once the Endpoint is running, the + first admin needs the `ndp_admin` role assigned in AAI — typically handled + by the NDP team during onboarding. After that, the admin grants roles to + other users through the Endpoint's **Access Requests** page. + +## What runs day to day + +- The Endpoint serves the **web app** and **HTTP API** at `…/ep-api/`. +- It validates user tokens against AAI on every request. +- Every ~55 minutes it sends a small **metrics payload** to the federation — + identity, EP version, host load (CPU/memory/disk), catalog activity + (`num_datasets`, `num_services`, service titles), and infrastructure flags + (S3 / Kafka / JupyterLab / Pre-CKAN enabled). **No tokens, user data or + dataset content** are sent. The Endpoint is non-blocking on this: if the + federation is unreachable, the Endpoint keeps working and retries. + +## Optional integrations + +- **JupyterLab** — surface a JupyterLab link in the Endpoint UI. +- **Kafka** — register and stream-publish Kafka topics as system datasets. +- **Pelican federation** — browse / download from external Pelican federations + (OSDF, etc.). +- **NetBird** ([netbird.io](https://netbird.io)) — when the Endpoint and its + backends run on different machines, run a NetBird mesh VPN so they talk over + a private encrypted overlay without exposing public ports. + +## Operational notes + +- **Secrets.** Tokens (CKAN, Pre-CKAN, S3) and the store encryption key live in + the host's `.env`. Back them up; rotate them through the AAI team if they are + ever exposed. +- **Backups.** Back up the catalog database (MongoDB or CKAN) on your normal + institutional schedule. The Endpoint itself is stateless. +- **Updates.** New Endpoint releases are published on Docker Hub. Upgrades are + a `docker compose pull && docker compose up -d` once your `.env` and version + pinning are in place. +- **Privacy.** Only metadata (titles, descriptions, counts) and infrastructure + flags are sent to the federation. Dataset content stays on the institution's + own storage. + +## Where to start + +The single most useful link for an admin is the Endpoint repository — it +contains the Docker image, the Compose definition, the configuration reference +and a step-by-step demo presentation: . diff --git a/docs/info_for_nationaldataplatform/README.md b/docs/info_for_nationaldataplatform/README.md new file mode 100644 index 0000000..bf6aca5 --- /dev/null +++ b/docs/info_for_nationaldataplatform/README.md @@ -0,0 +1,26 @@ +# `info_for_nationaldataplatform/` + +Drafts of documentation pages about the **NDP Endpoint (EP)**, written to slot +into the central NDP documentation site at +[nationaldataplatform.org/documentation/](https://nationaldataplatform.org/documentation/) +(suggested location: `/documentation/ndp-ep/...`). + +These files are plain Markdown — no site-specific front-matter — so they can be +included by any static site generator the documentation site uses. + +## Pages + +| # | File | Audience | What it covers | +|---|---|---|---| +| 00 | [`00-what-is-an-endpoint.md`](00-what-is-an-endpoint.md) | Everyone | What an NDP-EP is and where it fits in the federation. | +| 01 | [`01-using-an-endpoint.md`](01-using-an-endpoint.md) | Researchers, educators | Finding and using an Endpoint as a logged-in NDP user. | +| 02 | [`02-requesting-access-and-roles.md`](02-requesting-access-and-roles.md) | Users who want to publish | The access-request workflow and the viewer / writer / admin tiers. | +| 03 | [`03-publishing-data.md`](03-publishing-data.md) | Writers | The `+ New` flows — organizations, datasets, URL / S3 / Kafka resources, services. | +| 04 | [`04-automating-with-python.md`](04-automating-with-python.md) | Power users, data engineers | Using the `ndp-ep` Python library for automation and bulk loading. | +| 05 | [`05-for-institutional-admins.md`](05-for-institutional-admins.md) | Institutional IT, data-ops | How an institution obtains an Endpoint, what NDP provides vs. what the institution provides. | + +## Status + +These are drafts maintained alongside the Endpoint API source, so changes to the +EP (UI, API, configuration) keep them in sync. When the central documentation +site picks them up, the canonical copy lives there.