Add NDP-EP documentation section for the central NDP docs site

docs/info_for_nationaldataplatform/00-what-is-an-endpoint.md

@@ -0,0 +1,84 @@
+# What is an NDP Endpoint?
+
+An **NDP Endpoint** (often shortened to **NDP-EP**, or just **EP**) is an
+institution's own data-publishing service that plugs into the National Data
+Platform. The platform is **federated by design**: instead of every institution
+uploading its data to one central place, each institution can run its own
+Endpoint that holds its own catalog and storage. An Endpoint's writers can
+selectively **publish** datasets and services upstream (through Pre-CKAN) so
+they also appear in the central NDP catalog — but the Endpoint itself remains
+independent.
+
+## What a single Endpoint provides
+
+An Endpoint is a small, self-contained service that offers:
+
+- **A web app** at `…/ep-api/ui/` where you search, browse, and (with the right
+  role) publish datasets, services, and resources.
+- **A REST API** at `…/ep-api/` (interactive docs at `…/ep-api/docs`) so the
+  same operations can be automated from code or from the
+  [`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library.
+- **A local catalog** — backed by **CKAN** or **MongoDB** — that stores the
+  metadata for the datasets, services, and resources the institution publishes.
+- **S3-compatible object storage** (optional) for large files, with a built-in
+  **S3 Management** UI for writers and admins.
+- **Identity, roles, and access requests** that reuse the user's NDP login
+  (CILogon-backed), so users do not create yet another account.
+- **Operational link to NDP**: the Endpoint periodically reports its presence
+  and operational metrics (status, version, host load, catalog counts) to the
+  platform. **Publishing data upstream** to the central catalog is a separate
+  per-item action initiated by writers (through Pre-CKAN).
+
+## Where the Endpoint fits in the platform
+
+```
+                ┌──────────────────────────────┐
+                │   National Data Platform     │   nationaldataplatform.org
+                │   (catalog · workspaces ·    │   (CILogon login, central
+                │    CollabStudio · …)         │    catalog, JupyterHub …)
+                └──────────────┬───────────────┘
+                               │ federates · discovers
+        ┌──────────────────────┼──────────────────────┐
+        ▼                      ▼                      ▼
+   ┌─────────┐            ┌─────────┐            ┌─────────┐
+   │  EP @   │            │  EP @   │            │  EP @   │
+   │ Inst. A │            │ Inst. B │            │ Inst. C │
+   └─────────┘            └─────────┘            └─────────┘
+```
+
+Each Endpoint keeps full control of **its own data and policies**. The central
+NDP catalog holds the items that EPs have chosen to publish upstream; the
+platform's operational registry simply knows which EPs exist (through their
+periodic status reports).
+
+## Who needs an Endpoint?
+
+| If you are… | You typically need… |
+|---|---|
+| A **researcher**, **educator**, or **student** consuming data on NDP | **No Endpoint.** Use the central NDP at [nationaldataplatform.org](https://nationaldataplatform.org/) to search the catalog, launch workspaces, and download data. |
+| A researcher **publishing your own data** through your institution's Endpoint | **No new Endpoint.** Use the EP that your institution already runs — see [Using an Endpoint](01-using-an-endpoint.md). |
+| An **institution, lab, or project** that wants to publish its own data into NDP under its own brand and policies | **An Endpoint of your own.** See [For institutional admins](05-for-institutional-admins.md). |
+
+## What the Endpoint is *not*
+
+- Not a place where you create users — identities come from NDP's central
+  authentication (CILogon → institutional login).
+- Not a copy of the central NDP catalog — the EP holds **your institution's**
+  catalog; items only reach the central catalog when a writer chooses to
+  publish them upstream.
+- Not a replacement for the workspaces / JupyterHub experience — those are
+  central NDP services and the EP integrates with them rather than duplicating
+  them.
+
+## Related pages
+
+- [Using an Endpoint](01-using-an-endpoint.md) — for researchers and educators.
+- [Requesting access and the role tiers](02-requesting-access-and-roles.md) —
+  how to be allowed to publish.
+- [Publishing data](03-publishing-data.md) — the `+ New` flows.
+- [Automating with Python](04-automating-with-python.md) — the `ndp-ep` library.
+- [For institutional admins](05-for-institutional-admins.md) — getting an
+  Endpoint for your organization.
+
+The source code for the Endpoint API and web app lives at
+[github.com/national-data-platform/ep-api](https://github.com/national-data-platform/ep-api).

docs/info_for_nationaldataplatform/01-using-an-endpoint.md

@@ -0,0 +1,72 @@
+# Using an Endpoint
+
+If you are a researcher, educator or student on NDP, you do not need to install
+anything to use an Endpoint: you reach it through your normal NDP login and use
+its web interface or its API.
+
+## Where you reach an Endpoint
+
+An Endpoint is **its own destination**: you reach it directly at its URL —
+typically `https://<your-institution>/ep-api/ui/`, given to you by your
+institution or by whoever shared a dataset hosted on it.
+
+Endpoints are **independent of the central NDP catalog**. The catalog at
+[nationaldataplatform.org](https://nationaldataplatform.org/) only contains
+items that an Endpoint's writers have **explicitly published upstream**, going
+through **Pre-CKAN** (staging) and into the central catalog. Datasets that
+live only on an Endpoint are not visible from the central catalog unless
+someone publishes them there.
+
+## Signing in
+
+The Endpoint reuses **your existing NDP identity**. You log in through
+**CI Logon** with your **institutional account** — the same flow as the central
+NDP. You do not create a separate Endpoint account, and commercial emails
+(Gmail, Outlook, etc.) are not eligible.
+
+Once you are logged in, the Endpoint reads your identity and your role (more on
+roles in [Requesting access and the role tiers](02-requesting-access-and-roles.md)).
+A user without a role can still **see and search public data**; write actions
+require an explicit role.
+
+## The Search experience
+
+The Endpoint's home page is **Search**. You can:
+
+- Type **free text** to match across **name, description and keywords**.
+- Switch the **category** between **All · Datasets · Services · Organizations**.
+- Choose the **catalog scope**:
+  - **Local** — only what this Endpoint publishes.
+  - **Global** — the federated NDP catalog (results from many Endpoints).
+- Filter by **organization** or toggle **Yours** to see only items you own.
+
+Results expand to show their description, tags, license, version, geographic
+extent (when present, drawn on a map) and the **resources** attached to each
+dataset.
+
+## Using a resource
+
+A dataset can carry several kinds of resources, and each kind opens in the way
+you expect:
+
+- **URL resources** — a link to a file or service (CSV, JSON, NetCDF, …). Click
+  to open or download.
+- **S3 resources** — a pointer to an object in S3-compatible storage. The
+  Endpoint can generate a temporary **presigned URL** so you can fetch the
+  object without having direct S3 credentials.
+- **Kafka topics** — a streaming data flow. The dataset page lists the broker
+  host, port and topic so you can connect with a Kafka client.
+
+## What if you want to publish?
+
+If your role is **read-only** ("viewer", or no role at all), the Endpoint will
+show your search results but will not show the **`+ New`** menu or the S3
+Management entry. To publish, you need to **request access** — see
+[Requesting access and the role tiers](02-requesting-access-and-roles.md).
+
+## Want to automate?
+
+Anything you do through the web app is also available through the Endpoint's
+HTTP API (`/ep-api/docs` for interactive documentation) and through the
+[`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library. See
+[Automating with Python](04-automating-with-python.md).

docs/info_for_nationaldataplatform/02-requesting-access-and-roles.md

@@ -0,0 +1,57 @@
+# Requesting access and the role tiers
+
+To **publish** or **modify** data on an Endpoint you need a role. Roles are
+**per-Endpoint** (you may be a writer on one Endpoint and a viewer on another)
+and they live in the central identity service, so they travel inside your NDP
+login. You do not maintain a separate password per Endpoint.
+
+## The three role tiers
+
+| Role | Can do |
+|---|---|
+| 👁️ **Viewer** | View and search the catalog (including public data anyone can see). Read-only. |
+| ✏️ **Writer** | Everything a viewer does, **plus** create / edit / delete organizations, datasets, services and resources; use the **S3 Management** tool. |
+| 🛠️ **Admin** | Everything a writer does, **plus** review access requests and reach admin-only pages (dashboard, settings). |
+
+The tiers are **hierarchical**: an admin does not also need a writer role
+assigned; a writer does not also need a viewer role assigned. You only need the
+**highest tier you want**.
+
+**Without a role**, an authenticated user can still search and read public data
+on the Endpoint, but cannot create or modify anything — this is the secure
+default.
+
+## Requesting access
+
+If you sign in and the Endpoint denies a write action — or if you simply want to
+contribute — request access from the Endpoint itself:
+
+1. Sign in with your NDP (CI Logon) account at your institution's Endpoint URL.
+2. If you have no role yet, the Endpoint shows a **Request access** form with a
+   short **justification** field — describe what you want to do and which group
+   or project you belong to.
+3. Submit. The request now appears as **pending** to the Endpoint's
+   administrators.
+
+## What the administrator does
+
+An Endpoint administrator opens the **Access Requests** page, reviews pending
+requests and either **approves** the request with a tier — **Viewer**, **Writer**
+or **Admin** — or **rejects** it with a brief reason.
+
+Approving grants the role to your account on this Endpoint.
+
+## After approval
+
+The new role takes effect when your **next token is issued** — typically on
+your next sign-in. If the Endpoint still acts as if you had no role, sign out
+and back in to refresh.
+
+If your role is removed or changed later, the same applies: the change is
+visible after the next sign-in.
+
+## Once you have a writer role
+
+You can publish through the **`+ New`** menu and manage your data on the
+**Search** page. See [Publishing data](03-publishing-data.md) for the available
+flows and the fields each one requires.

docs/info_for_nationaldataplatform/03-publishing-data.md

@@ -0,0 +1,87 @@
+# Publishing data
+
+Publishing on an Endpoint requires the **writer** (or admin) role — see
+[Requesting access and the role tiers](02-requesting-access-and-roles.md).
+
+Once you have a role, the navigation bar shows a **`+ New`** menu. The menu
+groups every creation flow the Endpoint offers — six in total, split into
+**catalog containers** and **data resources**.
+
+## At a glance
+
+| Kind | What it is |
+|---|---|
+| **Organization** | A top-level group that owns datasets and services. |
+| **Dataset** | A logical container of related resources, owned by an organization. |
+| **Service** | A network-accessible service (REST API, web app, etc.) owned by an organization. |
+| **URL resource** | A link to a file or service (CSV, JSON, NetCDF, stream, …). |
+| **S3 resource** | An object in S3-compatible storage. |
+| **Kafka topic** | A streaming data flow registered as a system dataset. |
+
+Catalog containers (Organization, Dataset, Service) are the things users
+**search and browse**. Data resources (URL, S3, Kafka) live **inside a dataset**
+and point at the actual data.
+
+## A typical publishing flow
+
+1. **Create an Organization** (once per group/lab/project).
+2. **Create a Dataset** under that Organization — give it a title, description
+   and tags.
+3. **Add one or more resources** to the dataset: a URL to a file, an object in
+   S3, or a Kafka topic — whichever describes where the data lives.
+
+You can also register **Services** (an API, a dashboard, a webhook…) under the
+special `services` organization — these appear in the catalog so other users
+can discover and call them.
+
+## Field cheat-sheet
+
+Required fields in **bold**. Slugs (`name`, `service_name`, `resource_name`,
+`dataset_name`) accept lowercase letters, digits, `_` and `-`.
+
+### Organization
+**`name`**, **`title`**, `description`.
+
+### Dataset
+**`name`**, **`title`**, **`owner_org`**, `notes`, `tags`, `groups`,
+`license_id`, `version`, `extras`, `resources`, `private`.
+
+### Service
+**`service_name`**, **`service_title`**, **`owner_org`** (must be `services`),
+**`service_url`**, `service_type` (one of **API**, **UI**, **Trigger** or
+free-text), `notes`, `health_check_url`, `documentation_url`, `extras`.
+
+### URL resource
+**`resource_name`**, **`resource_title`**, **`owner_org`**, **`resource_url`**,
+`file_type` (one of `stream`, `CSV`, `TXT`, `JSON`, `NetCDF` or custom),
+`processing` (type-specific: CSV delimiter and header line, JSON `data_key`,
+NetCDF `group`, …), `notes`, `mapping`, `extras`.
+
+### S3 resource
+**`resource_name`**, **`resource_title`**, **`owner_org`**, **`resource_s3`**
+(`s3://bucket/path` or `http(s)://…`), **`notes`** (may be empty), `extras`.
+
+### Kafka topic
+**`dataset_name`**, **`dataset_title`**, **`owner_org`**, **`kafka_topic`**,
+**`kafka_host`**, **`kafka_port`** (1–65535), **`dataset_description`**,
+`mapping`, `processing`, `extras`.
+
+## Managing what you publish
+
+On the **Search** page, your own items expose **Publish** and **Delete**
+actions. Use them to push a dataset upstream (when the Endpoint is connected
+to a staging Pre-CKAN or to the federation) or to retire it.
+
+## When the data is large
+
+For large object datasets, the writer can also use the **S3 Management** tool
+to create buckets, upload objects (drag-and-drop or file picker), generate
+**presigned URLs** for time-limited sharing, view metadata, and delete objects.
+S3 Management is restricted to writers and admins.
+
+## When you have many things to publish
+
+If you have many datasets or files to register, do it from code. The
+[`ndp-ep`](https://pypi.org/project/ndp-ep/) Python library performs every
+operation in this page from a script — see
+[Automating with Python](04-automating-with-python.md).