From 3311e75396ddd8ee1b2dec51c083c20ff3ccaccd Mon Sep 17 00:00:00 2001 From: DawMatt Date: Thu, 2 Jul 2026 23:31:59 +1000 Subject: [PATCH 01/19] Setup initial specification --- .specify/feature.json | 2 +- .../checklists/requirements.md | 37 ++++++ specs/004-lab-4-auto-registration/spec.md | 110 ++++++++++++++++++ 3 files changed, 148 insertions(+), 1 deletion(-) create mode 100644 specs/004-lab-4-auto-registration/checklists/requirements.md create mode 100644 specs/004-lab-4-auto-registration/spec.md diff --git a/.specify/feature.json b/.specify/feature.json index 1476533..aee6f97 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1,3 +1,3 @@ { - "feature_directory": "specs/003-lab-3-api-quality" + "feature_directory": "specs/004-lab-4-auto-registration" } diff --git a/specs/004-lab-4-auto-registration/checklists/requirements.md b/specs/004-lab-4-auto-registration/checklists/requirements.md new file mode 100644 index 0000000..35d33a7 --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Lab 4 - Auto Registration + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-02 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Items marked incomplete require spec updates before `/speckit-clarify` or `/speckit-plan`. +- No [NEEDS CLARIFICATION] markers were needed: reasonable, documented defaults were used for + mono-repo simulation, discovery cadence, and catalog-info.yaml vs. x-* precedence — all + recorded in the spec's Assumptions section per Constitution Principle VIII. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md new file mode 100644 index 0000000..4b273ca --- /dev/null +++ b/specs/004-lab-4-auto-registration/spec.md @@ -0,0 +1,110 @@ +# Feature Specification: Lab 4 - Auto Registration + +**Feature Branch**: `004-lab-4-auto-registration` + +**Created**: 2026-07-02 + +**Status**: Draft + +**Input**: User description: "Create new feature and specification from feature 4 in GOAL.md" — Lab 4 (Auto registration): Setup a process for automatic registration of APIs and creation of catalog metadata. Assume API definition files exist within a Git mono-repo. Include sourcing of metadata from x-* fields, within the API specifications themselves, into catalog metadata, fields and annotations. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) + +As a platform engineer, when a new API definition file (OpenAPI or AsyncAPI) is added anywhere within the designated mono-repo location, I want it to appear in the Backstage catalog as a registered API entity without anyone hand-authoring a `catalog-info.yaml` for it. + +**Why this priority**: This is the core value of the lab — removing manual catalog registration is the entire premise of "auto registration." Without this, there is no lab. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file to the mono-repo location, trigger a catalog refresh, and confirm a corresponding API entity appears in the Backstage catalog with no manual `catalog-info.yaml` written for that API. + +**Acceptance Scenarios**: + +1. **Given** a mono-repo containing one or more OpenAPI/AsyncAPI definition files with no matching catalog entity, **When** the catalog discovery process runs, **Then** a catalog API entity is created for each definition file found. +2. **Given** an API definition file that was previously registered, **When** its contents change (e.g., description or version updated) and discovery runs again, **Then** the existing catalog entity is updated in place rather than duplicated. +3. **Given** an API definition file is removed from the mono-repo, **When** discovery runs again, **Then** the corresponding catalog entity is no longer present as an active, discoverable entity. + +--- + +### User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or tags) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. + +**Why this priority**: This is what makes registration genuinely "automatic" rather than just file discovery — without metadata sourcing, every API would still need a hand-maintained catalog file for ownership and lifecycle, which is exactly what Lab 2's ownership model depends on. + +**Independent Test**: Author an API definition containing `x-*` fields for owning team and lifecycle, run discovery, and confirm the resulting catalog entity's `spec.owner`, `spec.lifecycle`, and equivalent fields/annotations reflect the values from the `x-*` fields — not placeholder or default values. + +**Acceptance Scenarios**: + +1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, tags, system grouping), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry. +3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. +4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. + +--- + +### User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +As a learner adapting this repository to my own organization, I want to understand which parts of the discovery and metadata-sourcing configuration are conventions I can change (e.g., file naming patterns, folder layout, `x-*` field names) versus fixed requirements, so I can apply this pattern to my own mono-repo layout. + +**Why this priority**: Reinforces Constitution Principle VIII (Support Experimentation) — the lab must remain valuable to learners whose mono-repo doesn't match the example layout exactly. Lower priority than P1/P2 because the mechanism must exist and work correctly before its adaptability can be documented. + +**Independent Test**: Follow the lab documentation's guidance to point discovery at a differently-structured example folder (different glob pattern, different `x-*` field names) and confirm registration still works as described. + +**Acceptance Scenarios**: + +1. **Given** the lab documentation, **When** a learner reads the configuration section, **Then** they can identify which settings (file location pattern, recognized `x-*` field names) are conventions they are expected to adapt, and which are fixed mechanics of the discovery process. + +--- + +### Edge Cases + +- What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. +- What happens when two API definition files resolve to the same catalog entity name? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. +- What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. +- What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. +- How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) located within a designated mono-repo folder structure, without requiring a hand-authored `catalog-info.yaml` per API. +- **FR-002**: The system MUST automatically create a catalog API entity for each newly discovered API definition file. +- **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. +- **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. +- **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields and map them into the corresponding catalog entity fields or annotations. +- **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. +- **FR-008**: The system MUST surface a visible, actionable error when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition — and MUST continue processing remaining, unaffected API definitions. +- **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. +- **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. +- **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. +- **FR-012**: The lab MUST run entirely on local, freely available tooling with $0 cost, requiring no paid service, hosted CI/CD, or webhook-reachable public endpoint (Constitution Principle V). +- **FR-013**: The lab MUST build upon the environment established by Labs 1-3 (base Backstage setup, users/roles/teams and ownership-based visibility, API quality tooling) without requiring those labs to be redone. + +### Key Entities + +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **Mono-repo Location**: The designated folder structure/convention within which API Definition Files are expected to live, used as the scope for discovery. +- **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. +- **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. +- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error, exactly which file and problem caused registration to fail, without needing to inspect system internals. +- **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. +- **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. + +## Assumptions + +- The "Git mono-repo" referenced in Lab 4 is simulated as a folder/convention within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. +- "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. +- Default `x-*` field names (e.g., for owning team, lifecycle, tags) will be defined by this lab as a documented convention; learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. +- When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. +- Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 04a48b063e45175f935a59fcffe7248227882896 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 07:09:11 +1000 Subject: [PATCH 02/19] API auto registration feature planned --- CLAUDE.md | 2 +- .../004-lab-4-auto-registration/data-model.md | 57 +++++++ specs/004-lab-4-auto-registration/plan.md | 148 ++++++++++++++++++ .../004-lab-4-auto-registration/quickstart.md | 45 ++++++ specs/004-lab-4-auto-registration/research.md | 136 ++++++++++++++++ specs/004-lab-4-auto-registration/spec.md | 28 ++-- 6 files changed, 406 insertions(+), 10 deletions(-) create mode 100644 specs/004-lab-4-auto-registration/data-model.md create mode 100644 specs/004-lab-4-auto-registration/plan.md create mode 100644 specs/004-lab-4-auto-registration/quickstart.md create mode 100644 specs/004-lab-4-auto-registration/research.md diff --git a/CLAUDE.md b/CLAUDE.md index adb4816..e465dfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,5 +1,5 @@ For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan -at specs/003-lab-3-api-quality/plan.md +at specs/004-lab-4-auto-registration/plan.md diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md new file mode 100644 index 0000000..abe48d4 --- /dev/null +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -0,0 +1,57 @@ +# Phase 1 Data Model: Lab 4 — Auto Registration + +This lab has no application database. "Data model" here means the shape of the catalog entities +produced, the config schema that drives discovery, and the mapping rules between an API +definition file and the resulting Backstage `API` entity. + +## Entity: API Definition File (input, not a catalog entity) + +A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`). + +| Field | Type | Notes | +|---|---|---| +| `openapi` / `asyncapi` | string | Version field; presence of either identifies the spec type. Exactly one is expected. | +| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. | +| `info.description` | string | Optional. Maps to `metadata.description` if present. | +| `info.x-backstage-owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration proceeds with no `spec.owner` override... actually **required by FR-005/FR-007's documented default** — see Mapping Rule 1 below. | +| `info.x-backstage-lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | +| `info.x-backstage-tags` | string[] | Optional. Default: `[]`. | + +## Entity: Catalog API Entity (output) + +Standard Backstage `API` kind entity, produced by the EntityProvider's full mutation. + +| Field | Source | Mapping Rule | +|---|---|---| +| `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | +| `metadata.title` | `info.title` | Verbatim. | +| `metadata.description` | `info.description` | Verbatim if present, else omitted. | +| `metadata.tags` | `info.x-backstage-tags` | Default `[]` if absent. | +| `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | +| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | +| `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | +| `spec.lifecycle` | `info.x-backstage-lifecycle` | Default `experimental` if absent (FR-007). | +| `spec.owner` | `info.x-backstage-owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | + +## Config: `autoApiRegistration` (app-config.yaml) + +| Key | Type | Default | Purpose | +|---|---|---|---| +| `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | +| `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | +| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | How often the EntityProvider re-scans. | +| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | + +## State transitions (per discovery cycle) + +1. Glob scan → candidate file list. +2. Parse + shape-check each file → valid candidates / log-only errors (unparseable files). +3. Slugify `info.title` → candidate entity name; detect duplicates among this cycle's candidates + → second-and-later duplicates become marker/error entities. +4. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners become + marker/error entities. +5. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored + `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). +6. Emit a `type: 'full'` mutation containing every entity produced by steps 2–5 (normal + + marker/error entities). Entities not included are retracted by the catalog (FR-004). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md new file mode 100644 index 0000000..72e0dbf --- /dev/null +++ b/specs/004-lab-4-auto-registration/plan.md @@ -0,0 +1,148 @@ +# Implementation Plan: Lab 4 — Auto Registration + +**Branch**: `004-lab-4-auto-registration` | **Date**: 2026-07-03 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `specs/004-lab-4-auto-registration/spec.md` + +## Summary + +Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replaces the +per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend +`EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / +`*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s +schedule — no catalog-info.yaml required per API (FR-001–FR-004). Catalog metadata (owner, +lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and +`x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented +defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) +are surfaced via backend logs and, where an entity can be identified, via Backstage's native +catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker +annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a +discovered file, takes precedence over auto-sourced metadata (FR-011). + +Two new sample files support the lab: a vendored copy of the MIT-licensed Scalar Galaxy API +(`galaxy-openapi.yaml`) with no pre-existing catalog registration, demonstrating genuine +"previously unregistered API" discovery (US1); and a small `precedence-demo-openapi.yaml` + +hand-authored `precedence-demo-catalog-info.yaml` pair demonstrating the precedence rule. +Lab documentation calls out which parts (glob patterns, root path, `x-*` field names) are +adaptable conventions per Constitution Principle VIII (FR-010, US3). + +## Technical Context + +**Language/Version**: TypeScript / Node.js 20 LTS (Backstage 1.51.0, pinned from Lab 1) + +**Primary Dependencies**: +- `fast-glob@3.3.3` — new direct dependency of `packages/backend`; already present transitively + in the workspace, promoted to an explicit dependency for the filesystem scan (research.md R2). +- `js-yaml@4.2.0` — new direct dependency of `packages/backend`; already present transitively, + promoted to explicit for parsing candidate spec files (research.md R3). +- `@backstage/plugin-catalog-node` (already installed, `2.2.1`) — supplies `EntityProvider`, + `CatalogProcessor`, and `catalogProcessingExtensionPoint` used to register the new backend + module, following the same `createBackendModule` pattern as + `packages/backend/src/extensions/permissionPolicy.ts`. +- No new frontend dependencies — this lab is backend-only; no new UI surfaces are added (errors + ride on Backstage's existing built-in processing-error UI). + +**Storage**: YAML files only — two new vendored API spec files, one new hand-authored +`catalog-info.yaml` (precedence demo), and one `app-config.yaml` addition (new +`autoApiRegistration` block). No database schema changes. + +**Testing**: Manual browser + log verification per lab README (no automated test suite — +consistent with Labs 1–3's tutorial-lab testing approach). + +**Target Platform**: Local development machine — Windows 10/11 and macOS 12+ (same as Labs 1–3). + +**Project Type**: Tutorial lab — Markdown documentation + YAML config/catalog files + one new +TypeScript backend module (EntityProvider + CatalogProcessor). + +**Performance Goals**: New/changed API definitions appear in the catalog within one discovery +cycle (≤30s, SC-001); scan cost is bounded by mono-repo size, acceptable at lab scale (a handful +of spec files). + +**Constraints**: Zero cost; cross-platform; no external network access beyond the one-time vendor +of the Scalar Galaxy API file at authoring time (the vendored copy itself requires no runtime +network access, per Constitution Principle VII); builds on Labs 1–3 without disrupting existing +registered APIs (museum, streetlights, train-travel) or the Lab 3 quality tooling. + +**Scale/Scope**: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored +`catalog-info.yaml`, 1 new backend module (~2 files: EntityProvider+processor and its +registration), 1 `app-config.yaml` config block, 3 existing registered APIs left untouched. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design — all gates hold.* + +| Principle | Gate | Status | +|-----------|------|--------| +| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: custom `EntityProvider` for filesystem-based catalog discovery, `x-*` vendor-extension metadata sourcing, native catalog processing-error surfacing via a custom `CatalogProcessor`, full-mutation-based create/update/retract lifecycle | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), and WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011) | ✅ Pass | +| III. Cross-Platform Compatibility | All new code is TypeScript/YAML; `fast-glob` and `js-yaml` are pure-JS, no native/platform-specific dependencies; `yarn add` and `yarn start` work identically on Windows and macOS; root-path resolution documented with platform-neutral relative paths | ✅ Pass | +| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–3's existing Node/Yarn setup — `fast-glob`/`js-yaml` are installed via the existing `yarn add` workflow; README documents the install step | ✅ Pass | +| V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | +| VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | +| VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `x-backstage-*` field names as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | + +No violations. Complexity Tracking section omitted. + +## Project Structure + +### Documentation (this feature) + +```text +specs/004-lab-4-auto-registration/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +└── tasks.md # Phase 2 output (/speckit-tasks — NOT created here) +``` + +No `contracts/` directory — this lab produces no external API contracts. The `autoApiRegistration` +config shape and the entity mapping rules are documented in data-model.md instead. + +### Source Code (repository root) + +```text +labs/ +└── lab-04-auto-registration/ + ├── README.md # Lab documentation (overview, prereqs, steps, + │ # verification, troubleshooting, adaptable + │ # conventions per US3/FR-010) + └── apis/ + ├── galaxy/ + │ └── galaxy-openapi.yaml # New: vendored Scalar Galaxy API (MIT), + │ # x-backstage-owner/lifecycle/tags added; + │ # no catalog-info.yaml — pure auto-discovery + └── precedence-demo/ + ├── precedence-demo-openapi.yaml # New: minimal spec for the precedence demo + └── precedence-demo-catalog-info.yaml # New: hand-authored entity that must win + # over the auto-sourced one (FR-011) +``` + +Changes to the student's Backstage instance (guided by README, not committed as generated output, +but the module source itself IS committed since it is lab teaching content): + +```text +# In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): +app-config.yaml # Modified: add autoApiRegistration config block +packages/backend/ +├── package.json # Modified: add fast-glob, js-yaml +└── src/ + ├── index.ts # Modified: backend.add(import('./extensions/autoApiRegistration')) + └── extensions/ + └── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend + # module (discovery, parsing, mapping, owner + # validation, precedence check, error marking) +``` + +**Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–3. All +student-facing content lives under `labs/lab-04-auto-registration/`. The two new sample API +files (and the one hand-authored precedence-demo catalog file) are committed alongside the lab +README. The student's Backstage instance (from Lab 1) is modified in-place per the README +instructions, same as Lab 3's pattern. Speckit artefacts live under +`specs/004-lab-4-auto-registration/` per convention. + +## Complexity Tracking + +No violations recorded — table omitted per template instructions. diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md new file mode 100644 index 0000000..053a942 --- /dev/null +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -0,0 +1,45 @@ +# Quickstart: Lab 4 — Auto Registration + +This is the design-time quickstart used to validate the plan; the learner-facing README (built in +`/speckit-tasks` + `/speckit-implement`) will follow the same flow with full explanatory prose per +Constitution Principle II. + +## Prerequisites + +- Labs 1–3 completed (running Backstage 1.51.0 instance, Lab 2 teams/users, Lab 3 quality tooling). +- No new external accounts or services. + +## Steps + +1. **Install dependencies**: add `fast-glob` and `js-yaml` as direct dependencies of + `packages/backend`. +2. **Add the auto-registration backend module** + (`packages/backend/src/extensions/autoApiRegistration.ts`): implements the `EntityProvider` + + companion `CatalogProcessor` described in research.md R1–R4. Registered in + `packages/backend/src/index.ts` via `backend.add(import('./extensions/autoApiRegistration'))`. +3. **Configure discovery** in `app-config.yaml` under a new `autoApiRegistration` block (see + data-model.md for keys/defaults). +4. **Add the two new sample files**: + - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API + with `x-backstage-*` fields added) — no pre-existing `catalog-info.yaml`. + - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + + `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. +5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: + - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually + written catalog file (US1, SC-001). + - Its `spec.owner`, `spec.lifecycle`, and tags match the `x-backstage-*` values in the spec + file (US2, SC-002). + - The precedence-demo API's catalog entity reflects the hand-authored + `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). +6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, + confirm the existing entity updates rather than duplicating (FR-003). +7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait + one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. +8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line + identifies the file; temporarily set `x-backstage-owner` to a nonexistent team, confirm a + catalog processing error is visible on that entity (FR-008, SC-003). + +## Out of scope for this quickstart + +- Full JSON-Schema validation of spec files (documented as a learner extension point, not built). +- Real-time file-watching (documented as a deliberate simplification — 30s poll cycle). diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md new file mode 100644 index 0000000..68b2515 --- /dev/null +++ b/specs/004-lab-4-auto-registration/research.md @@ -0,0 +1,136 @@ +# Phase 0 Research: Lab 4 — Auto Registration + +## R1: Discovery mechanism + +**Decision**: A custom Backstage **EntityProvider** (not a `CatalogProcessor`), registered as a +backend module via `catalogProcessingExtensionPoint.addEntityProvider(...)`, following the same +`createBackendModule` pattern already used for the permission policy +(`labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/permissionPolicy.ts`). + +**Rationale**: An `EntityProvider` owns a full, versioned mutation of "the set of entities this +provider currently knows about." On each scheduled run it can emit a `type: 'full'` mutation +containing exactly the API entities currently discoverable — anything omitted from that mutation +is automatically retracted by the catalog. This gives create (FR-002), update (FR-003), and +removal-on-delete (FR-004) for free from one code path, with no separate cleanup logic needed. +A `CatalogProcessor` only transforms entities that already exist via some `Location`; it cannot +originate new entities from an arbitrary filesystem scan on its own. + +**Alternatives considered**: +- GitHub discovery processor (`catalog-backend-module-github`) — rejected: requires a real GitHub + App/token and a reachable remote repo, violating the zero-cost/no-external-network assumption + already recorded in the spec (Assumptions: mono-repo is simulated locally). +- `type: file` catalog locations with glob targets pointing at `catalog-info.yaml` — rejected: + this still requires a hand-authored `catalog-info.yaml` per API, which is precisely what Lab 4 + must eliminate (FR-001). + +## R2: File discovery within the mono-repo + +**Decision**: Use `fast-glob` (already present in the workspace `node_modules`, will be added as +an explicit `dependencies` entry in `packages/backend/package.json`) to scan a configured root +path for `**/*-openapi.yaml` and `**/*-asyncapi.yaml`, on a scheduled interval via +`coreServices.scheduler` (default every 30s — fast enough for interactive lab feedback without +hammering the filesystem). + +**Root path**: configurable via `autoApiRegistration.rootPath` in `app-config.yaml`; documented +as an adaptable convention (FR-010). Default value resolves from the backend process's working +directory (`packages/backend` under `backstage-cli package start`) up to the repository root — +exact relative segment count will be confirmed experimentally during implementation and recorded +here as a Run update, consistent with how Lab 3's research.md captured verified findings. + +**Alternatives considered**: +- `chokidar` file-watcher for real-time push updates — rejected: adds a persistent watch process + and cross-platform edge cases (network drives, WSL) for marginal benefit over a 30s poll in a + local tutorial; the spec's Assumptions already state discovery is a "periodic/on-demand" cycle, + not real-time push. + +## R3: Parsing OpenAPI / AsyncAPI files and extracting `x-*` metadata + +**Decision**: Use `js-yaml` (already present, will be added as an explicit dependency) to parse +each matched file into a plain object. A file is treated as valid input if it parses as +YAML/JSON **and** contains either an `openapi` or `asyncapi` top-level version field **and** an +`info.title` string. Anything else (parse failure, missing both version fields, missing title) is +treated as malformed per the Edge Cases section. + +Recognized `x-*` fields are read from the top-level `info` object for both OpenAPI and AsyncAPI +(per the Clarifications session): +- `info.x-backstage-owner` → `spec.owner` +- `info.x-backstage-lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) +- `info.x-backstage-tags` → `metadata.tags` (default: `[]`, if absent) + +**Rationale**: Full JSON-Schema validation against the OpenAPI/AsyncAPI meta-schemas (e.g. via +`@apidevtools/swagger-parser`, also present in `node_modules`) is available but deliberately not +used for the primary discovery path — it would reject specs with minor structural issues that are +otherwise perfectly usable for catalog registration, and doing full schema validation is out of +scope for what this lab teaches (catalog metadata sourcing, not spec authoring correctness). The +lightweight shape check above is sufficient to distinguish "not an API spec at all" from "a real +spec with metadata problems," which is the distinction FR-008's error handling cares about. + +**Alternatives considered**: +- `@apidevtools/swagger-parser` full validation — rejected as primary path (see rationale above); + noted as a documented extension point for learners in quickstart.md. + +## R4: Owner validation and "visible, actionable error" surfacing + +**Decision**: Errors are surfaced through **two independent, existing Backstage mechanisms** — +no bespoke error-reporting system is built: + +1. **Backend logs**: every discovery cycle logs one line per skipped/errored file (parse failure, + invalid owner reference, name collision) via the standard backend `logger`, already made + visible through the installed `@backstage/plugin-catalog-backend-module-logs`. +2. **Backstage's native processing-error UI**: for errors that occur *after* a candidate entity + has enough shape to be identified (invalid owner ref, name collision), the provider still emits + a minimal entity for that file, tagged with an annotation + (`apiportal-lab.io/registration-error: `). A small companion `CatalogProcessor` + (`preProcessEntity` hook, added to the same backend module) inspects that annotation and throws + a `InputError` when present — Backstage's core catalog processing engine natively records + processor errors per-entity and surfaces them in the entity's own "Inspect entity" / + unprocessed-entities view, which is exactly the built-in "catalog processing errors UI" the + Clarifications session specified. One message, written once, drives both surfaces. + + Files that fail the **shape check in R3** (not parseable as an API spec at all — no + `openapi`/`asyncapi` field, no `info.title`) cannot produce even a minimal entity (no name to + register under) and are therefore logged only; this is called out explicitly in quickstart.md + as the one category of error that is log-only rather than log+UI, and is why FR-008 lists + "fails to parse" separately from the owner/collision cases. + +**Owner validation**: before emitting the full mutation, the provider fetches all existing +`User`/`Group` entities via the in-process `CatalogService` and checks each candidate's +`x-backstage-owner` value resolves to a known entity ref (defaulting kind to `group:default/` if +no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the +repo). Unresolvable owners produce the marker-entity + processor-error path above. + +**Name collisions**: if two discovered files slugify to the same candidate entity name, the first +(sorted by file path for determinism) is registered normally; the second is emitted as a marker +entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the +conflict is independently visible without ever silently overwriting the first. + +**Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a +candidate in its full mutation, the provider queries the catalog for an existing entity of the +same `kind: API` and `metadata.name` that did **not** originate from this provider (i.e. whose +`metadata.annotations['backstage.io/managed-by-origin-location']` differs from our provider's +entity source). If one exists, the auto-sourced candidate is skipped for that name — the +hand-authored entity wins, matching the documented Assumption. + +## R5: New sample API for the "previously unregistered" discovery demo + +**Decision**: Vendor a trimmed copy of the **Scalar Galaxy API** (MIT-licensed, +github.com/scalar/scalar, published build at +`https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as +`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with `x-backstage-owner`, +`x-backstage-lifecycle`, and `x-backstage-tags` fields added under `info` directly in the vendored +copy. The upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, +all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), +covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and +Streetlights examples already used in Labs 1–3, satisfying Constitution Principle VII. + +A second, small hand-crafted OpenAPI file (`precedence-demo-openapi.yaml`, ~2 paths) is added +alongside a hand-authored `precedence-demo-catalog-info.yaml` in +`labs/lab-04-auto-registration/apis/precedence-demo/`, specifically to exercise the FR-011 +precedence rule end-to-end without touching any Lab 1–3 registered API. + +**Alternatives considered**: +- Reusing an existing Lab 1–3 API for the discovery demo — rejected: it is already registered via + a hand-authored `catalog-info.yaml`, so it cannot authentically demonstrate "a previously + unregistered API gets discovered" (User Story 1's core scenario). +- Redocly's other example repos (TicTacToe, etc.) as backup — not needed; Scalar Galaxy confirmed + suitable on inspection. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index 4b273ca..b914754 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -8,6 +8,16 @@ **Input**: User description: "Create new feature and specification from feature 4 in GOAL.md" — Lab 4 (Auto registration): Setup a process for automatic registration of APIs and creation of catalog metadata. Assume API definition files exist within a Git mono-repo. Include sourcing of metadata from x-* fields, within the API specifications themselves, into catalog metadata, fields and annotations. +## Clarifications + +### Session 2026-07-02 + +- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) +- Q: How is the mono-repo discovery scope defined — fixed folder(s) or filename pattern? → A: No fixed folder; discovery matches a filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) anywhere in the repo +- Q: Where should discovery/registration errors be surfaced to the learner? → A: Both Backstage's built-in catalog processing errors UI (entity-level) and backend log output +- Q: How is the catalog entity's name derived from the API definition file? → A: From the spec's own `info.title` field, slugified (not the file path, not a separate `x-*` name field) +- Q: Where do recognized `x-*` metadata fields live within an AsyncAPI document, relative to OpenAPI? → A: Same placement as OpenAPI — under the document's top-level `info` object for both formats + ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) @@ -60,7 +70,7 @@ As a learner adapting this repository to my own organization, I want to understa ### Edge Cases - What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. -- What happens when two API definition files resolve to the same catalog entity name? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. +- What happens when two API definition files resolve to the same catalog entity name (i.e., the same slugified `info.title`)? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. - What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. - What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. - How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. @@ -69,14 +79,14 @@ As a learner adapting this repository to my own organization, I want to understa ### Functional Requirements -- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) located within a designated mono-repo folder structure, without requiring a hand-authored `catalog-info.yaml` per API. +- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) anywhere within the mono-repo by matching a documented filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) rather than requiring a fixed folder location, and without requiring a hand-authored `catalog-info.yaml` per API. - **FR-002**: The system MUST automatically create a catalog API entity for each newly discovered API definition file. - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields and map them into the corresponding catalog entity fields or annotations. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields — located under the document's top-level `info` object for both OpenAPI and AsyncAPI — and map them into the corresponding catalog entity fields or annotations. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. -- **FR-008**: The system MUST surface a visible, actionable error when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition — and MUST continue processing remaining, unaffected API definitions. +- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. - **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. - **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. @@ -86,8 +96,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities - **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. -- **Mono-repo Location**: The designated folder structure/convention within which API Definition Files are expected to live, used as the scope for discovery. +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name is derived from the spec's own `info.title` field (slugified), and it carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. - **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. ## Success Criteria *(mandatory)* @@ -96,15 +106,15 @@ As a learner adapting this repository to my own organization, I want to understa - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. - **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. -- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error, exactly which file and problem caused registration to fail, without needing to inspect system internals. +- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. ## Assumptions -- The "Git mono-repo" referenced in Lab 4 is simulated as a folder/convention within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. +- The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- Default `x-*` field names (e.g., for owning team, lifecycle, tags) will be defined by this lab as a documented convention; learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- Default `x-*` field names are defined by this lab as `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` (namespaced under `x-backstage-*` to avoid collision with other vendor extensions); learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 1d17a2b737e47c23619e29ea4e93d50bc8544390 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 07:48:54 +1000 Subject: [PATCH 03/19] Update plan to support scaling to real world portal API and scan volumes --- .../004-lab-4-auto-registration/data-model.md | 49 +++++-- specs/004-lab-4-auto-registration/plan.md | 47 +++++-- .../004-lab-4-auto-registration/quickstart.md | 16 ++- specs/004-lab-4-auto-registration/research.md | 132 ++++++++++++++++-- 4 files changed, 207 insertions(+), 37 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index abe48d4..f75bcc4 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -40,18 +40,45 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta |---|---|---|---| | `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | | `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | -| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | How often the EntityProvider re-scans. | +| `autoApiRegistration.ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | +| `autoApiRegistration.mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). | +| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | +| `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | +| `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | | `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | +## Scan-state cache (persisted, `coreServices.database`) + +A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart +and (b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2). +Not a catalog entity, not written back into the mono-repo — internal backend state only. + +| Column | Type | Purpose | +|---|---|---| +| `file_path` | string (PK) | Absolute path of the discovered file. | +| `mtime_ms` | number | Last-observed file modification time; cheap first-pass change check on restart/reconciliation. | +| `content_hash` | string | sha1 of file bytes; only computed/compared when `mtime_ms` differs, since hashing every file on every check is itself costly at 1000+ files. | +| `entity_name` | string | The catalog entity name this file currently maps to — doubles as the slug→path collision index (research.md R4). | +| `last_error` | string \| null | Last registration error for this file, if any (mirrors the `apiportal-lab.io/registration-error` annotation; lets a restart re-emit the same marker entity without re-deriving the error). | + ## State transitions (per discovery cycle) -1. Glob scan → candidate file list. -2. Parse + shape-check each file → valid candidates / log-only errors (unparseable files). -3. Slugify `info.title` → candidate entity name; detect duplicates among this cycle's candidates - → second-and-later duplicates become marker/error entities. -4. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners become - marker/error entities. -5. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored - `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). -6. Emit a `type: 'full'` mutation containing every entity produced by steps 2–5 (normal + - marker/error entities). Entities not included are retracted by the catalog (FR-004). +1. **First run only**: glob scan (streamed, ignore patterns applied) → parse + shape-check every + matched file → build the initial scan-state cache → emit one `type: 'full'` mutation + establishing the baseline with the catalog (research.md R6). +2. **Every subsequent cycle** (`poll` mode: full sweep on `schedule.frequencySeconds`; `watch` + mode: per-event, plus a full sweep on `reconciliation.frequencySeconds`): + a. Glob scan (or, in `watch` mode, the single changed path from the watcher event). + b. For each candidate, `stat()` and compare `mtime_ms` against the cache; only files that + differ (or are new) proceed to content-hash comparison and re-parse — unchanged files are + skipped entirely (research.md R6, Problem 2). + c. Parse + shape-check changed/new files → valid candidates / log-only errors. + d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index + (O(1) lookup, not a full re-sort) → collisions become marker/error entities. + e. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners + become marker/error entities. + f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored + `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). + g. Diff the resulting candidate set against the cache → update cache rows → emit a + `type: 'delta'` mutation (`added`/`removed`) for just what changed (FR-002/003/004), + **not** a fresh full mutation (research.md R6, Problem 1). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 72e0dbf..802bec2 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -10,7 +10,14 @@ Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replac per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend `EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / `*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s -schedule — no catalog-info.yaml required per API (FR-001–FR-004). Catalog metadata (owner, +poll schedule for the lab's small sample set — no catalog-info.yaml required per API +(FR-001–FR-004). The same mechanism is designed to hold up at real-world mono-repo scale +(1000+ spec files, 1GB+ repos): a persisted scan-state cache (`coreServices.database`) lets a +restart skip re-parsing unchanged files, delta (not full) mutations are emitted after the +first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the +steady-state discovery signal — see research.md R6 for the full scaling rationale; only the +config *values* differ between the lab demo and a production-scale deployment, not the +architecture. Catalog metadata (owner, lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) @@ -35,6 +42,9 @@ adaptable conventions per Constitution Principle VIII (FR-010, US3). in the workspace, promoted to an explicit dependency for the filesystem scan (research.md R2). - `js-yaml@4.2.0` — new direct dependency of `packages/backend`; already present transitively, promoted to explicit for parsing candidate spec files (research.md R3). +- `chokidar` — new direct dependency of `packages/backend`; already present transitively; used + only when `autoApiRegistration.mode: 'watch'` (not the lab default), to give the discovery + mechanism a steady-state path that scales past the lab's poll-based default (research.md R6). - `@backstage/plugin-catalog-node` (already installed, `2.2.1`) — supplies `EntityProvider`, `CatalogProcessor`, and `catalogProcessingExtensionPoint` used to register the new backend module, following the same `createBackendModule` pattern as @@ -42,9 +52,13 @@ adaptable conventions per Constitution Principle VIII (FR-010, US3). - No new frontend dependencies — this lab is backend-only; no new UI surfaces are added (errors ride on Backstage's existing built-in processing-error UI). -**Storage**: YAML files only — two new vendored API spec files, one new hand-authored -`catalog-info.yaml` (precedence demo), and one `app-config.yaml` addition (new -`autoApiRegistration` block). No database schema changes. +**Storage**: YAML files (two new vendored API spec files, one new hand-authored +`catalog-info.yaml` for the precedence demo, one `app-config.yaml` addition) plus one new +backend-owned database table (`autoApiRegistration` scan-state cache, via `coreServices.database` +— same pluggable SQLite/Postgres service the catalog already uses) that persists per-file +mtime/hash/entity-name/error state across restarts so a real-world-scale mono-repo doesn't +require a full re-parse on every backend start (research.md R6, data-model.md). Not a schema +change to any existing Backstage table. **Testing**: Manual browser + log verification per lab README (no automated test suite — consistent with Labs 1–3's tutorial-lab testing approach). @@ -55,17 +69,21 @@ consistent with Labs 1–3's tutorial-lab testing approach). TypeScript backend module (EntityProvider + CatalogProcessor). **Performance Goals**: New/changed API definitions appear in the catalog within one discovery -cycle (≤30s, SC-001); scan cost is bounded by mono-repo size, acceptable at lab scale (a handful -of spec files). +cycle (≤30s at lab scale, SC-001). Mechanism-level goal (not exercised by the lab itself, but +designed for per research.md R6): a real mono-repo of 1000+ spec files across 1GB+ should incur +parse cost proportional to *changed* files, not total repo size, on both restart (persisted +scan-state cache) and steady-state discovery (watch mode + delta mutations). **Constraints**: Zero cost; cross-platform; no external network access beyond the one-time vendor of the Scalar Galaxy API file at authoring time (the vendored copy itself requires no runtime network access, per Constitution Principle VII); builds on Labs 1–3 without disrupting existing registered APIs (museum, streetlights, train-travel) or the Lab 3 quality tooling. -**Scale/Scope**: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored -`catalog-info.yaml`, 1 new backend module (~2 files: EntityProvider+processor and its -registration), 1 `app-config.yaml` config block, 3 existing registered APIs left untouched. +**Scale/Scope**: Lab demo: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored +`catalog-info.yaml`, 1 new backend module (~3 files: EntityProvider+processor, DB migration for +the scan-state cache, and registration), 1 `app-config.yaml` config block, 3 existing registered +APIs left untouched. Designed-for scale (config change only, no code change, per research.md R6): +1000+ spec files, 1GB+ mono-repo size. ## Constitution Check @@ -127,13 +145,16 @@ but the module source itself IS committed since it is lab teaching content): # In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): app-config.yaml # Modified: add autoApiRegistration config block packages/backend/ -├── package.json # Modified: add fast-glob, js-yaml +├── package.json # Modified: add fast-glob, js-yaml, chokidar └── src/ ├── index.ts # Modified: backend.add(import('./extensions/autoApiRegistration')) └── extensions/ - └── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend - # module (discovery, parsing, mapping, owner - # validation, precedence check, error marking) + ├── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend + │ # module (discovery, parsing, mapping, owner + │ # validation, precedence check, error marking, + │ # poll/watch mode, delta mutations) + └── autoApiRegistrationMigrations/ # New: coreServices.database migration creating + └── 001_scan_state_cache.ts # the scan-state cache table (research.md R6) ``` **Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–3. All diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 053a942..320c2e0 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -42,4 +42,18 @@ Constitution Principle II. ## Out of scope for this quickstart - Full JSON-Schema validation of spec files (documented as a learner extension point, not built). -- Real-time file-watching (documented as a deliberate simplification — 30s poll cycle). +- Exercising `watch` mode or the reconciliation safety net end-to-end (the lab's 2–3 sample files + don't demonstrate anything a poll cycle wouldn't; `mode: 'watch'` is documented, not walked + through step-by-step). + +## Scaling beyond the lab (documented, not walked through) + +The README includes a "Scaling to a real mono-repo" note pointing learners at: +- Why the default is `mode: 'poll'` + `type: 'full'`-then-`delta'` mutations, and what changes + (`mode: 'watch'`, larger `reconciliation.frequencySeconds`, mandatory `ignore` patterns) at + 1000+ files / 1GB+ (research.md R6). +- Why scan state is persisted in the backend's own database rather than written back into the + mono-repo as generated catalog files (data-model.md, research.md R6 Problem 2). +- That restart behavior at scale relies on the persisted cache, not a full re-scan — this is + called out explicitly since it's the part most likely to surprise a learner who scales this + lab up and then wonders why a restart is instant instead of taking minutes. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 68b2515..ba26fd6 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -7,13 +7,12 @@ backend module via `catalogProcessingExtensionPoint.addEntityProvider(...)`, fol `createBackendModule` pattern already used for the permission policy (`labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/permissionPolicy.ts`). -**Rationale**: An `EntityProvider` owns a full, versioned mutation of "the set of entities this -provider currently knows about." On each scheduled run it can emit a `type: 'full'` mutation -containing exactly the API entities currently discoverable — anything omitted from that mutation -is automatically retracted by the catalog. This gives create (FR-002), update (FR-003), and -removal-on-delete (FR-004) for free from one code path, with no separate cleanup logic needed. -A `CatalogProcessor` only transforms entities that already exist via some `Location`; it cannot -originate new entities from an arbitrary filesystem scan on its own. +**Rationale**: An `EntityProvider` owns a versioned mutation of "the set of entities this provider +currently knows about." Anything it does not currently claim is retracted by the catalog. This +gives create (FR-002), update (FR-003), and removal-on-delete (FR-004) for free from one code +path, with no separate cleanup logic needed. A `CatalogProcessor` only transforms entities that +already exist via some `Location`; it cannot originate new entities from an arbitrary filesystem +scan on its own. **Alternatives considered**: - GitHub discovery processor (`catalog-backend-module-github`) — rejected: requires a real GitHub @@ -23,6 +22,15 @@ originate new entities from an arbitrary filesystem scan on its own. this still requires a hand-authored `catalog-info.yaml` per API, which is precisely what Lab 4 must eliminate (FR-001). +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: the lab defaults use a `type: 'full'` +mutation on every cycle because it is the simplest thing to explain and the entity count is tiny. +That does **not** scale: a `full` mutation re-transmits and re-diffs every known entity on every +cycle, so its cost grows with total catalog size, not with how much actually changed. The +mechanism as designed (an `EntityProvider` driving mutations) scales fine — what must change is +*which mutation type it sends*. See R6 for the delta-mutation and persisted-cache design that +keeps the same `EntityProvider` architecture viable at scale; the lab's own config simply doesn't +need to turn those knobs on for 2–3 sample files. + ## R2: File discovery within the mono-repo **Decision**: Use `fast-glob` (already present in the workspace `node_modules`, will be added as @@ -37,11 +45,28 @@ directory (`packages/backend` under `backstage-cli package start`) up to the rep exact relative segment count will be confirmed experimentally during implementation and recorded here as a Run update, consistent with how Lab 3's research.md captured verified findings. +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: two changes are load-bearing at scale +and are therefore built into the mechanism now, not deferred as a "someday" rewrite, even though +the lab's default config doesn't need them: +1. **Ignore patterns are mandatory, not optional.** `fast-glob` is called with an `ignore` list + (`**/node_modules/**`, `**/.git/**`, `**/dist/**`, `**/build/**`, plus a configurable + `autoApiRegistration.ignore` extension point) from day one. Without this, a glob walk over a + 1GB tree spends nearly all its time inside dependency/build directories that will never contain + a hand-authored spec file. This is a correctness-adjacent default, not a performance tune-up — + it is documented and defaulted in the lab even though the lab repo itself has few such + directories at the discovery root, so learners inherit a safe default rather than discovering + the need for it the hard way at 1000 files. +2. **Streamed, not buffered, matching.** Use `fastGlob.stream(...)` and process results as they + arrive rather than `fastGlob(...)`'s array form, so peak memory during a scan is bounded by + in-flight parse concurrency (see R6), not by total file count. + **Alternatives considered**: -- `chokidar` file-watcher for real-time push updates — rejected: adds a persistent watch process - and cross-platform edge cases (network drives, WSL) for marginal benefit over a 30s poll in a - local tutorial; the spec's Assumptions already state discovery is a "periodic/on-demand" cycle, - not real-time push. +- `chokidar` file-watcher for real-time push updates — rejected *as the lab default* (adds a + persistent watch process and cross-platform edge cases — network drives, WSL, inotify limits on + very large trees — for marginal benefit over a 30s poll against 2–3 files). Reconsidered and + **adopted as the scale-up path** in R6: at 1000+ files, repeated full-tree poll walks are the + wrong steady-state mechanism, and a watcher layered over the same ignore-pattern set is what + makes the "how often do we rescan" question stop scaling with repo size. ## R3: Parsing OpenAPI / AsyncAPI files and extracting `x-*` metadata @@ -102,7 +127,10 @@ repo). Unresolvable owners produce the marker-entity + processor-error path abov **Name collisions**: if two discovered files slugify to the same candidate entity name, the first (sorted by file path for determinism) is registered normally; the second is emitted as a marker entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the -conflict is independently visible without ever silently overwriting the first. +conflict is independently visible without ever silently overwriting the first. Collision detection +is done against the persisted slug→path index described in R6, not by re-deriving and sorting the +full candidate list on every check — at 1000+ files, an O(N log N) sort per cycle (or per watch +event, in watch mode) is avoidable work; an index lookup is O(1). **Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a candidate in its full mutation, the provider queries the catalog for an existing entity of the @@ -134,3 +162,83 @@ precedence rule end-to-end without touching any Lab 1–3 registered API. unregistered API gets discovered" (User Story 1's core scenario). - Redocly's other example repos (TicTacToe, etc.) as backup — not needed; Scalar Galaxy confirmed suitable on inspection. + +## R6: Scalability & restart behavior at real-world mono-repo scale + +The lab itself discovers 2–3 files in a small repo, and its default config values (30s full poll, +no persistence) are chosen for that scale — they are explicitly **not** what a 1000+ file, 1GB+ +mono-repo should run with. This section records the mechanism-level decisions that keep the same +architecture (EntityProvider + fast-glob) viable at that scale, so scaling up is a config change +plus turning on code paths that already exist, not a rewrite. + +**Problem 1 — full mutations don't scale with catalog size.** A `type: 'full'` `EntityProvider` +mutation re-sends and re-diffs every currently-known entity on every cycle. At 1000+ APIs, most of +which are unchanged between cycles, this wastes network/serialization cost and catalog-processing +time proportional to total catalog size rather than to actual change volume. + +**Decision**: after the provider's first `full` mutation (needed once, to establish the baseline +with the catalog), all subsequent cycles emit `type: 'delta'` mutations (`added`/`removed` only), +computed by diffing the current scan against the persisted scan-state cache (Problem 2). This is +the standard Backstage `EntityProvider` pattern for large, incrementally-changing entity sets (the +same pattern the built-in GitHub discovery provider uses at organization scale) — no custom +protocol is invented. + +**Problem 2 — a cold backend restart re-derives everything from zero.** Backstage's catalog +database already persists previously-ingested *entities* across restarts (a provider's entities +are not deleted until that provider explicitly retracts them), so the catalog itself doesn't go +empty on restart. What is **not** persisted anywhere by default is the provider's own *scan +state* — which files it has seen, their content hashes, and what they mapped to — so without +additional persistence, every restart forces a full re-glob and full re-parse of the entire +mono-repo (1000+ files) before the provider can even confirm nothing changed. + +**Decision**: persist scan state — `filePath`, `mtimeMs`, `contentHash` (sha1 of file bytes), +`entityName`, `lastError` — as rows in a small table owned by this backend module, using +`coreServices.database` (the same pluggable database service Backstage's own catalog uses; +SQLite in the lab, Postgres in a real deployment — zero new infrastructure, consistent with +Principle V). On startup, the provider loads this cache and can validate it against the +filesystem far more cheaply than re-parsing everything: a `stat()` per known path compares +`mtimeMs` (and falls back to a content-hash check only when `mtimeMs` differs, since hashing +1000+ files is itself non-trivial I/O at 1GB scale) before deciding a file needs re-parsing. +Unchanged files contribute zero parse work to a restart. This directly answers "should scanner +results be persisted so restarts restore the catalog faster" — yes, but as internal scan-state +cache rows in the backend's own database, **not** as generated `catalog-info.yaml`-style files +written back into the mono-repo. Writing discovery output back into the source tree would (a) +reintroduce the hand-maintained-duplicate-file problem Lab 4 exists to eliminate, and (b) require +the discovery tool to have write access to a repo it should only need to read — a materially +larger and riskier permission footprint at real-world scale. + +**Problem 3 — polling a 1GB tree every 30s doesn't scale as the primary discovery loop.** Even +with ignore patterns (R2) and streaming (R2), a full `fast-glob` walk of a very large tree still +costs real I/O, and running that walk every 30s regardless of whether anything changed is wasted +work at scale — the lab's 30s interval exists for interactive demo feedback on 2–3 files, not +because 30s is an appropriate steady-state cadence for 1000+ files. + +**Decision**: at scale, discovery should run in **watch mode**: a `chokidar` watcher (rejected for +the lab default in R2, adopted here) observes `add`/`change`/`unlink` events under the same +ignore-pattern set, and each event triggers a targeted single-file re-parse + delta mutation +(update the scan-state cache row, emit one delta) rather than a full-tree rescan. The periodic +`fast-glob` sweep is retained but demoted to an infrequent **reconciliation safety net** +(independently configurable interval, e.g. minutes not seconds) that catches anything a watcher +event could plausibly miss (coalesced events under a large batch change such as a big `git pull`, +or watch reliability limits on some network/CI filesystems) — it is a correctness backstop, not +the primary discovery signal. Parsing is concurrency-limited (bounded worker count, default small) +so a large batch of simultaneous file-change events — e.g. after checking out a branch that +touches hundreds of spec files — doesn't spike memory or block the Node event loop. + +**Config surface** (see data-model.md): `autoApiRegistration.mode: 'poll' | 'watch'` (lab default: +`poll`, no watcher, matching the simple demo story in quickstart.md); `ignore` (glob exclusions, +always applied regardless of mode); `reconciliation.frequencySeconds` (decoupled from watch-event +responsiveness); `parseConcurrency`. Switching a real deployment to scale is: set `mode: 'watch'`, +raise `reconciliation.frequencySeconds` from 30 to something like 900–1800, and rely on the +already-built delta-mutation + persisted-cache path — no architectural change. + +**Alternatives considered**: +- Always persisting a `full` mutation every cycle regardless of scale — rejected: the cost model + is wrong for 1000+ entities (see Problem 1); would work fine at lab scale but silently becomes a + performance cliff at real-world scale, which is exactly the trap this section exists to avoid. +- Writing discovered metadata back into the mono-repo as generated catalog files, to "cache" scan + results — rejected (see Problem 2 rationale): wrong persistence layer, reintroduces duplication, + expands write-access footprint unnecessarily. +- Only a filesystem watcher, no periodic reconciliation sweep — rejected: watchers can miss events + (coalescing, restart races, some CI/network filesystem limitations); a slow safety-net sweep is + cheap insurance at any scale and is what makes watch mode trustworthy enough to recommend. From 2957f1191ff65415becd0e7d7f8fb0a53151581c Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:09:30 +1000 Subject: [PATCH 04/19] Revise x-* naming scheme for best practices --- .../004-lab-4-auto-registration/data-model.md | 21 ++++++------ specs/004-lab-4-auto-registration/plan.md | 14 ++++---- .../004-lab-4-auto-registration/quickstart.md | 8 ++--- specs/004-lab-4-auto-registration/research.md | 34 ++++++++++++++----- specs/004-lab-4-auto-registration/spec.md | 22 +++++++----- 5 files changed, 62 insertions(+), 37 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index f75bcc4..1744f2d 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -11,11 +11,11 @@ A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncap | Field | Type | Notes | |---|---|---| | `openapi` / `asyncapi` | string | Version field; presence of either identifies the spec type. Exactly one is expected. | -| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. | -| `info.description` | string | Optional. Maps to `metadata.description` if present. | -| `info.x-backstage-owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration proceeds with no `spec.owner` override... actually **required by FR-005/FR-007's documented default** — see Mapping Rule 1 below. | -| `info.x-backstage-lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | -| `info.x-backstage-tags` | string[] | Optional. Default: `[]`. | +| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. Not duplicated into `x-examplecorp`. | +| `info.description` | string | Optional. Maps to `metadata.description` if present. Not duplicated into `x-examplecorp`. | +| `tags[].name` | string[] | Optional, native top-level OpenAPI/AsyncAPI field (not under `info`, not under `x-examplecorp`). Default `[]` if absent. | +| `info.x-examplecorp.owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration still succeeds using a documented default — see Mapping Rule 1 below. | +| `info.x-examplecorp.lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | ## Entity: Catalog API Entity (output) @@ -26,12 +26,12 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | | `metadata.title` | `info.title` | Verbatim. | | `metadata.description` | `info.description` | Verbatim if present, else omitted. | -| `metadata.tags` | `info.x-backstage-tags` | Default `[]` if absent. | +| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | -| `spec.lifecycle` | `info.x-backstage-lifecycle` | Default `experimental` if absent (FR-007). | -| `spec.owner` | `info.x-backstage-owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | +| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | | `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | ## Config: `autoApiRegistration` (app-config.yaml) @@ -45,7 +45,8 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | | `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | | `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | -| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | +| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-examplecorp.owner` is absent (FR-007). | +| `autoApiRegistration.xNamespace` | string | `'examplecorp'` | The vendor namespace under `info.x-` that this discovery instance reads owner/lifecycle from. Adaptable per FR-010/US3 — this is the single setting a learner changes to rename the convention to their own company. | ## Scan-state cache (persisted, `coreServices.database`) @@ -75,7 +76,7 @@ Not a catalog entity, not written back into the mono-repo — internal backend s c. Parse + shape-check changed/new files → valid candidates / log-only errors. d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index (O(1) lookup, not a full re-sort) → collisions become marker/error entities. - e. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners + e. Resolve `x-examplecorp.owner` against known `User`/`Group` entities → unresolvable owners become marker/error entities. f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 802bec2..75c283d 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -17,10 +17,12 @@ restart skip re-parsing unchanged files, delta (not full) mutations are emitted first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the steady-state discovery signal — see research.md R6 for the full scaling rationale; only the config *values* differ between the lab demo and a production-scale deployment, not the -architecture. Catalog metadata (owner, -lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and -`x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented -defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) +architecture. Catalog owner/lifecycle metadata is sourced from a single vendor-namespaced +`info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one +namespace learners rename to their own company (FR-005–FR-006), with documented defaults when +absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), +description (`info.description`), tags (native top-level `tags`) — is sourced from there instead +of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, name collisions) are surfaced via backend logs and, where an entity can be identified, via Backstage's native catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a @@ -98,7 +100,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | | VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | | VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | -| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `x-backstage-*` field names as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | | IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | No violations. Complexity Tracking section omitted. @@ -130,7 +132,7 @@ labs/ └── apis/ ├── galaxy/ │ └── galaxy-openapi.yaml # New: vendored Scalar Galaxy API (MIT), - │ # x-backstage-owner/lifecycle/tags added; + │ # info.x-examplecorp.owner/lifecycle added; │ # no catalog-info.yaml — pure auto-discovery └── precedence-demo/ ├── precedence-demo-openapi.yaml # New: minimal spec for the precedence demo diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 320c2e0..c814db5 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -21,14 +21,14 @@ Constitution Principle II. data-model.md for keys/defaults). 4. **Add the two new sample files**: - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API - with `x-backstage-*` fields added) — no pre-existing `catalog-info.yaml`. + with an `info.x-examplecorp` object added) — no pre-existing `catalog-info.yaml`. - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. 5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually written catalog file (US1, SC-001). - - Its `spec.owner`, `spec.lifecycle`, and tags match the `x-backstage-*` values in the spec - file (US2, SC-002). + - Its `spec.owner` and `spec.lifecycle` match the `info.x-examplecorp` values, and its tags + match the spec's native `tags` array — not duplicated into `x-examplecorp` (US2, SC-002). - The precedence-demo API's catalog entity reflects the hand-authored `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). 6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, @@ -36,7 +36,7 @@ Constitution Principle II. 7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. 8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line - identifies the file; temporarily set `x-backstage-owner` to a nonexistent team, confirm a + identifies the file; temporarily set `info.x-examplecorp.owner` to a nonexistent team, confirm a catalog processing error is visible on that entity (FR-008, SC-003). ## Out of scope for this quickstart diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index ba26fd6..9f05b38 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -76,11 +76,26 @@ YAML/JSON **and** contains either an `openapi` or `asyncapi` top-level version f `info.title` string. Anything else (parse failure, missing both version fields, missing title) is treated as malformed per the Edge Cases section. -Recognized `x-*` fields are read from the top-level `info` object for both OpenAPI and AsyncAPI -(per the Clarifications session): -- `info.x-backstage-owner` → `spec.owner` -- `info.x-backstage-lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) -- `info.x-backstage-tags` → `metadata.tags` (default: `[]`, if absent) +Recognized `x-*` metadata is read from a single vendor-namespaced object nested under the +top-level `info` object, for both OpenAPI and AsyncAPI (per the Clarifications session, +2026-07-03 update): +- `info.x-examplecorp.owner` → `spec.owner` +- `info.x-examplecorp.lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) + +`examplecorp` is this lab's fictional company namespace (documented as the one thing every +learner renames first, per FR-010/US3). The field names deliberately omit `backstage` — this +metadata (who owns an API, what lifecycle stage it's in) is meaningful to any consumer of the +spec, not just this tool, so the namespace names the company, not the tool. + +Metadata with a natural home elsewhere in the spec is read from there instead, never duplicated +into `x-examplecorp`: +- `info.title` → `metadata.title` / entity name (already established, unchanged) +- `info.description` → `metadata.description` (already established, unchanged) +- top-level `tags[].name` → `metadata.tags` (default: `[]`, if the spec has no `tags` array) — + **changed from the original `x-backstage-tags` design**: both OpenAPI and AsyncAPI already + define a native top-level `tags` field for exactly this purpose, so sourcing tags from `x-*` as + well would create two places that could drift apart. `tags[].name` (ignoring + `tags[].description`, which has no catalog equivalent) is the source of truth. **Rationale**: Full JSON-Schema validation against the OpenAPI/AsyncAPI meta-schemas (e.g. via `@apidevtools/swagger-parser`, also present in `node_modules`) is available but deliberately not @@ -120,7 +135,7 @@ no bespoke error-reporting system is built: **Owner validation**: before emitting the full mutation, the provider fetches all existing `User`/`Group` entities via the in-process `CatalogService` and checks each candidate's -`x-backstage-owner` value resolves to a known entity ref (defaulting kind to `group:default/` if +`x-examplecorp.owner` value resolves to a known entity ref (defaulting kind to `group:default/` if no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the repo). Unresolvable owners produce the marker-entity + processor-error path above. @@ -144,9 +159,10 @@ hand-authored entity wins, matching the documented Assumption. **Decision**: Vendor a trimmed copy of the **Scalar Galaxy API** (MIT-licensed, github.com/scalar/scalar, published build at `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as -`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with `x-backstage-owner`, -`x-backstage-lifecycle`, and `x-backstage-tags` fields added under `info` directly in the vendored -copy. The upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, +`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with an `info.x-examplecorp` +object (`owner`, `lifecycle`) added directly in the vendored copy; its existing native `tags` +array is used as-is for `metadata.tags`. The upstream file is a single self-contained OpenAPI 3.1 +document (confirmed: 1469 lines, all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and Streetlights examples already used in Labs 1–3, satisfying Constitution Principle VII. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index b914754..60e98e2 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -12,12 +12,18 @@ ### Session 2026-07-02 -- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) +- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) — **superseded, see Session 2026-07-03 below** - Q: How is the mono-repo discovery scope defined — fixed folder(s) or filename pattern? → A: No fixed folder; discovery matches a filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) anywhere in the repo - Q: Where should discovery/registration errors be surfaced to the learner? → A: Both Backstage's built-in catalog processing errors UI (entity-level) and backend log output - Q: How is the catalog entity's name derived from the API definition file? → A: From the spec's own `info.title` field, slugified (not the file path, not a separate `x-*` name field) - Q: Where do recognized `x-*` metadata fields live within an AsyncAPI document, relative to OpenAPI? → A: Same placement as OpenAPI — under the document's top-level `info` object for both formats +### Session 2026-07-03 + +- Q: Should `x-*` fields sit directly in `info`, or nested under a single vendor-namespaced object? → A: Nested under a single `info.x-examplecorp` object (e.g. `info.x-examplecorp.owner`, `info.x-examplecorp.lifecycle`) — `examplecorp` is this lab's fictional company namespace, which learners are expected to rename to their own; a single grouping object avoids polluting `info` with multiple top-level `x-*` keys and makes "which fields are ours" unambiguous at a glance. +- Q: Should field names retain the `backstage` tool name (`x-backstage-owner`, etc.)? → A: No — the field names describe the metadata (owner, lifecycle), not the tool consuming it. Catalog metadata like ownership is tool-agnostic (equally meaningful to an API gateway or any other consumer), so `backstage` is dropped: `x-examplecorp.owner`, `x-examplecorp.lifecycle`. +- Q: Should `tags` remain a recognized `x-*` field? → A: No — removed. OpenAPI/AsyncAPI both already have a native top-level `tags` field; sourcing tags from `x-*` as well would duplicate data that already has a natural home in the spec, risking divergence. Catalog `metadata.tags` is now sourced from the spec's native `tags` array (`tags[].name`), not from any `x-*` field. The same reasoning already applied to `info.title` (entity name) and `info.description` (entity description), which were never duplicated into `x-*` fields. + ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) @@ -47,7 +53,7 @@ As an API owner, I want to declare metadata that Backstage needs (such as owning **Acceptance Scenarios**: 1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. -2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, tags, system grouping), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. 3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. 4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. @@ -84,7 +90,7 @@ As a learner adapting this repository to my own organization, I want to understa - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields — located under the document's top-level `info` object for both OpenAPI and AsyncAPI — and map them into the corresponding catalog entity fields or annotations. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. - **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. @@ -95,17 +101,17 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities -- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name is derived from the spec's own `info.title` field (slugified), and it carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle metadata sourced from `info.x-examplecorp`. - **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. -- **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. +- **`x-*` Metadata Mapping**: The documented mapping from the `info.x-examplecorp` object's fields to catalog entity fields/annotations, including default values applied when a field is absent, and the explicit list of fields sourced natively rather than from `x-*` (name, description, tags). ## Success Criteria *(mandatory)* ### Measurable Outcomes - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. -- **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. +- **SC-002**: 100% of the owning-team and lifecycle metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. - **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. @@ -114,7 +120,7 @@ As a learner adapting this repository to my own organization, I want to understa - The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- Default `x-*` field names are defined by this lab as `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` (namespaced under `x-backstage-*` to avoid collision with other vendor extensions); learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner` and `lifecycle` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 41b24620a25bc4732efb9685d2ab1f91f3ed9153 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:24:01 +1000 Subject: [PATCH 05/19] Added ability to support multiple repos --- .../004-lab-4-auto-registration/data-model.md | 112 ++++++++++++------ specs/004-lab-4-auto-registration/plan.md | 23 ++-- .../004-lab-4-auto-registration/quickstart.md | 15 +++ specs/004-lab-4-auto-registration/research.md | 99 +++++++++++++++- 4 files changed, 204 insertions(+), 45 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index 1744f2d..091ec77 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -31,55 +31,99 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | -| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | | `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | ## Config: `autoApiRegistration` (app-config.yaml) -| Key | Type | Default | Purpose | +The canonical shape is a list of sources (research.md R7), each independently configured. A flat +`autoApiRegistration.{rootPath,patterns,...}` (no `sources` key) is shorthand for a single +implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so a +single-repo setup never has to write the list form. + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo # required, unique; becomes part of the EntityProvider name + # and the scan-state cache's source_id column + rootPath: ... + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + ignore: ['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**'] + mode: poll # or 'watch' — independent per source (research.md R6/R7) + schedule: { frequencySeconds: 30 } + reconciliation: { frequencySeconds: 900 } # watch mode only + parseConcurrency: 4 + defaultOwner: group:default/platform-team + xNamespace: examplecorp + - id: team-checkout-api # a second source: a different team's repo, a local sibling + rootPath: ../team-checkout-api # checkout for the lab; a real remote repo's + patterns: ['**/*-openapi.yaml'] # synced local worktree in production (R7 + defaultOwner: group:default/checkout-team # Problem 4) + xNamespace: checkoutteam +``` + +| Key (per source, under `sources[]`, or flat for the single-source shorthand) | Type | Default | Purpose | |---|---|---|---| -| `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | -| `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | -| `autoApiRegistration.ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | -| `autoApiRegistration.mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). | -| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | -| `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | -| `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | -| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-examplecorp.owner` is absent (FR-007). | -| `autoApiRegistration.xNamespace` | string | `'examplecorp'` | The vendor namespace under `info.x-` that this discovery instance reads owner/lifecycle from. Adaptable per FR-010/US3 — this is the single setting a learner changes to rename the convention to their own company. | +| `id` | string | `'default'` (flat-config shorthand only) | Unique source identifier. Required when using the `sources[]` list form. Used as the `EntityProvider` name suffix and the scan-state cache's `source_id` (research.md R7). | +| `rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. In production, may point at a locally-synced worktree of a genuinely remote repo (research.md R7, Problem 4) — this schema does not change either way. | +| `patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010, per source. | +| `ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | +| `mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). Independent per source — e.g. a large mono-repo can run `watch` while a small team repo runs `poll` (research.md R7). | +| `schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | +| `reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | +| `parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | +| `defaultOwner` | string | *(no global default — each source must set its own, or inherit `autoApiRegistration.defaults.defaultOwner` if configured)* | Fallback owner ref applied when `x-.owner` is absent (FR-007). Deliberately per-source, not global — a team/pre-production repo plausibly has a different fallback owner than the platform mono-repo (research.md R7, Problem 2). | +| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | ## Scan-state cache (persisted, `coreServices.database`) -A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart -and (b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2). -Not a catalog entity, not written back into the mono-repo — internal backend state only. +A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart, +(b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2), and +(c) support collision/precedence checks across every configured source, not just the one currently +scanning (research.md R7, Problem 3). Not a catalog entity, not written back into any source +repo — internal backend state only. | Column | Type | Purpose | |---|---|---| -| `file_path` | string (PK) | Absolute path of the discovered file. | +| `source_id` | string (PK, part 1) | Which configured source (research.md R7) this row came from — `'default'` for the flat single-source config. Lets a collision/error message name *which repo*, not just which file. | +| `file_path` | string (PK, part 2) | Absolute path of the discovered file, within that source. | | `mtime_ms` | number | Last-observed file modification time; cheap first-pass change check on restart/reconciliation. | | `content_hash` | string | sha1 of file bytes; only computed/compared when `mtime_ms` differs, since hashing every file on every check is itself costly at 1000+ files. | -| `entity_name` | string | The catalog entity name this file currently maps to — doubles as the slug→path collision index (research.md R4). | +| `entity_name` | string, indexed | The catalog entity name this file currently maps to — doubles as the slug→(source, path) collision index (research.md R4), queried **without** filtering by `source_id` so collisions across sources are caught (research.md R7, Problem 3). | | `last_error` | string \| null | Last registration error for this file, if any (mirrors the `apiportal-lab.io/registration-error` annotation; lets a restart re-emit the same marker entity without re-deriving the error). | ## State transitions (per discovery cycle) -1. **First run only**: glob scan (streamed, ignore patterns applied) → parse + shape-check every - matched file → build the initial scan-state cache → emit one `type: 'full'` mutation - establishing the baseline with the catalog (research.md R6). -2. **Every subsequent cycle** (`poll` mode: full sweep on `schedule.frequencySeconds`; `watch` - mode: per-event, plus a full sweep on `reconciliation.frequencySeconds`): - a. Glob scan (or, in `watch` mode, the single changed path from the watcher event). - b. For each candidate, `stat()` and compare `mtime_ms` against the cache; only files that - differ (or are new) proceed to content-hash comparison and re-parse — unchanged files are - skipped entirely (research.md R6, Problem 2). - c. Parse + shape-check changed/new files → valid candidates / log-only errors. +Runs independently **per configured source** (research.md R7) — each source is its own +`EntityProvider` instance with its own schedule/mode, so one source's cycle does not block or +depend on another's. Steps (d)–(f) below query the scan-state cache and catalog **without** +filtering by `source_id`, since collision and precedence checks must be global (research.md R7, +Problem 3) even though discovery itself runs per source. + +1. **First run of a given source**: glob scan (streamed, ignore patterns applied) → parse + + shape-check every matched file → build that source's rows in the scan-state cache → emit one + `type: 'full'` mutation (from that source's `EntityProvider` instance) establishing the + baseline with the catalog (research.md R6). +2. **Every subsequent cycle for that source** (`poll` mode: full sweep on + `schedule.frequencySeconds`; `watch` mode: per-event, plus a full sweep on + `reconciliation.frequencySeconds`): + a. Glob scan of that source's `rootPath` (or, in `watch` mode, the single changed path from the + watcher event). + b. For each candidate, `stat()` and compare `mtime_ms` against that source's cache rows; only + files that differ (or are new) proceed to content-hash comparison and re-parse — unchanged + files are skipped entirely (research.md R6, Problem 2). + c. Parse + shape-check changed/new files, using that source's `xNamespace` → valid candidates / + log-only errors. d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index - (O(1) lookup, not a full re-sort) → collisions become marker/error entities. - e. Resolve `x-examplecorp.owner` against known `User`/`Group` entities → unresolvable owners - become marker/error entities. - f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored - `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). - g. Diff the resulting candidate set against the cache → update cache rows → emit a - `type: 'delta'` mutation (`added`/`removed`) for just what changed (FR-002/003/004), - **not** a fresh full mutation (research.md R6, Problem 1). + **across all sources** (O(1) lookup, not a full re-sort) → collisions become marker/error + entities, whether the colliding file is in this source or another one (research.md R7, + Problem 3). + e. Resolve `x-.owner` (falling back to this source's `defaultOwner` if absent) + against known `User`/`Group` entities → unresolvable owners become marker/error entities. + f. Check for a pre-existing, non-auto-sourced entity at the same name anywhere in the catalog + (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name + (FR-011). + g. Diff the resulting candidate set against this source's cache rows → update cache rows → + emit a `type: 'delta'` mutation (`added`/`removed`) from this source's `EntityProvider` + instance for just what changed (FR-002/003/004), **not** a fresh full mutation (research.md + R6, Problem 1). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 75c283d..5f685bf 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -8,16 +8,21 @@ Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replaces the per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend -`EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / +`EntityProvider` that scans a mono-repo for files matching `*-openapi.yaml` / `*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s poll schedule for the lab's small sample set — no catalog-info.yaml required per API -(FR-001–FR-004). The same mechanism is designed to hold up at real-world mono-repo scale -(1000+ spec files, 1GB+ repos): a persisted scan-state cache (`coreServices.database`) lets a -restart skip re-parsing unchanged files, delta (not full) mutations are emitted after the -first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the -steady-state discovery signal — see research.md R6 for the full scaling rationale; only the -config *values* differ between the lab demo and a production-scale deployment, not the -architecture. Catalog owner/lifecycle metadata is sourced from a single vendor-namespaced +(FR-001–FR-004). The same mechanism is designed to hold up at real-world scale in two +independent dimensions: **repo size** (1000+ spec files, 1GB+, research.md R6 — a persisted +scan-state cache lets a restart skip re-parsing unchanged files, delta-not-full mutations after +the first cycle, and an optional `chokidar` watch mode replacing repeated full-tree polling as +the steady-state discovery signal) and **repo count** (research.md R7 — config generalizes from +one `rootPath` to a list of independently-configured `sources[]`, each its own `EntityProvider` +instance with its own schedule, default owner, and `x-*` namespace, so a platform mono-repo and +one or more separate team/pre-production repos can each be discovered with their own settings +while collision and `catalog-info.yaml`-precedence checks stay global across all of them). Only +the config *values and entry count* differ between the lab demo (one small source) and a +production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle metadata +is sourced from a single vendor-namespaced `info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one namespace learners rename to their own company (FR-005–FR-006), with documented defaults when absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), @@ -100,7 +105,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | | VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | | VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | -| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; the `sources[]` config shape (research.md R7) is documented as the path to onboarding additional team/pre-production repos with their own defaults, without code changes; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | | IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | No violations. Complexity Tracking section omitted. diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index c814db5..11e29ab 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -57,3 +57,18 @@ The README includes a "Scaling to a real mono-repo" note pointing learners at: - That restart behavior at scale relies on the persisted cache, not a full re-scan — this is called out explicitly since it's the part most likely to surprise a learner who scales this lab up and then wonders why a restart is instant instead of taking minutes. + +A second "Scaling to multiple source repositories" note (research.md R7) covers: +- The `autoApiRegistration.sources[]` config list — the lab's own `app-config.yaml` uses the flat + single-source shorthand, but the README shows the equivalent explicit `sources: [{ id: default, + ... }]` form alongside a second example entry, so learners can see exactly what changes to add a + team or pre-production repo (a different `rootPath`, `defaultOwner`, and `xNamespace`). +- Why `defaultOwner` and `xNamespace` are per-source with no built-in global fallback (a repo + onboarding to auto-registration shouldn't have to adopt the platform mono-repo's team or vendor + namespace to participate). +- Why collision detection and `catalog-info.yaml` precedence checks are explicitly called out as + *global* across sources, not per-source — two teams' repos producing the same entity name must + still surface a visible conflict, not silently coexist. +- That onboarding a genuinely separate remote (not a local sibling checkout) requires a sync step + (e.g. scheduled shallow `git clone`/`pull`) ahead of the existing discovery pipeline, which the + lab does not build or require — `rootPath` always points at something already on local disk. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 9f05b38..9f201bc 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -44,6 +44,8 @@ as an adaptable convention (FR-010). Default value resolves from the backend pro directory (`packages/backend` under `backstage-cli package start`) up to the repository root — exact relative segment count will be confirmed experimentally during implementation and recorded here as a Run update, consistent with how Lab 3's research.md captured verified findings. +(This is the single-source shorthand; R7 generalizes `rootPath` into a per-entry field of an +`autoApiRegistration.sources[]` list for organizations with more than one source repository.) **Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: two changes are load-bearing at scale and are therefore built into the mechanism now, not deferred as a "someday" rewrite, even though @@ -145,14 +147,18 @@ entity (suffixed, e.g. `-collision`) carrying the registration-error annot conflict is independently visible without ever silently overwriting the first. Collision detection is done against the persisted slug→path index described in R6, not by re-deriving and sorting the full candidate list on every check — at 1000+ files, an O(N log N) sort per cycle (or per watch -event, in watch mode) is avoidable work; an index lookup is O(1). +event, in watch mode) is avoidable work; an index lookup is O(1). **This index and check are +global, across every configured source, not scoped to the source currently being processed** — see +R7 Problem 3 for why two different teams' repos colliding must not go undetected. **Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a candidate in its full mutation, the provider queries the catalog for an existing entity of the same `kind: API` and `metadata.name` that did **not** originate from this provider (i.e. whose `metadata.annotations['backstage.io/managed-by-origin-location']` differs from our provider's entity source). If one exists, the auto-sourced candidate is skipped for that name — the -hand-authored entity wins, matching the documented Assumption. +hand-authored entity wins, matching the documented Assumption. This check, like collision +detection, runs against catalog state as a whole and is unaffected by which source is being +processed — a hand-authored file anywhere wins over an auto-sourced candidate from any source. ## R5: New sample API for the "previously unregistered" discovery demo @@ -258,3 +264,92 @@ already-built delta-mutation + persisted-cache path — no architectural change. - Only a filesystem watcher, no periodic reconciliation sweep — rejected: watchers can miss events (coalescing, restart races, some CI/network filesystem limitations); a slow safety-net sweep is cheap insurance at any scale and is what makes watch mode trustworthy enough to recommend. + +## R7: Multiple source repositories (platform mono-repo + team/pre-production repos) + +The lab itself discovers files from one location. A real organization is unlikely to have every +API definition in the platform team's mono-repo — other teams, and pre-production/experimental +APIs, plausibly live in their own separate repositories, each wanting different discovery +settings (in particular, a different default owner — Problem 2 below). The mechanism must +generalize to N independently-configured sources without becoming N copies of the same code. + +**Problem 1 — a single `rootPath` config key can't express "scan these several, separately +configured locations."** The R1–R6 design (and the lab's `autoApiRegistration.*` config) assumes +exactly one root and one set of settings. + +**Decision**: restructure config around a list, `autoApiRegistration.sources: [...]`, where each +entry carries its own `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`/`reconciliation`, +`defaultOwner`, and `xNamespace` (data-model.md has the full per-source schema). At backend-module +init time, the module reads this list and instantiates **one `EntityProvider` instance per +source**, each with a distinct `getProviderName()` (`auto-api-registration:`) — not one +provider trying to juggle multiple roots internally. This is the same pattern Backstage's own +multi-target providers use (e.g. a GitHub discovery provider configured against several +org/repo targets each becomes its own provider instance): each source gets its own independent +scheduling, its own independent full-then-delta mutation lifecycle (R6), and a config or scanning +problem in one source (e.g. an unreadable path for a team repo that hasn't been checked out yet) +cannot stall or break discovery for any other source. For backward/simple compatibility, a flat +`autoApiRegistration.rootPath`/`.patterns`/etc. (no `sources` list) is treated as shorthand for a +single implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so +the single-repo lab config does not need to change shape. + +**Problem 2 — some settings are legitimately per-repo, not global.** The two called out +explicitly: `defaultOwner` (a pre-production or team repo plausibly has a different fallback team +than the platform mono-repo's `platform-team`) and, less obviously, `xNamespace` — a team repo +that already has its own established `x-*` vendor-extension convention before onboarding to +auto-registration shouldn't be forced to rename it to match the platform mono-repo's convention +just to participate. + +**Decision**: `defaultOwner` and `xNamespace` are per-source config keys with no global fallback +required (each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` +block if the learner wants shared values) — see data-model.md's per-source schema. `patterns`, +`ignore`, `mode`, and scheduling are also per-source for the same reason (a small pre-production +repo may reasonably use `mode: 'poll'` with a short interval while the platform mono-repo runs +`mode: 'watch'` at scale, per R6). + +**Problem 3 — collision detection and `catalog-info.yaml` precedence (R4) must stay global, not +per-source.** Two different teams' repos can trivially produce the same slugified entity name +(e.g. two teams each shipping a `payments-api`). If collision/precedence checks were scoped to a +single source's own scan-state rows, cross-source collisions would go undetected and both +entities would silently coexist as long as neither the collision index nor the "does a +non-auto-sourced entity already exist" precedence check restricts by source. The scan-state cache +table already keyed `entity_name` for O(1) collision lookups (R6) — the fix is simply that this +index (and the query behind the `catalog-info.yaml` precedence check) is queried **across all +sources**, not scoped to the source currently being processed. The `source_id` column added to +the cache table (data-model.md) exists precisely so a collision error message can say *which two +repos* collided — important for SC-003-style debuggability once "which file" becomes "which file, +in which repo." + +**Problem 4 — "other repos" may mean a genuinely separate Git remote, not a local sibling +directory.** The lab's own Assumptions section already anticipates a local sibling checkout ("a +local sibling repository the learner creates") for its own zero-network, zero-cost demo — that +covers the lab's own multi-source walkthrough (see quickstart.md). It does **not** cover a real +deployment where a team's repo is a separate remote the discovery backend has never had checked +out locally at all. + +**Decision (documented extension point, not built or demonstrated by the lab)**: each source's +`rootPath` config key is designed to remain a plain local filesystem path — the discovery, +parsing, and mapping logic (R1–R6) is entirely decoupled from *how bytes arrive on local disk*. +A production deployment onboarding a genuinely remote repo adds a lightweight sync step ahead of +the existing glob/parse pipeline — e.g. a shallow `git clone`/`git pull` into a local cache +directory on each scheduled cycle (or triggered by the same watch/reconciliation cadence as R6) — +and points that source's `rootPath` at the resulting local worktree. Every other mechanism (glob +patterns, ignore rules, delta mutations, scan-state cache, collision/precedence checks) is reused +unchanged; only the sync step differs per source. This keeps the zero-cost constraint intact for +the lab (no sync step is needed or built when `rootPath` already points at a local sibling +directory) while giving a real multi-repo deployment a documented, non-invasive path to genuinely +separate remotes — noted in quickstart.md as a "beyond the lab" extension point, consistent with +how R6's watch-mode scale-up is documented rather than exercised. + +**Alternatives considered**: +- One `EntityProvider` instance internally looping over multiple roots — rejected: conflates + scheduling, failure isolation, and mutation lifecycle across unrelated sources; a parse error or + slow filesystem in one team's repo could delay or corrupt discovery for every other source. Per + Backstage's own `EntityProvider` contract, provider identity is meant to be 1:1 with "one + coherent versioned view of a set of entities" — multiple sources are multiple such views. +- Making collision detection per-source (accept that two repos could produce the same name) — + rejected: silently produces whichever entity happens to process last as the "winner" with no + visible error, directly violating the Edge Cases requirement that name collisions must surface a + visible conflict rather than being silently resolved. +- Building real Git-remote syncing (clone/pull) as part of this lab — rejected: requires network + access and (for private repos) credentials, which conflicts with the lab's zero-network, + zero-cost constraint (Constitution V); documented as an extension point instead (Problem 4). From f90811de665f4f93ac3f3f33dd2fc07416d4bbc0 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:36:25 +1000 Subject: [PATCH 06/19] Added ability to configure API visibility via x-* fields and repo defaults --- .../004-lab-4-auto-registration/data-model.md | 14 +++- specs/004-lab-4-auto-registration/plan.md | 38 ++++++----- .../004-lab-4-auto-registration/quickstart.md | 25 +++++-- specs/004-lab-4-auto-registration/research.md | 67 ++++++++++++++----- specs/004-lab-4-auto-registration/spec.md | 23 ++++--- 5 files changed, 119 insertions(+), 48 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index 091ec77..e8c3c08 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -16,6 +16,7 @@ A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncap | `tags[].name` | string[] | Optional, native top-level OpenAPI/AsyncAPI field (not under `info`, not under `x-examplecorp`). Default `[]` if absent. | | `info.x-examplecorp.owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration still succeeds using a documented default — see Mapping Rule 1 below. | | `info.x-examplecorp.lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | +| `info.x-examplecorp.visibility` | string | Optional. One of `private` \| `shared` — the same two values Lab 2 established. Absent → falls back to that file's source's configured `defaultVisibility` (source-level default is `'private'` if the source doesn't set one — research.md R3/R7). Present-but-unrecognized → marker/error entity, not silently defaulted (research.md R4). | ## Entity: Catalog API Entity (output) @@ -28,7 +29,8 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.description` | `info.description` | Verbatim if present, else omitted. | | `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | -| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | +| `metadata.annotations['example.com/visibility']` | `info.x-examplecorp.visibility` | **The exact same annotation key the Lab 2/3 permission policy already reads** (`packages/backend/src/extensions/permissionPolicy.ts`) — no new annotation, no policy change. **Mapping Rule 2**: if absent, falls back to that file's source's configured `defaultVisibility` — itself optional per-source, falling back to the fixed `'private'` "default default" if the source doesn't set one either (research.md R3/R7, FR-006a). An unrecognized (non-`private`/`shared`) value is never silently defaulted — it's a marker/error entity instead (research.md R4). Reused verbatim rather than reinvented, per research.md R3. | +| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref, invalid visibility value, or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | | `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | @@ -54,12 +56,14 @@ autoApiRegistration: reconciliation: { frequencySeconds: 900 } # watch mode only parseConcurrency: 4 defaultOwner: group:default/platform-team + defaultVisibility: private # optional — falls back to the fixed 'private' default if unset xNamespace: examplecorp - id: team-checkout-api # a second source: a different team's repo, a local sibling rootPath: ../team-checkout-api # checkout for the lab; a real remote repo's patterns: ['**/*-openapi.yaml'] # synced local worktree in production (R7 defaultOwner: group:default/checkout-team # Problem 4) - xNamespace: checkoutteam + defaultVisibility: shared # e.g. a pre-production repo where every API should default + xNamespace: checkoutteam # to shared visibility unless a spec says otherwise ``` | Key (per source, under `sources[]`, or flat for the single-source shorthand) | Type | Default | Purpose | @@ -73,7 +77,8 @@ autoApiRegistration: | `reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | | `parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | | `defaultOwner` | string | *(no global default — each source must set its own, or inherit `autoApiRegistration.defaults.defaultOwner` if configured)* | Fallback owner ref applied when `x-.owner` is absent (FR-007). Deliberately per-source, not global — a team/pre-production repo plausibly has a different fallback owner than the platform mono-repo (research.md R7, Problem 2). | -| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | +| `defaultVisibility` | `'private' \| 'shared'` | `'private'` (the one fixed, hardcoded fallback in the whole config surface — the "default default") | Fallback visibility applied when `x-.visibility` is absent for a file in this source (research.md R3/R4). Per-source, like `defaultOwner` — e.g. a pre-production repo may reasonably default every undeclared API to `shared`, while the platform mono-repo keeps the conservative `private` fallback. Unlike `defaultOwner`/`xNamespace`, this key is optional even at the source level: a source that omits it gets `private`, so a learner adding a new source doesn't have to think about visibility defaults unless they want to change them. | +| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle/visibility from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | ## Scan-state cache (persisted, `coreServices.database`) @@ -120,6 +125,9 @@ Problem 3) even though discovery itself runs per source. Problem 3). e. Resolve `x-.owner` (falling back to this source's `defaultOwner` if absent) against known `User`/`Group` entities → unresolvable owners become marker/error entities. + Resolve `x-.visibility` (falling back to this source's `defaultVisibility`, or + `'private'` if the source doesn't set one, if absent) → validate against `{private, shared}` + → an unrecognized (present-but-invalid) value becomes a marker/error entity (research.md R4). f. Check for a pre-existing, non-auto-sourced entity at the same name anywhere in the catalog (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 5f685bf..64a5ba8 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -21,24 +21,30 @@ instance with its own schedule, default owner, and `x-*` namespace, so a platfor one or more separate team/pre-production repos can each be discovered with their own settings while collision and `catalog-info.yaml`-precedence checks stay global across all of them). Only the config *values and entry count* differ between the lab demo (one small source) and a -production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle metadata -is sourced from a single vendor-namespaced -`info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one -namespace learners rename to their own company (FR-005–FR-006), with documented defaults when -absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), +production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle/visibility +metadata is sourced from a single vendor-namespaced +`info.x-examplecorp` object (`owner`, `lifecycle`, `visibility`) — tool-agnostic field names, +grouped under one namespace learners rename to their own company (FR-005–FR-006, FR-006a), with +documented defaults when absent (FR-007). The `visibility` value reuses the exact +`example.com/visibility` annotation and `private`/`shared` values the Lab 2/3 permission policy +already enforces — no new visibility mechanism is introduced, only a new way of setting the +existing annotation. Metadata that already has a natural home in the spec — name (`info.title`), description (`info.description`), tags (native top-level `tags`) — is sourced from there instead -of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, name collisions) -are surfaced via backend logs and, where an entity can be identified, via Backstage's native -catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker -annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a -discovered file, takes precedence over auto-sourced metadata (FR-011). +of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, invalid +visibility values, name collisions) are surfaced via backend logs and, where an entity can be +identified, via Backstage's native catalog processing-error UI through a companion +`CatalogProcessor` that throws on a marker annotation (FR-008). A hand-authored +`catalog-info.yaml`, where one exists alongside a discovered file, takes precedence over +auto-sourced metadata (FR-011). Two new sample files support the lab: a vendored copy of the MIT-licensed Scalar Galaxy API -(`galaxy-openapi.yaml`) with no pre-existing catalog registration, demonstrating genuine -"previously unregistered API" discovery (US1); and a small `precedence-demo-openapi.yaml` + -hand-authored `precedence-demo-catalog-info.yaml` pair demonstrating the precedence rule. -Lab documentation calls out which parts (glob patterns, root path, `x-*` field names) are -adaptable conventions per Constitution Principle VIII (FR-010, US3). +(`galaxy-openapi.yaml`, `x-examplecorp.visibility: shared`) with no pre-existing catalog +registration, demonstrating genuine "previously unregistered API" discovery (US1) and, alongside +the precedence-demo API's differing visibility, a real private/shared contrast to verify (SC-006); +and a small `precedence-demo-openapi.yaml` + hand-authored `precedence-demo-catalog-info.yaml` +pair demonstrating the precedence rule. Lab documentation calls out which parts (glob patterns, +root path, `x-*` field names) are adaptable conventions per Constitution Principle VIII (FR-010, +US3). ## Technical Context @@ -99,7 +105,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | Principle | Gate | Status | |-----------|------|--------| | I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: custom `EntityProvider` for filesystem-based catalog discovery, `x-*` vendor-extension metadata sourcing, native catalog processing-error surfacing via a custom `CatalogProcessor`, full-mutation-based create/update/retract lifecycle | ✅ Pass | -| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), and WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011) | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011), and WHY visibility reuses the Lab 2/3 `example.com/visibility` annotation/policy verbatim instead of introducing a parallel mechanism (FR-006a) | ✅ Pass | | III. Cross-Platform Compatibility | All new code is TypeScript/YAML; `fast-glob` and `js-yaml` are pure-JS, no native/platform-specific dependencies; `yarn add` and `yarn start` work identically on Windows and macOS; root-path resolution documented with platform-neutral relative paths | ✅ Pass | | IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–3's existing Node/Yarn setup — `fast-glob`/`js-yaml` are installed via the existing `yarn add` workflow; README documents the install step | ✅ Pass | | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 11e29ab..799c3cd 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -21,23 +21,33 @@ Constitution Principle II. data-model.md for keys/defaults). 4. **Add the two new sample files**: - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API - with an `info.x-examplecorp` object added) — no pre-existing `catalog-info.yaml`. + with an `info.x-examplecorp` object added, `visibility: shared`) — no pre-existing + `catalog-info.yaml`. - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + - `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. + `precedence-demo-catalog-info.yaml` (hand-authored, `example.com/visibility: private`, + owner a non-platform team) — demonstrates FR-011 precedence and gives a real + private/shared contrast to check in step 5. 5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually written catalog file (US1, SC-001). - Its `spec.owner` and `spec.lifecycle` match the `info.x-examplecorp` values, and its tags match the spec's native `tags` array — not duplicated into `x-examplecorp` (US2, SC-002). + - The Galaxy API (`visibility: shared`) is visible to any authenticated user, while the + precedence-demo API (`visibility: private`, hand-authored) is visible only to its owning + team and platform-team — logging in as a non-owning, non-platform user confirms the + difference (SC-006). - The precedence-demo API's catalog entity reflects the hand-authored - `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). + `precedence-demo-catalog-info.yaml` throughout (owner, visibility), not the auto-sourced + values (FR-011). 6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, confirm the existing entity updates rather than duplicating (FR-003). 7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. 8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line identifies the file; temporarily set `info.x-examplecorp.owner` to a nonexistent team, confirm a - catalog processing error is visible on that entity (FR-008, SC-003). + catalog processing error is visible on that entity (FR-008, SC-003); temporarily set + `info.x-examplecorp.visibility` to an unrecognized value (e.g. `public`), confirm the same kind + of visible error rather than a silent default. ## Out of scope for this quickstart @@ -62,10 +72,13 @@ A second "Scaling to multiple source repositories" note (research.md R7) covers: - The `autoApiRegistration.sources[]` config list — the lab's own `app-config.yaml` uses the flat single-source shorthand, but the README shows the equivalent explicit `sources: [{ id: default, ... }]` form alongside a second example entry, so learners can see exactly what changes to add a - team or pre-production repo (a different `rootPath`, `defaultOwner`, and `xNamespace`). + team or pre-production repo (a different `rootPath`, `defaultOwner`, `defaultVisibility`, and + `xNamespace`). - Why `defaultOwner` and `xNamespace` are per-source with no built-in global fallback (a repo onboarding to auto-registration shouldn't have to adopt the platform mono-repo's team or vendor - namespace to participate). + namespace to participate), while `defaultVisibility` is per-source but *does* have a built-in + global fallback (`private`) — visibility is the one setting where "unconfigured" must still be + safe by default, not just adaptable. - Why collision detection and `catalog-info.yaml` precedence checks are explicitly called out as *global* across sources, not per-source — two teams' repos producing the same entity name must still surface a visible conflict, not silently coexist. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 9f201bc..51057e4 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -83,6 +83,30 @@ top-level `info` object, for both OpenAPI and AsyncAPI (per the Clarifications s 2026-07-03 update): - `info.x-examplecorp.owner` → `spec.owner` - `info.x-examplecorp.lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) +- `info.x-examplecorp.visibility` → `metadata.annotations['example.com/visibility']` (default: that + file's source's configured `defaultVisibility`, itself defaulting to `private` if the source + doesn't set one — see rationale below and R7) + +**Visibility is deliberately reused, not reinvented.** Lab 2/3 already established +`example.com/visibility: private | shared` as the annotation the permission policy +(`packages/backend/src/extensions/permissionPolicy.ts`) reads to decide access: `shared` bypasses +ownership entirely (visible to all authenticated users); anything else (including the annotation +being absent) falls through to the ownership-only rule. This lab adds a **third way to set that +same annotation** — sourced from `x-examplecorp.visibility` instead of hand-authored in +`catalog-info.yaml` — it does not add a new visibility concept, a new annotation key, or touch the +permission policy at all. The absent-field default is **per-source**, not a single hardcoded +value: each source in `autoApiRegistration.sources[]` may set its own `defaultVisibility` +(`private` or `shared`) for files that don't declare one — e.g. a pre-production repo where most +APIs are intentionally shared for cross-team testing might reasonably default to `shared`, while +the platform mono-repo keeps the conservative default. The "default default" — what a source gets +if it doesn't configure `defaultVisibility` at all — is `private`, mirroring the permission +policy's own existing fallback behavior when the annotation is missing entirely (Rule 3, +ownership-only), so a freshly-added source with no visibility configuration behaves identically to +a hand-authored entity with no visibility annotation. An `x-examplecorp.visibility` value that is +present but not `private` or `shared` is treated as malformed input (Edge Cases) — surfaced via +the same marker-entity + processor-error path as an invalid owner reference (R4), not silently +coerced to a default (source-level or otherwise), since silently defaulting a security-relevant +setting is exactly the kind of mistake that must stay visible. `examplecorp` is this lab's fictional company namespace (documented as the one thing every learner renames first, per FR-010/US3). The field names deliberately omit `backstage` — this @@ -141,6 +165,13 @@ no bespoke error-reporting system is built: no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the repo). Unresolvable owners produce the marker-entity + processor-error path above. +**Visibility validation**: `x-examplecorp.visibility`, if present, is checked against the fixed +set of values the existing permission policy understands (`private`, `shared`) — this is a static +enum check, not a catalog lookup, so it's cheap regardless of scale. An unrecognized value follows +the same marker-entity + processor-error path as an invalid owner reference. If absent, the value +comes from that file's source's `defaultVisibility` (falling back to `private` if the source +doesn't configure one) — see R3 for the full rationale and R7 for why this is per-source. + **Name collisions**: if two discovered files slugify to the same candidate entity name, the first (sorted by file path for determinism) is registered normally; the second is emitted as a marker entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the @@ -166,8 +197,10 @@ processed — a hand-authored file anywhere wins over an auto-sourced candidate github.com/scalar/scalar, published build at `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with an `info.x-examplecorp` -object (`owner`, `lifecycle`) added directly in the vendored copy; its existing native `tags` -array is used as-is for `metadata.tags`. The upstream file is a single self-contained OpenAPI 3.1 +object (`owner`, `lifecycle`, `visibility: shared` — chosen so the quickstart's SC-006 visibility +check has a real difference to demonstrate against the precedence-demo API below) added directly +in the vendored copy; its existing native `tags` array is used as-is for `metadata.tags`. The +upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and @@ -292,19 +325,23 @@ cannot stall or break discovery for any other source. For backward/simple compat single implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so the single-repo lab config does not need to change shape. -**Problem 2 — some settings are legitimately per-repo, not global.** The two called out -explicitly: `defaultOwner` (a pre-production or team repo plausibly has a different fallback team -than the platform mono-repo's `platform-team`) and, less obviously, `xNamespace` — a team repo -that already has its own established `x-*` vendor-extension convention before onboarding to -auto-registration shouldn't be forced to rename it to match the platform mono-repo's convention -just to participate. - -**Decision**: `defaultOwner` and `xNamespace` are per-source config keys with no global fallback -required (each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` -block if the learner wants shared values) — see data-model.md's per-source schema. `patterns`, -`ignore`, `mode`, and scheduling are also per-source for the same reason (a small pre-production -repo may reasonably use `mode: 'poll'` with a short interval while the platform mono-repo runs -`mode: 'watch'` at scale, per R6). +**Problem 2 — some settings are legitimately per-repo, not global.** Called out explicitly: +`defaultOwner` (a pre-production or team repo plausibly has a different fallback team than the +platform mono-repo's `platform-team`); `defaultVisibility` (a pre-production repo might reasonably +default undeclared APIs to `shared` for cross-team testing, while the platform mono-repo keeps a +conservative `private` default — R3); and, less obviously, `xNamespace` — a team repo that already +has its own established `x-*` vendor-extension convention before onboarding to auto-registration +shouldn't be forced to rename it to match the platform mono-repo's convention just to participate. + +**Decision**: `defaultOwner`, `defaultVisibility`, and `xNamespace` are per-source config keys — +see data-model.md's per-source schema. `defaultOwner` and `xNamespace` have no global fallback +(each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` block if +the learner wants shared values); `defaultVisibility` is the one exception with a genuine global +fallback (`private`, R3) precisely because leaving visibility completely unconfigured must still +be safe by default — a source shouldn't have to opt into a safe default, only opt out of it. +`patterns`, `ignore`, `mode`, and scheduling are also per-source for the same reason (a small +pre-production repo may reasonably use `mode: 'poll'` with a short interval while the platform +mono-repo runs `mode: 'watch'` at scale, per R6). **Problem 3 — collision detection and `catalog-info.yaml` precedence (R4) must stay global, not per-source.** Two different teams' repos can trivially produce the same slugified entity name diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index 60e98e2..cba7734 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -23,6 +23,7 @@ - Q: Should `x-*` fields sit directly in `info`, or nested under a single vendor-namespaced object? → A: Nested under a single `info.x-examplecorp` object (e.g. `info.x-examplecorp.owner`, `info.x-examplecorp.lifecycle`) — `examplecorp` is this lab's fictional company namespace, which learners are expected to rename to their own; a single grouping object avoids polluting `info` with multiple top-level `x-*` keys and makes "which fields are ours" unambiguous at a glance. - Q: Should field names retain the `backstage` tool name (`x-backstage-owner`, etc.)? → A: No — the field names describe the metadata (owner, lifecycle), not the tool consuming it. Catalog metadata like ownership is tool-agnostic (equally meaningful to an API gateway or any other consumer), so `backstage` is dropped: `x-examplecorp.owner`, `x-examplecorp.lifecycle`. - Q: Should `tags` remain a recognized `x-*` field? → A: No — removed. OpenAPI/AsyncAPI both already have a native top-level `tags` field; sourcing tags from `x-*` as well would duplicate data that already has a natural home in the spec, risking divergence. Catalog `metadata.tags` is now sourced from the spec's native `tags` array (`tags[].name`), not from any `x-*` field. The same reasoning already applied to `info.title` (entity name) and `info.description` (entity description), which were never duplicated into `x-*` fields. +- Q: Should Lab 2's API visibility concept (private/shared) be added to the `x-*`-managed metadata? → A: Yes — `info.x-examplecorp.visibility`, accepting the same `private` \| `shared` values Lab 2 already established, mapped to the same `example.com/visibility` annotation the Lab 2/3 permission policy already reads. Visibility has no natural home elsewhere in the spec (unlike name/description/tags), so it belongs in `x-examplecorp` alongside owner and lifecycle, not as a new mechanism. ## User Scenarios & Testing *(mandatory)* @@ -44,7 +45,7 @@ As a platform engineer, when a new API definition file (OpenAPI or AsyncAPI) is ### User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) -As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or tags) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. +As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or visibility) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. **Why this priority**: This is what makes registration genuinely "automatic" rather than just file discovery — without metadata sourcing, every API would still need a hand-maintained catalog file for ownership and lifecycle, which is exactly what Lab 2's ownership model depends on. @@ -53,9 +54,10 @@ As an API owner, I want to declare metadata that Backstage needs (such as owning **Acceptance Scenarios**: 1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. -2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, visibility), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. 3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. 4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. +5. **Given** an API definition containing an `x-*` visibility field, **When** the entity is registered, **Then** the catalog entity's visibility annotation matches that value and is enforced by the same permission policy introduced in Lab 2 (private → owning team + platform team only; shared → all authenticated users), with no separate visibility mechanism introduced for auto-registered APIs. --- @@ -78,6 +80,7 @@ As a learner adapting this repository to my own organization, I want to understa - What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. - What happens when two API definition files resolve to the same catalog entity name (i.e., the same slugified `info.title`)? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. - What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. +- What happens when the `x-*` visibility field is present but is not one of the recognized values (`private`, `shared`)? The system MUST surface a visible error for that entity rather than silently defaulting or registering it with an unenforceable visibility value. - What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. - How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. @@ -90,9 +93,10 @@ As a learner adapting this repository to my own organization, I want to understa - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and visibility) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. +- **FR-006a**: The system MUST extract API visibility from the documented `x-*` object and set the same visibility annotation the Lab 2 permission policy already enforces (`example.com/visibility`, values `private` or `shared`), reusing that existing policy mechanism rather than introducing a separate visibility concept or enforcement path for auto-registered APIs. When a given API definition omits the visibility field, the system MUST apply a default that is independently configurable per mono-repo/source location (so different source repositories can have different default visibility), falling back to `private` for any source that does not configure its own default. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. -- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. +- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, specifies an unrecognized visibility value, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. - **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. - **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. @@ -101,8 +105,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities -- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle metadata sourced from `info.x-examplecorp`. +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle, visibility). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle/visibility metadata sourced from `info.x-examplecorp`. - **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. - **`x-*` Metadata Mapping**: The documented mapping from the `info.x-examplecorp` object's fields to catalog entity fields/annotations, including default values applied when a field is absent, and the explicit list of fields sourced natively rather than from `x-*` (name, description, tags). @@ -111,7 +115,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Measurable Outcomes - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. -- **SC-002**: 100% of the owning-team and lifecycle metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. +- **SC-002**: 100% of the owning-team, lifecycle, and visibility metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. +- **SC-006**: A learner can set an API's visibility via its `x-*` field alone (no hand-authored catalog file) and observe the same access behavior Lab 2 documented for that visibility value (private vs. shared) when viewing the catalog as different users, confirming the auto-sourced value drives the existing permission policy rather than a separate mechanism. - **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. @@ -120,7 +125,9 @@ As a learner adapting this repository to my own organization, I want to understa - The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner` and `lifecycle` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. +- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner`, `lifecycle`, and `visibility` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. +- The visibility model established in Lab 2/3 (the `example.com/visibility` annotation and its `private`/`shared` values, enforced by the existing permission policy) is the source of truth referenced when mapping the `x-*` visibility field — Lab 4 does not introduce a new/parallel visibility mechanism, only a new way of setting the same annotation. +- The default visibility applied when a definition omits the `x-*` visibility field is configured per mono-repo/source location, not globally fixed — different source repositories (e.g. a pre-production repo vs. the platform mono-repo) may reasonably want different defaults. A source that does not configure its own default falls back to `private`, matching the permission policy's existing behavior when no visibility annotation is present at all. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From e2f3d96299ed7f0cc0f991fd20b54196180678e9 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:41:56 +1000 Subject: [PATCH 07/19] Implementation tasks planned --- specs/004-lab-4-auto-registration/tasks.md | 207 +++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 specs/004-lab-4-auto-registration/tasks.md diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md new file mode 100644 index 0000000..f232d5c --- /dev/null +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -0,0 +1,207 @@ +--- + +description: "Task list for Lab 4 — Auto Registration" +--- + +# Tasks: Lab 4 — Auto Registration + +**Input**: Design documents from `/specs/004-lab-4-auto-registration/` + +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md (all present; no `contracts/` — this lab produces no external API contracts) + +**Tests**: Not requested. Per plan.md's Testing section, verification is manual (browser + backend logs) against quickstart.md's steps — consistent with Labs 1–3. No automated test tasks are generated; quickstart validation is folded into each story's checkpoint and the final Polish phase. + +**Organization**: Tasks are grouped by user story (spec.md priorities P1/P2/P3) to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (US1, US2, US3) + +## Path Conventions + +This is a tutorial-lab project (not a generic web/mobile app). Two path roots are used: + +- **Backstage instance** (modified in place, from Lab 1): `labs/lab-01-base-backstage/backstage/` +- **Lab 4 teaching content** (new, committed alongside README): `labs/lab-04-auto-registration/` +- **Speckit artifacts**: `specs/004-lab-4-auto-registration/` (this directory — no code changes here) + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Add dependencies and scaffold the directories every later task writes into. + +- [ ] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) +- [ ] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` +- [ ] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` + +**Checkpoint**: Dependencies installed, directories exist — ready for foundational work. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Build the shared scanning/parsing/persistence/config plumbing that both US1 (discovery/CRUD) and US2 (metadata sourcing) sit on top of. No user story is independently testable until this phase is done — the EntityProvider skeleton it produces is what US1 wires up to catalog mutations and US2 extends with `x-*` extraction. + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete. + +- [ ] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section +- [ ] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) +- [ ] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) +- [ ] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) +- [ ] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) +- [ ] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) +- [ ] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) +- [ ] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) + +**Checkpoint**: Config loads, files are discoverable and parseable, scan-state cache persists — foundation ready for US1. + +--- + +## Phase 3: User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) 🎯 MVP + +**Goal**: A new OpenAPI/AsyncAPI file dropped anywhere under the mono-repo location appears in the Backstage catalog with no hand-authored `catalog-info.yaml`; edits update the entity in place; removing the file retracts it. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file, trigger discovery, confirm a corresponding API entity appears — with default owner/lifecycle values (no `x-*` extraction required for this story) — with no manual `catalog-info.yaml` written for it; edit the file and confirm update-in-place; remove the file and confirm the entity is no longer active. + +### Implementation for User Story 1 + +- [ ] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) +- [ ] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) +- [ ] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) +- [ ] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) +- [ ] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) +- [ ] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) + +**Checkpoint**: User Story 1 fully functional and independently testable — discovery, create, update, and removal all work end-to-end with default metadata. + +--- + +## Phase 4: User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +**Goal**: `spec.owner`, `spec.lifecycle`, and the `example.com/visibility` annotation are sourced from each file's `info.x-examplecorp` object (with documented defaults when absent), invalid values/references surface visible errors on both backend logs and the catalog processing-errors UI, name collisions and hand-authored `catalog-info.yaml` precedence are respected. + +**Independent Test**: Author an API definition with `x-examplecorp` owner/lifecycle/visibility fields, run discovery, confirm the resulting entity's `spec.owner`, `spec.lifecycle`, and `example.com/visibility` annotation reflect those values (not defaults); omit fields and confirm documented defaults apply without blocking registration; set an invalid owner/visibility and confirm a visible error on that entity without affecting other entities; confirm a hand-authored `catalog-info.yaml` wins over auto-sourced metadata for the same API. + +### Implementation for User Story 2 + +- [ ] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) +- [ ] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) +- [ ] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) +- [ ] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) +- [ ] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) +- [ ] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) +- [ ] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) +- [ ] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) +- [ ] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) +- [ ] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) +- [ ] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) + +**Checkpoint**: User Stories 1 AND 2 both work independently — metadata sourcing, defaults, validation errors, collisions, and precedence are all verified. + +--- + +## Phase 5: User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +**Goal**: Lab documentation clearly separates adaptable conventions (file location pattern, `x-*` field names, multi-source config) from fixed discovery mechanics, so a learner can repoint the lab at their own layout. + +**Independent Test**: Follow the lab README's configuration section to identify which settings are adaptable vs. fixed, and confirm the documented `sources[]` multi-repo example is internally consistent with data-model.md's schema. + +### Implementation for User Story 3 + +- [ ] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) +- [ ] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) + +**Checkpoint**: All three user stories independently functional; README fully documents adaptable vs. fixed conventions. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +**Purpose**: Final repo-wide consistency checks that span multiple stories. + +- [ ] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI +- [ ] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development +- [ ] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies — start immediately +- **Foundational (Phase 2)**: Depends on Setup completion — BLOCKS all user stories +- **User Story 1 (Phase 3)**: Depends on Foundational completion — no dependency on US2/US3 +- **User Story 2 (Phase 4)**: Depends on Foundational completion; extends the same file US1 built (T012), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency +- **User Story 3 (Phase 5)**: Depends on US1 (T015) and US2 (T024) being implemented, since the README documents both the discovery mechanism and the metadata-sourcing/error behavior +- **Polish (Phase 6)**: Depends on all user stories being complete + +### User Story Dependencies + +- **User Story 1 (P1)**: Can start after Foundational — no dependency on other stories +- **User Story 2 (P2)**: Can start after Foundational, but shares `autoApiRegistration.ts` with US1 (same file, sequential edits) — implement after US1 for a working MVP checkpoint first +- **User Story 3 (P3)**: Documentation-only; depends on US1 and US2 being functionally complete so the README describes real, working behavior + +### Within Each User Story + +- Mapping/extraction logic before mutation-emission logic before scheduling +- Sample files before manual verification steps +- Verification task last in each phase (checkpoint gate) + +### Parallel Opportunities + +- T002, T003 (Setup) in parallel +- T006, T007, T008 (Foundational: cache migration, discovery utility, parse utility) in parallel — distinct concerns, though ultimately land in the same file, so treat [P] as "logically independent," and serialize the actual edits if working solo +- T026, T027 (US2 sample files) in parallel with each other +- T031, T032 (US3 README sections) in parallel with each other +- T035, T037 (Polish) in parallel + +--- + +## Parallel Example: Foundational Phase + +```bash +# Logically independent foundational concerns (same target file — serialize edits if solo): +Task: "Implement scan-state cache DB migration in autoApiRegistrationMigrations/001_scan_state_cache.ts" +Task: "Implement streamed file discovery utility in autoApiRegistration.ts" +Task: "Implement YAML parse + shape-check utility in autoApiRegistration.ts" +``` + +## Parallel Example: User Story 2 Sample Files + +```bash +Task: "Create precedence-demo-openapi.yaml" +Task: "Create precedence-demo-catalog-info.yaml" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup +2. Complete Phase 2: Foundational (CRITICAL — blocks all stories) +3. Complete Phase 3: User Story 1 +4. **STOP and VALIDATE**: T017 confirms discovery/create/update/remove work end-to-end with default metadata +5. Demo: a new spec file appears in the catalog with zero hand-authored `catalog-info.yaml` + +### Incremental Delivery + +1. Setup + Foundational → scanning/parsing/persistence plumbing ready +2. Add User Story 1 → validate independently (T017) → MVP demo +3. Add User Story 2 → validate independently (T028) → owner/lifecycle/visibility sourcing, error surfacing, precedence all work +4. Add User Story 3 → README documents adaptability → lab is learner-ready +5. Polish → cold-start validation, constitution re-check + +### Notes + +- US1 and US2 share one implementation file (`autoApiRegistration.ts`) because the architecture (research.md R1) is a single `EntityProvider` — this is a deliberate single-module design, not an accidental cross-story coupling. Each story still has an independently verifiable acceptance test (T017 vs. T028) even though the code lands in the same file. +- No test tasks are included (Tests: Not requested — see header). Verification is manual, per quickstart.md, folded into each story's final task. +- `watch` mode (research.md R6) and the multi-source sync-step extension point (research.md R7 Problem 4) are documented (T031/T032) but not built or exercised end-to-end — consistent with quickstart.md's explicit "Out of scope for this quickstart" section. From fff3dd42107041eef7471f3dfe4b0b9a7bb78496 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 11:41:45 +1000 Subject: [PATCH 08/19] Implemented --- labs/lab-04-auto-registration/README.md | 1196 +++++++++++++ .../apis/galaxy/galaxy-openapi.yaml | 1476 +++++++++++++++++ .../precedence-demo-catalog-info.yaml | 21 + .../precedence-demo-openapi.yaml | 59 + .../004-lab-4-auto-registration/data-model.md | 4 +- specs/004-lab-4-auto-registration/research.md | 12 +- specs/004-lab-4-auto-registration/tasks.md | 74 +- 7 files changed, 2799 insertions(+), 43 deletions(-) create mode 100644 labs/lab-04-auto-registration/README.md create mode 100644 labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml create mode 100644 labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml create mode 100644 labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md new file mode 100644 index 0000000..955a8ed --- /dev/null +++ b/labs/lab-04-auto-registration/README.md @@ -0,0 +1,1196 @@ +# Lab 4 — Auto Registration + +## Overview + +Labs 1–3 registered every API by hand-authoring a `catalog-info.yaml` per API and adding it as a +`type: url` catalog location. That works for three APIs; it does not work for a mono-repo with +hundreds. This lab replaces the per-API hand-authored file with a custom Backstage backend module +that scans a directory tree for `*-openapi.yaml` / `*-asyncapi.yaml` files and turns each one +directly into a catalog `API` entity — no `catalog-info.yaml` required. + +By the end of this lab you will have: + +- A new backend module (`autoApiRegistration.ts` + a small database migration) that discovers, + parses, and registers API definitions on a poll schedule, using Backstage's `EntityProvider` + mechanism so create, update, and removal all come from the same code path. +- Owner, lifecycle, and visibility metadata sourced from each spec's own `info.x-examplecorp` + object, with documented defaults when a field is absent — no duplication of metadata that + already has a natural home in the spec (`info.title`, `info.description`, native `tags`). +- Errors (malformed files, unresolvable owners, invalid visibility values, name collisions) + surfaced through Backstage's own backend logs and its native catalog processing-error UI — no + bespoke error-reporting system. +- Two new sample APIs: a vendored copy of the Scalar Galaxy API (pure auto-discovery, no + `catalog-info.yaml`) and a small precedence-demo API pair that demonstrates a hand-authored + `catalog-info.yaml` winning over auto-sourced metadata for the same API. + +**What you will learn:** + +- Why a custom `EntityProvider` (not a `CatalogProcessor`) is the right mechanism for + filesystem-based catalog discovery, and how its `full`/`delta` mutation lifecycle gives you + create/update/remove for free +- How to source catalog metadata from a vendor-namespaced `x-*` extension instead of a parallel + hand-authored file, and how to fall back safely when a field is absent +- How to surface processing errors through Backstage's built-in mechanisms (backend logs + a + `CatalogProcessor` that throws on a marker annotation) instead of inventing a new one +- How a persisted scan-state cache and `sources[]` config generalize this mechanism from "one + small demo repo" to "1000+ files across multiple team repos" without an architecture change + +--- + +## Prerequisites + +- **Lab 3 completed** — Backstage running locally with Museum API, Streetlights API, and Train + Travel API registered; `eve` (platform team) able to sign in and see quality detail +- **Node.js 20 LTS** and **Yarn** — same versions as Labs 1–3 + +--- + +## Step 1 — Add Dependencies + +From `packages/backend`, add the three new direct dependencies (`fast-glob` for the filesystem +scan, `js-yaml` for parsing, `chokidar` for the optional `watch` mode used at real-world scale): + +``` +cd packages/backend +yarn add fast-glob@3.3.3 js-yaml@4.2.0 chokidar@3.6.0 +``` + +All three are already present transitively in the workspace — this promotes them to explicit +dependencies of the backend package, which is what actually imports them. + +--- + +## Step 2 — Add the Auto-Registration Backend Module + +This is the heart of the lab: a backend module that scans a directory for API definition files, +parses each one, and emits catalog `API` entities via a custom `EntityProvider`. A companion +`CatalogProcessor` surfaces registration errors (invalid owner, invalid visibility, name +collisions) through Backstage's native processing-error UI. + +**Why an `EntityProvider`, not a `CatalogProcessor`?** A `CatalogProcessor` only transforms +entities that already exist via some `Location` — it can't originate new entities from an +arbitrary filesystem scan on its own. An `EntityProvider` owns a versioned "set of entities I +currently know about"; anything it stops claiming is automatically retracted by the catalog. That +gives create, update, and removal-on-delete from one code path, with no separate cleanup logic. + +**Why is owner/lifecycle/visibility sourced from `info.x-examplecorp` instead of duplicated +fields?** Metadata that already has a natural home in the spec — `info.title` (name), +`info.description` (description), native top-level `tags` — is read from there, not duplicated +into a vendor extension. Only metadata with **no** natural OpenAPI/AsyncAPI home (who owns this +API, what lifecycle stage it's in, whether it's private or shared) goes under `x-examplecorp`. +`examplecorp` is a fictional company namespace — it's the first thing you should rename to your +own company when adapting this lab (see "Adaptable Conventions" below). + +**Why does visibility reuse the exact `example.com/visibility` annotation from Lab 2/3?** The +permission policy from Lab 3 (`packages/backend/src/extensions/permissionPolicy.ts`) already +reads that annotation to decide access: `shared` bypasses ownership entirely, anything else +(including absent) falls through to the ownership-only rule. This lab adds a **third way to set +that same annotation** — sourced from `x-examplecorp.visibility` — not a new visibility mechanism +or a change to the permission policy. + +**Why do errors ride on Backstage's own backend logs and processing-error UI, instead of a new +error-reporting system?** Every discovery cycle logs one line per skipped/errored file via the +standard backend logger. For errors on a file specific enough to have a candidate entity (invalid +owner, invalid visibility, name collision), the provider still emits a minimal entity tagged with +an `apiportal-lab.io/registration-error` annotation; a small companion `CatalogProcessor` throws an +`InputError` when it sees that annotation, and Backstage's catalog engine natively records and +surfaces per-entity processing errors — the same "Inspect entity" / processing-errors view you'd +see for any other catalog ingestion problem. One message, written once, drives both surfaces. +Files that fail even the basic shape check (not parseable as an API spec at all — no +`openapi`/`asyncapi` field, no `info.title`) can't produce a minimal entity (there's no name to +register under), so those are logged only, not surfaced in the UI. + +**Why does a hand-authored `catalog-info.yaml` win over auto-sourced metadata?** If a file already +has a hand-authored catalog entry (`kind: API`, matching `metadata.name`) that didn't come from +this provider, the provider skips its own auto-sourced candidate for that name entirely. This +lets you hand-author precise metadata for a specific API — pinning an owner, tightening +visibility — without the auto-registration mechanism fighting you for it every cycle. + +Create the database migration first, at +`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`: + +```typescript +// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts +// +// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal +// state private to this backend module, used to avoid re-parsing unchanged files on restart, to +// compute delta mutations, and to support collision/precedence checks across every configured +// source. +// +// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a +// filesystem `directory` migration source, since this file lives alongside the module's +// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. + +import type { Knex } from 'knex'; + +export const TABLE_NAME = 'auto_api_registration_scan_state'; + +export async function up(knex: Knex): Promise { + await knex.schema.createTable(TABLE_NAME, table => { + table.string('source_id').notNullable(); + table.string('file_path').notNullable(); + table.bigInteger('mtime_ms').notNullable(); + table.string('content_hash').notNullable(); + table.string('entity_name').notNullable(); + table.text('last_error').nullable(); + + table.primary(['source_id', 'file_path']); + table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); + }); +} + +export async function down(knex: Knex): Promise { + await knex.schema.dropTable(TABLE_NAME); +} +``` + +Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts`: + +```typescript +// packages/backend/src/extensions/autoApiRegistration.ts +// +// Lab 4: Auto Registration. +// +// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files +// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. + +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import fg from 'fast-glob'; +import chokidar from 'chokidar'; +import yaml from 'js-yaml'; +import { + coreServices, + createBackendModule, + resolvePackagePath, +} from '@backstage/backend-plugin-api'; +import { + catalogProcessingExtensionPoint, + catalogServiceRef, +} from '@backstage/plugin-catalog-node'; +import type { + CatalogProcessor, + CatalogProcessorEmit, + CatalogService, + DeferredEntity, + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-node'; +import type { + AuthService, + DatabaseService, + LoggerService, + RootConfigService, +} from '@backstage/backend-plugin-api'; +import { + ANNOTATION_LOCATION, + ANNOTATION_ORIGIN_LOCATION, + parseEntityRef, + stringifyEntityRef, +} from '@backstage/catalog-model'; +import type { Entity } from '@backstage/catalog-model'; +import { InputError } from '@backstage/errors'; +import type { Knex } from 'knex'; +import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; + +// --------------------------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------------------------- + +export const VISIBILITY_ANNOTATION = 'example.com/visibility'; +export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; +// Internal-only marker: identifies which provider instance produced an entity, so the +// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from +// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. +const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; + +const KNOWN_VISIBILITIES = new Set(['private', 'shared']); +const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; +const DEFAULT_IGNORE = [ + '**/node_modules/**', + '**/.git/**', + '**/dist/**', + '**/build/**', +]; + +export interface SourceConfig { + id: string; + rootPath: string; + patterns: string[]; + ignore: string[]; + mode: 'poll' | 'watch'; + scheduleFrequencySeconds: number; + reconciliationFrequencySeconds: number; + parseConcurrency: number; + defaultOwner: string; + defaultVisibility: 'private' | 'shared'; + xNamespace: string; +} + +function defaultRootPath(): string { + // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs + return path.resolve( + resolvePackagePath('backend'), + '../../../../lab-04-auto-registration/apis', + ); +} + +export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { + const raw = rootConfig.getOptionalConfig('autoApiRegistration'); + if (!raw) { + return []; + } + + const entries = raw.has('sources') + ? raw.getConfigArray('sources') + : [raw]; + + return entries.map(entry => { + const id = raw.has('sources') ? entry.getString('id') : 'default'; + const defaultOwner = entry.getOptionalString('defaultOwner'); + const xNamespace = entry.getOptionalString('xNamespace'); + if (!defaultOwner) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "defaultOwner"`, + ); + } + if (!xNamespace) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "xNamespace"`, + ); + } + const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; + if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { + throw new Error( + `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, + ); + } + + return { + id, + rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), + patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, + ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, + mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', + scheduleFrequencySeconds: + entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, + reconciliationFrequencySeconds: + entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, + parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, + defaultOwner, + defaultVisibility: defaultVisibility as 'private' | 'shared', + xNamespace, + }; + }); +} + +// --------------------------------------------------------------------------------------------- +// Discovery + parsing +// --------------------------------------------------------------------------------------------- + +async function* discoverFiles(source: SourceConfig): AsyncGenerator { + const stream = fg.stream(source.patterns, { + cwd: source.rootPath, + ignore: source.ignore, + absolute: true, + onlyFiles: true, + }); + for await (const entry of stream) { + yield entry.toString(); + } +} + +interface ParsedSpec { + kind: 'openapi' | 'asyncapi'; + title: string; + description?: string; + tags: string[]; + raw: Record; +} + +/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ +function parseSpecFile(contents: string): ParsedSpec | undefined { + let doc: unknown; + try { + doc = yaml.load(contents); + } catch { + return undefined; + } + if (!doc || typeof doc !== 'object') { + return undefined; + } + const obj = doc as Record; + const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; + if (!kind) { + return undefined; + } + const info = obj.info as Record | undefined; + const title = info?.title; + if (typeof title !== 'string' || title.length === 0) { + return undefined; + } + // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag + // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed + // through verbatim. + const tags = Array.isArray(obj.tags) + ? (obj.tags as Array>) + .map(t => t?.name) + .filter((t): t is string => typeof t === 'string') + .map(slugify) + .filter(t => t.length > 0) + : []; + + return { + kind, + title, + description: typeof info?.description === 'string' ? (info.description as string) : undefined, + tags, + raw: obj, + }; +} + +export function slugify(title: string): string { + return title + .toLowerCase() + .trim() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 63); +} + +// --------------------------------------------------------------------------------------------- +// Candidate mapping +// --------------------------------------------------------------------------------------------- + +interface MappingResult { + entity: Entity; + entityName: string; + error?: string; +} + +function xField(raw: Record, namespace: string, field: string): string | undefined { + const info = raw.info as Record | undefined; + const ns = info?.[`x-${namespace}`] as Record | undefined; + const value = ns?.[field]; + return typeof value === 'string' ? value : undefined; +} + +function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { + const value = rawOwner ?? defaultOwner; + return stringifyEntityRef( + parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), + ); +} + +function buildEntity(opts: { + providerName: string; + filePath: string; + contents: string; + parsed: ParsedSpec; + entityName: string; + owner: string; + lifecycle: string; + visibility: string; + registrationError?: string; +}): Entity { + const annotations: Record = { + // Identifies the source file for debugging and the catalog-info.yaml precedence check — not + // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no + // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider + // already holds the bytes; no fetch is needed). + [ANNOTATION_LOCATION]: `file:${opts.filePath}`, + [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, + [VISIBILITY_ANNOTATION]: opts.visibility, + [MANAGED_BY_ANNOTATION]: opts.providerName, + }; + if (opts.registrationError) { + annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; + } + + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'API', + metadata: { + name: opts.entityName, + title: opts.parsed.title, + description: opts.parsed.description, + tags: opts.parsed.tags, + annotations, + }, + spec: { + type: opts.parsed.kind, + lifecycle: opts.lifecycle, + owner: opts.owner, + definition: opts.contents, + }, + }; +} + +/** + * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility + * validation, and collision detection. Does not perform the catalog-info.yaml precedence check + * (that requires a catalog query and is done by the caller, once, per file). + */ +function mapCandidate(opts: { + providerName: string; + source: SourceConfig; + filePath: string; + contents: string; + parsed: ParsedSpec; + knownOwnerRefs: Set; + collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any +}): MappingResult { + const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; + const baseSlug = slugify(parsed.title); + + const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); + const owner = resolveOwnerRef(rawOwner, source.defaultOwner); + const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; + const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); + const visibility = rawVisibility ?? source.defaultVisibility; + + // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently + // defaulted. + if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility: source.defaultVisibility, + registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, + }), + error: 'invalid-visibility', + }; + } + + // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity + // instead of blocking the whole cycle. + if (!knownOwnerRefs.has(owner)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility, + registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, + }), + error: 'invalid-owner', + }; + } + + // Rule: a name collision with another file (in this source or another) is a suffixed + // marker/error entity — first-by-path wins, this one loses. + if (collidesWith) { + const collisionName = `${baseSlug}-collision`; + return { + entityName: collisionName, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: collisionName, + owner, + lifecycle, + visibility, + registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, + }), + error: 'name-collision', + }; + } + + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner, + lifecycle, + visibility, + }), + }; +} + +// --------------------------------------------------------------------------------------------- +// Scan-state cache +// --------------------------------------------------------------------------------------------- + +interface CacheRow { + source_id: string; + file_path: string; + mtime_ms: number; + content_hash: string; + entity_name: string; + last_error: string | null; +} + +class ScanStateCache { + constructor(private readonly knex: Knex) {} + + static async create(database: DatabaseService): Promise { + const knex = await database.getClient(); + // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered + // against the 'catalog' plugin shares that plugin's own database, so using knex's default + // `knex_migrations` table here would validate our one-migration `migrationSource` against the + // real catalog plugin's own (much longer) applied-migrations history and fail with + // "migration directory is corrupt". + await knex.migrate.latest({ + tableName: 'auto_api_registration_migrations', + migrationSource: { + async getMigrations() { + return ['001_scan_state_cache']; + }, + getMigrationName(migration: string) { + return migration; + }, + async getMigration() { + return scanStateCacheMigration; + }, + }, + }); + return new ScanStateCache(knex); + } + + async rowsForSource(sourceId: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); + } + + /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ + async rowsByEntityName(entityName: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); + } + + async upsert(row: CacheRow): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .insert(row) + .onConflict(['source_id', 'file_path']) + .merge(); + } + + async remove(sourceId: string, filePath: string): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .where({ source_id: sourceId, file_path: filePath }) + .delete(); + } +} + +function sha1(contents: string): string { + return crypto.createHash('sha1').update(contents).digest('hex'); +} + +// --------------------------------------------------------------------------------------------- +// EntityProvider (one instance per source) +// --------------------------------------------------------------------------------------------- + +async function mapLimit( + items: T[], + limit: number, + fn: (item: T) => Promise, +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + async function worker() { + while (next < items.length) { + const index = next++; + results[index] = await fn(items[index]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); + return results; +} + +class AutoApiRegistrationEntityProvider implements EntityProvider { + private connection?: EntityProviderConnection; + private hasRunOnce = false; + + constructor( + private readonly source: SourceConfig, + private readonly cache: ScanStateCache, + private readonly catalogApi: CatalogService, + private readonly auth: AuthService, + private readonly logger: LoggerService, + ) {} + + getProviderName(): string { + return `auto-api-registration:${this.source.id}`; + } + + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + async runCycle(): Promise { + if (!this.connection) { + this.logger.warn( + `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, + ); + return; + } + + const existingRows = await this.cache.rowsForSource(this.source.id); + const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); + + const currentPaths = new Set(); + for await (const filePath of discoverFiles(this.source)) { + currentPaths.add(filePath); + } + + const knownOwnerRefs = await this.fetchKnownOwnerRefs(); + + const changedPaths = [...currentPaths].filter(filePath => { + const cached = existingByPath.get(filePath); + if (!cached) return true; + const stat = fs.statSync(filePath); + return stat.mtimeMs !== cached.mtime_ms; + }); + + const added: DeferredEntity[] = []; + const removedRefs: string[] = []; + const upserts: CacheRow[] = []; + const deletes: string[] = []; + + await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { + const cached = existingByPath.get(filePath); + const stat = fs.statSync(filePath); + const contents = fs.readFileSync(filePath, 'utf8'); + const hash = sha1(contents); + + if (cached && cached.content_hash === hash) { + // mtime touched, content unchanged — refresh mtime only, no mutation needed. + upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); + return; + } + + const parsed = parseSpecFile(contents); + if (!parsed) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + const slug = slugify(parsed.title); + + // Precedence check: a hand-authored catalog-info.yaml for this name wins. + const precedenceWinner = await this.findNonAutoSourcedEntity(slug); + if (precedenceWinner) { + this.logger.info( + `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + // Global collision check: excludes this file's own prior row. + const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( + row => !(row.source_id === this.source.id && row.file_path === filePath), + ); + const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; + + const mapped = mapCandidate({ + providerName: this.getProviderName(), + source: this.source, + filePath, + contents, + parsed, + knownOwnerRefs, + collidesWith, + }); + + if (mapped.error) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, + ); + } + + added.push({ entity: mapped.entity }); + upserts.push({ + source_id: this.source.id, + file_path: filePath, + mtime_ms: stat.mtimeMs, + content_hash: hash, + entity_name: mapped.entityName, + last_error: mapped.error ?? null, + }); + }); + + for (const [filePath, row] of existingByPath) { + if (!currentPaths.has(filePath)) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); + deletes.push(filePath); + } + } + + if (!this.hasRunOnce) { + // First run: establish the baseline with one `full` mutation. + await this.connection.applyMutation({ type: 'full', entities: added }); + this.hasRunOnce = true; + } else if (added.length > 0 || removedRefs.length > 0) { + await this.connection.applyMutation({ + type: 'delta', + added, + removed: removedRefs.map(entityRef => ({ entityRef })), + }); + } + + for (const row of upserts) { + await this.cache.upsert(row); + } + for (const filePath of deletes) { + await this.cache.remove(this.source.id, filePath); + } + } + + private async fetchKnownOwnerRefs(): Promise> { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, + { credentials }, + ); + return new Set(response.items.map(entity => stringifyEntityRef(entity))); + } + + private async findNonAutoSourcedEntity(entityName: string): Promise { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: 'API', 'metadata.name': entityName } }, + { credentials }, + ); + return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); + } +} + +// --------------------------------------------------------------------------------------------- +// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI +// --------------------------------------------------------------------------------------------- + +class AutoApiRegistrationErrorProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'auto-api-registration-error-processor'; + } + + async preProcessEntity( + entity: Entity, + _location: unknown, + _emit: CatalogProcessorEmit, + ): Promise { + const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; + if (message) { + throw new InputError(message); + } + return entity; + } +} + +// --------------------------------------------------------------------------------------------- +// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) +// --------------------------------------------------------------------------------------------- + +export default createBackendModule({ + pluginId: 'catalog', + moduleId: 'auto-api-registration', + register(reg) { + reg.registerInit({ + deps: { + catalog: catalogProcessingExtensionPoint, + database: coreServices.database, + scheduler: coreServices.scheduler, + logger: coreServices.logger, + rootConfig: coreServices.rootConfig, + catalogApi: catalogServiceRef, + auth: coreServices.auth, + }, + async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { + const sources = normalizeConfig(rootConfig); + if (sources.length === 0) { + logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); + return; + } + + const cache = await ScanStateCache.create(database); + + const providers = sources.map( + source => + new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), + ); + + catalog.addEntityProvider(providers); + catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); + + for (const [index, source] of sources.entries()) { + const provider = providers[index]; + + if (source.mode === 'poll') { + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:poll`, + frequency: { seconds: source.scheduleFrequencySeconds }, + timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); + } + }, + }); + } else { + // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of + // just this source on add/change/unlink, coalesced by chokidar's own event batching. + // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer + // `reconciliation.frequencySeconds` safety-net sweep below. + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:reconciliation`, + frequency: { seconds: source.reconciliationFrequencySeconds }, + timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); + } + }, + }); + + const watcher = chokidar.watch(source.patterns, { + cwd: source.rootPath, + ignored: source.ignore, + ignoreInitial: true, + }); + let debounce: NodeJS.Timeout | undefined; + const triggerRescan = () => { + clearTimeout(debounce); + debounce = setTimeout(() => { + provider + .runCycle() + .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); + }, 500); + }; + watcher.on('add', triggerRescan); + watcher.on('change', triggerRescan); + watcher.on('unlink', triggerRescan); + } + } + }, + }); + }, +}); +``` + +--- + +## Step 3 — Register the Module + +Open `packages/backend/src/index.ts` and add one line after the existing catalog module +registrations: + +```typescript +// See https://backstage.io/docs/features/software-catalog/configuration#subscribing-to-catalog-errors +backend.add(import('@backstage/plugin-catalog-backend-module-logs')); + +// Lab 4: auto-discover *-openapi.yaml / *-asyncapi.yaml files and register them as API entities +// with no hand-authored catalog-info.yaml required — see labs/lab-04-auto-registration/README.md +backend.add(import('./extensions/autoApiRegistration')); +``` + +Like `permissionPolicy.ts` in Lab 3, the module self-registers and declares its own dependencies +via `createBackendModule` — no other wiring is needed. + +--- + +## Step 4 — Configure Discovery + +Add an `autoApiRegistration` block to `app-config.yaml`. This is the **single-source shorthand** +— equivalent to a one-entry `sources: [{ id: default, ... }]` list — which is all a single-repo +setup like this lab needs: + +```yaml +autoApiRegistration: + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp +``` + +`rootPath` is deliberately left unset here — the module resolves a default of +`labs/lab-04-auto-registration/apis` (relative to the backend package) on its own. `patterns`, +`ignore`, `mode`, and `schedule.frequencySeconds` all default sensibly too (see "Adaptable +Conventions vs. Fixed Mechanics" below for the full list). `defaultOwner` and `xNamespace` have no +built-in default and must be set explicitly — a source shouldn't silently inherit a fallback team +or vendor namespace it never opted into. + +--- + +## Step 5 — Add the Sample API Files + +Two sample files are already committed alongside this README (nothing to create in this step — +just take a look): + +- **`apis/galaxy/galaxy-openapi.yaml`** — a vendored copy of the MIT-licensed + [Scalar Galaxy API](https://github.com/scalar/scalar), with an `info.x-examplecorp` object + (`owner`, `lifecycle`, `visibility: shared`) added. It has **no** `catalog-info.yaml` — this is + the "previously unregistered API" that Step 6 will confirm gets discovered automatically. +- **`apis/precedence-demo/precedence-demo-openapi.yaml`** + **`precedence-demo-catalog-info.yaml`** + — a minimal API paired with a *hand-authored* catalog entry (owner `museum-team`, + `visibility: private`) that deliberately differs from the spec file's own `x-examplecorp` + values (owner `platform-team`, `visibility: shared`). This demonstrates the precedence rule: + the hand-authored file wins. + +The hand-authored `precedence-demo-catalog-info.yaml` needs its own catalog location, the same way +Lab 2/3's hand-authored files do — add it to `app-config.yaml`'s `catalog.locations`, replacing +`` and `` with your fork and the current branch name: + +```yaml +catalog: + locations: + # --- Lab 4: Precedence-demo API (hand-authored — must win over auto-sourced metadata) --- + - type: url + target: https://raw.githubusercontent.com//backstage-apiportal-lab//labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml + rules: + - allow: [API] +``` + +This only works once you've pushed your branch — the raw URL has to actually resolve. If you +haven't pushed yet, push now before continuing to Step 6. + +--- + +## Step 6 — Start Backstage and Verify Discovery + +``` +yarn start +``` + +Within one discovery cycle (default 30s), open the catalog and confirm: + +- **The Galaxy API appears with no hand-authored `catalog-info.yaml`.** Search for "Scalar + Galaxy" — it should show up as an `API` entity even though you never wrote a `catalog-info.yaml` + for it. +- **Its owner, lifecycle, and tags match the spec.** `spec.owner` and `spec.lifecycle` should + match the `info.x-examplecorp` values in `galaxy-openapi.yaml`; `metadata.tags` should match its + native `tags` array (slugified — kebab-case, not the original mixed-case names). +- **Visibility contrast.** Sign in as a non-owning, non-platform user (e.g. `charlie`, from Lab + 2 — see `app-config.local.yaml`). The Galaxy API (`visibility: shared`) should be visible; the + precedence-demo API (`visibility: private`, hand-authored, owned by `museum-team`) should not be. +- **Precedence.** The precedence-demo API's entity should reflect the hand-authored + `precedence-demo-catalog-info.yaml` (owner `museum-team`, `visibility: private`) — **not** the + auto-sourced `x-examplecorp` values in its own spec file (owner `platform-team`, + `visibility: shared`). + +--- + +## Step 7 — Verify Update-in-Place + +Edit `apis/galaxy/galaxy-openapi.yaml`'s `info.description` and wait one poll cycle. Reload the +Galaxy API's entity page — the description should update in place. The entity's `metadata.uid` +does not change; this is an update, not a duplicate registration. + +--- + +## Step 8 — Verify Removal + +Temporarily rename `galaxy-openapi.yaml` to something outside the discovery pattern (e.g. +`galaxy-openapi.yaml.bak`) and wait one poll cycle. The Galaxy API should disappear from the +catalog. Rename it back — it reappears within one more cycle, as a fresh registration (a new +`metadata.uid`, since from the provider's perspective this is indistinguishable from a new file). + +--- + +## Step 9 — Verify Error Handling + +Three failure modes, each temporary — revert after checking: + +1. **Malformed YAML.** Break the syntax in one sample file (e.g. an unclosed quote). Check the + backend logs — you should see a line identifying the specific file, and the entity for that + file should simply stop appearing in the catalog (not appear broken; log-only, since a file + that doesn't parse has no name to register an error entity under). +2. **Unresolvable owner.** Set `info.x-examplecorp.owner` to a nonexistent team (e.g. + `group:default/nonexistent-team`). Within one cycle, the entity's "Inspect entity" view should + show a catalog processing error naming the bad owner reference. Other entities are unaffected. +3. **Invalid visibility.** Set `info.x-examplecorp.visibility` to an unrecognized value (e.g. + `public`). You should see the same kind of processing error — not a silent fallback to + `private`. + +--- + +## Adaptable Conventions vs. Fixed Mechanics + +**Adaptable — change these to fit your own repo layout:** + +- `rootPath` — where the scan starts. Point it at your own mono-repo root. +- `patterns` — the filename glob(s) that identify an API definition file. The default + (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`) is this lab's convention, not Backstage's; use + whatever your team already follows. +- `xNamespace` — the vendor namespace under `info.x-` that owner/lifecycle/visibility + are read from. `examplecorp` is a fictional placeholder — **rename this to your own company or + team first** when adapting this lab. +- `defaultOwner` / `defaultVisibility` — the fallback values used when a spec doesn't declare its + own. Per-source, so different repos can have different fallbacks. +- `ignore` — glob exclusions, always applied. Extend (don't replace) the defaults + (`node_modules`, `.git`, `dist`, `build`) for your own build-output directories. + +**Fixed — this is how the mechanism works, not a config knob:** + +- The `EntityProvider` full-then-delta mutation lifecycle (Step 2's rationale above). +- The scan-state cache schema and its role in restart/delta behavior (see "Scaling to a Real + Mono-Repo" below). +- Errors riding on Backstage's native backend-log + processing-error-UI mechanisms, rather than a + bespoke error system. +- The catalog-info.yaml precedence rule: a hand-authored entity for a name always wins over an + auto-sourced candidate for that same name. + +--- + +## Scaling to a Real Mono-Repo + +This lab's defaults (`mode: 'poll'`, 30-second full sweep, no reconciliation) are tuned for 2–3 +sample files and interactive feedback — they are **not** what a 1000+ file, 1GB+ mono-repo should +run with. The architecture (an `EntityProvider` driving mutations) scales fine as-is; only the +config values need to change: + +- **`mode: 'watch'`** replaces the repeated full-tree poll with a `chokidar` watcher that reacts + to individual file add/change/unlink events, so "how often do we rescan" stops scaling with + repo size. A much longer `reconciliation.frequencySeconds` (minutes, not seconds) runs + alongside it as a safety-net full sweep, catching anything a watcher event could plausibly miss + (coalesced events under a big `git pull`, watcher reliability limits on some filesystems). +- **The scan-state cache is what makes restarts fast at scale.** It's a small table in the + backend's own database (`coreServices.database` — SQLite in this lab, Postgres in a real + deployment), storing each file's path, modification time, content hash, and mapped entity name. + On restart, the provider compares this cache against the filesystem instead of re-parsing + everything from zero — unchanged files cost nothing. This is deliberately **not** written back + into the mono-repo as generated `catalog-info.yaml` files: that would reintroduce the + hand-maintained-duplicate-file problem this lab exists to eliminate, and would require the + discovery tool to have *write* access to a repo it should only need to read. +- After the first `full` mutation establishes the baseline, every later cycle sends a `delta` + mutation (`added`/`removed` only) — cost proportional to what changed, not to total catalog + size. This is the same pattern Backstage's own GitHub discovery provider uses at organization + scale. +- `parseConcurrency` bounds how many files are parsed at once, so a large batch of simultaneous + changes (e.g. checking out a branch that touches hundreds of files) doesn't spike memory or + block the event loop. + +If you scale this lab up, expect restarts to be fast (cache-driven, not a full re-scan) — that's +the part most likely to surprise you the first time. + +--- + +## Scaling to Multiple Source Repositories + +The config schema generalizes from one `rootPath` to a list of independently-configured sources: + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo + rootPath: ../platform-monorepo/apis + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + mode: watch + reconciliation: + frequencySeconds: 900 + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp + - id: team-checkout-api + rootPath: ../team-checkout-api + patterns: ['**/*-openapi.yaml'] + mode: poll + schedule: + frequencySeconds: 60 + defaultOwner: group:default/checkout-team + defaultVisibility: shared + xNamespace: checkoutteam +``` + +Each entry becomes its own `EntityProvider` instance with its own schedule, own mode, own default +owner, and own `x-*` namespace — a slow filesystem or config problem in one source can't stall or +break discovery for any other source. + +`defaultOwner` and `xNamespace` are per-source with **no** built-in global fallback: a team +onboarding their own repo shouldn't have to adopt the platform mono-repo's team or vendor +namespace just to participate. `defaultVisibility` is the one exception — it's per-source but +*does* fall back to `private` if a source doesn't set it, because leaving visibility completely +unconfigured must still be safe by default, not just adaptable. + +Collision detection and the catalog-info.yaml precedence check are **global across every source**, +not scoped to whichever source is currently being scanned — two different teams' repos can easily +produce the same slugified entity name (two `payments-api`s), and that must surface a visible +conflict rather than silently coexisting or overwriting. + +Onboarding a genuinely separate Git remote (not a local sibling checkout, which is what this lab's +own multi-source walkthrough would use) needs one more piece this lab doesn't build: a sync step +(e.g. a scheduled shallow `git clone`/`pull` into a local cache directory) ahead of the existing +glob/parse pipeline, pointing that source's `rootPath` at the resulting local worktree. Every other +mechanism — glob patterns, ignore rules, delta mutations, scan-state cache, collision/precedence +checks — is reused unchanged; only how bytes arrive on local disk differs. + +--- + +## Verification Checklist + +- [ ] Galaxy API appears in the catalog within one poll cycle, with no hand-authored + `catalog-info.yaml` +- [ ] Galaxy API's owner/lifecycle match its `x-examplecorp` values; tags match its native `tags` + array +- [ ] Editing `galaxy-openapi.yaml`'s description updates the existing entity, not a duplicate +- [ ] Renaming `galaxy-openapi.yaml` out of the discovery pattern retracts the entity; renaming it + back re-registers it +- [ ] Galaxy (`shared`) is visible to a non-owning, non-platform user; precedence-demo + (`private`, hand-authored) is not +- [ ] The precedence-demo API reflects `precedence-demo-catalog-info.yaml`'s values, not its own + spec file's `x-examplecorp` values +- [ ] A broken-YAML file logs an identifying error and its entity does not appear +- [ ] A nonexistent owner reference surfaces a catalog processing error on that entity only +- [ ] An unrecognized visibility value surfaces a processing error rather than a silent default + +--- + +## Troubleshooting + +- **Nothing appears after 30+ seconds.** Check the backend logs for + `auto-api-registration:default:` lines. A "skipped a cycle — provider not yet connected" + warning on the very first tick is normal (the scheduler's first run can fire slightly before + the catalog engine finishes wiring up the provider); it should not repeat. +- **"Owner ... does not resolve to a known User or Group entity" for an owner you're sure + exists.** Confirm that group is actually loaded in the catalog (check its entity page directly) + — this error means the *catalog* doesn't know about the group yet, not that your YAML is wrong. + A common cause locally is a `catalog.locations` URL that hasn't been pushed yet (see Step 5). +- **An entity you expect to see is silently missing, with no error logged.** Check the backend + logs for a `Policy check failed for api:default/` warning — this means the entity was + built and emitted, but failed Backstage's own entity-schema validation (for example, + `metadata.tags` containing something that isn't a valid kebab-case slug) during processing, so + it never got "stitched" into the catalog you can query. This is different from a registration + error (which the error processor surfaces intentionally) — it's a schema mismatch the entity + never should have had in the first place. +- **A stale entity keeps reappearing after you thought you removed it.** The scan-state cache + persists across backend restarts (it's a real database table, not in-memory). If you manually + edited the database or copied it between environments, the cache may be out of sync with the + filesystem. Deleting the `auto_api_registration_scan_state` table (or just wiping the local + SQLite file, per this lab's `':memory:'` dev config, restarting achieves the same thing) forces + a fresh full rescan. +- **Glob pattern typos.** If a file you expect to be discovered isn't, double check `patterns` — + a mismatched suffix (`-openapi.yml` vs `-openapi.yaml`) is a common culprit, and the discovery + mechanism has no way to tell you "this almost matched." +- **Wrong `xNamespace`.** If owner/lifecycle/visibility all come back as defaults even though your + spec file has an `x-*` object, check that the object's key matches `xNamespace` exactly — + `x-examplecorp` in config but `x-example-corp` (extra hyphen) in the spec file will silently + fall through to defaults rather than error, since an absent `x-*` object is valid input. diff --git a/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml new file mode 100644 index 0000000..053975a --- /dev/null +++ b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml @@ -0,0 +1,1476 @@ +openapi: 3.1.1 +info: + title: Scalar Galaxy + description: | + The Scalar Galaxy is an example OpenAPI document to test OpenAPI tools and libraries. It's a fictional universe with fictional planets and fictional data. + + ## Resources + + * https://github.com/scalar/scalar + * https://github.com/OAI/OpenAPI-Specification + * https://scalar.com + + ## Markdown Support + + All descriptions *can* contain ~~tons of text~~ **Markdown**. [If GitHub supports the syntax](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax), chances are we're supporting it, too. + +
+ Examples + + **Blockquotes** + + > I love OpenAPI. <3 + + **Tables** + + | Feature | Availability | + | ---------------- | ------------ | + | Markdown Support | ✓ | + + **Accordion** + + ```html +
+ Using Details Tags +

HTML Example

+
+ ``` + + **Images** + + Yes, there's support for images, too! + + ![Empty placeholder image showing the width/height](https://images.placeholders.dev/?width=1280&height=720) + + **Alerts** + + > [!tip] + > You can use Markdown alerts in your descriptions. + +
+ version: 0.6.8 + contact: + name: Scalar Support + url: https://scalar.com + email: support@scalar.com + license: + name: MIT + url: https://opensource.org/license/MIT + # Lab 4: catalog metadata sourced by autoApiRegistration (owner/lifecycle/visibility) — + # see labs/lab-04-auto-registration/README.md. No catalog-info.yaml exists for this API; + # it is discovered and registered purely from this file. + x-examplecorp: + owner: group:default/platform-team + lifecycle: production + visibility: shared +externalDocs: + description: Documentation + url: https://github.com/scalar/scalar +servers: + - url: https://galaxy.scalar.com + - url: '{protocol}://void.scalar.com/{path}' + description: Responds with your request data + variables: + protocol: + enum: + - https + - http + default: https + path: + default: '' +security: + - bearerAuth: [] + - basicAuth: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + - apiKeyHeader: [] + apiKeyQuery: [] + - apiKeyCookie: [] + - oAuth2: [] + - openIdConnect: [] +x-speakeasy-webhooks: + security: + type: signature + headerName: x-signature + signatureTextEncoding: base64 + algorithm: hmac-sha256 +tags: + - name: Authentication + description: Some endpoints are public, but some require authentication. We + provide all the required endpoints to create an account and authorize + yourself. + - name: Planets + description: Everything about planets + - name: Celestial Bodies + description: Celestial bodies are the planets and satellites in the Scalar Galaxy. +paths: + /planets: + get: + tags: + - Planets + summary: Get all planets + description: + It's easy to say you know them all, but do you really? Retrieve all + the planets and check whether you missed one. + operationId: getAllData + security: [] + parameters: + - $ref: '#/components/parameters/limit' + - $ref: '#/components/parameters/offset' + responses: + '200': + description: OK + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Pagination-Total: + $ref: '#/components/headers/X-Pagination-Total' + X-Pagination-Page: + $ref: '#/components/headers/X-Pagination-Page' + X-Pagination-Per-Page: + $ref: '#/components/headers/X-Pagination-Per-Page' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanets' + # Bind the generic Paginated template's item type to Planet via a $dynamicAnchor. + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + application/xml: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanetsXml' + xml: + name: planets + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has data array and meta", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("data"); + pm.expect(jsonData.data).to.be.an("array"); + pm.expect(jsonData).to.have.property("meta"); + }); + post: + tags: + - Planets + summary: Create a planet + description: + Time to play god and create a new planet. What do you think? Ah, + don't think too much. What could go wrong anyway? + operationId: createPlanet + callbacks: + planetCreated: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly created planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the callback + '204': + description: + Your server should return this HTTP status code if no longer + interested in further updates + planetCreationFailed: + '{$request.body#/failureCallbackUrl}': + post: + security: [] + requestBody: + description: Information about which fields failed to validate + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the failure callback notification + planetExploded: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly exploded planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the planet explosion callback notification + requestBody: + description: Planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + responses: + '201': + description: Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /planets/{planetId}: + get: + tags: + - Planets + summary: Get a planet + description: + You'll better learn a little bit more about the planets. It might + come in handy once space travel is available for everyone. + operationId: getPlanet + security: [] + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet Found + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + put: + tags: + - Planets + summary: Update a planet + description: Sometimes you make mistakes, that's fine. No worries, you can + update all planets. + operationId: updatePlanet + requestBody: + description: New information about the planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet updated successfully + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + delete: + tags: + - Planets + summary: Delete a planet + operationId: deletePlanet + description: This endpoint was used to delete planets. Unfortunately, that + caused a lot of trouble for planets with life. So, this endpoint is now + deprecated and should not be used anymore. + x-scalar-stability: experimental + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '204': + description: No Content + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 204", () => { + pm.response.to.have.status(204); + }); + /planets/{planetId}/image: + post: + tags: + - Planets + summary: Upload an image to a planet + description: Got a crazy good photo of a planet? Share it with the world! + operationId: uploadImage + parameters: + - $ref: '#/components/parameters/planetId' + requestBody: + description: Image to upload + content: + multipart/form-data: + schema: + type: object + properties: + image: + type: string + format: binary + description: The image file to upload + examples: + - '@mars.jpg' + - '@jupiter.png' + responses: + '200': + $ref: '#/components/responses/ImageUploaded' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has message and imageUrl", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("message"); + pm.expect(jsonData).to.have.property("imageUrl"); + }); + /celestial-bodies: + post: + tags: + - Celestial Bodies + summary: Create a celestial body + description: Stars, moons, comets, the occasional rogue asteroid — if it + glows or drifts through the void, you can add it here. + operationId: createCelestialBody + requestBody: + description: Celestial body to create + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + responses: + '201': + description: Celestial body created + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a celestial body with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /user/signup: + post: + tags: + - Authentication + summary: Create a user + description: Time to create a user account, eh? + operationId: createUser + security: [] + requestBody: + description: User to create + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + application/xml: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + responses: + '201': + description: User account created successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '422': + $ref: '#/components/responses/UnprocessableEntity' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /auth/token: + post: + tags: + - Authentication + summary: Get a token + description: + Yeah, this is the boring security stuff. Just get your super secret + token and move on. + operationId: getToken + security: [] + requestBody: + description: Body for credentials to authenticate a user + content: + application/json: + schema: + $ref: '#/components/schemas/Credentials' + application/xml: + schema: + $ref: '#/components/schemas/Credentials' + responses: + '201': + description: Token Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + content: + application/json: + schema: + $ref: '#/components/schemas/Token' + application/xml: + schema: + $ref: '#/components/schemas/Token' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '429': + $ref: '#/components/responses/TooManyRequests' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response contains token", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("token"); + pm.expect(jsonData.token).to.be.an("string"); + }); + /me: + get: + tags: + - Authentication + summary: Get authenticated user + description: Find yourself they say. That's what you can do here. + operationId: getMe + security: + - basicAuth: [] + - oAuth2: + - read:account + - bearerAuth: [] + - apiKeyHeader: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + apiKeyQuery: [] + responses: + '200': + description: Authenticated user information retrieved successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); +webhooks: + newPlanet: + post: + tags: + - Planets + security: [] + requestBody: + description: Information about a new planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: + Return a 200 status to indicate that the data was received + successfully + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: JWT Bearer token authentication + basicAuth: + type: http + scheme: basic + description: Basic HTTP authentication + apiKeyHeader: + type: apiKey + in: header + name: X-API-Key + description: API key request header + apiKeyQuery: + type: apiKey + in: query + name: api_key + description: API key query parameter + apiKeyCookie: + type: apiKey + in: cookie + name: api_key + description: API key browser cookie + oAuth2: + type: oauth2 + description: OAuth 2.0 authentication + flows: + authorizationCode: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + clientCredentials: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + implicit: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + password: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + openIdConnect: + type: openIdConnect + openIdConnectUrl: https://galaxy.scalar.com/.well-known/openid-configuration + description: OpenID Connect Authentication + parameters: + planetId: + name: planetId + description: The ID of the planet to get + in: path + required: true + schema: + type: integer + format: int64 + examples: + - 1 + limit: + name: limit + description: The number of items to return + in: query + required: false + schema: + type: integer + format: int64 + default: 10 + offset: + name: offset + description: The number of items to skip before starting to collect the result set + in: query + required: false + schema: + type: integer + format: int64 + default: 0 + headers: + X-RateLimit-Limit: + description: The number of allowed requests in the current period + schema: + type: integer + example: 100 + X-RateLimit-Remaining: + description: The number of remaining requests in the current period + schema: + type: integer + example: 95 + X-RateLimit-Reset: + description: The number of seconds left in the current period + schema: + type: integer + example: 3600 + X-Request-ID: + description: Unique identifier for the request + schema: + type: string + format: uuid + example: 123e4567-e89b-12d3-a456-426614174000 + X-Pagination-Total: + description: Total number of items available + schema: + type: integer + example: 1000 + X-Pagination-Page: + description: Current page number + schema: + type: integer + example: 1 + X-Pagination-Per-Page: + description: Number of items per page + schema: + type: integer + example: 10 + X-Cache-Control: + description: Cache control directives + schema: + type: string + example: max-age=3600, public + ETag: + description: Entity tag for caching + schema: + type: string + example: '"33a64df551"' + Last-Modified: + description: The last modification date of the resource + schema: + type: string + format: date-time + example: 2023-12-01T12:00:00Z + Location: + description: The URL of the newly created resource + schema: + type: string + format: uri + example: https://galaxy.scalar.com/planets/123 + X-Processing-Time: + description: The time taken to process the request in milliseconds + schema: + type: integer + example: 150 + responses: + ImageUploaded: + description: Image uploaded + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + application/xml: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + BadRequest: + description: Bad Request + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + content: + application/json: + schema: + $ref: '#/components/schemas/BadRequestError' + application/xml: + schema: + $ref: '#/components/schemas/BadRequestError' + Forbidden: + description: Forbidden + content: + application/json: + schema: + $ref: '#/components/schemas/ForbiddenError' + application/xml: + schema: + $ref: '#/components/schemas/ForbiddenError' + NotFound: + description: Not Found + content: + application/json: + schema: + $ref: '#/components/schemas/NotFoundError' + application/xml: + schema: + $ref: '#/components/schemas/NotFoundError' + Unauthorized: + description: Unauthorized + content: + application/json: + schema: + $ref: '#/components/schemas/UnauthorizedError' + application/xml: + schema: + $ref: '#/components/schemas/UnauthorizedError' + Conflict: + description: Conflict + content: + application/json: + schema: + $ref: '#/components/schemas/Conflict' + application/xml: + schema: + $ref: '#/components/schemas/Conflict' + UnprocessableEntity: + description: Unprocessable Entity + content: + application/json: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + application/xml: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + TooManyRequests: + description: Too Many Requests + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + Retry-After: + description: The number of seconds to wait before retrying + schema: + type: integer + example: 60 + content: + application/json: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + application/xml: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + schemas: + User: + description: A user + type: object + xml: + name: user + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Marc + - Cam + - Hans + Credentials: + description: Credentials to authenticate a user + type: object + required: + - email + - password + properties: + email: + type: string + format: email + examples: + - marc@scalar.com + - cam@scalar.com + - hans@scalar.com + password: + type: string + writeOnly: true + examples: + - i-love-scalar + - i-love-oss + - qwerty123 + Token: + description: A token to authenticate a user + type: object + properties: + token: + type: string + examples: + - eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c + CelestialBody: + oneOf: + - $ref: '#/components/schemas/Planet' + - $ref: '#/components/schemas/Satellite' + discriminator: + propertyName: type + mapping: + terrestrial: '#/components/schemas/Planet' + gas_giant: '#/components/schemas/Planet' + ice_giant: '#/components/schemas/Planet' + dwarf: '#/components/schemas/Planet' + super_earth: '#/components/schemas/Planet' + moon: '#/components/schemas/Satellite' + asteroid: '#/components/schemas/Satellite' + comet: '#/components/schemas/Satellite' + description: A celestial body which can be either a planet or a satellite + Planet: + description: A planet in the Scalar Galaxy + type: object + required: + - id + - name + additionalProperties: false + xml: + name: planet + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + x-variable: planetId + name: + type: string + examples: + - Mars + - Jupiter + - HD 40307g + description: + type: + - string + - 'null' + examples: + - The red planet + - A gas giant with a great red spot + type: + type: string + enum: + - terrestrial + - gas_giant + - ice_giant + - dwarf + - super_earth + x-enum-varnames: + - Terrestrial + - GasGiant + - IceGiant + - Dwarf + - SuperEarth + x-enum-descriptions: + terrestrial: Rocky planets with solid surfaces, like Earth and Mars + gas_giant: + Large planets composed mainly of hydrogen and helium, like Jupiter + and Saturn + ice_giant: + Planets composed of water, ammonia, and methane ices, like Uranus and + Neptune + dwarf: Small planetary bodies that don't meet full planet criteria, like Pluto + super_earth: Rocky planets larger than Earth but smaller than gas giants + examples: + - terrestrial + habitabilityIndex: + type: number + format: float + minimum: 0 + maximum: 1 + description: A score from 0 to 1 indicating potential habitability + examples: + - 0.68 + physicalProperties: + type: object + additionalProperties: + x-additionalPropertiesName: measurement + type: number + format: float + description: Additional physical measurements for the planet + properties: + mass: + type: number + format: float + exclusiveMinimum: 0 + description: Mass in Earth masses (must be greater than 0) + examples: + - 0.107 + radius: + type: number + format: float + exclusiveMinimum: 0 + description: Radius in Earth radii (must be greater than 0) + examples: + - 0.532 + gravity: + type: number + format: float + description: Surface gravity in Earth g + examples: + - 0.378 + temperature: + type: object + additionalProperties: + x-additionalPropertiesName: temperatureMetric + type: number + format: float + description: Additional temperature-related measurements in Kelvin + properties: + min: + type: number + format: float + description: Minimum temperature in Kelvin + examples: + - 130 + max: + type: number + format: float + description: Maximum temperature in Kelvin + examples: + - 308 + average: + type: number + format: float + description: Average temperature in Kelvin + examples: + - 210 + atmosphere: + type: array + description: Atmospheric composition + items: + type: object + additionalProperties: + x-additionalPropertiesName: atmosphericData + type: string + description: Additional atmospheric composition data + properties: + compound: + type: string + examples: + - CO2 + - N2 + percentage: + type: number + format: float + exclusiveMaximum: 100 + examples: + - 95.3 + discoveredAt: + type: string + format: date-time + examples: + - 1610-01-07T00:00:00Z + image: + type: + - string + - 'null' + examples: + - https://cdn.scalar.com/photos/mars.jpg + satellites: + type: array + items: + $ref: '#/components/schemas/Satellite' + creator: + $ref: '#/components/schemas/User' + tags: + type: array + items: + type: string + examples: + - - solar-system + - rocky + - explored + lastUpdated: + type: string + format: date-time + readOnly: true + examples: + - 2024-01-15T14:30:00Z + successCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a successful operation + examples: + - https://example.com/webhook + failureCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a failed operation + examples: + - https://example.com/webhook + Satellite: + description: Every satellite in the Scalar Galaxy + type: object + required: + - name + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Phobos + description: + type: + - string + - 'null' + examples: + - Phobos is the larger and innermost of the two moons of Mars. + diameter: + type: number + format: float + description: Diameter in kilometers + examples: + - 22.2 + type: + type: string + enum: + - moon + - asteroid + - comet + x-enum-varnames: + - Moon + - Asteroid + - Comet + x-enum-descriptions: + moon: Natural satellites that orbit planets + asteroid: Rocky objects that orbit the sun, typically found in the asteroid belt + comet: + Icy bodies that release gas when approaching the sun, creating visible + tails + examples: + - moon + orbit: + type: object + properties: + planet: + $ref: '#/components/schemas/Planet' + orbitalPeriod: + type: number + format: float + description: Orbital period in Earth days + examples: + - 0.319 + distance: + type: number + format: float + description: Average distance from the planet in kilometers + examples: + - 9376 + Paginated: + description: + A generic paginated response. Specializing schemas bind the item type by declaring a + `$dynamicAnchor` named `itemType`, the JSON Schema 2020-12 way to express `Paginated`. + $id: 'https://galaxy.scalar.com/schemas/Paginated' + $defs: + itemType: + $dynamicAnchor: itemType + not: {} + type: object + properties: + data: + type: array + items: + $dynamicRef: '#itemType' + allOf: + - $ref: '#/components/schemas/PaginatedResource' + PaginatedResource: + description: A paginated resource + type: object + properties: + meta: + type: object + properties: + limit: + type: integer + format: int64 + examples: + - 10 + offset: + type: integer + format: int64 + examples: + - 0 + total: + type: integer + format: int64 + examples: + - 100 + next: + type: + - string + - 'null' + examples: + - /planets?limit=10&offset=10 + ImageUploadedMessage: + x-scalar-ignore: true + description: Message about an image upload + type: object + properties: + message: + type: string + examples: + - Image uploaded successfully + imageUrl: + type: string + description: The URL where the uploaded image can be accessed + examples: + - https://cdn.scalar.com/images/8f47c132-9d1f-4f83-b5a4-91db5ee757ab.jpg + uploadedAt: + type: string + format: date-time + description: Timestamp when the image was uploaded + examples: + - 2024-01-15T14:30:00Z + fileSize: + type: integer + description: Size of the uploaded image in bytes + examples: + - 1048576 + mimeType: + type: string + description: The content type of the uploaded image + examples: + - image/jpeg + - image/png + BadRequestError: + x-scalar-ignore: true + description: RFC 7807 (https://datatracker.ietf.org/doc/html/rfc7807) + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/bad-request + title: + type: string + examples: + - Bad Request + status: + type: integer + format: int64 + examples: + - 400 + detail: + type: string + examples: + - The request was invalid. + ForbiddenError: + x-scalar-ignore: true + description: Error response for forbidden access (RFC 7807). Returned when the user does not have permission to access the requested resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/forbidden + title: + type: string + examples: + - Forbidden + status: + type: integer + format: int64 + examples: + - 403 + detail: + type: string + examples: + - You are not authorized to access this resource. + NotFoundError: + x-scalar-ignore: true + description: Error response for resource not found (RFC 7807). Returned when the requested resource does not exist. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Not Found + status: + type: integer + format: int64 + examples: + - 404 + detail: + type: string + examples: + - The resource you are trying to access does not exist. + UnauthorizedError: + x-scalar-ignore: true + description: Error response for unauthorized access (RFC 7807). Returned when authentication is required or has failed. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Unauthorized + status: + type: integer + format: int64 + examples: + - 401 + detail: + type: string + examples: + - You are not authorized to access this resource. + Conflict: + x-scalar-ignore: true + description: Error response for resource conflicts (RFC 7807). Returned when the request conflicts with the current state of the resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/conflict + title: + type: string + examples: + - Conflict + status: + type: integer + format: int64 + examples: + - 409 + detail: + type: string + examples: + - The resource you are trying to access is in conflict. + UnprocessableEntity: + x-scalar-ignore: true + description: Error response for unprocessable entity (RFC 7807). Returned when the request is well-formed but contains semantic errors. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/unprocessable-entity + title: + type: string + examples: + - Unprocessable Entity + status: + type: integer + format: int64 + examples: + - 422 + detail: + type: string + examples: + - The request was invalid. + TooManyRequestsError: + x-scalar-ignore: true + description: Error response for rate limiting (RFC 7807). Returned when the client has exceeded the rate limit for requests. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/too-many-requests + title: + type: string + examples: + - Too Many Requests + status: + type: integer + format: int64 + examples: + - 429 + detail: + type: string + examples: + - Rate limit exceeded. Please try again later. diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml new file mode 100644 index 0000000..6e79bf3 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml @@ -0,0 +1,21 @@ +# Hand-authored entity for the precedence-demo API — deliberately NOT auto-discovered from +# precedence-demo-openapi.yaml's own x-examplecorp values. This file demonstrates FR-011: a +# hand-authored catalog-info.yaml takes precedence over auto-sourced metadata for the same API. +# See labs/lab-04-auto-registration/README.md. +apiVersion: backstage.io/v1alpha1 +kind: API +metadata: + name: precedence-demo-api + title: Precedence Demo API + description: > + Hand-authored registration for the precedence-demo API — this entity, not the auto-sourced + candidate from precedence-demo-openapi.yaml's x-examplecorp object, is what appears in the + catalog (FR-011). + annotations: + example.com/visibility: private +spec: + type: openapi + lifecycle: production + owner: group:default/museum-team + definition: + $text: https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/004-lab-4-auto-registration/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml new file mode 100644 index 0000000..0650e37 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml @@ -0,0 +1,59 @@ +openapi: 3.1.0 +info: + title: Precedence Demo API + description: > + A minimal API used only to demonstrate FR-011: a hand-authored catalog-info.yaml + (precedence-demo-catalog-info.yaml, alongside this file) takes precedence over the metadata + auto-sourced from this file's own x-examplecorp object. See labs/lab-04-auto-registration/README.md. + version: 1.0.0 + # Deliberately different from the hand-authored catalog-info.yaml's owner/visibility, so the + # precedence check has something real to demonstrate (FR-011). + x-examplecorp: + owner: group:default/platform-team + lifecycle: experimental + visibility: shared +tags: + - name: Demo + description: Precedence demonstration endpoints +paths: + /widgets: + get: + tags: [Demo] + summary: List widgets + operationId: listWidgets + responses: + '200': + description: A list of widgets + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/Widget' + /widgets/{widgetId}: + get: + tags: [Demo] + summary: Get a widget + operationId: getWidget + parameters: + - name: widgetId + in: path + required: true + schema: + type: string + responses: + '200': + description: A single widget + content: + application/json: + schema: + $ref: '#/components/schemas/Widget' +components: + schemas: + Widget: + type: object + properties: + id: + type: string + name: + type: string diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index e8c3c08..6070187 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -27,14 +27,14 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | | `metadata.title` | `info.title` | Verbatim. | | `metadata.description` | `info.description` | Verbatim if present, else omitted. | -| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | +| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. **Confirmed during implementation**: Backstage's `metadata.tags` format requires kebab-case slugs (`[a-z0-9+#]` separated by `-`); native spec tag names (e.g. `"Celestial Bodies"`) are free text, so each is slugified with the same slug function used for `metadata.name` before being written — otherwise the entity fails catalog policy validation and is silently dropped during processing (never reaches the catalog, no user-visible error). | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | | `metadata.annotations['example.com/visibility']` | `info.x-examplecorp.visibility` | **The exact same annotation key the Lab 2/3 permission policy already reads** (`packages/backend/src/extensions/permissionPolicy.ts`) — no new annotation, no policy change. **Mapping Rule 2**: if absent, falls back to that file's source's configured `defaultVisibility` — itself optional per-source, falling back to the fixed `'private'` "default default" if the source doesn't set one either (research.md R3/R7, FR-006a). An unrecognized (non-`private`/`shared`) value is never silently defaulted — it's a marker/error entity instead (research.md R4). Reused verbatim rather than reinvented, per research.md R3. | | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref, invalid visibility value, or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | | `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | -| `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | +| `spec.definition` | discovery | **Revised during implementation**: `spec.definition` is a plain `string` field (not an object), and Backstage's default `UrlReaderService` stack has no `file:`-scheme reader — a `{ $text: 'file:...' }` placeholder (the originally-planned approach, mirroring Labs 1–3's hand-authored `$text` + `type: url` pattern) fails with `NotAllowedError` for a local filesystem path with no matching `reading.allow` host entry. Since the provider already holds the file's bytes in memory at parse time, `spec.definition` is populated with that raw file content directly — no read-back, no placeholder resolution, no duplication (the file itself isn't a second copy; it's the copy). The `$text`/`type: url` pattern remains correct and unchanged for genuinely hand-authored `catalog-info.yaml` files (e.g. `precedence-demo-catalog-info.yaml`), which reference a *different* file over a real network-reachable URL — this revision only applies to entities this provider constructs programmatically from a file it already read. | ## Config: `autoApiRegistration` (app-config.yaml) diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 51057e4..c950573 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -40,10 +40,14 @@ path for `**/*-openapi.yaml` and `**/*-asyncapi.yaml`, on a scheduled interval v hammering the filesystem). **Root path**: configurable via `autoApiRegistration.rootPath` in `app-config.yaml`; documented -as an adaptable convention (FR-010). Default value resolves from the backend process's working -directory (`packages/backend` under `backstage-cli package start`) up to the repository root — -exact relative segment count will be confirmed experimentally during implementation and recorded -here as a Run update, consistent with how Lab 3's research.md captured verified findings. +as an adaptable convention (FR-010). Default value resolves via `resolvePackagePath('backend')` +(from `@backstage/backend-plugin-api`) rather than `process.cwd()` directly — `resolvePackagePath` +locates the `packages/backend` directory via Node module resolution of its own `package.json` +(`name: "backend"`), which is robust regardless of which working directory the backend process was +actually launched from, unlike a `process.cwd()`-relative assumption. From there, 4 `..` segments +(`packages/backend` → `packages` → `backstage` → `lab-01-base-backstage` → `labs`) reach `labs/`, +then into `lab-04-auto-registration/apis` — confirmed experimentally during implementation +(`packages/backend/src/extensions/autoApiRegistration.ts`'s `defaultRootPath()`). (This is the single-source shorthand; R7 generalizes `rootPath` into a per-entry field of an `autoApiRegistration.sources[]` list for organizations with more than one source repository.) diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md index f232d5c..ac48f3f 100644 --- a/specs/004-lab-4-auto-registration/tasks.md +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -32,9 +32,9 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **Purpose**: Add dependencies and scaffold the directories every later task writes into. -- [ ] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) -- [ ] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` -- [ ] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` +- [X] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) +- [X] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` +- [X] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` **Checkpoint**: Dependencies installed, directories exist — ready for foundational work. @@ -46,14 +46,14 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **⚠️ CRITICAL**: No user story work can begin until this phase is complete. -- [ ] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section -- [ ] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) -- [ ] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) -- [ ] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) -- [ ] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) -- [ ] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) -- [ ] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) -- [ ] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) +- [X] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section +- [X] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) +- [X] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) +- [X] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) +- [X] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) +- [X] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) +- [X] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) +- [X] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) **Checkpoint**: Config loads, files are discoverable and parseable, scan-state cache persists — foundation ready for US1. @@ -67,12 +67,12 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 1 -- [ ] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) -- [ ] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) -- [ ] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) -- [ ] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) -- [ ] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) -- [ ] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) +- [X] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) +- [X] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) +- [X] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) +- [X] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) +- [X] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) +- [X] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) **Checkpoint**: User Story 1 fully functional and independently testable — discovery, create, update, and removal all work end-to-end with default metadata. @@ -86,17 +86,17 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 2 -- [ ] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) -- [ ] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) -- [ ] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) -- [ ] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) -- [ ] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) -- [ ] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) -- [ ] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) -- [ ] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) -- [ ] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) -- [ ] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) -- [ ] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) +- [X] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) +- [X] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) +- [X] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) +- [X] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) +- [X] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) +- [X] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) +- [X] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) +- [X] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) +- [X] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) +- [X] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) +- [X] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) **Checkpoint**: User Stories 1 AND 2 both work independently — metadata sourcing, defaults, validation errors, collisions, and precedence are all verified. @@ -110,12 +110,12 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 3 -- [ ] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) -- [ ] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) +- [X] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) +- [X] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) **Checkpoint**: All three user stories independently functional; README fully documents adaptable vs. fixed conventions. @@ -125,9 +125,9 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **Purpose**: Final repo-wide consistency checks that span multiple stories. -- [ ] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI -- [ ] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development -- [ ] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built +- [X] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI +- [X] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development +- [X] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built --- From db35fb821709cccc94e10af81a410e2201de4289 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 12:17:06 +1000 Subject: [PATCH 09/19] Extracted code from documentation, to make implementation and docs more usable --- labs/lab-02-users-roles/README.md | 155 +--- .../apiVisibility/ApiVisibilityCard.tsx | 41 + .../app/src/modules/apiVisibility/index.ts | 18 + .../src/extensions/permissionPolicy.ts | 66 ++ labs/lab-03-api-quality/README.md | 133 +-- .../app/src/modules/apiGrade/index.ts | 18 + .../spectralLinter/SpectralLinterContent.tsx | 41 + .../app/src/modules/spectralLinter/index.ts | 29 + .../spectral-linter-content.d.ts | 4 + labs/lab-04-auto-registration/README.md | 806 +----------------- .../src/extensions/autoApiRegistration.ts | 749 ++++++++++++++++ .../001_scan_state_cache.ts | 32 + 12 files changed, 1055 insertions(+), 1037 deletions(-) create mode 100644 labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx create mode 100644 labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts create mode 100644 labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts create mode 100644 labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts create mode 100644 labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts diff --git a/labs/lab-02-users-roles/README.md b/labs/lab-02-users-roles/README.md index d13a808..c1ea96a 100644 --- a/labs/lab-02-users-roles/README.md +++ b/labs/lab-02-users-roles/README.md @@ -319,77 +319,19 @@ Run this command from inside your Backstage root directory #### Create `ApiVisibilityCard.tsx` -Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` with the following -content: - -```typescript -import React from 'react'; -import { useEntity } from '@backstage/plugin-catalog-react'; -import { InfoCard } from '@backstage/core-components'; -import { Typography, Box, Chip } from '@material-ui/core'; - -const VISIBILITY_ANNOTATION = 'example.com/visibility'; - -export function ApiVisibilityCard() { - const { entity } = useEntity(); - const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; - - if (!visibility) { - return ( - - - No visibility designation set. Under the permission policy, this API is visible - only to members of the owning team (treated as private by default). - - - ); - } - - const isShared = visibility === 'shared'; - - return ( - - - - - {isShared - ? 'Visible to all authenticated users regardless of team membership.' - : 'Visible only to members of the owning team.'} - - - - ); -} -``` +Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx`](./code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx); +copy it in as-is. It reads the `example.com/visibility` annotation off the entity and renders a +"Shared"/"Private" chip, with a fallback message when the annotation is absent (treated as +private by the permission policy). #### Create `index.ts` -Create `packages/app/src/modules/apiVisibility/index.ts` with the following content: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiVisibilityCard } from './ApiVisibilityCard'; - -const apiVisibilityCard = EntityCardBlueprint.make({ - name: 'api-visibility', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiVisibilityCard), - }, -}); - -export const apiVisibilityModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiVisibilityCard], -}); -``` +Create `packages/app/src/modules/apiVisibility/index.ts` — its full content is committed at +[`code/packages/app/src/modules/apiVisibility/index.ts`](./code/packages/app/src/modules/apiVisibility/index.ts); +copy it in as-is. It wraps `ApiVisibilityCard` in an `EntityCardBlueprint` extension and bundles +it into a `createFrontendModule`. > **Note on Alpha imports**: `EntityCardBlueprint` is imported from > `@backstage/plugin-catalog-react/alpha`. This is the officially supported extension @@ -599,76 +541,13 @@ For all non-catalog permissions (scaffolding, TechDocs, search, etc.), the polic Create the directory `packages/backend/src/extensions/` inside your Backstage root (`labs/lab-01-base-backstage/backstage/`), then create a new file called -`permissionPolicy.ts` inside it with the following content: - -```typescript -// packages/backend/src/extensions/permissionPolicy.ts -import { createBackendModule } from '@backstage/backend-plugin-api'; -import { - PolicyDecision, - AuthorizeResult, - isResourcePermission, -} from '@backstage/plugin-permission-common'; -import { - PermissionPolicy, - PolicyQuery, - PolicyQueryUser, -} from '@backstage/plugin-permission-node'; -import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; -import { - catalogConditions, - createCatalogConditionalDecision, -} from '@backstage/plugin-catalog-backend/alpha'; - -class CatalogOwnershipPolicy implements PermissionPolicy { - async handle( - request: PolicyQuery, - user?: PolicyQueryUser, - ): Promise { - if (isResourcePermission(request.permission, 'catalog-entity')) { - return createCatalogConditionalDecision( - request.permission, - { - anyOf: [ - // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. - // This keeps the org chart and team pages open for everyone. - { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, - - // Rule 2: APIs annotated as shared are visible to all authenticated users, - // regardless of which team they belong to. - catalogConditions.hasAnnotation({ - annotation: 'example.com/visibility', - value: 'shared', - }), - - // Rule 3: Private APIs are visible only to members of the owning team. - // The user's ownershipEntityRefs contains their user ref plus all their - // group refs, resolved from the catalog User entity's memberOf list. - catalogConditions.isEntityOwner({ - claims: user?.info.ownershipEntityRefs ?? [], - }), - ], - }, - ); - } - // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. - return { result: AuthorizeResult.ALLOW }; - } -} - -export default createBackendModule({ - pluginId: 'permission', - moduleId: 'permission-policy', - register(reg) { - reg.registerInit({ - deps: { policy: policyExtensionPoint }, - async init({ policy }) { - policy.setPolicy(new CatalogOwnershipPolicy()); - }, - }); - }, -}); -``` +`permissionPolicy.ts` inside it. Its full content is committed alongside this README at +[`code/packages/backend/src/extensions/permissionPolicy.ts`](./code/packages/backend/src/extensions/permissionPolicy.ts); +copy it in as-is. In short: it's a `CatalogOwnershipPolicy` that, for the `catalog-entity` +resource permission, allows non-API entities unconditionally, allows APIs annotated +`example.com/visibility: shared` for everyone, and otherwise falls back to an ownership check +via `catalogConditions.isEntityOwner`; every other permission is unconditionally allowed. It's +registered as a `permission-policy` backend module via `createBackendModule`. **Import sources** (all packages are already installed in Lab 1 — no `npm install` needed): diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx new file mode 100644 index 0000000..0fcff5c --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx @@ -0,0 +1,41 @@ +import React from 'react'; +import { useEntity } from '@backstage/plugin-catalog-react'; +import { InfoCard } from '@backstage/core-components'; +import { Typography, Box, Chip } from '@material-ui/core'; + +const VISIBILITY_ANNOTATION = 'example.com/visibility'; + +export function ApiVisibilityCard() { + const { entity } = useEntity(); + const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; + + if (!visibility) { + return ( + + + No visibility designation set. Under the permission policy, this API is visible + only to members of the owning team (treated as private by default). + + + ); + } + + const isShared = visibility === 'shared'; + + return ( + + + + + {isShared + ? 'Visible to all authenticated users regardless of team membership.' + : 'Visible only to members of the owning team.'} + + + + ); +} diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts new file mode 100644 index 0000000..e21dd81 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiVisibilityCard } from './ApiVisibilityCard'; + +const apiVisibilityCard = EntityCardBlueprint.make({ + name: 'api-visibility', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiVisibilityCard), + }, +}); + +export const apiVisibilityModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiVisibilityCard], +}); diff --git a/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts new file mode 100644 index 0000000..e5b52e8 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts @@ -0,0 +1,66 @@ +// packages/backend/src/extensions/permissionPolicy.ts +import { createBackendModule } from '@backstage/backend-plugin-api'; +import { + PolicyDecision, + AuthorizeResult, + isResourcePermission, +} from '@backstage/plugin-permission-common'; +import { + PermissionPolicy, + PolicyQuery, + PolicyQueryUser, +} from '@backstage/plugin-permission-node'; +import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; +import { + catalogConditions, + createCatalogConditionalDecision, +} from '@backstage/plugin-catalog-backend/alpha'; + +class CatalogOwnershipPolicy implements PermissionPolicy { + async handle( + request: PolicyQuery, + user?: PolicyQueryUser, + ): Promise { + if (isResourcePermission(request.permission, 'catalog-entity')) { + return createCatalogConditionalDecision( + request.permission, + { + anyOf: [ + // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. + // This keeps the org chart and team pages open for everyone. + { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, + + // Rule 2: APIs annotated as shared are visible to all authenticated users, + // regardless of which team they belong to. + catalogConditions.hasAnnotation({ + annotation: 'example.com/visibility', + value: 'shared', + }), + + // Rule 3: Private APIs are visible only to members of the owning team. + // The user's ownershipEntityRefs contains their user ref plus all their + // group refs, resolved from the catalog User entity's memberOf list. + catalogConditions.isEntityOwner({ + claims: user?.info.ownershipEntityRefs ?? [], + }), + ], + }, + ); + } + // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. + return { result: AuthorizeResult.ALLOW }; + } +} + +export default createBackendModule({ + pluginId: 'permission', + moduleId: 'permission-policy', + register(reg) { + reg.registerInit({ + deps: { policy: policyExtensionPoint }, + async init({ policy }) { + policy.setPolicy(new CatalogOwnershipPolicy()); + }, + }); + }, +}); diff --git a/labs/lab-03-api-quality/README.md b/labs/lab-03-api-quality/README.md index fe4447b..f5d59d8 100644 --- a/labs/lab-03-api-quality/README.md +++ b/labs/lab-03-api-quality/README.md @@ -134,28 +134,11 @@ value `'info'` places the card in the right-hand Info column (the same column as card). Without it, the card defaults to the main content area at the bottom of the page — far less visible. -Create the file `packages/app/src/modules/apiGrade/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; - -const apiGradeCard = EntityCardBlueprint.make({ - name: 'api-grade', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiGradeCard), - }, -}); - -export const apiGradeModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiGradeCard], -}); -``` +Create the file `packages/app/src/modules/apiGrade/index.ts` — its full content is committed +alongside this README at +[`code/packages/app/src/modules/apiGrade/index.ts`](./code/packages/app/src/modules/apiGrade/index.ts); +copy it in as-is. It wraps the plugin's `ApiGradeCard` in an `EntityCardBlueprint` extension and +bundles it into a `createFrontendModule`, the same pattern as Lab 2's `apiVisibility` module. `filter: 'kind:API'` ensures the card only appears on API entity pages — not on Component, System, or other entity pages. @@ -354,60 +337,17 @@ mode. Importing it directly sidesteps the incompatibility entirely instead of wo it. Because this subpath has no TypeScript declaration file, add a small ambient module -declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`: - -```typescript -declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { - import { ComponentType } from 'react'; - export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; -} -``` - -Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`: - -```typescript -import React, { useEffect, useState } from 'react'; -import { useApi, identityApiRef } from '@backstage/core-plugin-api'; -import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; -import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; -import { InfoCard } from '@backstage/core-components'; -import { Typography } from '@material-ui/core'; - -export function SpectralLinterContent() { - const { entity } = useEntity(); - const identityApi = useApi(identityApiRef); - const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); - const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); - const [identityLoading, setIdentityLoading] = useState(true); - - useEffect(() => { - identityApi.getBackstageIdentity().then(identity => { - setIsPlatformTeamMember( - identity.ownershipEntityRefs.includes('group:default/platform-team'), - ); - setIdentityLoading(false); - }); - }, [identityApi]); - - if (ownershipLoading || identityLoading) return null; - - // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate - // FUNCTION, not a boolean — it must be called with the current entity. - if (!isOwnedEntity(entity) && !isPlatformTeamMember) { - return ( - - - Detailed quality information is restricted to members of the API's owning team - and the platform team. Sign in as a member of the owning team or the platform - team to view linting results. - - - ); - } - - return ; -} -``` +declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts` — its +full content is committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`](./code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts); +copy it in as-is. + +Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx` — its full +content is committed at +[`code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`](./code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx); +copy it in as-is. In short: it renders the plugin's inner `EntityApiDocsSpectralLinterContent` +component, gated behind an ownership/platform-team check — the owning team or a platform-team +member sees the lint results; everyone else sees an access-restricted message. **Why is `isOwnedEntity` called as a function?** `useEntityOwnership()` (from `@backstage/plugin-catalog-react`) returns `{ loading, isOwnedEntity }` where `isOwnedEntity` @@ -428,39 +368,14 @@ permission policy. ## Step 9 — Create the spectralLinter Frontend Module -Create the file `packages/app/src/modules/spectralLinter/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import { convertLegacyPlugin } from '@backstage/core-compat-api'; -import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; -import React from 'react'; -import { SpectralLinterContent } from './SpectralLinterContent'; - -const spectralLinterContent = EntityContentBlueprint.make({ - name: 'spectral-linter', - params: { - defaultPath: '/spectral', - defaultTitle: 'Spectral', - filter: 'kind:API', - loader: async () => React.createElement(SpectralLinterContent), - }, -}); - -export const spectralLinterModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [spectralLinterContent], -}); - -// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend -// system's API registry. `extensions: []` means we deliberately do NOT bring in the -// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped -// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. -export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { - extensions: [], -}); -``` +Create the file `packages/app/src/modules/spectralLinter/index.ts` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/index.ts`](./code/packages/app/src/modules/spectralLinter/index.ts); +copy it in as-is. In short: it wraps `SpectralLinterContent` in an `EntityContentBlueprint` +extension (the "Spectral" tab) and, separately, uses `convertLegacyPlugin` to register the +plugin's `linterApiRef` (backed by its `LinterClient`) with the new frontend system's API +registry, with `extensions: []` since only the API — not the plugin's own page/route +extensions — is needed. `defaultPath: '/spectral'` and `defaultTitle: 'Spectral'` define the tab's URL path and display label. `filter: 'kind:API'` ensures the tab appears only on API entity pages. diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts new file mode 100644 index 0000000..826d22c --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; + +const apiGradeCard = EntityCardBlueprint.make({ + name: 'api-grade', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiGradeCard), + }, +}); + +export const apiGradeModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiGradeCard], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx new file mode 100644 index 0000000..90760ac --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx @@ -0,0 +1,41 @@ +import React, { useEffect, useState } from 'react'; +import { useApi, identityApiRef } from '@backstage/core-plugin-api'; +import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; +import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; +import { InfoCard } from '@backstage/core-components'; +import { Typography } from '@material-ui/core'; + +export function SpectralLinterContent() { + const { entity } = useEntity(); + const identityApi = useApi(identityApiRef); + const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); + const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); + const [identityLoading, setIdentityLoading] = useState(true); + + useEffect(() => { + identityApi.getBackstageIdentity().then(identity => { + setIsPlatformTeamMember( + identity.ownershipEntityRefs.includes('group:default/platform-team'), + ); + setIdentityLoading(false); + }); + }, [identityApi]); + + if (ownershipLoading || identityLoading) return null; + + // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate + // FUNCTION, not a boolean — it must be called with the current entity. + if (!isOwnedEntity(entity) && !isPlatformTeamMember) { + return ( + + + Detailed quality information is restricted to members of the API's owning team + and the platform team. Sign in as a member of the owning team or the platform + team to view linting results. + + + ); + } + + return ; +} diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts new file mode 100644 index 0000000..aa32826 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts @@ -0,0 +1,29 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import { convertLegacyPlugin } from '@backstage/core-compat-api'; +import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; +import React from 'react'; +import { SpectralLinterContent } from './SpectralLinterContent'; + +const spectralLinterContent = EntityContentBlueprint.make({ + name: 'spectral-linter', + params: { + defaultPath: '/spectral', + defaultTitle: 'Spectral', + filter: 'kind:API', + loader: async () => React.createElement(SpectralLinterContent), + }, +}); + +export const spectralLinterModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [spectralLinterContent], +}); + +// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend +// system's API registry. `extensions: []` means we deliberately do NOT bring in the +// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped +// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. +export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { + extensions: [], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts new file mode 100644 index 0000000..c71e3a1 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts @@ -0,0 +1,4 @@ +declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { + import { ComponentType } from 'react'; + export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; +} diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md index 955a8ed..11786bd 100644 --- a/labs/lab-04-auto-registration/README.md +++ b/labs/lab-04-auto-registration/README.md @@ -107,796 +107,22 @@ lets you hand-author precise metadata for a specific API — pinning an owner, t visibility — without the auto-registration mechanism fighting you for it every cycle. Create the database migration first, at -`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`: - -```typescript -// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts -// -// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal -// state private to this backend module, used to avoid re-parsing unchanged files on restart, to -// compute delta mutations, and to support collision/precedence checks across every configured -// source. -// -// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a -// filesystem `directory` migration source, since this file lives alongside the module's -// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. - -import type { Knex } from 'knex'; - -export const TABLE_NAME = 'auto_api_registration_scan_state'; - -export async function up(knex: Knex): Promise { - await knex.schema.createTable(TABLE_NAME, table => { - table.string('source_id').notNullable(); - table.string('file_path').notNullable(); - table.bigInteger('mtime_ms').notNullable(); - table.string('content_hash').notNullable(); - table.string('entity_name').notNullable(); - table.text('last_error').nullable(); - - table.primary(['source_id', 'file_path']); - table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); - }); -} - -export async function down(knex: Knex): Promise { - await knex.schema.dropTable(TABLE_NAME); -} -``` - -Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts`: - -```typescript -// packages/backend/src/extensions/autoApiRegistration.ts -// -// Lab 4: Auto Registration. -// -// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files -// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. - -import crypto from 'crypto'; -import fs from 'fs'; -import path from 'path'; -import fg from 'fast-glob'; -import chokidar from 'chokidar'; -import yaml from 'js-yaml'; -import { - coreServices, - createBackendModule, - resolvePackagePath, -} from '@backstage/backend-plugin-api'; -import { - catalogProcessingExtensionPoint, - catalogServiceRef, -} from '@backstage/plugin-catalog-node'; -import type { - CatalogProcessor, - CatalogProcessorEmit, - CatalogService, - DeferredEntity, - EntityProvider, - EntityProviderConnection, -} from '@backstage/plugin-catalog-node'; -import type { - AuthService, - DatabaseService, - LoggerService, - RootConfigService, -} from '@backstage/backend-plugin-api'; -import { - ANNOTATION_LOCATION, - ANNOTATION_ORIGIN_LOCATION, - parseEntityRef, - stringifyEntityRef, -} from '@backstage/catalog-model'; -import type { Entity } from '@backstage/catalog-model'; -import { InputError } from '@backstage/errors'; -import type { Knex } from 'knex'; -import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; - -// --------------------------------------------------------------------------------------------- -// Config -// --------------------------------------------------------------------------------------------- - -export const VISIBILITY_ANNOTATION = 'example.com/visibility'; -export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; -// Internal-only marker: identifies which provider instance produced an entity, so the -// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from -// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. -const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; - -const KNOWN_VISIBILITIES = new Set(['private', 'shared']); -const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; -const DEFAULT_IGNORE = [ - '**/node_modules/**', - '**/.git/**', - '**/dist/**', - '**/build/**', -]; - -export interface SourceConfig { - id: string; - rootPath: string; - patterns: string[]; - ignore: string[]; - mode: 'poll' | 'watch'; - scheduleFrequencySeconds: number; - reconciliationFrequencySeconds: number; - parseConcurrency: number; - defaultOwner: string; - defaultVisibility: 'private' | 'shared'; - xNamespace: string; -} - -function defaultRootPath(): string { - // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs - return path.resolve( - resolvePackagePath('backend'), - '../../../../lab-04-auto-registration/apis', - ); -} - -export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { - const raw = rootConfig.getOptionalConfig('autoApiRegistration'); - if (!raw) { - return []; - } - - const entries = raw.has('sources') - ? raw.getConfigArray('sources') - : [raw]; - - return entries.map(entry => { - const id = raw.has('sources') ? entry.getString('id') : 'default'; - const defaultOwner = entry.getOptionalString('defaultOwner'); - const xNamespace = entry.getOptionalString('xNamespace'); - if (!defaultOwner) { - throw new Error( - `autoApiRegistration source "${id}" is missing required "defaultOwner"`, - ); - } - if (!xNamespace) { - throw new Error( - `autoApiRegistration source "${id}" is missing required "xNamespace"`, - ); - } - const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; - if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { - throw new Error( - `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, - ); - } - - return { - id, - rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), - patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, - ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, - mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', - scheduleFrequencySeconds: - entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, - reconciliationFrequencySeconds: - entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, - parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, - defaultOwner, - defaultVisibility: defaultVisibility as 'private' | 'shared', - xNamespace, - }; - }); -} - -// --------------------------------------------------------------------------------------------- -// Discovery + parsing -// --------------------------------------------------------------------------------------------- - -async function* discoverFiles(source: SourceConfig): AsyncGenerator { - const stream = fg.stream(source.patterns, { - cwd: source.rootPath, - ignore: source.ignore, - absolute: true, - onlyFiles: true, - }); - for await (const entry of stream) { - yield entry.toString(); - } -} - -interface ParsedSpec { - kind: 'openapi' | 'asyncapi'; - title: string; - description?: string; - tags: string[]; - raw: Record; -} - -/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ -function parseSpecFile(contents: string): ParsedSpec | undefined { - let doc: unknown; - try { - doc = yaml.load(contents); - } catch { - return undefined; - } - if (!doc || typeof doc !== 'object') { - return undefined; - } - const obj = doc as Record; - const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; - if (!kind) { - return undefined; - } - const info = obj.info as Record | undefined; - const title = info?.title; - if (typeof title !== 'string' || title.length === 0) { - return undefined; - } - // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag - // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed - // through verbatim. - const tags = Array.isArray(obj.tags) - ? (obj.tags as Array>) - .map(t => t?.name) - .filter((t): t is string => typeof t === 'string') - .map(slugify) - .filter(t => t.length > 0) - : []; - - return { - kind, - title, - description: typeof info?.description === 'string' ? (info.description as string) : undefined, - tags, - raw: obj, - }; -} - -export function slugify(title: string): string { - return title - .toLowerCase() - .trim() - .replace(/[^a-z0-9]+/g, '-') - .replace(/^-+|-+$/g, '') - .slice(0, 63); -} - -// --------------------------------------------------------------------------------------------- -// Candidate mapping -// --------------------------------------------------------------------------------------------- - -interface MappingResult { - entity: Entity; - entityName: string; - error?: string; -} - -function xField(raw: Record, namespace: string, field: string): string | undefined { - const info = raw.info as Record | undefined; - const ns = info?.[`x-${namespace}`] as Record | undefined; - const value = ns?.[field]; - return typeof value === 'string' ? value : undefined; -} - -function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { - const value = rawOwner ?? defaultOwner; - return stringifyEntityRef( - parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), - ); -} - -function buildEntity(opts: { - providerName: string; - filePath: string; - contents: string; - parsed: ParsedSpec; - entityName: string; - owner: string; - lifecycle: string; - visibility: string; - registrationError?: string; -}): Entity { - const annotations: Record = { - // Identifies the source file for debugging and the catalog-info.yaml precedence check — not - // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no - // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider - // already holds the bytes; no fetch is needed). - [ANNOTATION_LOCATION]: `file:${opts.filePath}`, - [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, - [VISIBILITY_ANNOTATION]: opts.visibility, - [MANAGED_BY_ANNOTATION]: opts.providerName, - }; - if (opts.registrationError) { - annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; - } - - return { - apiVersion: 'backstage.io/v1alpha1', - kind: 'API', - metadata: { - name: opts.entityName, - title: opts.parsed.title, - description: opts.parsed.description, - tags: opts.parsed.tags, - annotations, - }, - spec: { - type: opts.parsed.kind, - lifecycle: opts.lifecycle, - owner: opts.owner, - definition: opts.contents, - }, - }; -} - -/** - * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility - * validation, and collision detection. Does not perform the catalog-info.yaml precedence check - * (that requires a catalog query and is done by the caller, once, per file). - */ -function mapCandidate(opts: { - providerName: string; - source: SourceConfig; - filePath: string; - contents: string; - parsed: ParsedSpec; - knownOwnerRefs: Set; - collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any -}): MappingResult { - const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; - const baseSlug = slugify(parsed.title); - - const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); - const owner = resolveOwnerRef(rawOwner, source.defaultOwner); - const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; - const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); - const visibility = rawVisibility ?? source.defaultVisibility; - - // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently - // defaulted. - if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner: source.defaultOwner, - lifecycle, - visibility: source.defaultVisibility, - registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, - }), - error: 'invalid-visibility', - }; - } - - // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity - // instead of blocking the whole cycle. - if (!knownOwnerRefs.has(owner)) { - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner: source.defaultOwner, - lifecycle, - visibility, - registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, - }), - error: 'invalid-owner', - }; - } - - // Rule: a name collision with another file (in this source or another) is a suffixed - // marker/error entity — first-by-path wins, this one loses. - if (collidesWith) { - const collisionName = `${baseSlug}-collision`; - return { - entityName: collisionName, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: collisionName, - owner, - lifecycle, - visibility, - registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, - }), - error: 'name-collision', - }; - } - - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner, - lifecycle, - visibility, - }), - }; -} - -// --------------------------------------------------------------------------------------------- -// Scan-state cache -// --------------------------------------------------------------------------------------------- - -interface CacheRow { - source_id: string; - file_path: string; - mtime_ms: number; - content_hash: string; - entity_name: string; - last_error: string | null; -} - -class ScanStateCache { - constructor(private readonly knex: Knex) {} - - static async create(database: DatabaseService): Promise { - const knex = await database.getClient(); - // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered - // against the 'catalog' plugin shares that plugin's own database, so using knex's default - // `knex_migrations` table here would validate our one-migration `migrationSource` against the - // real catalog plugin's own (much longer) applied-migrations history and fail with - // "migration directory is corrupt". - await knex.migrate.latest({ - tableName: 'auto_api_registration_migrations', - migrationSource: { - async getMigrations() { - return ['001_scan_state_cache']; - }, - getMigrationName(migration: string) { - return migration; - }, - async getMigration() { - return scanStateCacheMigration; - }, - }, - }); - return new ScanStateCache(knex); - } - - async rowsForSource(sourceId: string): Promise { - return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); - } - - /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ - async rowsByEntityName(entityName: string): Promise { - return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); - } - - async upsert(row: CacheRow): Promise { - await this.knex(scanStateCacheMigration.TABLE_NAME) - .insert(row) - .onConflict(['source_id', 'file_path']) - .merge(); - } - - async remove(sourceId: string, filePath: string): Promise { - await this.knex(scanStateCacheMigration.TABLE_NAME) - .where({ source_id: sourceId, file_path: filePath }) - .delete(); - } -} - -function sha1(contents: string): string { - return crypto.createHash('sha1').update(contents).digest('hex'); -} - -// --------------------------------------------------------------------------------------------- -// EntityProvider (one instance per source) -// --------------------------------------------------------------------------------------------- - -async function mapLimit( - items: T[], - limit: number, - fn: (item: T) => Promise, -): Promise { - const results: R[] = new Array(items.length); - let next = 0; - async function worker() { - while (next < items.length) { - const index = next++; - results[index] = await fn(items[index]); - } - } - await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); - return results; -} - -class AutoApiRegistrationEntityProvider implements EntityProvider { - private connection?: EntityProviderConnection; - private hasRunOnce = false; - - constructor( - private readonly source: SourceConfig, - private readonly cache: ScanStateCache, - private readonly catalogApi: CatalogService, - private readonly auth: AuthService, - private readonly logger: LoggerService, - ) {} - - getProviderName(): string { - return `auto-api-registration:${this.source.id}`; - } - - async connect(connection: EntityProviderConnection): Promise { - this.connection = connection; - } - - async runCycle(): Promise { - if (!this.connection) { - this.logger.warn( - `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, - ); - return; - } - - const existingRows = await this.cache.rowsForSource(this.source.id); - const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); - - const currentPaths = new Set(); - for await (const filePath of discoverFiles(this.source)) { - currentPaths.add(filePath); - } - - const knownOwnerRefs = await this.fetchKnownOwnerRefs(); - - const changedPaths = [...currentPaths].filter(filePath => { - const cached = existingByPath.get(filePath); - if (!cached) return true; - const stat = fs.statSync(filePath); - return stat.mtimeMs !== cached.mtime_ms; - }); - - const added: DeferredEntity[] = []; - const removedRefs: string[] = []; - const upserts: CacheRow[] = []; - const deletes: string[] = []; - - await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { - const cached = existingByPath.get(filePath); - const stat = fs.statSync(filePath); - const contents = fs.readFileSync(filePath, 'utf8'); - const hash = sha1(contents); - - if (cached && cached.content_hash === hash) { - // mtime touched, content unchanged — refresh mtime only, no mutation needed. - upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); - return; - } - - const parsed = parseSpecFile(contents); - if (!parsed) { - this.logger.error( - `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, - ); - if (cached) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); - deletes.push(filePath); - } - return; - } - - const slug = slugify(parsed.title); - - // Precedence check: a hand-authored catalog-info.yaml for this name wins. - const precedenceWinner = await this.findNonAutoSourcedEntity(slug); - if (precedenceWinner) { - this.logger.info( - `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, - ); - if (cached) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); - deletes.push(filePath); - } - return; - } - - // Global collision check: excludes this file's own prior row. - const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( - row => !(row.source_id === this.source.id && row.file_path === filePath), - ); - const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; - - const mapped = mapCandidate({ - providerName: this.getProviderName(), - source: this.source, - filePath, - contents, - parsed, - knownOwnerRefs, - collidesWith, - }); - - if (mapped.error) { - this.logger.error( - `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, - ); - } - - added.push({ entity: mapped.entity }); - upserts.push({ - source_id: this.source.id, - file_path: filePath, - mtime_ms: stat.mtimeMs, - content_hash: hash, - entity_name: mapped.entityName, - last_error: mapped.error ?? null, - }); - }); - - for (const [filePath, row] of existingByPath) { - if (!currentPaths.has(filePath)) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); - deletes.push(filePath); - } - } - - if (!this.hasRunOnce) { - // First run: establish the baseline with one `full` mutation. - await this.connection.applyMutation({ type: 'full', entities: added }); - this.hasRunOnce = true; - } else if (added.length > 0 || removedRefs.length > 0) { - await this.connection.applyMutation({ - type: 'delta', - added, - removed: removedRefs.map(entityRef => ({ entityRef })), - }); - } - - for (const row of upserts) { - await this.cache.upsert(row); - } - for (const filePath of deletes) { - await this.cache.remove(this.source.id, filePath); - } - } - - private async fetchKnownOwnerRefs(): Promise> { - const credentials = await this.auth.getOwnServiceCredentials(); - const response = await this.catalogApi.getEntities( - { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, - { credentials }, - ); - return new Set(response.items.map(entity => stringifyEntityRef(entity))); - } - - private async findNonAutoSourcedEntity(entityName: string): Promise { - const credentials = await this.auth.getOwnServiceCredentials(); - const response = await this.catalogApi.getEntities( - { filter: { kind: 'API', 'metadata.name': entityName } }, - { credentials }, - ); - return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); - } -} - -// --------------------------------------------------------------------------------------------- -// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI -// --------------------------------------------------------------------------------------------- - -class AutoApiRegistrationErrorProcessor implements CatalogProcessor { - getProcessorName(): string { - return 'auto-api-registration-error-processor'; - } - - async preProcessEntity( - entity: Entity, - _location: unknown, - _emit: CatalogProcessorEmit, - ): Promise { - const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; - if (message) { - throw new InputError(message); - } - return entity; - } -} - -// --------------------------------------------------------------------------------------------- -// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) -// --------------------------------------------------------------------------------------------- - -export default createBackendModule({ - pluginId: 'catalog', - moduleId: 'auto-api-registration', - register(reg) { - reg.registerInit({ - deps: { - catalog: catalogProcessingExtensionPoint, - database: coreServices.database, - scheduler: coreServices.scheduler, - logger: coreServices.logger, - rootConfig: coreServices.rootConfig, - catalogApi: catalogServiceRef, - auth: coreServices.auth, - }, - async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { - const sources = normalizeConfig(rootConfig); - if (sources.length === 0) { - logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); - return; - } - - const cache = await ScanStateCache.create(database); - - const providers = sources.map( - source => - new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), - ); - - catalog.addEntityProvider(providers); - catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); - - for (const [index, source] of sources.entries()) { - const provider = providers[index]; - - if (source.mode === 'poll') { - await scheduler.scheduleTask({ - id: `auto-api-registration:${source.id}:poll`, - frequency: { seconds: source.scheduleFrequencySeconds }, - timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, - fn: async () => { - try { - await provider.runCycle(); - } catch (error) { - logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); - } - }, - }); - } else { - // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of - // just this source on add/change/unlink, coalesced by chokidar's own event batching. - // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer - // `reconciliation.frequencySeconds` safety-net sweep below. - await scheduler.scheduleTask({ - id: `auto-api-registration:${source.id}:reconciliation`, - frequency: { seconds: source.reconciliationFrequencySeconds }, - timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, - fn: async () => { - try { - await provider.runCycle(); - } catch (error) { - logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); - } - }, - }); - - const watcher = chokidar.watch(source.patterns, { - cwd: source.rootPath, - ignored: source.ignore, - ignoreInitial: true, - }); - let debounce: NodeJS.Timeout | undefined; - const triggerRescan = () => { - clearTimeout(debounce); - debounce = setTimeout(() => { - provider - .runCycle() - .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); - }, 500); - }; - watcher.on('add', triggerRescan); - watcher.on('change', triggerRescan); - watcher.on('unlink', triggerRescan); - } - } - }, - }); - }, -}); -``` +`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`. Its full +content is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`](./code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts) +— copy it in as-is. It creates the `auto_api_registration_scan_state` table (`source_id`, +`file_path`, `mtime_ms`, `content_hash`, `entity_name`, `last_error`), applied via a knex custom +`migrationSource` (see the module below) rather than a filesystem `directory` migration source, +since this file lives alongside the module's TypeScript source rather than in a package-level +`migrations/` folder resolved at runtime. + +Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts` — this is the +heart of the lab. Its full content (~750 lines: config normalization, filesystem discovery and +YAML parsing, `x-*` candidate mapping with owner/visibility/collision validation, the scan-state +cache, the `EntityProvider` itself with its poll/watch scheduling, and the companion +`CatalogProcessor`) is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistration.ts`](./code/packages/backend/src/extensions/autoApiRegistration.ts) +— copy it in as-is. --- diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts new file mode 100644 index 0000000..45529f2 --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts @@ -0,0 +1,749 @@ +// packages/backend/src/extensions/autoApiRegistration.ts +// +// Lab 4: Auto Registration. +// +// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files +// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. + +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import fg from 'fast-glob'; +import chokidar from 'chokidar'; +import yaml from 'js-yaml'; +import { + coreServices, + createBackendModule, + resolvePackagePath, +} from '@backstage/backend-plugin-api'; +import { + catalogProcessingExtensionPoint, + catalogServiceRef, +} from '@backstage/plugin-catalog-node'; +import type { + CatalogProcessor, + CatalogProcessorEmit, + CatalogService, + DeferredEntity, + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-node'; +import type { + AuthService, + DatabaseService, + LoggerService, + RootConfigService, +} from '@backstage/backend-plugin-api'; +import { + ANNOTATION_LOCATION, + ANNOTATION_ORIGIN_LOCATION, + parseEntityRef, + stringifyEntityRef, +} from '@backstage/catalog-model'; +import type { Entity } from '@backstage/catalog-model'; +import { InputError } from '@backstage/errors'; +import type { Knex } from 'knex'; +import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; + +// --------------------------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------------------------- + +export const VISIBILITY_ANNOTATION = 'example.com/visibility'; +export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; +// Internal-only marker: identifies which provider instance produced an entity, so the +// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from +// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. +const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; + +const KNOWN_VISIBILITIES = new Set(['private', 'shared']); +const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; +const DEFAULT_IGNORE = [ + '**/node_modules/**', + '**/.git/**', + '**/dist/**', + '**/build/**', +]; + +export interface SourceConfig { + id: string; + rootPath: string; + patterns: string[]; + ignore: string[]; + mode: 'poll' | 'watch'; + scheduleFrequencySeconds: number; + reconciliationFrequencySeconds: number; + parseConcurrency: number; + defaultOwner: string; + defaultVisibility: 'private' | 'shared'; + xNamespace: string; +} + +function defaultRootPath(): string { + // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs + return path.resolve( + resolvePackagePath('backend'), + '../../../../lab-04-auto-registration/apis', + ); +} + +export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { + const raw = rootConfig.getOptionalConfig('autoApiRegistration'); + if (!raw) { + return []; + } + + const entries = raw.has('sources') + ? raw.getConfigArray('sources') + : [raw]; + + return entries.map(entry => { + const id = raw.has('sources') ? entry.getString('id') : 'default'; + const defaultOwner = entry.getOptionalString('defaultOwner'); + const xNamespace = entry.getOptionalString('xNamespace'); + if (!defaultOwner) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "defaultOwner"`, + ); + } + if (!xNamespace) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "xNamespace"`, + ); + } + const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; + if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { + throw new Error( + `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, + ); + } + + return { + id, + rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), + patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, + ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, + mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', + scheduleFrequencySeconds: + entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, + reconciliationFrequencySeconds: + entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, + parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, + defaultOwner, + defaultVisibility: defaultVisibility as 'private' | 'shared', + xNamespace, + }; + }); +} + +// --------------------------------------------------------------------------------------------- +// Discovery + parsing +// --------------------------------------------------------------------------------------------- + +async function* discoverFiles(source: SourceConfig): AsyncGenerator { + const stream = fg.stream(source.patterns, { + cwd: source.rootPath, + ignore: source.ignore, + absolute: true, + onlyFiles: true, + }); + for await (const entry of stream) { + yield entry.toString(); + } +} + +interface ParsedSpec { + kind: 'openapi' | 'asyncapi'; + title: string; + description?: string; + tags: string[]; + raw: Record; +} + +/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ +function parseSpecFile(contents: string): ParsedSpec | undefined { + let doc: unknown; + try { + doc = yaml.load(contents); + } catch { + return undefined; + } + if (!doc || typeof doc !== 'object') { + return undefined; + } + const obj = doc as Record; + const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; + if (!kind) { + return undefined; + } + const info = obj.info as Record | undefined; + const title = info?.title; + if (typeof title !== 'string' || title.length === 0) { + return undefined; + } + // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag + // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed + // through verbatim. + const tags = Array.isArray(obj.tags) + ? (obj.tags as Array>) + .map(t => t?.name) + .filter((t): t is string => typeof t === 'string') + .map(slugify) + .filter(t => t.length > 0) + : []; + + return { + kind, + title, + description: typeof info?.description === 'string' ? (info.description as string) : undefined, + tags, + raw: obj, + }; +} + +export function slugify(title: string): string { + return title + .toLowerCase() + .trim() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 63); +} + +// --------------------------------------------------------------------------------------------- +// Candidate mapping +// --------------------------------------------------------------------------------------------- + +interface MappingResult { + entity: Entity; + entityName: string; + error?: string; +} + +function xField(raw: Record, namespace: string, field: string): string | undefined { + const info = raw.info as Record | undefined; + const ns = info?.[`x-${namespace}`] as Record | undefined; + const value = ns?.[field]; + return typeof value === 'string' ? value : undefined; +} + +function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { + const value = rawOwner ?? defaultOwner; + return stringifyEntityRef( + parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), + ); +} + +function buildEntity(opts: { + providerName: string; + filePath: string; + contents: string; + parsed: ParsedSpec; + entityName: string; + owner: string; + lifecycle: string; + visibility: string; + registrationError?: string; +}): Entity { + const annotations: Record = { + // Identifies the source file for debugging and the catalog-info.yaml precedence check — not + // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no + // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider + // already holds the bytes; no fetch is needed). + [ANNOTATION_LOCATION]: `file:${opts.filePath}`, + [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, + [VISIBILITY_ANNOTATION]: opts.visibility, + [MANAGED_BY_ANNOTATION]: opts.providerName, + }; + if (opts.registrationError) { + annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; + } + + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'API', + metadata: { + name: opts.entityName, + title: opts.parsed.title, + description: opts.parsed.description, + tags: opts.parsed.tags, + annotations, + }, + spec: { + type: opts.parsed.kind, + lifecycle: opts.lifecycle, + owner: opts.owner, + definition: opts.contents, + }, + }; +} + +/** + * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility + * validation, and collision detection. Does not perform the catalog-info.yaml precedence check + * (that requires a catalog query and is done by the caller, once, per file). + */ +function mapCandidate(opts: { + providerName: string; + source: SourceConfig; + filePath: string; + contents: string; + parsed: ParsedSpec; + knownOwnerRefs: Set; + collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any +}): MappingResult { + const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; + const baseSlug = slugify(parsed.title); + + const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); + const owner = resolveOwnerRef(rawOwner, source.defaultOwner); + const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; + const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); + const visibility = rawVisibility ?? source.defaultVisibility; + + // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently + // defaulted. + if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility: source.defaultVisibility, + registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, + }), + error: 'invalid-visibility', + }; + } + + // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity + // instead of blocking the whole cycle. + if (!knownOwnerRefs.has(owner)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility, + registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, + }), + error: 'invalid-owner', + }; + } + + // Rule: a name collision with another file (in this source or another) is a suffixed + // marker/error entity — first-by-path wins, this one loses. + if (collidesWith) { + const collisionName = `${baseSlug}-collision`; + return { + entityName: collisionName, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: collisionName, + owner, + lifecycle, + visibility, + registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, + }), + error: 'name-collision', + }; + } + + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner, + lifecycle, + visibility, + }), + }; +} + +// --------------------------------------------------------------------------------------------- +// Scan-state cache +// --------------------------------------------------------------------------------------------- + +interface CacheRow { + source_id: string; + file_path: string; + mtime_ms: number; + content_hash: string; + entity_name: string; + last_error: string | null; +} + +class ScanStateCache { + constructor(private readonly knex: Knex) {} + + static async create(database: DatabaseService): Promise { + const knex = await database.getClient(); + // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered + // against the 'catalog' plugin shares that plugin's own database, so using knex's default + // `knex_migrations` table here would validate our one-migration `migrationSource` against the + // real catalog plugin's own (much longer) applied-migrations history and fail with + // "migration directory is corrupt". + await knex.migrate.latest({ + tableName: 'auto_api_registration_migrations', + migrationSource: { + async getMigrations() { + return ['001_scan_state_cache']; + }, + getMigrationName(migration: string) { + return migration; + }, + async getMigration() { + return scanStateCacheMigration; + }, + }, + }); + return new ScanStateCache(knex); + } + + async rowsForSource(sourceId: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); + } + + /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ + async rowsByEntityName(entityName: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); + } + + async upsert(row: CacheRow): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .insert(row) + .onConflict(['source_id', 'file_path']) + .merge(); + } + + async remove(sourceId: string, filePath: string): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .where({ source_id: sourceId, file_path: filePath }) + .delete(); + } +} + +function sha1(contents: string): string { + return crypto.createHash('sha1').update(contents).digest('hex'); +} + +// --------------------------------------------------------------------------------------------- +// EntityProvider (one instance per source) +// --------------------------------------------------------------------------------------------- + +async function mapLimit( + items: T[], + limit: number, + fn: (item: T) => Promise, +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + async function worker() { + while (next < items.length) { + const index = next++; + results[index] = await fn(items[index]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); + return results; +} + +class AutoApiRegistrationEntityProvider implements EntityProvider { + private connection?: EntityProviderConnection; + private hasRunOnce = false; + + constructor( + private readonly source: SourceConfig, + private readonly cache: ScanStateCache, + private readonly catalogApi: CatalogService, + private readonly auth: AuthService, + private readonly logger: LoggerService, + ) {} + + getProviderName(): string { + return `auto-api-registration:${this.source.id}`; + } + + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + async runCycle(): Promise { + if (!this.connection) { + this.logger.warn( + `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, + ); + return; + } + + const existingRows = await this.cache.rowsForSource(this.source.id); + const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); + + const currentPaths = new Set(); + for await (const filePath of discoverFiles(this.source)) { + currentPaths.add(filePath); + } + + const knownOwnerRefs = await this.fetchKnownOwnerRefs(); + + const changedPaths = [...currentPaths].filter(filePath => { + const cached = existingByPath.get(filePath); + if (!cached) return true; + const stat = fs.statSync(filePath); + return stat.mtimeMs !== cached.mtime_ms; + }); + + const added: DeferredEntity[] = []; + const removedRefs: string[] = []; + const upserts: CacheRow[] = []; + const deletes: string[] = []; + + await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { + const cached = existingByPath.get(filePath); + const stat = fs.statSync(filePath); + const contents = fs.readFileSync(filePath, 'utf8'); + const hash = sha1(contents); + + if (cached && cached.content_hash === hash) { + // mtime touched, content unchanged — refresh mtime only, no mutation needed. + upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); + return; + } + + const parsed = parseSpecFile(contents); + if (!parsed) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + const slug = slugify(parsed.title); + + // Precedence check: a hand-authored catalog-info.yaml for this name wins. + const precedenceWinner = await this.findNonAutoSourcedEntity(slug); + if (precedenceWinner) { + this.logger.info( + `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + // Global collision check: excludes this file's own prior row. + const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( + row => !(row.source_id === this.source.id && row.file_path === filePath), + ); + const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; + + const mapped = mapCandidate({ + providerName: this.getProviderName(), + source: this.source, + filePath, + contents, + parsed, + knownOwnerRefs, + collidesWith, + }); + + if (mapped.error) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, + ); + } + + added.push({ entity: mapped.entity }); + upserts.push({ + source_id: this.source.id, + file_path: filePath, + mtime_ms: stat.mtimeMs, + content_hash: hash, + entity_name: mapped.entityName, + last_error: mapped.error ?? null, + }); + }); + + for (const [filePath, row] of existingByPath) { + if (!currentPaths.has(filePath)) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); + deletes.push(filePath); + } + } + + if (!this.hasRunOnce) { + // First run: establish the baseline with one `full` mutation. + await this.connection.applyMutation({ type: 'full', entities: added }); + this.hasRunOnce = true; + } else if (added.length > 0 || removedRefs.length > 0) { + await this.connection.applyMutation({ + type: 'delta', + added, + removed: removedRefs.map(entityRef => ({ entityRef })), + }); + } + + for (const row of upserts) { + await this.cache.upsert(row); + } + for (const filePath of deletes) { + await this.cache.remove(this.source.id, filePath); + } + } + + private async fetchKnownOwnerRefs(): Promise> { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, + { credentials }, + ); + return new Set(response.items.map(entity => stringifyEntityRef(entity))); + } + + private async findNonAutoSourcedEntity(entityName: string): Promise { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: 'API', 'metadata.name': entityName } }, + { credentials }, + ); + return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); + } +} + +// --------------------------------------------------------------------------------------------- +// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI +// --------------------------------------------------------------------------------------------- + +class AutoApiRegistrationErrorProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'auto-api-registration-error-processor'; + } + + async preProcessEntity( + entity: Entity, + _location: unknown, + _emit: CatalogProcessorEmit, + ): Promise { + const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; + if (message) { + throw new InputError(message); + } + return entity; + } +} + +// --------------------------------------------------------------------------------------------- +// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) +// --------------------------------------------------------------------------------------------- + +export default createBackendModule({ + pluginId: 'catalog', + moduleId: 'auto-api-registration', + register(reg) { + reg.registerInit({ + deps: { + catalog: catalogProcessingExtensionPoint, + database: coreServices.database, + scheduler: coreServices.scheduler, + logger: coreServices.logger, + rootConfig: coreServices.rootConfig, + catalogApi: catalogServiceRef, + auth: coreServices.auth, + }, + async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { + const sources = normalizeConfig(rootConfig); + if (sources.length === 0) { + logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); + return; + } + + const cache = await ScanStateCache.create(database); + + const providers = sources.map( + source => + new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), + ); + + catalog.addEntityProvider(providers); + catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); + + for (const [index, source] of sources.entries()) { + const provider = providers[index]; + + if (source.mode === 'poll') { + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:poll`, + frequency: { seconds: source.scheduleFrequencySeconds }, + timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); + } + }, + }); + } else { + // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of + // just this source on add/change/unlink, coalesced by chokidar's own event batching. + // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer + // `reconciliation.frequencySeconds` safety-net sweep below. + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:reconciliation`, + frequency: { seconds: source.reconciliationFrequencySeconds }, + timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); + } + }, + }); + + const watcher = chokidar.watch(source.patterns, { + cwd: source.rootPath, + ignored: source.ignore, + ignoreInitial: true, + }); + let debounce: NodeJS.Timeout | undefined; + const triggerRescan = () => { + clearTimeout(debounce); + debounce = setTimeout(() => { + provider + .runCycle() + .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); + }, 500); + }; + watcher.on('add', triggerRescan); + watcher.on('change', triggerRescan); + watcher.on('unlink', triggerRescan); + } + } + }, + }); + }, +}); diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts new file mode 100644 index 0000000..78f6a7b --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts @@ -0,0 +1,32 @@ +// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts +// +// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal +// state private to this backend module, used to avoid re-parsing unchanged files on restart, to +// compute delta mutations, and to support collision/precedence checks across every configured +// source. +// +// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a +// filesystem `directory` migration source, since this file lives alongside the module's +// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. + +import type { Knex } from 'knex'; + +export const TABLE_NAME = 'auto_api_registration_scan_state'; + +export async function up(knex: Knex): Promise { + await knex.schema.createTable(TABLE_NAME, table => { + table.string('source_id').notNullable(); + table.string('file_path').notNullable(); + table.bigInteger('mtime_ms').notNullable(); + table.string('content_hash').notNullable(); + table.string('entity_name').notNullable(); + table.text('last_error').nullable(); + + table.primary(['source_id', 'file_path']); + table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); + }); +} + +export async function down(knex: Knex): Promise { + await knex.schema.dropTable(TABLE_NAME); +} From 7198868523d180f1ec2a708fed47c79f6c0cda54 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 12:24:17 +1000 Subject: [PATCH 10/19] Prevent large source files being inlined within lab documentation --- .specify/memory/constitution.md | 41 +++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 843f761..ce7ff1f 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,9 +1,12 @@ @@ -35,8 +48,22 @@ Documentation MUST explain the *why* and *how* of each step, not just the *what* The journey of setting up Backstage is as important as the final running instance. Steps that are non-obvious, error-prone, or surprising MUST include explanatory context. +When a lab step instructs the user to create a file, and that file's full content either +(a) cannot reasonably fit on one screen (as a rough guide, more than ~40–50 lines) or +(b) is a complete, reusable source file rather than a short edit/diff snippet — regardless of +length — the file's full content MUST be committed to the repository under the lab's own +directory (convention: a `code/` subdirectory mirroring the target relative path, e.g. +`labs/lab-0N-*/code/packages/app/src/modules/foo/Bar.tsx`), and the README MUST link to that +committed file instead of embedding the whole content inline in a fenced code block. Short +edit-in-place snippets that show surrounding context for a targeted diff against a file created +in an earlier step (not a full new file) are exempt and may remain inline. + **Rationale**: A user who follows steps blindly cannot troubleshoot or adapt. Understanding -the process produces durable knowledge; following a script does not. +the process produces durable knowledge; following a script does not. Embedding entire source +files inline inflates page length without adding explanatory value — the prose around the file +(the *why*) is what teaches; the file content itself is better consulted, copied, or diffed as +a real file. A committed, linked file is also directly reusable by a learner adapting the lab +to their own repo, which an inline fence is not. ### III. Cross-Platform Compatibility @@ -162,6 +189,10 @@ Labs that use simplified security practices (Principle IX) MUST also include: Labs MUST NOT require external network access beyond downloading freely available software and dependencies. All API samples and test data MUST be included in the repository. +Labs that instruct the user to create a large or reusable source file MUST commit that file's +content under a `code/` subdirectory within the lab's own directory and link to it from the +README, per Principle II, rather than embedding it inline. + ## Development Workflow - Each lab corresponds to one speckit feature branch following the `###-lab-name` convention. @@ -193,4 +224,4 @@ All plan `Constitution Check` gates MUST reference the principles by Roman numer - `yarn dev` does not exist. Use `yarn start` instead. -**Version**: 1.3.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-06-08 +**Version**: 1.4.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-07-03 From cb2a283c71a5017582ca340bcacaa2f946f6768f Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 23:20:27 +1000 Subject: [PATCH 11/19] Auto-registration lab functional --- labs/lab-04-auto-registration/README.md | 8 +++++ .../checklists/issues.md | 33 +++++++++++++++++++ specs/004-lab-4-auto-registration/tasks.md | 19 +++++++++++ 3 files changed, 60 insertions(+) create mode 100644 specs/004-lab-4-auto-registration/checklists/issues.md diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md index 11786bd..8ca2198 100644 --- a/labs/lab-04-auto-registration/README.md +++ b/labs/lab-04-auto-registration/README.md @@ -400,6 +400,14 @@ checks — is reused unchanged; only how bytes arrive on local disk differs. exists.** Confirm that group is actually loaded in the catalog (check its entity page directly) — this error means the *catalog* doesn't know about the group yet, not that your YAML is wrong. A common cause locally is a `catalog.locations` URL that hasn't been pushed yet (see Step 5). + This can also show up as "no APIs appear in the catalog at all" rather than a single owner + error: if the `catalog.locations` entry for your org data (`teams.yaml`/`users.yaml`) points at + a branch name that's since been merged and deleted (a stale reference from an earlier lab, + rather than your current branch), that location 404s, the group it would have defined never + registers, and *every* auto-sourced API whose owner references that group fails validation — + which looks like the discovery mechanism itself is broken. Check the backend logs for "Unable + to read url, no matching files found" lines against your `catalog.locations` URLs first; if any + point at a branch other than `main` or your current branch, that's almost always the real cause. - **An entity you expect to see is silently missing, with no error logged.** Check the backend logs for a `Policy check failed for api:default/` warning — this means the entity was built and emitted, but failed Backstage's own entity-schema validation (for example, diff --git a/specs/004-lab-4-auto-registration/checklists/issues.md b/specs/004-lab-4-auto-registration/checklists/issues.md new file mode 100644 index 0000000..d6cfebe --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/issues.md @@ -0,0 +1,33 @@ +## Run 1 - 2026/07/03 + +- [X] No APIs were available in the catalog after starting backstage. Relevant log entries are included below. + + **Resolved** (specs/004-lab-4-auto-registration/tasks.md T038–T040): root cause was six + `catalog.locations` URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` still + pinned to the stale, merged-and-gone `003-api-quality` branch. `teams.yaml` (defining + `platform-team`) 404'd as a result, so the Lab 4 owner validator correctly rejected + `galaxy-openapi.yaml`'s `owner: group:default/platform-team` — the actual cause of the empty + catalog. Fixed by repointing those six URLs at `main`, where the content is already merged. + Verified via a clean `yarn start`: museum/streetlights/train-travel/scalar-galaxy/ + precedence-demo-api all register with no `AutoApiRegistrationErrorProcessor` errors and no + "Unable to read url" warnings. + +``` +2026-07-03T12:38:51.027Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:00.820Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml entity="location:default/generated-4c23d1d635d435e80cc8b698fc86956f9fffff3c" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml" +2026-07-03T12:39:00.895Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml entity="location:default/generated-2cde689609df331e9267009ea6c02850d6b1603e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml" +2026-07-03T12:39:00.896Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml entity="location:default/generated-cc3b19c3e787fc46d27f583f1d53dd2e8ec1a456" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml" +2026-07-03T12:39:00.899Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml entity="location:default/generated-ae8dd75945a327a0201813834829ec035913de8e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml" +2026-07-03T12:39:00.907Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml entity="location:default/generated-1390492c77b8f266a9f6608c0595f96efa060611" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml" +2026-07-03T12:39:00.915Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml entity="location:default/generated-9ecb0d8da3f46f025b2e5e23de21f385b1a49388" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml" +2026-07-03T12:39:07.085Z catalog warn Processor AutoApiRegistrationErrorProcessor threw an error while preprocessing; caused by InputError: Owner "group:default/platform-team" does not resolve to a known User or Group entity entity="api:default/scalar-galaxy" location="file:/Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml" +2026-07-03T12:39:21.056Z rootHttpRouter info [2026-07-03T12:39:21.056Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.056Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.060Z rootHttpRouter info [2026-07-03T12:39:21.060Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.060Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.061Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:51.087Z rootHttpRouter info [2026-07-03T12:39:51.087Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.087Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.092Z rootHttpRouter info [2026-07-03T12:39:51.092Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.092Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.093Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:40:21.123Z rootHttpRouter info [2026-07-03T12:40:21.123Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.123Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.127Z rootHttpRouter info [2026-07-03T12:40:21.127Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.127Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.128Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +``` \ No newline at end of file diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md index ac48f3f..2ae6c85 100644 --- a/specs/004-lab-4-auto-registration/tasks.md +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -131,6 +131,24 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar --- +## Phase 7: Bug Fixes (from checklists/issues.md — Run 1, 2026/07/03) + +**Purpose**: Fix the reported "No APIs were available in the catalog after starting backstage" issue. Root cause analysis of the pasted log: + +- The `group:default/platform-team` owner-resolution failure (T020's validator, `InputError` on `galaxy-openapi.yaml`) is a *symptom*: `teams.yaml` — which defines `platform-team` — never loads, because its catalog location URL is 404ing. +- That 404, and the four other "Unable to read url" warnings (`teams.yaml`, `users.yaml`, `museum-api.yaml`, `streetlights-api.yaml`, `train-travel-api.yaml`... actually `platform-user.yaml` too), all point at `raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/...` — a stale feature-branch name left over from when Lab 3's instructions were followed. That content has since been merged to `main` (verified: present at `main:labs/lab-02-users-roles/catalog/teams.yaml` and `main:labs/lab-03-api-quality/catalog/platform-user.yaml`), so the branch-pinned URLs should point at `main` instead. +- With `teams.yaml` unreachable, `platform-team` never registers as a `Group`, so the Lab 4 owner validator (correctly) rejects `galaxy-openapi.yaml`'s `owner: group:default/platform-team` and emits a marker/error entity instead of a real one — leaving the catalog with no visible APIs. +- The one remaining branch-pinned location (`precedence-demo-catalog-info.yaml`, pointing at `004-lab-4-auto-registration`) is *not* a bug — it is new-on-this-branch content that genuinely requires the branch to be pushed to GitHub first, exactly as documented in `labs/lab-04-auto-registration/README.md` (the "This only works once you've pushed your branch" note). No code or config change fixes that one; it's a sequencing step the learner has to perform. + +- [X] T038 Fix the five stale `003-api-quality`-branch catalog location URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` (lines ~130, 136, 142, 148, 154, 160, plus the two `spectralLinter` ruleset URLs at ~206–207) to point at `main` instead, since that content is already merged to `main` +- [X] T039 [P] Add a troubleshooting entry to `labs/lab-04-auto-registration/README.md`'s Troubleshooting section explaining the observed failure chain (stale/unpushed branch in an org-data or API catalog location → that entity never registers → any auto-sourced API whose `x-examplecorp.owner` references a group defined only in that unreachable location fails owner validation → looks like "no APIs in the catalog" even though the real cause is upstream of Lab 4's own code) so learners can self-diagnose instead of assuming the `EntityProvider` is broken +- [X] T040 Restart `labs/lab-01-base-backstage/backstage/` from a clean `yarn start` after T038 and confirm via backend logs and the catalog UI: `platform-team` resolves (no more `InputError` on `galaxy-openapi.yaml`), museum/streetlights/train-travel APIs and the `platform-team`/`platform-user` org entities all load without "Unable to read url" warnings, and the Galaxy API entity appears with `spec.owner: group:default/platform-team` (SC-001/SC-002); note that `precedence-demo-api` will still be absent unless the current branch has been pushed to GitHub, per T039's documented caveat +- [X] T041 Update `specs/004-lab-4-auto-registration/checklists/issues.md`: mark the Run 1 item resolved with a one-line pointer to T038–T040 once T040's re-verification passes + +**Checkpoint**: Catalog populates on a clean start; the only remaining "gap" (precedence-demo-api before the branch is pushed) is expected and documented, not a bug. + +--- + ## Dependencies & Execution Order ### Phase Dependencies @@ -141,6 +159,7 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar - **User Story 2 (Phase 4)**: Depends on Foundational completion; extends the same file US1 built (T012), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency - **User Story 3 (Phase 5)**: Depends on US1 (T015) and US2 (T024) being implemented, since the README documents both the discovery mechanism and the metadata-sourcing/error behavior - **Polish (Phase 6)**: Depends on all user stories being complete +- **Bug Fixes (Phase 7)**: Independent of Phases 3–6's code (config/docs-only fix) but logically follows Polish since it was found during Polish-phase (T036) cold-start validation ### User Story Dependencies From e7edc032d1aea85078c26a69da19ddd06aaae19e Mon Sep 17 00:00:00 2001 From: DawMatt Date: Sat, 4 Jul 2026 05:00:38 +1000 Subject: [PATCH 12/19] Add lab to readme. Ensure readme updated automatically in future. --- .github/workflows/readme-sync.yml | 14 ++++++++ .specify/memory/constitution.md | 48 ++++++++++++++++++---------- .specify/templates/tasks-template.md | 3 ++ CONTRIBUTING.md | 9 +++++- README.md | 20 ++++++++---- scripts/check-readme.sh | 31 ++++++++++++++++++ 6 files changed, 100 insertions(+), 25 deletions(-) create mode 100644 .github/workflows/readme-sync.yml create mode 100755 scripts/check-readme.sh diff --git a/.github/workflows/readme-sync.yml b/.github/workflows/readme-sync.yml new file mode 100644 index 0000000..32e9c73 --- /dev/null +++ b/.github/workflows/readme-sync.yml @@ -0,0 +1,14 @@ +name: README sync check + +on: + push: + branches: [main] + pull_request: + +jobs: + check-readme: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Verify root README references every labs/ and specs/ directory + run: bash scripts/check-readme.sh diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index ce7ff1f..5b4f204 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,27 +1,27 @@ For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan -at specs/004-lab-4-auto-registration/plan.md +at specs/005-lab-5-mocking-testing/plan.md diff --git a/specs/005-lab-5-mocking-testing/data-model.md b/specs/005-lab-5-mocking-testing/data-model.md new file mode 100644 index 0000000..30c4de3 --- /dev/null +++ b/specs/005-lab-5-mocking-testing/data-model.md @@ -0,0 +1,78 @@ +# Phase 1 Data Model: Lab 5 - Mocking and Testing + +This lab has no application database entities. "Data model" here means the configuration and +runtime shapes introduced by the new mock-orchestration script and frontend module. + +## Mock Registry Entry (in-memory, computed at each `start-mocks.mjs` run) + +Computed once per run from the OpenAPI files matched by Lab 4's `autoApiRegistration` +`rootPath`/`patterns` config; not persisted between runs (recomputed deterministically each start). + +| Field | Type | Description | +|---|---|---| +| `specFilePath` | string | Absolute path to the discovered `*-openapi.yaml` file | +| `entityNameSlug` | string | Slugified `info.title`, using the same algorithm as Lab 4's entity-name derivation, so the frontend module can compute a matching proxy path from `entity.metadata.name` alone | +| `port` | number | Assigned deterministically: `4010 + index`, where `index` is the 0-based position in the list of matched files sorted by path | +| `proxyPath` | string | `/api/proxy/mock/` | + +## Generated Config File: `app-config.mocks.yaml` + +Written by `scripts/start-mocks.mjs` before Prism child processes are spawned; not committed to +the repository (generated artifact, added to `.gitignore`). Shape: + +```yaml +proxy: + endpoints: + /mock/: + target: 'http://localhost:' + changeOrigin: true +``` + +One entry per Mock Registry Entry. Merged with the committed `app-config.yaml` via +`backstage-cli repo start --config app-config.yaml --config app-config.mocks.yaml`. + +## New committed config: `mocking.defaultCredential` (in `app-config.yaml`) + +```yaml +mocking: + defaultCredential: + scheme: 'bearer' # matches the securityScheme type used in this lab's example APIs + value: 'lab-mock-token-do-not-use' # obviously fictional, per Constitution Principle IX +``` + +Read client-side via `configApiRef` by the `apiMocking` frontend module (research.md R6). Not a +secret — deliberately fictional and safe to ship in the frontend bundle, same as any other +documented lab-only example credential. + +## Frontend runtime shape: merged `servers` list + +At render time, for any `openapi`-type entity, the `apiMocking` module computes: + +```text +effectiveServers = entity.spec.definition.servers (native, unmodified) + + [{ url: , description: 'Local mock (Lab 5)' }] +``` + +No persistence; recomputed per render from the entity already loaded by the catalog page. The +proxy path is derived purely from `entityNameSlug(entity.metadata.name)` — no per-entity +configuration is required for a new API to get a mock server entry, satisfying the "generic by +default" requirement from this session's clarification. + +## Relationships + +```text +OpenAPI spec file (*-openapi.yaml) + │ (discovered via Lab 4's autoApiRegistration config) + ▼ +Mock Registry Entry ──► Prism child process (serves on `port`) + │ + ▼ +app-config.mocks.yaml proxy endpoint ──► merged into running Backstage config + │ + ▼ +Catalog API entity's rendered `servers` list (apiMocking frontend module) + │ + ▼ +Learner selects target in Swagger UI "Try it out" ──► request routed through +Backstage's existing proxy (mock) or directly to the sandbox host (Galaxy's native servers) +``` diff --git a/specs/005-lab-5-mocking-testing/plan.md b/specs/005-lab-5-mocking-testing/plan.md new file mode 100644 index 0000000..f25de19 --- /dev/null +++ b/specs/005-lab-5-mocking-testing/plan.md @@ -0,0 +1,169 @@ +# Implementation Plan: Lab 5 — Mocking and Testing + +**Branch**: `005-lab-5-mocking-testing` | **Date**: 2026-07-04 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `specs/005-lab-5-mocking-testing/spec.md` + +## Summary + +Lab 5 builds on the running Backstage 1.51.0 instance from Labs 1–4. It adds two capabilities to +every registered OpenAPI API's page: a locally running, dynamically generated mock implementation +(via Prism, one instance per discovered `*-openapi.yaml` file, reusing Lab 4's discovery config as +the single source of truth — FR-001, FR-005) and a documented public sandbox target for APIs that +already have one natively (the Galaxy API's `galaxy.scalar.com`/`void.scalar.com` servers — FR-003). +Both are selectable from the API's existing "Try it out" viewer without editing any committed spec +file's `servers` array — instead, a new frontend module (`packages/app/src/modules/apiMocking/`) +overrides the `apiDocsConfigRef` API, the same wrapper-module pattern Lab 3 used for +`apiGrade`/`spectralLinter`, since Backstage's default OpenAPI viewer has no built-in +config-level toggle for extra servers or credential injection (research.md R1). The mock server(s) +start automatically alongside `yarn start` via a new orchestrator script and `concurrently`, with +`wait-on` sequencing to avoid a config-file race (FR-001, research.md R4). Requests to a mock are +routed through Backstage's existing proxy mechanism (a generated `app-config.mocks.yaml`, one +endpoint per discovered API) to avoid CORS; Galaxy's sandbox is called directly since its native +`servers` entries already work (FR-004, research.md R5). A documented, fictional default +credential lives in `app-config.yaml` and is applied by pre-authorizing Swagger UI's own +"Authorize" dialog on mount — not a proxy header rewrite, since a static header can't yield to a +learner's own credential — giving "default that works out of the box, freely overridable" for +free from the same UI a learner would use anyway (FR-006, FR-007, research.md R6). Errors (mock +not running, sandbox unreachable, bad credential) surface through the viewer's own request/response +handling, not a bespoke error system (FR-008). A `## Security Note` section (Constitution +Principle IX / Lab Structure Standards) documents the shortcut and the production alternative +before the learner encounters the default credential. + +## Technical Context + +**Language/Version**: TypeScript / Node.js 20 LTS (Backstage 1.51.0, pinned from Lab 1) + +**Primary Dependencies**: +- `@stoplight/prism-cli@5.15.11` (Apache-2.0) — new root devDependency; provides the `prism mock` + command used to dynamically serve each discovered OpenAPI file (research.md R2). +- `concurrently` (new root devDependency) — runs the mock orchestrator alongside + `backstage-cli repo start` (research.md R4); not previously present anywhere in the workspace. +- `wait-on` (new root devDependency) — sequences Backstage's startup to wait for the generated + `app-config.mocks.yaml` to exist, avoiding a config-read race (research.md R4). +- `@backstage/plugin-api-docs` (already installed, `^0.14.1`) — supplies `apiDocsConfigRef` and + the default `OpenApiDefinitionWidget`/`swagger-ui-react` rendering this lab wraps, not replaces + (research.md R1). +- No new backend Backstage module — unlike Lab 4, this lab needs no `EntityProvider`/catalog + changes; it only reads Lab 4's existing discovery config. + +**Storage**: One new generated (not committed) config file, `app-config.mocks.yaml` (proxy +endpoints, one per discovered OpenAPI file — data-model.md), plus one new committed config key, +`mocking.defaultCredential`, in `app-config.yaml`. No database changes. + +**Testing**: Manual browser + orchestrator-log verification per lab README (no automated test +suite — consistent with Labs 1–4's tutorial-lab testing approach). + +**Target Platform**: Local development machine — Windows 10/11 and macOS 12+ (same as Labs 1–4). + +**Project Type**: Tutorial lab — Markdown documentation + a small Node orchestrator script + one +new TypeScript frontend module (`apiDocsConfigRef` override) + config additions. + +**Performance Goals**: A learner can select the mock target and receive a response within the same +interaction as any other "Try it out" request (SC-001); mock startup adds at most a few seconds to +`yarn start` (Prism boot time per instance), not exercised against real-world repo scale — port +assignment (research.md R4, data-model.md) is a lab-scale convenience, documented as needing a +persisted port map at real mono-repo scale (quickstart.md). + +**Constraints**: Zero cost; cross-platform; no external network access beyond the optional Galaxy +sandbox calls (User Story 2), which remain optional relative to the fully local mock path (FR-011); +builds on Labs 1–4 without editing any previously committed spec file's native `servers` array +(this session's clarification) or disrupting existing registered APIs. + +**Scale/Scope**: Lab demo: 1 new orchestrator script, 1 new frontend module (~2-3 files), 2 +new/modified config keys (`mocking.defaultCredential`, root `start` script), 3 new root +devDependencies, 0 new sample API files (reuses Museum + Galaxy from Labs 1 & 4), 0 changes to any +existing committed API spec file. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design — all gates hold.* + +| Principle | Gate | Status | +|-----------|------|--------| +| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: `apiDocsConfigRef` customization for target-server injection and credential pre-fill, dynamic OpenAPI mocking via Prism, and Backstage proxy config for CORS-safe frontend calls | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY `apiDocsConfigRef` override was necessary (no built-in toggle, research.md R1), WHY a static proxy header was rejected in favor of an Authorize-dialog pre-fill (research.md R6), and WHY Galaxy needs no additional proxy config while Museum's mock does (research.md R5). The orchestrator script and frontend module are each likely >50 lines and MUST be committed under `labs/lab-05-mocking-testing/code/` and linked from the README, not inlined | ✅ Pass (with file-size rule flagged for tasks) | +| III. Cross-Platform Compatibility | Orchestrator script, `concurrently`/`wait-on`/`prism` are pure JS, no native/platform-specific dependencies; `yarn start` continues to work identically on Windows and macOS; README documents any PowerShell vs. bash quoting differences for the modified `start` script string | ✅ Pass | +| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–4's existing Node/Yarn setup — all three new dependencies install via the existing `yarn add` workflow; README documents the install step | ✅ Pass | +| V. Zero-Cost Operation | `@stoplight/prism-cli` is Apache-2.0 (free/open-source, satisfies Principle V's "freely available under open-source or freeware terms" though not MIT specifically); `concurrently`/`wait-on` are MIT; Galaxy's public sandbox is free; no paid service, and the fully local mock path (User Story 1) has zero dependency on external network availability (FR-011) | ✅ Pass | +| VI. Progressive Lab Structure | Requires Labs 1–4 completion; reuses Lab 4's discovery config and Lab 1/4's Museum/Galaxy specs unmodified; no prior lab's committed spec file is edited | ✅ Pass | +| VII. Modern & Purposeful API Examples | No new sample API is introduced — Museum and Galaxy, both already compliant per their own labs' plans, are reused as-is | ✅ Pass (N/A, no new example) | +| VIII. Support Experimentation | README documents adaptable conventions: mock port base (`4010`), reliance on Lab 4's discovery glob, the default credential value, and the proxy endpoint naming scheme; verification tests outcomes (a response comes back, the credential used is the one displayed), not exact port numbers | ✅ Pass | +| IX. Pragmatic Security for Learning Environments | `mocking.defaultCredential` is a fictional, clearly-labelled example value stored in committed `app-config.yaml`; a `## Security Note` section (before the learner encounters the credential config step) explains the shortcut and the production alternative (secrets manager / vault-injected config, per-user credentials, never committing real credentials) | ✅ Pass (new obligation: Security Note section required, first for this lab series to introduce a credential) | + +No violations. Complexity Tracking section omitted. + +## Project Structure + +### Documentation (this feature) + +```text +specs/005-lab-5-mocking-testing/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +└── tasks.md # Phase 2 output (/speckit-tasks — NOT created here) +``` + +No `contracts/` directory — this lab produces no external API contracts of its own. The generated +`app-config.mocks.yaml` shape and the `mocking.defaultCredential` config key are documented in +data-model.md instead. + +### Source Code (repository root) + +```text +labs/ +└── lab-05-mocking-testing/ + ├── README.md # Lab documentation (overview, prereqs, steps, + │ # verification, troubleshooting, Security Note, + │ # adaptable conventions per US-adaptation goal) + └── code/ + ├── scripts/ + │ └── start-mocks.mjs # New: mock orchestrator (discovery, port assignment, + │ # app-config.mocks.yaml generation, Prism child + │ # process supervision) — committed here, linked + │ # from README per Constitution Principle II + └── packages/ + └── app/ + └── src/ + └── modules/ + └── apiMocking/ + └── index.tsx # New: apiDocsConfigRef override — merges mock/ + # sandbox servers, pre-authorizes Swagger UI with + # the default credential (research.md R1, R6) +``` + +Changes to the student's Backstage instance (guided by README, not committed as generated output, +but the module/script source itself IS committed since it is lab teaching content): + +```text +# In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): +package.json # Modified: add @stoplight/prism-cli, concurrently, + # wait-on; change "start" script (research.md R4) +app-config.yaml # Modified: add mocking.defaultCredential block +.gitignore # Modified: add app-config.mocks.yaml (generated, + # not committed) +scripts/ +└── start-mocks.mjs # New: copied from labs/lab-05-.../code/scripts/ +packages/app/ +└── src/ + ├── apis.ts # Modified: register the apiMocking module + └── modules/ + └── apiMocking/ + └── index.tsx # New: copied from labs/lab-05-.../code/packages/ + # app/src/modules/apiMocking/ +``` + +**Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–4. All +student-facing content lives under `labs/lab-05-mocking-testing/`; the orchestrator script and +frontend module are committed under its `code/` subdirectory (both exceed the ~40-50 line inline +threshold per Constitution Principle II) and linked from the README rather than embedded inline. +The student's Backstage instance (from Lab 1) is modified in place per the README instructions, +same pattern as Labs 3–4. No new sample API files are added — Museum (Lab 1) and Galaxy (Lab 4) +are reused unmodified. Speckit artefacts live under `specs/005-lab-5-mocking-testing/` per +convention. + +## Complexity Tracking + +No violations recorded — table omitted per template instructions. diff --git a/specs/005-lab-5-mocking-testing/quickstart.md b/specs/005-lab-5-mocking-testing/quickstart.md new file mode 100644 index 0000000..bb1a1af --- /dev/null +++ b/specs/005-lab-5-mocking-testing/quickstart.md @@ -0,0 +1,75 @@ +# Quickstart: Lab 5 — Mocking and Testing + +This is the design-time quickstart used to validate the plan; the learner-facing README (built in +`/speckit-tasks` + `/speckit-implement`) will follow the same flow with full explanatory prose per +Constitution Principle II. + +## Prerequisites + +- Labs 1–4 completed (running Backstage 1.51.0 instance, Lab 2 teams/visibility, Lab 3 quality + tooling, Lab 4 auto-registration with Museum, Streetlights, Galaxy, and precedence-demo APIs + registered). +- No new external accounts or services. No new OS-level prerequisites beyond the existing + Node/Yarn toolchain. + +## Steps + +1. **Install dependencies**: add `@stoplight/prism-cli`, `concurrently`, and `wait-on` as root + devDependencies of the Lab 1 Backstage workspace. +2. **Add the mock orchestrator script** + (`scripts/start-mocks.mjs`): discovers OpenAPI files via Lab 4's existing + `autoApiRegistration.rootPath`/`patterns` config, assigns each a deterministic port from + `4010`, writes `app-config.mocks.yaml` with one proxy endpoint per file, then spawns one + `prism mock -p ` child process per file (research.md R2–R5, data-model.md). +3. **Update the root `start` script** to run the orchestrator and Backstage concurrently, with + `wait-on` sequencing so Backstage only boots once `app-config.mocks.yaml` exists + (research.md R4). +4. **Add `mocking.defaultCredential`** to `app-config.yaml` (data-model.md) — a documented, + obviously fictional bearer token — and add `app-config.mocks.yaml` to `.gitignore` (it's a + generated artifact, not committed). +5. **Add the `apiMocking` frontend module** + (`packages/app/src/modules/apiMocking/index.tsx`): overrides `apiDocsConfigRef` for the + `openapi` widget type to (a) merge a `Local mock (Lab 5)` server entry — computed from + `entity.metadata.name`, no per-entity config needed — into the rendered `servers` list, and + (b) pre-authorize Swagger UI's "Authorize" dialog with `mocking.defaultCredential` on mount + (research.md R1, R6). Registered in `packages/app/src/apis.ts` (or equivalent frontend module + registration point). +6. **Start Backstage** (`yarn start`) and confirm: + - The mock orchestrator log shows a Prism instance started for the Museum API (and the other + discovered OpenAPI files) on its assigned port. + - Opening the Museum API's page in Backstage shows a `Local mock (Lab 5)` entry in the server + dropdown alongside its native (non-resolvable) production URL. +7. **User Story 1 — mock**: select the `Local mock (Lab 5)` server for the Museum API, execute a + sample "Try it out" request, and confirm a realistic example response comes back (US1, SC-001). + Stop the mock orchestrator process, retry the same request, and confirm a clear connection + error is shown (not a silent failure). +8. **User Story 2 — sandbox**: open the Galaxy API's page, confirm its native `servers` dropdown + already offers `galaxy.scalar.com` / `void.scalar.com` with no additional Lab 5 configuration, + select one, execute a request, and confirm the response comes from the real sandbox (US2, + SC-002). +9. **User Story 3 — credentials**: on an authenticated request (mock or sandbox), confirm the + "Authorize" dialog is already pre-filled with the `mocking.defaultCredential` value and the + request succeeds with zero setup (SC-003). Replace it with a different value in the same + dialog, confirm the replacement is what's actually sent (SC-004), and confirm + `app-config.mocks.yaml`/no other file ever contains a personal credential. +10. **Verify a second, unconfigured API gets a mock automatically**: temporarily add a third + `*-openapi.yaml` file matching Lab 4's discovery pattern, restart `yarn start`, and confirm a + `Local mock (Lab 5)` entry appears for it too with zero Lab-5-specific configuration — proving + the mechanism is generic, not hardcoded to Museum (this session's clarification). Remove the + temporary file afterward. + +## Out of scope for this quickstart + +- AsyncAPI mocking (Streetlights) — explicitly out of scope per this session's clarification; + README states this as a known limitation. +- Exercising a genuinely unreachable/CORS-blocked sandbox end-to-end (the lab's own Galaxy example + is reachable and CORS-permissive); the resulting error-handling behavior (FR-008) is documented, + not walked through against a real failure. + +## Scaling beyond the lab (documented, not walked through) + +The README includes a note on why default-port assignment (`4010 + index`, sorted by path) is +stable enough for a handful of lab APIs but would need an explicit, persisted port map for a large +mono-repo where files are added/removed frequently (to avoid every learner's mock ports shifting +on every restart) — mirroring the same "lab-scale convenience vs. real-scale mechanism" framing +Lab 4 used for its scan-state cache. diff --git a/specs/005-lab-5-mocking-testing/research.md b/specs/005-lab-5-mocking-testing/research.md new file mode 100644 index 0000000..8aff52e --- /dev/null +++ b/specs/005-lab-5-mocking-testing/research.md @@ -0,0 +1,130 @@ +# Phase 0 Research: Lab 5 - Mocking and Testing + +## R1: How to override the API viewer's target servers and inject a default credential + +**Decision**: Add a small custom frontend module (`packages/app/src/modules/apiMocking/`) that overrides +the `apiDocsConfigRef` API — the same `ApiBlueprint`/frontend-module pattern Lab 3 already established +for `apiGrade` and `spectralLinter` (`packages/app/src/modules/apiGrade/index.ts`, +`packages/app/src/modules/spectralLinter/index.ts`). The module supplies a custom +`defaultDefinitionWidgets()` entry for the `openapi` type that wraps +`@backstage/plugin-api-docs`'s `OpenApiDefinitionWidget`, passing through two extra +`swagger-ui-react` props it does not otherwise expose to callers: a merged `spec.servers` array +(mock + any lab-configured sandbox target, in addition to whatever the spec natively declares) and +an `onComplete` hook that pre-authorizes Swagger UI's "Authorize" state with a default credential. + +**Rationale**: Confirmed by reading the installed `@backstage/plugin-api-docs@^0.14.1` source +(`OpenApiDefinition.esm.js`) that "Try it out" is enabled by default (`SwaggerUI` is rendered +directly, no `supportedSubmitMethods` restriction) and that `swaggerUiProps` are spread onto +`SwaggerUI` unmodified — but the entry point (`ApiDefinitionCard.esm.js`) only ever calls +`definitionWidget.component(entity.spec.definition)`, and the default widget factory hardcodes +`component: (definition) => ` with no extra props threaded +through and no app-config-level toggle for `servers` or auth. There is no third-party Backstage +community plugin that already does this. Overriding `apiDocsConfigRef` is the smallest change that +reaches the render path; it is also directly analogous to Lab 3's precedent, so it introduces no new +architectural pattern to the series. + +**Alternatives considered**: +- Forking/patching `@backstage/plugin-api-docs` in place — rejected: unmaintainable across + Backstage upgrades, contradicts Constitution Principle II's teaching-through-real-mechanisms intent. +- Replacing `plugin-api-docs` entirely with a bespoke viewer — rejected: massive scope increase, + would also need to reimplement AsyncAPI rendering used by Streetlights, unrelated to this lab's goal. + +## R2: Mock server tooling + +**Decision**: `@stoplight/prism-cli` (currently `5.15.11`, Apache-2.0), added as a root +`devDependency` and invoked as a child process (`prism mock -p `) per discovered +OpenAPI file. Prism generates responses dynamically from a spec's own examples/schemas — matching +GOAL.md's "dynamically create... a mock implementation" requirement — and works against the +Museum API's self-contained spec (no external `$ref`s, per Constitution Principle VII) with no +further setup. + +**Rationale**: Free, actively maintained, npm-installable (no JVM/Docker prerequisite), and purpose +-built for OpenAPI mocking. + +**Alternatives considered**: +- WireMock — rejected: requires a JVM prerequisite not otherwise needed by any prior lab, adding + friction (Constitution Principle IV). +- Hand-rolled static-JSON fixture server — rejected: loses "dynamic" generation from the spec's own + examples/schemas; would drift from the spec instead of reflecting it. + +## R3: Which files to mock — discovery convention + +**Decision**: Reuse Lab 4's existing `autoApiRegistration.rootPath`/`patterns` config +(`*-openapi.yaml`) as the single source of truth for which OpenAPI files exist, rather than +introducing a second glob configuration surface for mocking. + +**Rationale**: Avoids config drift between "what's registered in the catalog" and "what's mockable"; +directly satisfies this session's clarification that mocking must be generic/available by default to +any registered OpenAPI API, not hardcoded to one example. + +**Alternatives considered**: A separate `mocking.include` glob — rejected as duplicate, driftable +configuration for the same underlying file set. + +## R4: Running mock server(s) alongside `yarn start` + +**Decision**: Add `concurrently` and `wait-on` as new root devDependencies and a new +`scripts/start-mocks.mjs` orchestrator script. The script synchronously discovers OpenAPI files, +assigns each a deterministic port (sequential from `4010`, sorted by file path), and writes a +generated `app-config.mocks.yaml` containing one proxy endpoint per file — all before spawning the +long-lived Prism child processes. The root `start` script becomes: + +```json +"start": "concurrently -k -n app,mocks \"wait-on ./app-config.mocks.yaml && backstage-cli repo start --config app-config.yaml --config app-config.mocks.yaml\" \"node scripts/start-mocks.mjs\"" +``` + +`wait-on` guarantees the generated config file exists before `backstage-cli repo start` reads it, +avoiding a startup race between the two concurrent processes. + +**Rationale**: `backstage-cli repo start` already owns frontend/backend orchestration internally; +adding a third independent process via `concurrently` (not currently a dependency anywhere in the +workspace) is the smallest change that achieves "mock starts automatically with `yarn start`," per +this session's clarification, without forking Backstage's own CLI. + +**Alternatives considered**: Implementing mocking as a Backstage backend plugin/module (the pattern +Lab 4 used for catalog discovery) — rejected: heavier (new backend module + migration), and a poor +fit for supervising long-lived external child processes, which a plain Node script does more simply. + +## R5: Proxy routing and CORS + +**Decision**: Every discovered OpenAPI file gets a generated proxy endpoint +(`/api/proxy/mock/`, using the same slugification as Lab 4's entity-name +derivation) targeting its local Prism instance, satisfying FR-004 (avoid browser CORS restrictions +via Backstage's existing proxy mechanism). The Galaxy API's pre-existing native `servers` entries +(`galaxy.scalar.com`, `void.scalar.com`) are called directly by the browser, without an additional +proxy hop — Scalar's public example services are designed for direct external/browser calls. + +**Rationale**: Matches FR-003's "native list MAY continue to satisfy target selection without +additional configuration" and avoids proxying traffic that doesn't need it. + +**Alternatives considered**: Proxying all sandbox traffic uniformly — rejected as unnecessary +complexity when the native `servers` mechanism already works for Galaxy. + +## R6: Default non-production credential delivery + +**Decision**: The default credential value is declared in `app-config.yaml` under a new +`mocking.defaultCredential` key (a documented, obviously fictional value, e.g. +`lab-mock-token-do-not-use`, per Constitution Principle IX). The frontend module (R1) reads it via +`configApiRef` and applies it by pre-authorizing Swagger UI's internal auth state on mount +(`onComplete(system) => system.authActions.authorize(...)`) — a genuine pre-fill of the same +"Authorize" dialog a learner would otherwise fill in themselves, not a server-side header rewrite. + +**Rationale**: A static Backstage proxy `headers:` entry unconditionally sets/overwrites the +outgoing request's Authorization header on every proxied call — it cannot conditionally yield to a +learner's own credential, which would silently break FR-007 ("supply your own credential, +overriding the default"). A client-side pre-fill is naturally editable through the exact same UI +control the learner would use anyway, so "default that's easy to override" falls out of the +mechanism for free. + +**Alternatives considered**: Unconditional proxy-level header injection — rejected, breaks override. +Requiring learners to always enter a credential manually — rejected, fails FR-006 ("succeeds by +default without any learner setup"). + +## Deviation from the specify-session Clarifications wording + +The spec's Clarifications/Assumptions describe the default credential as "injected via Backstage's +proxy configuration." Per R6, the credential *value* is still declared in `app-config.yaml` (i.e. +still "Backstage configuration," not a personal secret or a proxy-secret store), but the mechanism +that applies it to a request is a client-side Swagger UI pre-fill rather than a proxy header +rewrite — the literal "injected via proxy" implementation was found, during this research, to be +incompatible with the override requirement (FR-007). The functional outcome (default present, +sourced from documented lab config, learner can override) is unchanged. From b734874d5a33d1d012cc75b6ab134e300dae48b0 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Sat, 4 Jul 2026 07:05:21 +1000 Subject: [PATCH 15/19] Constitution update to support future scaling --- .specify/memory/constitution.md | 71 +++++++++++++++++++++++---------- 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 5b4f204..07fa8ec 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,31 +1,43 @@ @@ -125,7 +137,7 @@ Every sample API MUST meet all of the following criteria: good API design. Learners leave with a correct Backstage setup but no intuition for what a well-designed API looks like. Using purposeful examples makes both lessons land simultaneously. -### VIII. Support Experimentation +### VIII. Support Experimentation & Scale Labs MUST be designed to support forking, adaptation, and deviation. Documentation MUST NOT assume users will follow every step exactly as written. @@ -133,10 +145,27 @@ Where users may reasonably substitute tools, configurations, or approaches, the acknowledge this and explain the trade-offs rather than presenting one path as the only path. Verification steps MUST test outcomes, not prescribed implementation details. +Labs MUST also choose technical approaches that can scale to real-world implementation sizes +(as a reference point, 500+ APIs) via configuration and resource-sizing changes rather than a +different architecture. A lab MAY defer building or exercising the at-scale behavior itself — +the lab demo may run against a handful of sample APIs — as long as: (a) the chosen mechanism has +no fundamental scaling ceiling that would force a redesign, and (b) the lab's documentation +identifies what changes (config, not code) between lab-demo scale and a large-scale deployment, +and why. Design choices whose sole purpose is to simplify the lab, and which do not generalize to +a larger scale even in principle, MUST be flagged explicitly as a known lab-only simplification +rather than left implicit. + **Rationale**: Users fork this repository to adapt it to their own needs, teams, and -constraints. A lab that only works when followed to the letter fails the majority of its -audience. Outcome-based verification and acknowledged variation points make labs resilient -to the experimentation that real learning requires. +constraints — including scale far beyond the lab's own small demo. A lab that only works when +followed to the letter fails the majority of its audience; a lab whose only path to real-world +scale is a rewrite fails learners evaluating whether the approach will actually hold up in their +organization. Outcome-based verification and acknowledged variation points make labs resilient to +experimentation; requiring the *mechanism* — not necessarily the lab's exercised scenario — to +scale keeps labs honest about what's a genuine design decision versus a shortcut taken purely for +tutorial simplicity. Lab 4's `autoApiRegistration` design (a persisted scan-state cache and +multi-source config built for 1000+ files/1GB+ repos, though the lab itself exercises only two +small sample files) already established this pattern; this principle codifies that precedent for +all labs going forward. ### IX. Pragmatic Security for Learning Environments @@ -238,4 +267,4 @@ All plan `Constitution Check` gates MUST reference the principles by Roman numer - `yarn dev` does not exist. Use `yarn start` instead. -**Version**: 1.5.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-07-04 +**Version**: 1.6.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-07-04 From 339758100b0cb17e89781b08282e2680c114aa09 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Sat, 4 Jul 2026 10:04:26 +1000 Subject: [PATCH 16/19] Replanned and retasked around scalable API mocking solution --- .specify/memory/constitution.md | 14 +- specs/005-lab-5-mocking-testing/data-model.md | 90 +++++--- specs/005-lab-5-mocking-testing/plan.md | 140 +++++++----- specs/005-lab-5-mocking-testing/quickstart.md | 90 +++++--- specs/005-lab-5-mocking-testing/research.md | 162 ++++++++++---- specs/005-lab-5-mocking-testing/tasks.md | 209 ++++++++++++++++++ 6 files changed, 525 insertions(+), 180 deletions(-) create mode 100644 specs/005-lab-5-mocking-testing/tasks.md diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 07fa8ec..0555d43 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -24,10 +24,16 @@ Instance files requiring updates (in-progress/completed feature work): `autoApiRegistration` design (persisted scan-state cache, multi-source config) was already documented as scaling to 1000+ files/1GB+ via config only, ahead of this amendment; this is the precedent the new principle text codifies - ✅ specs/005-lab-5-mocking-testing/{research.md,quickstart.md} — already compliant without - changes: research.md R4 and quickstart.md's "Scaling beyond the lab" section already flag - lab-scale port assignment as a documented simplification needing a persisted port map at - real scale, ahead of this amendment + ⚠→✅ specs/005-lab-5-mocking-testing/{research.md,data-model.md,plan.md,quickstart.md,tasks.md} + — CORRECTION (2026-07-04, post-amendment): this report originally claimed Lab 5 was "already + compliant without changes" at v1.6.0's adoption. That was wrong — its then-current design + spawned one `prism` CLI process per discovered API, which does not scale (500+ APIs → 500+ + permanently-running processes/ports regardless of use; a real ceiling, not a config knob). + Caught on user review immediately after this amendment and reworked to a single in-process, + lazily-loading gateway (research.md R2/R7) before any implementation began. Left here as a + record that the "already compliant" pass at amendment time was not rigorous enough — it read + the existing scaling *prose* as sufficient without checking whether the underlying mechanism + actually had a scaling ceiling ⚠ specs/001-lab-1-base-backstage/plan.md — Constitution Check table missing Principle IX (pre-existing gap, unrelated to this amendment) and does not yet address Principle VIII's new scale requirement (new gap, low priority — Lab 1 is base setup, not a scaling-sensitive diff --git a/specs/005-lab-5-mocking-testing/data-model.md b/specs/005-lab-5-mocking-testing/data-model.md index 30c4de3..98387eb 100644 --- a/specs/005-lab-5-mocking-testing/data-model.md +++ b/specs/005-lab-5-mocking-testing/data-model.md @@ -1,48 +1,63 @@ # Phase 1 Data Model: Lab 5 - Mocking and Testing This lab has no application database entities. "Data model" here means the configuration and -runtime shapes introduced by the new mock-orchestration script and frontend module. +runtime shapes introduced by the single mock gateway process and the frontend module. -## Mock Registry Entry (in-memory, computed at each `start-mocks.mjs` run) +## Discovery Map (in-memory, built once at gateway startup) -Computed once per run from the OpenAPI files matched by Lab 4's `autoApiRegistration` -`rootPath`/`patterns` config; not persisted between runs (recomputed deterministically each start). +Computed once when `scripts/mock-gateway.mjs` starts, from the OpenAPI files matched by Lab 4's +`autoApiRegistration` `rootPath`/`patterns` config (research.md R3). A cheap glob, not a parse — +safe to build eagerly even at 500+ files. Not refreshed while the process is running (a newly added +file requires a gateway restart to become mockable — research.md R7, documented as an intentional, +Lab-4-style poll/watch-mode tradeoff, not silently unsupported). | Field | Type | Description | |---|---|---| -| `specFilePath` | string | Absolute path to the discovered `*-openapi.yaml` file | -| `entityNameSlug` | string | Slugified `info.title`, using the same algorithm as Lab 4's entity-name derivation, so the frontend module can compute a matching proxy path from `entity.metadata.name` alone | -| `port` | number | Assigned deterministically: `4010 + index`, where `index` is the 0-based position in the list of matched files sorted by path | -| `proxyPath` | string | `/api/proxy/mock/` | +| `entityNameSlug` | string | Slugified `info.title`, using the same algorithm as Lab 4's entity-name derivation, so the frontend module can compute a matching gateway path from `entity.metadata.name` alone | +| `specFilePath` | string | Absolute path to the matched `*-openapi.yaml` file | -## Generated Config File: `app-config.mocks.yaml` +## Loaded Spec Cache (in-memory, lazy, LRU-bounded) -Written by `scripts/start-mocks.mjs` before Prism child processes are spawned; not committed to -the repository (generated artifact, added to `.gitignore`). Shape: +Populated on first request for a given `entityNameSlug`; empty at gateway startup. Not persisted — +rebuilt (lazily, again) on every gateway restart. -```yaml -proxy: - endpoints: - /mock/: - target: 'http://localhost:' - changeOrigin: true -``` +| Field | Type | Description | +|---|---|---| +| `entityNameSlug` | string | Cache key | +| `operations` | `IHttpOperation[]` | Result of `@stoplight/http-spec`'s `transformOas3Operations(document)` (research.md R2) — the shape Prism's `.request()` accepts | +| `loadedAt` | timestamp | Set on cache insert; used only for logging/troubleshooting, not eviction | -One entry per Mock Registry Entry. Merged with the committed `app-config.yaml` via -`backstage-cli repo start --config app-config.yaml --config app-config.mocks.yaml`. +**Eviction**: bounded by `mocking.gateway.maxCachedSpecs` (config, default `100`). On a cache miss +that would exceed the bound, the least-recently-used entry is evicted before inserting the new one. +This is the single knob documented as changing between lab-demo scale and a 500+ API deployment +(research.md R7) — no code change required to raise it. -## New committed config: `mocking.defaultCredential` (in `app-config.yaml`) +## New committed config: `mocking` block (in `app-config.yaml`) ```yaml mocking: + gateway: + port: 4010 # single fixed port for the whole gateway (not per-API) + maxCachedSpecs: 100 # LRU bound (research.md R7) defaultCredential: - scheme: 'bearer' # matches the securityScheme type used in this lab's example APIs - value: 'lab-mock-token-do-not-use' # obviously fictional, per Constitution Principle IX + scheme: 'bearer' # matches the securityScheme type used in this lab's example APIs + value: 'lab-mock-token-do-not-use' # obviously fictional, per Constitution Principle IX + +proxy: + endpoints: + /mock: + target: 'http://localhost:4010' # matches mocking.gateway.port — single static entry, + changeOrigin: true # no generated config file (research.md R5) ``` -Read client-side via `configApiRef` by the `apiMocking` frontend module (research.md R6). Not a -secret — deliberately fictional and safe to ship in the frontend bundle, same as any other -documented lab-only example credential. +Read by two consumers: +- `scripts/mock-gateway.mjs` reads `mocking.gateway.*` to know which port to bind and how large a + cache to keep. +- The `apiMocking` frontend module reads `mocking.defaultCredential` client-side via `configApiRef` + (research.md R6) — not a secret, deliberately fictional and safe to ship in the frontend bundle. + +No generated/uncommitted config file is produced by this lab (unlike the superseded per-API-process +design, which generated `app-config.mocks.yaml` at every startup — removed, research.md R4/R5). ## Frontend runtime shape: merged `servers` list @@ -50,11 +65,12 @@ At render time, for any `openapi`-type entity, the `apiMocking` module computes: ```text effectiveServers = entity.spec.definition.servers (native, unmodified) - + [{ url: , description: 'Local mock (Lab 5)' }] + + [{ url: '/api/proxy/mock/' + entityNameSlug(entity.metadata.name), + description: 'Local mock (Lab 5)' }] ``` No persistence; recomputed per render from the entity already loaded by the catalog page. The -proxy path is derived purely from `entityNameSlug(entity.metadata.name)` — no per-entity +gateway path is derived purely from `entityNameSlug(entity.metadata.name)` — no per-entity configuration is required for a new API to get a mock server entry, satisfying the "generic by default" requirement from this session's clarification. @@ -62,17 +78,23 @@ default" requirement from this session's clarification. ```text OpenAPI spec file (*-openapi.yaml) - │ (discovered via Lab 4's autoApiRegistration config) + │ (discovered via Lab 4's autoApiRegistration config, once, at gateway startup) ▼ -Mock Registry Entry ──► Prism child process (serves on `port`) +Discovery Map entry (entityNameSlug → specFilePath) │ + │ (first request for this slug only) ▼ -app-config.mocks.yaml proxy endpoint ──► merged into running Backstage config +Loaded Spec Cache entry (transformOas3Operations, LRU-bounded) │ ▼ +Prism instance.request(...) ──► mocked HTTP response + ▲ + │ (every request, single static proxy hop) +Backstage proxy `/api/proxy/mock/` ──► gateway port (mocking.gateway.port) + ▲ + │ Catalog API entity's rendered `servers` list (apiMocking frontend module) + ▲ │ - ▼ -Learner selects target in Swagger UI "Try it out" ──► request routed through -Backstage's existing proxy (mock) or directly to the sandbox host (Galaxy's native servers) +Learner selects target in Swagger UI "Try it out" ``` diff --git a/specs/005-lab-5-mocking-testing/plan.md b/specs/005-lab-5-mocking-testing/plan.md index f25de19..2557175 100644 --- a/specs/005-lab-5-mocking-testing/plan.md +++ b/specs/005-lab-5-mocking-testing/plan.md @@ -8,48 +8,65 @@ Lab 5 builds on the running Backstage 1.51.0 instance from Labs 1–4. It adds two capabilities to every registered OpenAPI API's page: a locally running, dynamically generated mock implementation -(via Prism, one instance per discovered `*-openapi.yaml` file, reusing Lab 4's discovery config as -the single source of truth — FR-001, FR-005) and a documented public sandbox target for APIs that -already have one natively (the Galaxy API's `galaxy.scalar.com`/`void.scalar.com` servers — FR-003). -Both are selectable from the API's existing "Try it out" viewer without editing any committed spec -file's `servers` array — instead, a new frontend module (`packages/app/src/modules/apiMocking/`) -overrides the `apiDocsConfigRef` API, the same wrapper-module pattern Lab 3 used for -`apiGrade`/`spectralLinter`, since Backstage's default OpenAPI viewer has no built-in -config-level toggle for extra servers or credential injection (research.md R1). The mock server(s) -start automatically alongside `yarn start` via a new orchestrator script and `concurrently`, with -`wait-on` sequencing to avoid a config-file race (FR-001, research.md R4). Requests to a mock are -routed through Backstage's existing proxy mechanism (a generated `app-config.mocks.yaml`, one -endpoint per discovered API) to avoid CORS; Galaxy's sandbox is called directly since its native -`servers` entries already work (FR-004, research.md R5). A documented, fictional default -credential lives in `app-config.yaml` and is applied by pre-authorizing Swagger UI's own -"Authorize" dialog on mount — not a proxy header rewrite, since a static header can't yield to a -learner's own credential — giving "default that works out of the box, freely overridable" for -free from the same UI a learner would use anyway (FR-006, FR-007, research.md R6). Errors (mock -not running, sandbox unreachable, bad credential) surface through the viewer's own request/response -handling, not a bespoke error system (FR-008). A `## Security Note` section (Constitution -Principle IX / Lab Structure Standards) documents the shortcut and the production alternative -before the learner encounters the default credential. +and a documented public sandbox target for APIs that already have one natively (the Galaxy API's +`galaxy.scalar.com`/`void.scalar.com` servers — FR-003). Mocking is served by a **single, in-process +gateway** (`scripts/mock-gateway.mjs`) rather than one process per API: it reuses Lab 4's discovery +config to build a cheap `slug → spec file` map at startup (FR-001, FR-005, research.md R3), then +lazily parses and caches (LRU-bounded) each API's operations — via Stoplight Prism's programmatic +library (`@stoplight/prism-http` + `@stoplight/http-spec`, not the `prism` CLI) — only on that API's +first request (research.md R2). This design was revised mid-plan after review against Constitution +Principle VIII's amended scale requirement: an earlier one-process-per-API design would have meant +500+ permanently-running processes/ports for a 500-API deployment regardless of use, which fails the +"mechanism doesn't need to change at scale, only sizing" bar; the single lazy-loading gateway's only +scale knob is a cache-size config value (`mocking.gateway.maxCachedSpecs`, research.md R7). Both mock +and sandbox targets are selectable from the API's existing "Try it out" viewer without editing any +committed spec file's `servers` array — instead, a new frontend module +(`packages/app/src/modules/apiMocking/`) overrides the `apiDocsConfigRef` API, the same +wrapper-module pattern Lab 3 used for `apiGrade`/`spectralLinter`, since Backstage's default OpenAPI +viewer has no built-in config-level toggle for extra servers or credential injection (research.md +R1). The gateway starts automatically alongside `yarn start` via `concurrently` (FR-001, research.md +R4). Requests to the mock are routed through a single, static Backstage proxy endpoint (committed +directly in `app-config.yaml`, no generated config file) to avoid CORS; Galaxy's sandbox is called +directly since its native `servers` entries already work (FR-004, research.md R5). A documented, +fictional default credential lives in `app-config.yaml` and is applied by pre-authorizing Swagger +UI's own "Authorize" dialog on mount — not a proxy header rewrite, since a static header can't yield +to a learner's own credential — giving "default that works out of the box, freely overridable" for +free from the same UI a learner would use anyway (FR-006, FR-007, research.md R6). Errors (mock not +running, sandbox unreachable, bad credential) surface through the viewer's own request/response +handling, not a bespoke error system (FR-008). A `## Security Note` section (Constitution Principle +IX / Lab Structure Standards) documents the shortcut and the production alternative before the +learner encounters the default credential. ## Technical Context **Language/Version**: TypeScript / Node.js 20 LTS (Backstage 1.51.0, pinned from Lab 1) **Primary Dependencies**: -- `@stoplight/prism-cli@5.15.11` (Apache-2.0) — new root devDependency; provides the `prism mock` - command used to dynamically serve each discovered OpenAPI file (research.md R2). -- `concurrently` (new root devDependency) — runs the mock orchestrator alongside - `backstage-cli repo start` (research.md R4); not previously present anywhere in the workspace. -- `wait-on` (new root devDependency) — sequences Backstage's startup to wait for the generated - `app-config.mocks.yaml` to exist, avoiding a config-read race (research.md R4). +- `@stoplight/prism-http@5.15.11` (Apache-2.0) — new root devDependency; programmatic mock-response + library (`createInstance(...).request(...)`), used directly by the gateway instead of the `prism` + CLI, so a single process can serve every API without binding one port per API (research.md R2). +- `@stoplight/http-spec@7.1.0` (Apache-2.0) — new root devDependency; `transformOas3Operations()` + (subpath `@stoplight/http-spec/oas3`) converts a parsed OpenAPI document into the operations shape + `prism-http` consumes (research.md R2). +- `js-yaml` (new root devDependency) — parses each spec file's YAML before conversion (same package + Lab 4 already uses in `packages/backend`, added here as a separate root devDependency since this + script runs outside that package). +- `concurrently` (new root devDependency) — runs the mock gateway alongside `backstage-cli repo + start` (research.md R4); not previously present anywhere in the workspace. `wait-on` is + deliberately **not** added — the revised design has no generated config file to wait for + (research.md R4/R5). - `@backstage/plugin-api-docs` (already installed, `^0.14.1`) — supplies `apiDocsConfigRef` and the default `OpenApiDefinitionWidget`/`swagger-ui-react` rendering this lab wraps, not replaces (research.md R1). - No new backend Backstage module — unlike Lab 4, this lab needs no `EntityProvider`/catalog changes; it only reads Lab 4's existing discovery config. -**Storage**: One new generated (not committed) config file, `app-config.mocks.yaml` (proxy -endpoints, one per discovered OpenAPI file — data-model.md), plus one new committed config key, -`mocking.defaultCredential`, in `app-config.yaml`. No database changes. +**Storage**: No generated/uncommitted config file (the superseded per-API-process design generated +`app-config.mocks.yaml`; the revised single-gateway design needs only static, committed config — data +-model.md). Two new committed config blocks in `app-config.yaml`: `mocking.gateway` +(`port`, `maxCachedSpecs`) and `mocking.defaultCredential`, plus one static `proxy.endpoints['/mock']` +entry. All in-memory state (discovery map, LRU spec cache) is rebuilt on every gateway restart — no +database changes. **Testing**: Manual browser + orchestrator-log verification per lab README (no automated test suite — consistent with Labs 1–4's tutorial-lab testing approach). @@ -60,20 +77,24 @@ suite — consistent with Labs 1–4's tutorial-lab testing approach). new TypeScript frontend module (`apiDocsConfigRef` override) + config additions. **Performance Goals**: A learner can select the mock target and receive a response within the same -interaction as any other "Try it out" request (SC-001); mock startup adds at most a few seconds to -`yarn start` (Prism boot time per instance), not exercised against real-world repo scale — port -assignment (research.md R4, data-model.md) is a lab-scale convenience, documented as needing a -persisted port map at real mono-repo scale (quickstart.md). +interaction as any other "Try it out" request (SC-001); the gateway itself boots near-instantly +(only a cheap file-glob at startup, no per-API parsing — parsing is deferred to first request per +API, research.md R2). At real mono-repo scale (500+ APIs), process/port count remains exactly 1 +regardless of catalog size (research.md R7) — the only scale-sensitive knob is +`mocking.gateway.maxCachedSpecs`, documented as needing to be raised (config only) for a +deployment where many distinct APIs are actively exercised concurrently (quickstart.md). **Constraints**: Zero cost; cross-platform; no external network access beyond the optional Galaxy sandbox calls (User Story 2), which remain optional relative to the fully local mock path (FR-011); builds on Labs 1–4 without editing any previously committed spec file's native `servers` array (this session's clarification) or disrupting existing registered APIs. -**Scale/Scope**: Lab demo: 1 new orchestrator script, 1 new frontend module (~2-3 files), 2 -new/modified config keys (`mocking.defaultCredential`, root `start` script), 3 new root -devDependencies, 0 new sample API files (reuses Museum + Galaxy from Labs 1 & 4), 0 changes to any -existing committed API spec file. +**Scale/Scope**: Lab demo: 1 new gateway script, 1 new frontend module (~2-3 files), 3 +new/modified config blocks (`mocking.gateway`, `mocking.defaultCredential`, one static `proxy` +entry, plus the root `start` script), 4 new root devDependencies, 0 new sample API files (reuses +Museum + Galaxy from Labs 1 & 4), 0 changes to any existing committed API spec file. Designed-for +scale (config change only, no code/architecture change, per research.md R7): 500+ APIs, O(1) +processes/ports, memory bounded by `maxCachedSpecs` rather than catalog size. ## Constitution Check @@ -81,14 +102,14 @@ existing committed API spec file. | Principle | Gate | Status | |-----------|------|--------| -| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: `apiDocsConfigRef` customization for target-server injection and credential pre-fill, dynamic OpenAPI mocking via Prism, and Backstage proxy config for CORS-safe frontend calls | ✅ Pass | -| II. Process-Oriented Documentation | README must explain WHY `apiDocsConfigRef` override was necessary (no built-in toggle, research.md R1), WHY a static proxy header was rejected in favor of an Authorize-dialog pre-fill (research.md R6), and WHY Galaxy needs no additional proxy config while Museum's mock does (research.md R5). The orchestrator script and frontend module are each likely >50 lines and MUST be committed under `labs/lab-05-mocking-testing/code/` and linked from the README, not inlined | ✅ Pass (with file-size rule flagged for tasks) | -| III. Cross-Platform Compatibility | Orchestrator script, `concurrently`/`wait-on`/`prism` are pure JS, no native/platform-specific dependencies; `yarn start` continues to work identically on Windows and macOS; README documents any PowerShell vs. bash quoting differences for the modified `start` script string | ✅ Pass | -| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–4's existing Node/Yarn setup — all three new dependencies install via the existing `yarn add` workflow; README documents the install step | ✅ Pass | -| V. Zero-Cost Operation | `@stoplight/prism-cli` is Apache-2.0 (free/open-source, satisfies Principle V's "freely available under open-source or freeware terms" though not MIT specifically); `concurrently`/`wait-on` are MIT; Galaxy's public sandbox is free; no paid service, and the fully local mock path (User Story 1) has zero dependency on external network availability (FR-011) | ✅ Pass | +| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: `apiDocsConfigRef` customization for target-server injection and credential pre-fill, dynamic OpenAPI mocking via Prism's programmatic library, and Backstage proxy config for CORS-safe frontend calls | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY `apiDocsConfigRef` override was necessary (no built-in toggle, research.md R1), WHY a single lazy-loading gateway replaced an initial one-process-per-API design after Principle VIII review (research.md R2/R7 — a concrete worked example of the principle, worth calling out explicitly to learners), WHY a static proxy header was rejected in favor of an Authorize-dialog pre-fill (research.md R6), and WHY Galaxy needs no additional proxy config while Museum's mock does (research.md R5). The gateway script and frontend module are each likely >50 lines and MUST be committed under `labs/lab-05-mocking-testing/code/` and linked from the README, not inlined | ✅ Pass (with file-size rule flagged for tasks) | +| III. Cross-Platform Compatibility | Gateway script, `concurrently`, `@stoplight/prism-http`/`@stoplight/http-spec`/`js-yaml` are pure JS, no native/platform-specific dependencies; `yarn start` continues to work identically on Windows and macOS; README documents any PowerShell vs. bash quoting differences for the modified `start` script string | ✅ Pass | +| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–4's existing Node/Yarn setup — all four new dependencies install via the existing `yarn add` workflow; README documents the install step | ✅ Pass | +| V. Zero-Cost Operation | `@stoplight/prism-http`/`@stoplight/http-spec` are Apache-2.0 (free/open-source, satisfies Principle V's "freely available under open-source or freeware terms" though not MIT specifically); `js-yaml`/`concurrently` are MIT; Galaxy's public sandbox is free; no paid service, and the fully local mock path (User Story 1) has zero dependency on external network availability (FR-011) | ✅ Pass | | VI. Progressive Lab Structure | Requires Labs 1–4 completion; reuses Lab 4's discovery config and Lab 1/4's Museum/Galaxy specs unmodified; no prior lab's committed spec file is edited | ✅ Pass | | VII. Modern & Purposeful API Examples | No new sample API is introduced — Museum and Galaxy, both already compliant per their own labs' plans, are reused as-is | ✅ Pass (N/A, no new example) | -| VIII. Support Experimentation | README documents adaptable conventions: mock port base (`4010`), reliance on Lab 4's discovery glob, the default credential value, and the proxy endpoint naming scheme; verification tests outcomes (a response comes back, the credential used is the one displayed), not exact port numbers | ✅ Pass | +| VIII. Support Experimentation & Scale | README documents adaptable conventions: gateway port, cache size, reliance on Lab 4's discovery glob, the default credential value; verification tests outcomes (a response comes back, the credential used is the one displayed), not implementation internals. **Scale** (this row was corrected mid-plan — see research.md R2/R7): the mechanism is a single gateway process handling every discovered API via lazy load + a bounded LRU cache, so process/port count is O(1) regardless of catalog size and memory is bounded by `mocking.gateway.maxCachedSpecs`, not by total API count — the *only* thing that changes between the lab's 3–4-API demo and a 500+-API deployment is that config value. The one flagged lab-only simplification (discovery is boot-time-only, not live-watched) is explicitly documented, mirroring Lab 4's poll/watch framing, per the principle's requirement to call out non-generalizing simplifications rather than leave them implicit | ✅ Pass | | IX. Pragmatic Security for Learning Environments | `mocking.defaultCredential` is a fictional, clearly-labelled example value stored in committed `app-config.yaml`; a `## Security Note` section (before the learner encounters the credential config step) explains the shortcut and the production alternative (secrets manager / vault-injected config, per-user credentials, never committing real credentials) | ✅ Pass (new obligation: Security Note section required, first for this lab series to introduce a credential) | No violations. Complexity Tracking section omitted. @@ -106,9 +127,8 @@ specs/005-lab-5-mocking-testing/ └── tasks.md # Phase 2 output (/speckit-tasks — NOT created here) ``` -No `contracts/` directory — this lab produces no external API contracts of its own. The generated -`app-config.mocks.yaml` shape and the `mocking.defaultCredential` config key are documented in -data-model.md instead. +No `contracts/` directory — this lab produces no external API contracts of its own. The static +`mocking`/`proxy` config shapes are documented in data-model.md instead. ### Source Code (repository root) @@ -120,10 +140,11 @@ labs/ │ # adaptable conventions per US-adaptation goal) └── code/ ├── scripts/ - │ └── start-mocks.mjs # New: mock orchestrator (discovery, port assignment, - │ # app-config.mocks.yaml generation, Prism child - │ # process supervision) — committed here, linked - │ # from README per Constitution Principle II + │ └── mock-gateway.mjs # New: single in-process mock gateway (discovery map, + │ # lazy load + LRU-bounded spec cache, Prism + │ # programmatic request handling) — committed here, + │ # linked from README per Constitution Principle II + │ # (research.md R2, R7) └── packages/ └── app/ └── src/ @@ -139,13 +160,14 @@ but the module/script source itself IS committed since it is lab teaching conten ```text # In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): -package.json # Modified: add @stoplight/prism-cli, concurrently, - # wait-on; change "start" script (research.md R4) -app-config.yaml # Modified: add mocking.defaultCredential block -.gitignore # Modified: add app-config.mocks.yaml (generated, - # not committed) +package.json # Modified: add @stoplight/prism-http, + # @stoplight/http-spec, js-yaml, concurrently; + # change "start" script (research.md R4) +app-config.yaml # Modified: add mocking.gateway, + # mocking.defaultCredential, and the static + # proxy.endpoints['/mock'] entry (data-model.md) scripts/ -└── start-mocks.mjs # New: copied from labs/lab-05-.../code/scripts/ +└── mock-gateway.mjs # New: copied from labs/lab-05-.../code/scripts/ packages/app/ └── src/ ├── apis.ts # Modified: register the apiMocking module @@ -155,8 +177,10 @@ packages/app/ # app/src/modules/apiMocking/ ``` +No `.gitignore` change is needed — the revised design generates no runtime config file (data-model.md). + **Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–4. All -student-facing content lives under `labs/lab-05-mocking-testing/`; the orchestrator script and +student-facing content lives under `labs/lab-05-mocking-testing/`; the gateway script and frontend module are committed under its `code/` subdirectory (both exceed the ~40-50 line inline threshold per Constitution Principle II) and linked from the README rather than embedded inline. The student's Backstage instance (from Lab 1) is modified in place per the README instructions, diff --git a/specs/005-lab-5-mocking-testing/quickstart.md b/specs/005-lab-5-mocking-testing/quickstart.md index bb1a1af..24ccec0 100644 --- a/specs/005-lab-5-mocking-testing/quickstart.md +++ b/specs/005-lab-5-mocking-testing/quickstart.md @@ -14,35 +14,42 @@ Constitution Principle II. ## Steps -1. **Install dependencies**: add `@stoplight/prism-cli`, `concurrently`, and `wait-on` as root - devDependencies of the Lab 1 Backstage workspace. -2. **Add the mock orchestrator script** - (`scripts/start-mocks.mjs`): discovers OpenAPI files via Lab 4's existing - `autoApiRegistration.rootPath`/`patterns` config, assigns each a deterministic port from - `4010`, writes `app-config.mocks.yaml` with one proxy endpoint per file, then spawns one - `prism mock -p ` child process per file (research.md R2–R5, data-model.md). -3. **Update the root `start` script** to run the orchestrator and Backstage concurrently, with - `wait-on` sequencing so Backstage only boots once `app-config.mocks.yaml` exists - (research.md R4). -4. **Add `mocking.defaultCredential`** to `app-config.yaml` (data-model.md) — a documented, - obviously fictional bearer token — and add `app-config.mocks.yaml` to `.gitignore` (it's a - generated artifact, not committed). +1. **Install dependencies**: add `@stoplight/prism-http`, `@stoplight/http-spec`, `js-yaml`, and + `concurrently` as root devDependencies of the Lab 1 Backstage workspace. +2. **Add the mock gateway script** + (`scripts/mock-gateway.mjs`): a single long-lived Node HTTP server, listening on + `mocking.gateway.port` (default `4010`), that (a) at startup builds a cheap + `entityNameSlug → spec file path` map from Lab 4's existing `autoApiRegistration` + `rootPath`/`patterns` config, and (b) on each request to `//*`, lazily loads + parses + + converts that API's spec (`js-yaml` + `@stoplight/http-spec`'s `transformOas3Operations`) on + first use only, caches the result in a `mocking.gateway.maxCachedSpecs`-bounded LRU map, and + serves a mocked response via `@stoplight/prism-http`'s `createInstance(...).request(...)` + (research.md R2, R3, R7; data-model.md). +3. **Add static config** to `app-config.yaml`: `mocking.gateway` (`port`, `maxCachedSpecs`), + `mocking.defaultCredential`, and one static `proxy.endpoints['/mock']` entry pointing at the + gateway's port (data-model.md) — no generated/uncommitted config file is produced by this lab. +4. **Update the root `start` script** to run the gateway and Backstage concurrently via + `concurrently` (research.md R4) — no sequencing dependency needed, since the gateway's proxy + target is a static config value valid from the moment Backstage starts. 5. **Add the `apiMocking` frontend module** (`packages/app/src/modules/apiMocking/index.tsx`): overrides `apiDocsConfigRef` for the - `openapi` widget type to (a) merge a `Local mock (Lab 5)` server entry — computed from - `entity.metadata.name`, no per-entity config needed — into the rendered `servers` list, and - (b) pre-authorize Swagger UI's "Authorize" dialog with `mocking.defaultCredential` on mount - (research.md R1, R6). Registered in `packages/app/src/apis.ts` (or equivalent frontend module - registration point). + `openapi` widget type to (a) merge a `Local mock (Lab 5)` server entry — URL + `/api/proxy/mock/`, computed from `entity.metadata.name`, no per-entity config + needed — into the rendered `servers` list, and (b) pre-authorize Swagger UI's "Authorize" + dialog with `mocking.defaultCredential` on mount (research.md R1, R6). Registered in + `packages/app/src/apis.ts` (or equivalent frontend module registration point). 6. **Start Backstage** (`yarn start`) and confirm: - - The mock orchestrator log shows a Prism instance started for the Museum API (and the other - discovered OpenAPI files) on its assigned port. + - The gateway logs that it has built its discovery map (e.g. "N APIs discoverable") without + parsing any of them yet. - Opening the Museum API's page in Backstage shows a `Local mock (Lab 5)` entry in the server dropdown alongside its native (non-resolvable) production URL. -7. **User Story 1 — mock**: select the `Local mock (Lab 5)` server for the Museum API, execute a - sample "Try it out" request, and confirm a realistic example response comes back (US1, SC-001). - Stop the mock orchestrator process, retry the same request, and confirm a clear connection - error is shown (not a silent failure). +7. **User Story 1 — mock, including lazy-load behavior**: select the `Local mock (Lab 5)` server + for the Museum API, execute a sample "Try it out" request, and confirm a realistic example + response comes back (US1, SC-001). Check the gateway log: the *first* request for Museum should + log a "loading spec" line (and its measured parse latency — confirming research.md R2's + reasoned-but-unverified latency claim in practice); subsequent requests to the same API should + not re-log a load, confirming the cache is working. Stop the gateway process, retry the same + request, and confirm a clear connection error is shown (not a silent failure). 8. **User Story 2 — sandbox**: open the Galaxy API's page, confirm its native `servers` dropdown already offers `galaxy.scalar.com` / `void.scalar.com` with no additional Lab 5 configuration, select one, execute a request, and confirm the response comes from the real sandbox (US2, @@ -50,13 +57,15 @@ Constitution Principle II. 9. **User Story 3 — credentials**: on an authenticated request (mock or sandbox), confirm the "Authorize" dialog is already pre-filled with the `mocking.defaultCredential` value and the request succeeds with zero setup (SC-003). Replace it with a different value in the same - dialog, confirm the replacement is what's actually sent (SC-004), and confirm - `app-config.mocks.yaml`/no other file ever contains a personal credential. + dialog, confirm the replacement is what's actually sent (SC-004), and confirm no committed file + ever contains a personal credential. 10. **Verify a second, unconfigured API gets a mock automatically**: temporarily add a third - `*-openapi.yaml` file matching Lab 4's discovery pattern, restart `yarn start`, and confirm a - `Local mock (Lab 5)` entry appears for it too with zero Lab-5-specific configuration — proving - the mechanism is generic, not hardcoded to Museum (this session's clarification). Remove the - temporary file afterward. + `*-openapi.yaml` file matching Lab 4's discovery pattern, restart `yarn start` (the gateway's + discovery map is boot-time-only, so a restart is expected/documented — research.md R7), and + confirm a `Local mock (Lab 5)` entry appears for it too with zero Lab-5-specific + configuration, and that its spec is only parsed on its own first request (not eagerly at + gateway startup) — proving the mechanism is both generic (this session's clarification) and + lazy (research.md R2/R7). Remove the temporary file afterward. ## Out of scope for this quickstart @@ -65,11 +74,22 @@ Constitution Principle II. - Exercising a genuinely unreachable/CORS-blocked sandbox end-to-end (the lab's own Galaxy example is reachable and CORS-permissive); the resulting error-handling behavior (FR-008) is documented, not walked through against a real failure. +- Exercising actual LRU eviction under load (the lab's 3–4 sample APIs never approach the default + `maxCachedSpecs: 100` bound); the eviction *mechanism* is unit-testable in principle but not + demonstrated live in this lab. ## Scaling beyond the lab (documented, not walked through) -The README includes a note on why default-port assignment (`4010 + index`, sorted by path) is -stable enough for a handful of lab APIs but would need an explicit, persisted port map for a large -mono-repo where files are added/removed frequently (to avoid every learner's mock ports shifting -on every restart) — mirroring the same "lab-scale convenience vs. real-scale mechanism" framing -Lab 4 used for its scan-state cache. +The README includes a note on why this lab's mocking mechanism was deliberately redesigned, during +planning, away from an initial "one process per API" approach once checked against Constitution +Principle VIII's scale requirement — using this as a concrete, worked example for learners of what +"design for scale, defer the exercise of it" means in practice: +- Why process/port count is fixed at 1 regardless of catalog size (a single gateway, not one per + API), and why that specific property — not just "it's configurable" — is what makes the design + actually satisfy the principle (research.md R2, R7). +- Why `mocking.gateway.maxCachedSpecs` is the one value that needs to grow for a real, heavily-used + 500+ API deployment, and why raising it is cheap (bounded memory per cached spec) compared to the + superseded design's fixed per-API process/port cost. +- Why the gateway's discovery map is boot-time-only (a documented, Lab-4-style poll/watch + tradeoff) rather than live-refreshed, and what a `watch`-based refresh mode would look like if a + learner's fork needs it (mirroring Lab 4's own `mode: 'poll'` vs. `mode: 'watch'` framing). diff --git a/specs/005-lab-5-mocking-testing/research.md b/specs/005-lab-5-mocking-testing/research.md index 8aff52e..92f040b 100644 --- a/specs/005-lab-5-mocking-testing/research.md +++ b/specs/005-lab-5-mocking-testing/research.md @@ -29,29 +29,61 @@ architectural pattern to the series. - Replacing `plugin-api-docs` entirely with a bespoke viewer — rejected: massive scope increase, would also need to reimplement AsyncAPI rendering used by Streetlights, unrelated to this lab's goal. -## R2: Mock server tooling - -**Decision**: `@stoplight/prism-cli` (currently `5.15.11`, Apache-2.0), added as a root -`devDependency` and invoked as a child process (`prism mock -p `) per discovered -OpenAPI file. Prism generates responses dynamically from a spec's own examples/schemas — matching -GOAL.md's "dynamically create... a mock implementation" requirement — and works against the -Museum API's self-contained spec (no external `$ref`s, per Constitution Principle VII) with no -further setup. - -**Rationale**: Free, actively maintained, npm-installable (no JVM/Docker prerequisite), and purpose --built for OpenAPI mocking. +## R2: Mock serving mechanism — single in-process gateway, not one process per API + +**Decision (revised)**: A single, long-lived Node process (`scripts/mock-gateway.mjs`) serves mocks +for *every* discovered OpenAPI file on one fixed port, using Stoplight Prism's programmatic library +API directly — not the `prism` CLI, and not one child process per API. On each incoming request to +`//*`: +1. Look up `entityNameSlug` in an in-memory `slug → specFilePath` map built once at gateway startup + from Lab 4's existing `autoApiRegistration` discovery config (research.md R3) — cheap (a glob), + not a parse, so this is fine to do eagerly even at 500+ files. +2. If this slug's spec has not been loaded yet, read + `js-yaml`-parse the file and convert it to + Prism's operation format via `@stoplight/http-spec`'s `transformOas3Operations(document)` + (`@stoplight/http-spec@7.1.0`, subpath `@stoplight/http-spec/oas3`) — synchronous, in-memory, + requires no external `$ref` bundling step for this lab's self-contained specs (Constitution + Principle VII). Cache the resulting `IHttpOperation[]` in a bounded, LRU-evicted `Map`. +3. Call `.request()` on a single, gateway-lifetime `Prism.createInstance({ mock: { dynamic: false } })` + instance (`@stoplight/prism-http@5.15.11`) with the translated request and the (now-cached) + operations array, translate the returned mock response back to a real HTTP response. + +Confirmed via direct inspection of `@stoplight/prism-http`'s README and `npm pack`/`npm view` on +both packages (not guessed): `createInstance(...).request(requestDescriptor, operations)` returns a +mocked response without binding any HTTP server/port of its own — it's a plain function call. No +`@stoplight/prism-cli` or `@stoplight/prism-http-server` dependency is needed; `@stoplight/prism-core` +is pulled in automatically as a dependency of `prism-http`. + +**Rationale**: The original per-file `prism mock -p ` CLI-child-process design (superseded +below) does not satisfy Constitution Principle VIII's amended scale requirement. At 500+ APIs it means +500+ permanently-running Node/Prism processes and 500+ held ports, regardless of whether any given +mock is ever actually requested — this is not "config that changes at scale," it's a mechanism with a +real, low ceiling (OS process limits, port exhaustion, tens of GB of idle memory). A single gateway +process with **lazy** loading means cost is paid only for APIs a learner (or, at real scale, an +engineer) actually exercises, and the **only** thing that needs to change between a 3-API lab demo and +a 500-API deployment is a cache-size number (R7) — exactly what Principle VIII asks for. **Alternatives considered**: -- WireMock — rejected: requires a JVM prerequisite not otherwise needed by any prior lab, adding - friction (Constitution Principle IV). -- Hand-rolled static-JSON fixture server — rejected: loses "dynamic" generation from the spec's own - examples/schemas; would drift from the spec instead of reflecting it. +- One `prism mock` CLI process per file (original decision, this session) — rejected on re-review: + linear process/port growth with no bound, fails the scale gate even though it "just works" via + more config entries; flagged as the mistake this revision corrects. +- A bounded *pool* of live CLI child processes (spawn on demand, kill least-recently-used) — rejected + as unnecessarily complex compared to the in-process library call once R1/R2's programmatic API was + confirmed available; process spawn/teardown overhead is strictly worse than an in-memory cache hit. +- Hand-rolled static-JSON fixture server — still rejected for the same reason as before: loses + "dynamic" generation from the spec's own examples/schemas. + +**Open item flagged, not blocking**: first-request parse latency for the `transformOas3Operations` +call is reasoned to be low (tens of ms for a self-contained, few-dozen-path spec — a synchronous, +in-memory tree walk, no I/O beyond the initial file read) but has not been independently measured. +Tasks.md includes a step to log actual first-load latency during implementation rather than asserting +this figure as verified fact. ## R3: Which files to mock — discovery convention **Decision**: Reuse Lab 4's existing `autoApiRegistration.rootPath`/`patterns` config (`*-openapi.yaml`) as the single source of truth for which OpenAPI files exist, rather than -introducing a second glob configuration surface for mocking. +introducing a second glob configuration surface for mocking. The gateway (R2) builds its +`slug → specFilePath` map from this config once at startup. **Rationale**: Avoids config drift between "what's registered in the catalog" and "what's mockable"; directly satisfies this session's clarification that mocking must be generic/available by default to @@ -60,44 +92,58 @@ any registered OpenAPI API, not hardcoded to one example. **Alternatives considered**: A separate `mocking.include` glob — rejected as duplicate, driftable configuration for the same underlying file set. -## R4: Running mock server(s) alongside `yarn start` +## R4: Running the gateway alongside `yarn start` -**Decision**: Add `concurrently` and `wait-on` as new root devDependencies and a new -`scripts/start-mocks.mjs` orchestrator script. The script synchronously discovers OpenAPI files, -assigns each a deterministic port (sequential from `4010`, sorted by file path), and writes a -generated `app-config.mocks.yaml` containing one proxy endpoint per file — all before spawning the -long-lived Prism child processes. The root `start` script becomes: +**Decision (revised)**: Add `concurrently` as a root devDependency and run +`scripts/mock-gateway.mjs` alongside `backstage-cli repo start`: ```json -"start": "concurrently -k -n app,mocks \"wait-on ./app-config.mocks.yaml && backstage-cli repo start --config app-config.yaml --config app-config.mocks.yaml\" \"node scripts/start-mocks.mjs\"" +"start": "concurrently -k -n app,mocks \"backstage-cli repo start\" \"node scripts/mock-gateway.mjs\"" ``` -`wait-on` guarantees the generated config file exists before `backstage-cli repo start` reads it, -avoiding a startup race between the two concurrent processes. +`wait-on` is **no longer needed** and has been dropped from the design. The original design used it +to sequence Backstage's startup against a *generated* proxy config file that had to exist first +(`app-config.mocks.yaml`); the revised design (R2, R5) needs no generated config at all — the gateway's +single port and single proxy endpoint are static, committed values in `app-config.yaml` that are valid +whether or not the gateway process has finished booting yet. The gateway only needs to be listening by +the time a learner actually clicks "Try it out," which is well after both processes have started. **Rationale**: `backstage-cli repo start` already owns frontend/backend orchestration internally; -adding a third independent process via `concurrently` (not currently a dependency anywhere in the -workspace) is the smallest change that achieves "mock starts automatically with `yarn start`," per -this session's clarification, without forking Backstage's own CLI. +`concurrently` is the smallest addition that achieves "mock starts automatically with `yarn start`." +Removing `wait-on` and the generated-config race it existed to solve is a direct, welcome consequence +of moving to a single static gateway rather than N generated proxy entries. **Alternatives considered**: Implementing mocking as a Backstage backend plugin/module (the pattern -Lab 4 used for catalog discovery) — rejected: heavier (new backend module + migration), and a poor -fit for supervising long-lived external child processes, which a plain Node script does more simply. +Lab 4 used for catalog discovery) — rejected: heavier (new backend module + registration), and this +lab's plan already avoids introducing a second backend module; a plain Node process is simpler to +explain and — usefully — works standalone outside Backstage too (a learner could curl the gateway +directly), which a backend-embedded route would not. ## R5: Proxy routing and CORS -**Decision**: Every discovered OpenAPI file gets a generated proxy endpoint -(`/api/proxy/mock/`, using the same slugification as Lab 4's entity-name -derivation) targeting its local Prism instance, satisfying FR-004 (avoid browser CORS restrictions -via Backstage's existing proxy mechanism). The Galaxy API's pre-existing native `servers` entries -(`galaxy.scalar.com`, `void.scalar.com`) are called directly by the browser, without an additional -proxy hop — Scalar's public example services are designed for direct external/browser calls. +**Decision (revised)**: A single, static proxy endpoint, committed directly in `app-config.yaml`: + +```yaml +proxy: + endpoints: + /mock: + target: 'http://localhost:${mocking.gateway.port}' + changeOrigin: true +``` + +(interpolated from the same `mocking.gateway.port` config key the gateway itself reads — single +source of truth, no drift). The frontend module computes each entity's mock URL as +`/api/proxy/mock/` — one static prefix plus the same slug used throughout. The Galaxy +API's pre-existing native `servers` entries (`galaxy.scalar.com`, `void.scalar.com`) continue to be +called directly by the browser, without a proxy hop — Scalar's public example services are designed +for direct external/browser calls. -**Rationale**: Matches FR-003's "native list MAY continue to satisfy target selection without -additional configuration" and avoids proxying traffic that doesn't need it. +**Rationale**: With one gateway instead of N mock processes, there is exactly one proxy target to +declare — no generated config, no per-API entries, no runtime file-write step at all. This is a +straightforward simplification that falls out of R2's redesign. -**Alternatives considered**: Proxying all sandbox traffic uniformly — rejected as unnecessary -complexity when the native `servers` mechanism already works for Galaxy. +**Alternatives considered**: Per-API generated proxy entries (original decision, superseded) — +rejected along with the per-process design it existed to support. ## R6: Default non-production credential delivery @@ -119,12 +165,30 @@ mechanism for free. Requiring learners to always enter a credential manually — rejected, fails FR-006 ("succeeds by default without any learner setup"). -## Deviation from the specify-session Clarifications wording - -The spec's Clarifications/Assumptions describe the default credential as "injected via Backstage's -proxy configuration." Per R6, the credential *value* is still declared in `app-config.yaml` (i.e. -still "Backstage configuration," not a personal secret or a proxy-secret store), but the mechanism -that applies it to a request is a client-side Swagger UI pre-fill rather than a proxy header -rewrite — the literal "injected via proxy" implementation was found, during this research, to be -incompatible with the override requirement (FR-007). The functional outcome (default present, -sourced from documented lab config, learner can override) is unchanged. +## R7: Constitution Principle VIII — why this mechanism actually scales + +**Decision**: Adopt an explicit, documented cache bound, `mocking.gateway.maxCachedSpecs` (default +100 in the lab's `app-config.yaml`), governing the LRU cache in R2. Document this as the *one* knob +that changes between the lab's demo scale (3–4 APIs) and a real deployment (500+ APIs) — no code or +architecture change. + +**Rationale — mapped directly against the amended Principle VIII**: +- **Process/port count is O(1)**, not O(APIs): exactly one Node process and one port, regardless of + whether the catalog has 4 APIs or 5,000. This is the property the original per-file-process design + (R2, superseded) lacked, and is the actual scale fix, not a documentation-only reframing. +- **Memory cost is O(actively-used APIs), not O(catalog size)**: an unused API's spec is never parsed + or held in memory at all (lazy load). Even a fully "warm" cache of `maxCachedSpecs: 100` mid-size + OpenAPI specs (a few hundred KB of parsed operations each, generously) is on the order of tens of + MB — versus tens of GB for 500 permanently-running Prism processes under the original design. +- **The only lab-only simplification that does not itself generalize**: discovery of the + `slug → specFilePath` map happens once at gateway startup (R3), not on a live-reload/watch basis — + a newly added API requires a gateway restart to become mockable. This mirrors Lab 4's own + `mode: 'poll'` vs. `mode: 'watch'` framing (Lab 4's research.md R6) and is flagged the same way here + per Principle VIII's requirement to call out simplifications explicitly: the README documents that a + real, frequently-changing 500+ API mono-repo would want the same kind of watch-based refresh Lab 4 + already introduces for catalog discovery, applied here to the gateway's slug map — a config/mode + change, not a redesign. + +This principle-mapping section exists specifically because the original (superseded) R2/R4/R5 +decisions were written and approved before Constitution Principle VIII's scale requirement was +amended in, and did not actually satisfy it on review — this file's revision is that correction. diff --git a/specs/005-lab-5-mocking-testing/tasks.md b/specs/005-lab-5-mocking-testing/tasks.md new file mode 100644 index 0000000..72817aa --- /dev/null +++ b/specs/005-lab-5-mocking-testing/tasks.md @@ -0,0 +1,209 @@ +--- + +description: "Task list for Lab 5 — Mocking and Testing" +--- + +# Tasks: Lab 5 — Mocking and Testing + +**Input**: Design documents from `/specs/005-lab-5-mocking-testing/` + +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md (all present; no `contracts/` — this lab produces no external API contracts) + +**Tests**: Not requested. Per plan.md's Testing section, verification is manual (browser + gateway logs) against quickstart.md's steps — consistent with Labs 1–4. No automated test tasks are generated; quickstart validation is folded into each story's checkpoint and the final Polish phase. + +**Organization**: Tasks are grouped by user story (spec.md priorities P1/P2/P2) to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (US1, US2, US3) + +## Path Conventions + +This is a tutorial-lab project (not a generic web/mobile app). Three path roots are used: + +- **Backstage instance** (modified in place, from Lab 1): `labs/lab-01-base-backstage/backstage/` +- **Lab 5 teaching content** (new, committed alongside README): `labs/lab-05-mocking-testing/` +- **Speckit artifacts**: `specs/005-lab-5-mocking-testing/` (this directory — no code changes here) + +**Note**: This task list reflects a mid-plan architecture revision (research.md R2/R7). The original +design spawned one `prism` CLI process per discovered API; on review against Constitution Principle +VIII's amended scale requirement, that was found not to scale (500+ APIs → 500+ permanently-running +processes/ports). The design below instead uses a single, long-lived, lazily-loading mock gateway +process built on Prism's programmatic library. + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Add dependencies and scaffold the directories every later task writes into. + +- [ ] T001 Add `@stoplight/prism-http@5.15.11`, `@stoplight/http-spec@7.1.0`, `js-yaml`, and `concurrently` as root devDependencies of `labs/lab-01-base-backstage/backstage/package.json` (`yarn add -D` from the workspace root; research.md R2/R4) +- [ ] T002 [P] Create the lab content directories `labs/lab-05-mocking-testing/code/scripts/` and `labs/lab-05-mocking-testing/code/packages/app/src/modules/apiMocking/` + +**Checkpoint**: Dependencies installed, directories exist — ready for foundational work. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Establish the module-registration point, the static gateway/proxy config, and the gateway script's discovery skeleton that both US1 (mock target injection) and US3 (credential pre-fill) extend. No user story is independently testable until this phase is done. + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete. + +- [ ] T003 Scaffold the `apiMocking` frontend module as a passthrough override of `apiDocsConfigRef` for the `openapi` widget type — renders the default `OpenApiDefinitionWidget` unchanged, no behavior added yet — in `labs/lab-05-mocking-testing/code/packages/app/src/modules/apiMocking/index.tsx` (research.md R1) +- [ ] T004 Copy the scaffolded module to `labs/lab-01-base-backstage/backstage/packages/app/src/modules/apiMocking/index.tsx` and register it in `labs/lab-01-base-backstage/backstage/packages/app/src/apis.ts` (depends on T003) +- [ ] T005 Add the static `mocking.gateway` config block (`port: 4010`, `maxCachedSpecs: 100`) and the single static `proxy.endpoints['/mock']` entry (`target: 'http://localhost:4010'`, matching `mocking.gateway.port`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md, research.md R5) +- [ ] T006 Scaffold `scripts/mock-gateway.mjs`: read `mocking.gateway.port` from `app-config.yaml`, bind an HTTP server on it, and build the discovery map (`entityNameSlug → specFilePath`) once at startup from Lab 4's `autoApiRegistration.rootPath`/`patterns` config — no parsing of any spec file yet (research.md R2/R3) — in `labs/lab-05-mocking-testing/code/scripts/mock-gateway.mjs` +- [ ] T007 Wire the root `start` script (`concurrently` running the gateway alongside `backstage-cli repo start`; no `wait-on` needed since there is no generated config file to sequence against) in `labs/lab-01-base-backstage/backstage/package.json` (research.md R4, depends on T001) + +**Checkpoint**: Module registration point exists and renders unchanged; gateway binds its port and builds the discovery map; static config and start-script wiring are in place — ready for US1. + +--- + +## Phase 3: User Story 1 - Dynamically mock an API and try it from Backstage (Priority: P1) 🎯 MVP + +**Goal**: A learner can start the local dev environment, see a "Local mock (Lab 5)" server option on any OpenAPI API's page, and execute a working "Try it out" request against it with no real backend involved — served lazily by a single gateway process regardless of how many APIs are registered. + +**Independent Test**: Start `yarn start`, open the Museum API's page (its native `servers` entry is fictitious/non-resolvable), select the mock target, execute a sample request, and confirm a realistic example response comes back; confirm the gateway only parsed Museum's spec (not every discovered API) by checking its log; stop the gateway and confirm a clear connection error instead of a silent failure. + +### Implementation for User Story 1 + +- [ ] T008 [US1] Implement the request handler for `//*`: on a cache miss, read + `js-yaml`-parse the matched spec file and convert it via `@stoplight/http-spec`'s `transformOas3Operations(document)`; store the result in an LRU map bounded by `mocking.gateway.maxCachedSpecs` (evict least-recently-used on overflow); log the load and, for troubleshooting per research.md's flagged open item, the measured parse latency — in `labs/lab-05-mocking-testing/code/scripts/mock-gateway.mjs` (data-model.md Loaded Spec Cache, research.md R2/R7, depends on T006) +- [ ] T009 [US1] Implement request/response translation and mock generation: build one gateway-lifetime `Prism.createInstance({ mock: { dynamic: false } })`, translate each incoming HTTP request into Prism's request descriptor, call `.request(descriptor, operations)` using the (now-cached) operations from T008, and translate the mock response back into a real HTTP response; return a clear 404 for an unknown slug — in `labs/lab-05-mocking-testing/code/scripts/mock-gateway.mjs` (research.md R2, depends on T008) +- [ ] T010 [US1] Copy the completed `mock-gateway.mjs` into `labs/lab-01-base-backstage/backstage/scripts/mock-gateway.mjs` (depends on T009) +- [ ] T011 [US1] Extend the `apiMocking` module to merge a `Local mock (Lab 5)` server entry into the rendered `servers` list for `openapi`-type entities — the entry's URL is `/api/proxy/mock/`, computed purely from `entity.metadata.name` (via the same slug algorithm as the gateway's discovery map), so no per-entity configuration is required for a new API to get one — in `labs/lab-05-mocking-testing/code/packages/app/src/modules/apiMocking/index.tsx` (data-model.md, FR-001, depends on T003) +- [ ] T012 [US1] Copy the updated module into `labs/lab-01-base-backstage/backstage/packages/app/src/modules/apiMocking/index.tsx` (depends on T011, T004) +- [ ] T013 [US1] Manually verify against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 6–7): the gateway log shows the discovery map built (N APIs discoverable) but nothing parsed yet; Museum's page shows `Local mock (Lab 5)` alongside its native URL; selecting it and executing a request returns a realistic example response (SC-001) and logs a first-load line with measured parse latency; a second request does not re-log a load (cache hit); stopping the gateway and retrying produces a clear connection error, not a silent failure (FR-008) (depends on T010, T012) + +**Checkpoint**: User Story 1 fully functional and independently testable — this is the MVP. + +--- + +## Phase 4: User Story 2 - Interact with an existing test implementation (Priority: P2) + +**Goal**: A learner can select a real, already-running public sandbox server for an API (Galaxy) as the active request target, distinct from the local mock, with zero Lab 5 configuration required. + +**Independent Test**: Open the Galaxy API's page, select one of its native sandbox servers instead of the mock, execute a request, and confirm the response comes from the real sandbox. + +### Implementation for User Story 2 + +- [ ] T014 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md step 8): the Galaxy API's page already offers `galaxy.scalar.com` and `void.scalar.com` in its server dropdown with zero Lab 5 configuration, because they are already native to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`'s `servers` list (FR-003, research.md R5); select one, execute a request, and confirm the response comes from the live sandbox, not the local mock (SC-002) +- [ ] T015 [P] [US2] Verify the sandbox-unreachable edge case: temporarily point a test request at an invalid path/host for the sandbox target and confirm a clear, visible error is shown (not a silent failure or a broken-looking lab), per the edge case documented in spec.md + +**Checkpoint**: User Stories 1 AND 2 both work independently — mock and sandbox targets are both selectable and functional. + +--- + +## Phase 5: User Story 3 - Supply and manage non-production credentials (Priority: P2) + +**Goal**: A learner can execute an authenticated request against a mock/test API with zero credential setup (a documented default is pre-filled), and can replace it with their own credential when they have one, without ever committing a personal credential to the repository. + +**Independent Test**: Execute an authenticated request with no credential entered and confirm the documented default succeeds automatically; replace the pre-filled value in the Authorize dialog and confirm the replacement is what's actually sent. + +### Implementation for User Story 3 + +- [ ] T016 [US3] Add the `mocking.defaultCredential` config block (`scheme: bearer`, an obviously fictional value, e.g. `lab-mock-token-do-not-use`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md, Constitution Principle IX) +- [ ] T017 [US3] Extend the `apiMocking` module to read `mocking.defaultCredential` via `configApiRef` and pre-authorize Swagger UI's "Authorize" state on mount (`onComplete(system) => system.authActions.authorize(...)`), leaving the value fully editable/overridable through the same dialog — in `labs/lab-05-mocking-testing/code/packages/app/src/modules/apiMocking/index.tsx` (research.md R6, FR-006/FR-007, depends on T011) +- [ ] T018 [US3] Copy the updated module into `labs/lab-01-base-backstage/backstage/packages/app/src/modules/apiMocking/index.tsx` (depends on T017, T012) +- [ ] T019 [US3] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md step 9): an authenticated request against the mock/sandbox succeeds with zero learner setup, using the pre-filled default (SC-003); replacing the pre-filled value in the Authorize dialog and re-running confirms the replacement — not the default — is what's sent (SC-004); confirm no personal credential ever appears in `app-config.yaml` or any other committed file (depends on T016, T018) + +**Checkpoint**: All three user stories independently functional — mock, sandbox, and credentials all verified. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +**Purpose**: Learner-facing documentation and final repo-wide consistency checks that span multiple stories. + +- [ ] T020 Write `labs/lab-05-mocking-testing/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II) covering: install deps, scaffold/register the `apiMocking` module, add static `mocking`/`proxy` config, wire the `start` script, start Backstage, verify mock/sandbox/credential behavior — following quickstart.md steps 1–9, and linking to the committed `labs/lab-05-mocking-testing/code/scripts/mock-gateway.mjs` and `code/packages/app/src/modules/apiMocking/index.tsx` rather than inlining them (both exceed the ~40–50 line threshold) (depends on T013, T014, T019) +- [ ] T021 Add the README's `## Security Note` section (Constitution Principle IX / Lab Structure Standards), placed before the `mocking.defaultCredential` configuration step, explaining the shortcut (a shared fictional credential committed to `app-config.yaml`) and the production alternative (secrets manager / vault-injected config, per-user credentials, never committing real credentials) (depends on T016) +- [ ] T022 Add the README's "Why one gateway process, not one per API" section (Constitution Principle VIII, research.md R2/R7): a worked example, for learners, of designing a lab mechanism to scale — explain the superseded per-API-process design, why it failed the scale check at 500+ APIs, and why lazy load + a single bounded LRU cache fixes it without changing the lab's own demo behavior +- [ ] T023 [P] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010): gateway port, `maxCachedSpecs`, reliance on Lab 4's discovery glob, and the default credential value are adaptable; the `apiDocsConfigRef` override mechanism and the single-gateway architecture itself are fixed +- [ ] T024 [P] Add the README's note on why AsyncAPI (Streetlights) has no mock option and will not get one from this mechanism (FR-005, out-of-scope edge case) +- [ ] T025 Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T013/T014/T015/T019 plus common failure modes (gateway port already in use, a spec file that fails to parse on first request, `maxCachedSpecs` too low for a learner's own larger fork causing unexpected cache churn) +- [ ] T026 Update root `README.md` (Lab Series table entry, Getting Started tree, Repository Structure tree) to reference `labs/lab-05-mocking-testing/` and `specs/005-lab-5-mocking-testing/`, per the Constitution's Lab Structure Standards — required for this lab to be considered complete +- [ ] T027 [P] Verify the "generic by default" mechanism (quickstart.md step 10): temporarily add a third `*-openapi.yaml` file matching Lab 4's discovery pattern, restart `yarn start` (discovery is boot-time-only, so a restart is expected — research.md R7), confirm it automatically gets a `Local mock (Lab 5)` entry with zero Lab-5-specific configuration, confirm its spec is parsed only on its own first request (not eagerly), then remove the temporary file +- [ ] T028 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles, with particular attention to Principle VIII's scale claims) and correct plan.md if any gate's justification no longer matches what was built +- [ ] T029 Run the full quickstart.md validation sequence (steps 1–10) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies — start immediately +- **Foundational (Phase 2)**: Depends on Setup completion — BLOCKS all user stories +- **User Story 1 (Phase 3)**: Depends on Foundational completion — no dependency on US2/US3 +- **User Story 2 (Phase 4)**: Depends on Foundational completion only (uses Galaxy's pre-existing native `servers`, no shared code with US1) +- **User Story 3 (Phase 5)**: Depends on Foundational completion; extends the same file US1 built (T011), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency +- **Polish (Phase 6)**: Depends on all three user stories being complete + +### User Story Dependencies + +- **User Story 1 (P1)**: Can start after Foundational — no dependency on other stories +- **User Story 2 (P2)**: Can start after Foundational — fully independent of US1/US3 (no shared code) +- **User Story 3 (P2)**: Can start after Foundational, but shares `apiMocking/index.tsx` with US1 (same file, sequential edits) — implement after US1 for a working MVP checkpoint first + +### Within Each User Story + +- Gateway/module extension before the copy-into-Backstage-instance task +- Manual verification task last in each phase (checkpoint gate) + +### Parallel Opportunities + +- T015 (US2 edge case) in parallel with T014 (US2 primary verification) once both are ready to run +- T023, T024 (Polish README sections) in parallel with each other +- T027, T028 (Polish) in parallel + +--- + +## Parallel Example: User Story 2 Verification + +```bash +Task: "Verify Galaxy sandbox target selection and response" +Task: "Verify sandbox-unreachable edge case" +``` + +## Parallel Example: Polish README Sections + +```bash +Task: "Add Adaptable conventions vs. fixed mechanics section" +Task: "Add AsyncAPI-out-of-scope note" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup +2. Complete Phase 2: Foundational (CRITICAL — blocks all stories) +3. Complete Phase 3: User Story 1 +4. **STOP and VALIDATE**: T013 confirms gateway discovery, lazy load + caching, target selection, and error handling all work end-to-end +5. Demo: the Museum API's page offers a working local mock with no real backend involved, served by a single gateway process that never touches specs nobody is testing + +### Incremental Delivery + +1. Setup + Foundational → module-registration point, static config, and gateway discovery skeleton ready +2. Add User Story 1 → validate independently (T013) → MVP demo +3. Add User Story 2 → validate independently (T014/T015) → sandbox target selection confirmed, zero new code +4. Add User Story 3 → validate independently (T019) → default credential + override confirmed +5. Polish → README (including the scale-design worked example), Security Note, root README update, cold-start validation, constitution re-check + +### Notes + +- US1 and US3 share one implementation file (`apiMocking/index.tsx`) because both extend the same + `apiDocsConfigRef` override introduced in Foundational — a deliberate single-module design, not + an accidental cross-story coupling. Each story still has an independently verifiable acceptance + test (T013 vs. T019) even though the code lands in the same file. +- US2 requires no code changes at all — Galaxy's native `servers` list already satisfies it + (research.md R5) — so its phase is verification-only, confirming the "no additional + configuration" claim in FR-003 rather than building anything new. +- No test tasks are included (Tests: Not requested — see header). Verification is manual, per + quickstart.md, folded into each story's final task(s). +- This task list supersedes an earlier draft built around one `prism` CLI process per API; that + design was reworked (research.md R2/R7) after review against Constitution Principle VIII's scale + requirement, before any implementation tasks were executed — see the note under Path Conventions. From dd8d20d9cbeb712ba5f1796c29d4862291d860a5 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Sat, 4 Jul 2026 14:02:50 +1000 Subject: [PATCH 17/19] Mocking implemented --- README.md | 21 +- labs/lab-05-mocking-testing/README.md | 391 ++++++++++++++++++ .../app/src/modules/apiMocking/index.tsx | 155 +++++++ .../code/scripts/mock-gateway.mjs | 379 +++++++++++++++++ specs/005-lab-5-mocking-testing/data-model.md | 22 +- specs/005-lab-5-mocking-testing/plan.md | 10 +- specs/005-lab-5-mocking-testing/research.md | 84 +++- specs/005-lab-5-mocking-testing/tasks.md | 58 +-- 8 files changed, 1065 insertions(+), 55 deletions(-) create mode 100644 labs/lab-05-mocking-testing/README.md create mode 100644 labs/lab-05-mocking-testing/code/packages/app/src/modules/apiMocking/index.tsx create mode 100644 labs/lab-05-mocking-testing/code/scripts/mock-gateway.mjs diff --git a/README.md b/README.md index 79cbc6f..56241b5 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Each lab is self-contained, cross-platform (Windows and macOS), and costs nothin | [Lab 2](labs/lab-02-users-roles/) | Users, Roles & API Visibility | Add users, teams, and a custom permission policy; private APIs are visible only to their owning team, shared APIs are visible to everyone | | [Lab 3](labs/lab-03-api-quality/) | API Quality | Add a shared Spectral ruleset, the api-grade quality plugin, and the Spectral linter plugin; API owners and a platform team see detailed quality/lint results, everyone else sees a summary grade | | [Lab 4](labs/lab-04-auto-registration/) | Auto Registration | Auto-discover and register APIs from a Git mono-repo via a custom `EntityProvider`; source owner, lifecycle, and visibility metadata from `x-*` fields in the spec itself | -| Lab 5 *(coming soon)* | Mocking & Testing | Dynamically mock or exercise a test implementation of any registered API, with support for user-supplied non-production credentials | +| [Lab 5](labs/lab-05-mocking-testing/) | Mocking & Testing | Dynamically mock any registered OpenAPI API via a single lazy-loading gateway process, or exercise a real sandbox; pre-filled, overridable non-production credentials | | Lab 6 *(coming soon)* | API Lifecycle Management | Register multiple major versions of an API in parallel; track lifecycle state (development/test/production) and deprecation/retirement per version | | Lab 7 *(coming soon)* | Other Documentation | Add the Thoughtworks Tech Radar plugin; register blips to plot your API landscape | @@ -70,10 +70,15 @@ labs/ │ ├── README.md ← continue here after Lab 2 │ ├── .spectral.yaml ← shared Spectral ruleset used by both quality plugins │ └── catalog/ ← platform team member (eve) added in this lab -└── lab-04-auto-registration/ - ├── README.md ← continue here after Lab 3 - ├── autoApiRegistration.ts ← backend EntityProvider that scans and registers APIs - └── apis/ ← auto-discovered specs, incl. the Scalar Galaxy vendor copy +├── lab-04-auto-registration/ +│ ├── README.md ← continue here after Lab 3 +│ ├── autoApiRegistration.ts ← backend EntityProvider that scans and registers APIs +│ └── apis/ ← auto-discovered specs, incl. the Scalar Galaxy vendor copy +└── lab-05-mocking-testing/ + ├── README.md ← continue here after Lab 4 + └── code/ + ├── scripts/mock-gateway.mjs ← single lazy-loading Prism mock gateway + └── packages/app/src/modules/apiMocking/ ← apiDocsConfigRef override (mock + credentials) ``` --- @@ -86,12 +91,14 @@ backstage-apiportal-lab/ │ ├── lab-01-base-backstage/ │ ├── lab-02-users-roles/ │ ├── lab-03-api-quality/ -│ └── lab-04-auto-registration/ +│ ├── lab-04-auto-registration/ +│ └── lab-05-mocking-testing/ ├── specs/ ← SDD artifacts (spec, plan, tasks per lab) │ ├── 001-lab-1-base-backstage/ │ ├── 002-lab-2-users-roles/ │ ├── 003-lab-3-api-quality/ -│ └── 004-lab-4-auto-registration/ +│ ├── 004-lab-4-auto-registration/ +│ └── 005-lab-5-mocking-testing/ ├── .specify/ ← Speckit configuration and templates ├── GOAL.md ← high-level goals for the full lab series ├── CONTRIBUTING.md ← how to contribute new labs diff --git a/labs/lab-05-mocking-testing/README.md b/labs/lab-05-mocking-testing/README.md new file mode 100644 index 0000000..35a52f4 --- /dev/null +++ b/labs/lab-05-mocking-testing/README.md @@ -0,0 +1,391 @@ +# Lab 5 — Mocking and Testing + +## Overview + +Every API registered in Labs 1–4 renders a "Try it out" panel — but only if a real backend is +sitting behind the API's declared `servers` URL. Museum's production server +(`https://api.example.museum/v1`) is fictitious and will never resolve; Galaxy's sandbox servers +are real, but not every API a learner (or a real engineer) registers will have one. This lab adds +a locally running, dynamically generated mock to every registered OpenAPI API's page, plus a +documented pattern for APIs that already ship a public sandbox (Galaxy), so "Try it out" always has +something real to talk to. + +By the end of this lab you will have: + +- A single, long-lived mock gateway process (`scripts/mock-gateway.mjs`) that serves mocks for + *every* discovered OpenAPI file on one fixed port, using Stoplight Prism's programmatic library + — not one process per API. It builds a cheap `slug → spec file` map at startup and only parses + and converts a given API's spec the first time that API is actually requested, caching the + result in a bounded LRU map. +- A new frontend module (`packages/app/src/modules/apiMocking/`) that overrides + `apiDocsConfigRef` — the same pattern Lab 3 used for `apiGrade`/`spectralLinter` — to merge a + "Local mock (Lab 5)" server entry into every OpenAPI API's server dropdown, computed purely from + the entity's own name, with zero per-entity configuration. +- A documented, fictional default credential, pre-filled into Swagger UI's own "Authorize" dialog + on page load, so an authenticated request succeeds with zero learner setup — fully editable + through that same dialog if a learner wants to supply their own. + +**What you will learn:** + +- Why a single, lazily-loading gateway process — not one process per API — is the only design that + actually satisfies "works the same at 500+ APIs as it does at 3," and what happens to a + first-draft design that gets this wrong (see "Why one gateway process, not one per API" below) +- Why the API viewer needed a frontend-module override at all — `@backstage/plugin-api-docs` has + no built-in config toggle for extra servers or credential injection +- Why a client-side "Authorize" pre-fill was chosen over a proxy-level header rewrite for the + default credential, and why that choice is what makes "override with your own credential" work + at all +- Why Galaxy needs zero Lab 5 configuration to get a sandbox option, while Museum needs one small + addition to become mockable + +--- + +## Prerequisites + +- **Labs 1–4 completed** — Backstage running locally with Museum, Streetlights, Train Travel, + Galaxy, and the precedence-demo API all registered (Lab 4's auto-registration in place) +- **Node.js 20+ (or 22/24) and Yarn** — same toolchain as Labs 1–4 +- No new external accounts, services, or OS-level prerequisites + +--- + +## Security Note + +This lab introduces the series' first credential: `mocking.defaultCredential`, committed in +`app-config.yaml` as an obviously fictional bearer token (`lab-mock-token-do-not-use`). It exists +purely so a learner can exercise an authenticated request with zero setup — it is pre-filled into +Swagger UI's own "Authorize" dialog (fully visible and editable there), never sent as a hidden, +unconditional header a learner couldn't see or override. + +**Do not follow this pattern in production.** A committed, shared, fictional credential is a +teaching shortcut for a local sandbox with no real data behind it. In a real deployment: + +- Credentials belong in a secrets manager or vault-injected config, never a committed file. +- Credentials should be scoped per user or per team, not shared across everyone who clones the repo. +- A real credential should never be able to reach a real backend from a public or shared + environment — this lab's mock gateway has no real data or side effects to protect in the first + place, which is precisely why a shortcut is acceptable here and nowhere else. + +--- + +## Step 1 — Install Dependencies + +From the Lab 1 Backstage workspace root: + +``` +cd labs/lab-01-base-backstage/backstage +yarn add -D @stoplight/prism-http@5.15.11 @stoplight/http-spec@7.1.0 js-yaml concurrently +``` + +`@stoplight/prism-http` is Stoplight Prism's *programmatic* mock-response library — not the +`prism` CLI. It exposes `createInstance(...).request(...)` as a plain function call that returns a +mocked HTTP response without binding any port of its own, which is what makes a single process +able to serve every API. `@stoplight/http-spec` is pulled in by `prism-http` and used internally to +convert an OpenAPI document into the operation shape Prism consumes. `js-yaml` reads +`app-config.yaml` and each spec file's `info.title`. `concurrently` runs the gateway alongside +`backstage-cli repo start`. + +The frontend module also needs its own copy of `js-yaml`, since it parses +`entity.spec.definition` (always delivered as a raw string) client-side: + +``` +yarn workspace app add js-yaml +``` + +## Step 2 — Add the Mock Gateway Script + +Copy [`code/scripts/mock-gateway.mjs`](code/scripts/mock-gateway.mjs) to +`scripts/mock-gateway.mjs` in the Backstage workspace. At a high level, it: + +1. Reads `app-config.yaml` directly (no Backstage config service — this is a standalone Node + script) for `mocking.gateway.port`/`maxCachedSpecs` and Lab 4's own + `autoApiRegistration.rootPath`/`patterns`. +2. At startup, builds an in-memory `entityNameSlug → specFilePath` map from every file matching + Lab 4's discovery config, **plus** one additional source covering Lab 1's `apis/` directory + (see "Why Museum needs one extra source" below) — a cheap directory walk plus a lightweight + `info.title` read, not the expensive spec-to-operations conversion, so this is safe to do + eagerly even at hundreds of files. +3. On each request to `//*`, looks up the slug, and on a cache miss, converts that one + spec via `@stoplight/prism-http`'s `getHttpOperationsFromSpec(filePath)` — which dereferences + the spec's own internal `$ref`s before converting, a step required for anything beyond the + most trivial spec (see "A correction worth knowing about" below) — caching the result in an + LRU map bounded by `mocking.gateway.maxCachedSpecs`. +4. Calls a single, gateway-lifetime `Prism.createInstance({ mock: { dynamic: false } })` instance's + `.request()` with the translated request and cached operations, and translates the mocked + response back into a real HTTP response. + +### A correction worth knowing about + +The natural first attempt is to `js-yaml`-parse a spec file and hand it directly to +`@stoplight/http-spec`'s `transformOas3Operations(document)`. That works right up until a response +has no static example and Prism falls back to schema-driven generation — at which point it fails +with `Invalid reference token: components`, because `transformOas3Operations` alone does not +resolve the document's own internal `$ref`s (e.g. `#/components/schemas/Foo`), which almost every +non-trivial OpenAPI document has. `getHttpOperationsFromSpec` (exported directly by +`@stoplight/prism-http`) wraps that same call with the dereference step it actually needs — this +was found by testing against Museum's real spec, not by reading documentation more carefully in +advance, which is exactly the kind of thing you only learn by running the code. + +## Step 3 — Add Static Configuration + +Add to `app-config.yaml`: + +```yaml +proxy: + endpoints: + '/mock': + target: 'http://localhost:4010' + changeOrigin: true + +mocking: + gateway: + port: 4010 + maxCachedSpecs: 100 + defaultCredential: + scheme: bearer + value: lab-mock-token-do-not-use +``` + +The proxy target and `mocking.gateway.port` are the same number, committed twice — Backstage's +`app-config.yaml` has no config-to-config interpolation (only `${ENV_VAR}` substitution), so +keeping them in sync is a one-line manual step, not a recurring one. + +## Step 4 — Wire the Start Script + +In the root `package.json`: + +```json +"start": "concurrently -n app,mocks \"backstage-cli repo start\" \"node scripts/mock-gateway.mjs\"" +``` + +No `wait-on` sequencing is needed — the gateway's port and proxy target are static, committed +values valid the moment Backstage starts, whether or not the gateway has finished booting yet; the +gateway only needs to be listening by the time a learner clicks "Try it out". `concurrently`'s +`-k`/`--kill-others` flag is deliberately **not** used: if the gateway process dies or is stopped, +Backstage keeps running and reports a clear connection error on the next mock request (Step 7), +which is exactly the behavior you want to be able to demonstrate and exactly what `-k` would +prevent by tearing down the whole session the moment the gateway exits. + +## Step 5 — Add the `apiMocking` Frontend Module + +Copy [`code/packages/app/src/modules/apiMocking/index.tsx`](code/packages/app/src/modules/apiMocking/index.tsx) +to `packages/app/src/modules/apiMocking/index.tsx`, then register it in `packages/app/src/App.tsx` +alongside Lab 3's modules: + +```ts +import { apiMockingModule } from './modules/apiMocking'; + +export default createApp({ + features: [ + // ...existing features + apiMockingModule, + ], +}); +``` + +The module overrides the `config` extension of the `api-docs` plugin (the same `pluginId` + +extension name the plugin itself registers for `apiDocsConfigRef`) to supply a custom +`getApiDefinitionWidget`. For any `openapi`-type entity, it: + +- Parses `entity.spec.definition` (a raw YAML/JSON string — that's the only form the widget's + `component(definition)` signature ever receives) into an object, appends a + `{ url: '/api/proxy/mock/', description: 'Local mock (Lab 5)' }` entry to + its `servers` array, and passes the merged object through as an extra `spec` prop. +- Reads every `http`/`bearer`-type security scheme the spec declares and pre-authorizes each one + via `onComplete(system) => system.authActions.authorize(...)`, using + `mocking.defaultCredential`. + +Two mechanisms make this possible without any change to `@backstage/plugin-api-docs` itself, +confirmed by reading its source rather than assumed: + +- `getApiDefinitionWidget(apiEntity)` — unlike the widget's own `component(definition)` — receives + the **full** entity, which is what lets this module compute a per-entity mock URL with no + per-entity configuration. +- `OpenApiDefinitionWidget` forwards every prop it receives down to `swagger-ui-react`'s own + ``, *after* its own internal `spec={definitionString}` — so an extra `spec` prop + passed through the widget wins over the string-derived one. + +This new config key also needs to be declared visible to the frontend bundle. Add +`packages/app/config.d.ts`: + +```ts +export interface Config { + mocking?: { + defaultCredential?: { + /** @visibility frontend */ + scheme: string; + /** @visibility frontend */ + value: string; + }; + }; +} +``` + +and add `"configSchema": "config.d.ts"` to `packages/app/package.json`. Without both, Backstage's +config-visibility filter silently strips `mocking.defaultCredential` before it reaches the +browser — `configApi.getOptional(...)` will just return `undefined`, with no error anywhere. This +was found by checking the `