From 3311e75396ddd8ee1b2dec51c083c20ff3ccaccd Mon Sep 17 00:00:00 2001 From: DawMatt Date: Thu, 2 Jul 2026 23:31:59 +1000 Subject: [PATCH 01/12] Setup initial specification --- .specify/feature.json | 2 +- .../checklists/requirements.md | 37 ++++++ specs/004-lab-4-auto-registration/spec.md | 110 ++++++++++++++++++ 3 files changed, 148 insertions(+), 1 deletion(-) create mode 100644 specs/004-lab-4-auto-registration/checklists/requirements.md create mode 100644 specs/004-lab-4-auto-registration/spec.md diff --git a/.specify/feature.json b/.specify/feature.json index 1476533..aee6f97 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1,3 +1,3 @@ { - "feature_directory": "specs/003-lab-3-api-quality" + "feature_directory": "specs/004-lab-4-auto-registration" } diff --git a/specs/004-lab-4-auto-registration/checklists/requirements.md b/specs/004-lab-4-auto-registration/checklists/requirements.md new file mode 100644 index 0000000..35d33a7 --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Lab 4 - Auto Registration + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-02 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Items marked incomplete require spec updates before `/speckit-clarify` or `/speckit-plan`. +- No [NEEDS CLARIFICATION] markers were needed: reasonable, documented defaults were used for + mono-repo simulation, discovery cadence, and catalog-info.yaml vs. x-* precedence — all + recorded in the spec's Assumptions section per Constitution Principle VIII. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md new file mode 100644 index 0000000..4b273ca --- /dev/null +++ b/specs/004-lab-4-auto-registration/spec.md @@ -0,0 +1,110 @@ +# Feature Specification: Lab 4 - Auto Registration + +**Feature Branch**: `004-lab-4-auto-registration` + +**Created**: 2026-07-02 + +**Status**: Draft + +**Input**: User description: "Create new feature and specification from feature 4 in GOAL.md" — Lab 4 (Auto registration): Setup a process for automatic registration of APIs and creation of catalog metadata. Assume API definition files exist within a Git mono-repo. Include sourcing of metadata from x-* fields, within the API specifications themselves, into catalog metadata, fields and annotations. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) + +As a platform engineer, when a new API definition file (OpenAPI or AsyncAPI) is added anywhere within the designated mono-repo location, I want it to appear in the Backstage catalog as a registered API entity without anyone hand-authoring a `catalog-info.yaml` for it. + +**Why this priority**: This is the core value of the lab — removing manual catalog registration is the entire premise of "auto registration." Without this, there is no lab. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file to the mono-repo location, trigger a catalog refresh, and confirm a corresponding API entity appears in the Backstage catalog with no manual `catalog-info.yaml` written for that API. + +**Acceptance Scenarios**: + +1. **Given** a mono-repo containing one or more OpenAPI/AsyncAPI definition files with no matching catalog entity, **When** the catalog discovery process runs, **Then** a catalog API entity is created for each definition file found. +2. **Given** an API definition file that was previously registered, **When** its contents change (e.g., description or version updated) and discovery runs again, **Then** the existing catalog entity is updated in place rather than duplicated. +3. **Given** an API definition file is removed from the mono-repo, **When** discovery runs again, **Then** the corresponding catalog entity is no longer present as an active, discoverable entity. + +--- + +### User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or tags) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. + +**Why this priority**: This is what makes registration genuinely "automatic" rather than just file discovery — without metadata sourcing, every API would still need a hand-maintained catalog file for ownership and lifecycle, which is exactly what Lab 2's ownership model depends on. + +**Independent Test**: Author an API definition containing `x-*` fields for owning team and lifecycle, run discovery, and confirm the resulting catalog entity's `spec.owner`, `spec.lifecycle`, and equivalent fields/annotations reflect the values from the `x-*` fields — not placeholder or default values. + +**Acceptance Scenarios**: + +1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, tags, system grouping), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry. +3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. +4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. + +--- + +### User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +As a learner adapting this repository to my own organization, I want to understand which parts of the discovery and metadata-sourcing configuration are conventions I can change (e.g., file naming patterns, folder layout, `x-*` field names) versus fixed requirements, so I can apply this pattern to my own mono-repo layout. + +**Why this priority**: Reinforces Constitution Principle VIII (Support Experimentation) — the lab must remain valuable to learners whose mono-repo doesn't match the example layout exactly. Lower priority than P1/P2 because the mechanism must exist and work correctly before its adaptability can be documented. + +**Independent Test**: Follow the lab documentation's guidance to point discovery at a differently-structured example folder (different glob pattern, different `x-*` field names) and confirm registration still works as described. + +**Acceptance Scenarios**: + +1. **Given** the lab documentation, **When** a learner reads the configuration section, **Then** they can identify which settings (file location pattern, recognized `x-*` field names) are conventions they are expected to adapt, and which are fixed mechanics of the discovery process. + +--- + +### Edge Cases + +- What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. +- What happens when two API definition files resolve to the same catalog entity name? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. +- What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. +- What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. +- How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) located within a designated mono-repo folder structure, without requiring a hand-authored `catalog-info.yaml` per API. +- **FR-002**: The system MUST automatically create a catalog API entity for each newly discovered API definition file. +- **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. +- **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. +- **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields and map them into the corresponding catalog entity fields or annotations. +- **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. +- **FR-008**: The system MUST surface a visible, actionable error when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition — and MUST continue processing remaining, unaffected API definitions. +- **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. +- **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. +- **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. +- **FR-012**: The lab MUST run entirely on local, freely available tooling with $0 cost, requiring no paid service, hosted CI/CD, or webhook-reachable public endpoint (Constitution Principle V). +- **FR-013**: The lab MUST build upon the environment established by Labs 1-3 (base Backstage setup, users/roles/teams and ownership-based visibility, API quality tooling) without requiring those labs to be redone. + +### Key Entities + +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **Mono-repo Location**: The designated folder structure/convention within which API Definition Files are expected to live, used as the scope for discovery. +- **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. +- **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. +- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error, exactly which file and problem caused registration to fail, without needing to inspect system internals. +- **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. +- **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. + +## Assumptions + +- The "Git mono-repo" referenced in Lab 4 is simulated as a folder/convention within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. +- "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. +- Default `x-*` field names (e.g., for owning team, lifecycle, tags) will be defined by this lab as a documented convention; learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. +- When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. +- Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 04a48b063e45175f935a59fcffe7248227882896 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 07:09:11 +1000 Subject: [PATCH 02/12] API auto registration feature planned --- CLAUDE.md | 2 +- .../004-lab-4-auto-registration/data-model.md | 57 +++++++ specs/004-lab-4-auto-registration/plan.md | 148 ++++++++++++++++++ .../004-lab-4-auto-registration/quickstart.md | 45 ++++++ specs/004-lab-4-auto-registration/research.md | 136 ++++++++++++++++ specs/004-lab-4-auto-registration/spec.md | 28 ++-- 6 files changed, 406 insertions(+), 10 deletions(-) create mode 100644 specs/004-lab-4-auto-registration/data-model.md create mode 100644 specs/004-lab-4-auto-registration/plan.md create mode 100644 specs/004-lab-4-auto-registration/quickstart.md create mode 100644 specs/004-lab-4-auto-registration/research.md diff --git a/CLAUDE.md b/CLAUDE.md index adb4816..e465dfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,5 +1,5 @@ For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan -at specs/003-lab-3-api-quality/plan.md +at specs/004-lab-4-auto-registration/plan.md diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md new file mode 100644 index 0000000..abe48d4 --- /dev/null +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -0,0 +1,57 @@ +# Phase 1 Data Model: Lab 4 — Auto Registration + +This lab has no application database. "Data model" here means the shape of the catalog entities +produced, the config schema that drives discovery, and the mapping rules between an API +definition file and the resulting Backstage `API` entity. + +## Entity: API Definition File (input, not a catalog entity) + +A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`). + +| Field | Type | Notes | +|---|---|---| +| `openapi` / `asyncapi` | string | Version field; presence of either identifies the spec type. Exactly one is expected. | +| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. | +| `info.description` | string | Optional. Maps to `metadata.description` if present. | +| `info.x-backstage-owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration proceeds with no `spec.owner` override... actually **required by FR-005/FR-007's documented default** — see Mapping Rule 1 below. | +| `info.x-backstage-lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | +| `info.x-backstage-tags` | string[] | Optional. Default: `[]`. | + +## Entity: Catalog API Entity (output) + +Standard Backstage `API` kind entity, produced by the EntityProvider's full mutation. + +| Field | Source | Mapping Rule | +|---|---|---| +| `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | +| `metadata.title` | `info.title` | Verbatim. | +| `metadata.description` | `info.description` | Verbatim if present, else omitted. | +| `metadata.tags` | `info.x-backstage-tags` | Default `[]` if absent. | +| `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | +| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | +| `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | +| `spec.lifecycle` | `info.x-backstage-lifecycle` | Default `experimental` if absent (FR-007). | +| `spec.owner` | `info.x-backstage-owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | + +## Config: `autoApiRegistration` (app-config.yaml) + +| Key | Type | Default | Purpose | +|---|---|---|---| +| `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | +| `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | +| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | How often the EntityProvider re-scans. | +| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | + +## State transitions (per discovery cycle) + +1. Glob scan → candidate file list. +2. Parse + shape-check each file → valid candidates / log-only errors (unparseable files). +3. Slugify `info.title` → candidate entity name; detect duplicates among this cycle's candidates + → second-and-later duplicates become marker/error entities. +4. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners become + marker/error entities. +5. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored + `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). +6. Emit a `type: 'full'` mutation containing every entity produced by steps 2–5 (normal + + marker/error entities). Entities not included are retracted by the catalog (FR-004). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md new file mode 100644 index 0000000..72e0dbf --- /dev/null +++ b/specs/004-lab-4-auto-registration/plan.md @@ -0,0 +1,148 @@ +# Implementation Plan: Lab 4 — Auto Registration + +**Branch**: `004-lab-4-auto-registration` | **Date**: 2026-07-03 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `specs/004-lab-4-auto-registration/spec.md` + +## Summary + +Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replaces the +per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend +`EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / +`*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s +schedule — no catalog-info.yaml required per API (FR-001–FR-004). Catalog metadata (owner, +lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and +`x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented +defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) +are surfaced via backend logs and, where an entity can be identified, via Backstage's native +catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker +annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a +discovered file, takes precedence over auto-sourced metadata (FR-011). + +Two new sample files support the lab: a vendored copy of the MIT-licensed Scalar Galaxy API +(`galaxy-openapi.yaml`) with no pre-existing catalog registration, demonstrating genuine +"previously unregistered API" discovery (US1); and a small `precedence-demo-openapi.yaml` + +hand-authored `precedence-demo-catalog-info.yaml` pair demonstrating the precedence rule. +Lab documentation calls out which parts (glob patterns, root path, `x-*` field names) are +adaptable conventions per Constitution Principle VIII (FR-010, US3). + +## Technical Context + +**Language/Version**: TypeScript / Node.js 20 LTS (Backstage 1.51.0, pinned from Lab 1) + +**Primary Dependencies**: +- `fast-glob@3.3.3` — new direct dependency of `packages/backend`; already present transitively + in the workspace, promoted to an explicit dependency for the filesystem scan (research.md R2). +- `js-yaml@4.2.0` — new direct dependency of `packages/backend`; already present transitively, + promoted to explicit for parsing candidate spec files (research.md R3). +- `@backstage/plugin-catalog-node` (already installed, `2.2.1`) — supplies `EntityProvider`, + `CatalogProcessor`, and `catalogProcessingExtensionPoint` used to register the new backend + module, following the same `createBackendModule` pattern as + `packages/backend/src/extensions/permissionPolicy.ts`. +- No new frontend dependencies — this lab is backend-only; no new UI surfaces are added (errors + ride on Backstage's existing built-in processing-error UI). + +**Storage**: YAML files only — two new vendored API spec files, one new hand-authored +`catalog-info.yaml` (precedence demo), and one `app-config.yaml` addition (new +`autoApiRegistration` block). No database schema changes. + +**Testing**: Manual browser + log verification per lab README (no automated test suite — +consistent with Labs 1–3's tutorial-lab testing approach). + +**Target Platform**: Local development machine — Windows 10/11 and macOS 12+ (same as Labs 1–3). + +**Project Type**: Tutorial lab — Markdown documentation + YAML config/catalog files + one new +TypeScript backend module (EntityProvider + CatalogProcessor). + +**Performance Goals**: New/changed API definitions appear in the catalog within one discovery +cycle (≤30s, SC-001); scan cost is bounded by mono-repo size, acceptable at lab scale (a handful +of spec files). + +**Constraints**: Zero cost; cross-platform; no external network access beyond the one-time vendor +of the Scalar Galaxy API file at authoring time (the vendored copy itself requires no runtime +network access, per Constitution Principle VII); builds on Labs 1–3 without disrupting existing +registered APIs (museum, streetlights, train-travel) or the Lab 3 quality tooling. + +**Scale/Scope**: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored +`catalog-info.yaml`, 1 new backend module (~2 files: EntityProvider+processor and its +registration), 1 `app-config.yaml` config block, 3 existing registered APIs left untouched. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design — all gates hold.* + +| Principle | Gate | Status | +|-----------|------|--------| +| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: custom `EntityProvider` for filesystem-based catalog discovery, `x-*` vendor-extension metadata sourcing, native catalog processing-error surfacing via a custom `CatalogProcessor`, full-mutation-based create/update/retract lifecycle | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), and WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011) | ✅ Pass | +| III. Cross-Platform Compatibility | All new code is TypeScript/YAML; `fast-glob` and `js-yaml` are pure-JS, no native/platform-specific dependencies; `yarn add` and `yarn start` work identically on Windows and macOS; root-path resolution documented with platform-neutral relative paths | ✅ Pass | +| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–3's existing Node/Yarn setup — `fast-glob`/`js-yaml` are installed via the existing `yarn add` workflow; README documents the install step | ✅ Pass | +| V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | +| VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | +| VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `x-backstage-*` field names as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | + +No violations. Complexity Tracking section omitted. + +## Project Structure + +### Documentation (this feature) + +```text +specs/004-lab-4-auto-registration/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +└── tasks.md # Phase 2 output (/speckit-tasks — NOT created here) +``` + +No `contracts/` directory — this lab produces no external API contracts. The `autoApiRegistration` +config shape and the entity mapping rules are documented in data-model.md instead. + +### Source Code (repository root) + +```text +labs/ +└── lab-04-auto-registration/ + ├── README.md # Lab documentation (overview, prereqs, steps, + │ # verification, troubleshooting, adaptable + │ # conventions per US3/FR-010) + └── apis/ + ├── galaxy/ + │ └── galaxy-openapi.yaml # New: vendored Scalar Galaxy API (MIT), + │ # x-backstage-owner/lifecycle/tags added; + │ # no catalog-info.yaml — pure auto-discovery + └── precedence-demo/ + ├── precedence-demo-openapi.yaml # New: minimal spec for the precedence demo + └── precedence-demo-catalog-info.yaml # New: hand-authored entity that must win + # over the auto-sourced one (FR-011) +``` + +Changes to the student's Backstage instance (guided by README, not committed as generated output, +but the module source itself IS committed since it is lab teaching content): + +```text +# In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): +app-config.yaml # Modified: add autoApiRegistration config block +packages/backend/ +├── package.json # Modified: add fast-glob, js-yaml +└── src/ + ├── index.ts # Modified: backend.add(import('./extensions/autoApiRegistration')) + └── extensions/ + └── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend + # module (discovery, parsing, mapping, owner + # validation, precedence check, error marking) +``` + +**Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–3. All +student-facing content lives under `labs/lab-04-auto-registration/`. The two new sample API +files (and the one hand-authored precedence-demo catalog file) are committed alongside the lab +README. The student's Backstage instance (from Lab 1) is modified in-place per the README +instructions, same as Lab 3's pattern. Speckit artefacts live under +`specs/004-lab-4-auto-registration/` per convention. + +## Complexity Tracking + +No violations recorded — table omitted per template instructions. diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md new file mode 100644 index 0000000..053a942 --- /dev/null +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -0,0 +1,45 @@ +# Quickstart: Lab 4 — Auto Registration + +This is the design-time quickstart used to validate the plan; the learner-facing README (built in +`/speckit-tasks` + `/speckit-implement`) will follow the same flow with full explanatory prose per +Constitution Principle II. + +## Prerequisites + +- Labs 1–3 completed (running Backstage 1.51.0 instance, Lab 2 teams/users, Lab 3 quality tooling). +- No new external accounts or services. + +## Steps + +1. **Install dependencies**: add `fast-glob` and `js-yaml` as direct dependencies of + `packages/backend`. +2. **Add the auto-registration backend module** + (`packages/backend/src/extensions/autoApiRegistration.ts`): implements the `EntityProvider` + + companion `CatalogProcessor` described in research.md R1–R4. Registered in + `packages/backend/src/index.ts` via `backend.add(import('./extensions/autoApiRegistration'))`. +3. **Configure discovery** in `app-config.yaml` under a new `autoApiRegistration` block (see + data-model.md for keys/defaults). +4. **Add the two new sample files**: + - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API + with `x-backstage-*` fields added) — no pre-existing `catalog-info.yaml`. + - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + + `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. +5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: + - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually + written catalog file (US1, SC-001). + - Its `spec.owner`, `spec.lifecycle`, and tags match the `x-backstage-*` values in the spec + file (US2, SC-002). + - The precedence-demo API's catalog entity reflects the hand-authored + `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). +6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, + confirm the existing entity updates rather than duplicating (FR-003). +7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait + one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. +8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line + identifies the file; temporarily set `x-backstage-owner` to a nonexistent team, confirm a + catalog processing error is visible on that entity (FR-008, SC-003). + +## Out of scope for this quickstart + +- Full JSON-Schema validation of spec files (documented as a learner extension point, not built). +- Real-time file-watching (documented as a deliberate simplification — 30s poll cycle). diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md new file mode 100644 index 0000000..68b2515 --- /dev/null +++ b/specs/004-lab-4-auto-registration/research.md @@ -0,0 +1,136 @@ +# Phase 0 Research: Lab 4 — Auto Registration + +## R1: Discovery mechanism + +**Decision**: A custom Backstage **EntityProvider** (not a `CatalogProcessor`), registered as a +backend module via `catalogProcessingExtensionPoint.addEntityProvider(...)`, following the same +`createBackendModule` pattern already used for the permission policy +(`labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/permissionPolicy.ts`). + +**Rationale**: An `EntityProvider` owns a full, versioned mutation of "the set of entities this +provider currently knows about." On each scheduled run it can emit a `type: 'full'` mutation +containing exactly the API entities currently discoverable — anything omitted from that mutation +is automatically retracted by the catalog. This gives create (FR-002), update (FR-003), and +removal-on-delete (FR-004) for free from one code path, with no separate cleanup logic needed. +A `CatalogProcessor` only transforms entities that already exist via some `Location`; it cannot +originate new entities from an arbitrary filesystem scan on its own. + +**Alternatives considered**: +- GitHub discovery processor (`catalog-backend-module-github`) — rejected: requires a real GitHub + App/token and a reachable remote repo, violating the zero-cost/no-external-network assumption + already recorded in the spec (Assumptions: mono-repo is simulated locally). +- `type: file` catalog locations with glob targets pointing at `catalog-info.yaml` — rejected: + this still requires a hand-authored `catalog-info.yaml` per API, which is precisely what Lab 4 + must eliminate (FR-001). + +## R2: File discovery within the mono-repo + +**Decision**: Use `fast-glob` (already present in the workspace `node_modules`, will be added as +an explicit `dependencies` entry in `packages/backend/package.json`) to scan a configured root +path for `**/*-openapi.yaml` and `**/*-asyncapi.yaml`, on a scheduled interval via +`coreServices.scheduler` (default every 30s — fast enough for interactive lab feedback without +hammering the filesystem). + +**Root path**: configurable via `autoApiRegistration.rootPath` in `app-config.yaml`; documented +as an adaptable convention (FR-010). Default value resolves from the backend process's working +directory (`packages/backend` under `backstage-cli package start`) up to the repository root — +exact relative segment count will be confirmed experimentally during implementation and recorded +here as a Run update, consistent with how Lab 3's research.md captured verified findings. + +**Alternatives considered**: +- `chokidar` file-watcher for real-time push updates — rejected: adds a persistent watch process + and cross-platform edge cases (network drives, WSL) for marginal benefit over a 30s poll in a + local tutorial; the spec's Assumptions already state discovery is a "periodic/on-demand" cycle, + not real-time push. + +## R3: Parsing OpenAPI / AsyncAPI files and extracting `x-*` metadata + +**Decision**: Use `js-yaml` (already present, will be added as an explicit dependency) to parse +each matched file into a plain object. A file is treated as valid input if it parses as +YAML/JSON **and** contains either an `openapi` or `asyncapi` top-level version field **and** an +`info.title` string. Anything else (parse failure, missing both version fields, missing title) is +treated as malformed per the Edge Cases section. + +Recognized `x-*` fields are read from the top-level `info` object for both OpenAPI and AsyncAPI +(per the Clarifications session): +- `info.x-backstage-owner` → `spec.owner` +- `info.x-backstage-lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) +- `info.x-backstage-tags` → `metadata.tags` (default: `[]`, if absent) + +**Rationale**: Full JSON-Schema validation against the OpenAPI/AsyncAPI meta-schemas (e.g. via +`@apidevtools/swagger-parser`, also present in `node_modules`) is available but deliberately not +used for the primary discovery path — it would reject specs with minor structural issues that are +otherwise perfectly usable for catalog registration, and doing full schema validation is out of +scope for what this lab teaches (catalog metadata sourcing, not spec authoring correctness). The +lightweight shape check above is sufficient to distinguish "not an API spec at all" from "a real +spec with metadata problems," which is the distinction FR-008's error handling cares about. + +**Alternatives considered**: +- `@apidevtools/swagger-parser` full validation — rejected as primary path (see rationale above); + noted as a documented extension point for learners in quickstart.md. + +## R4: Owner validation and "visible, actionable error" surfacing + +**Decision**: Errors are surfaced through **two independent, existing Backstage mechanisms** — +no bespoke error-reporting system is built: + +1. **Backend logs**: every discovery cycle logs one line per skipped/errored file (parse failure, + invalid owner reference, name collision) via the standard backend `logger`, already made + visible through the installed `@backstage/plugin-catalog-backend-module-logs`. +2. **Backstage's native processing-error UI**: for errors that occur *after* a candidate entity + has enough shape to be identified (invalid owner ref, name collision), the provider still emits + a minimal entity for that file, tagged with an annotation + (`apiportal-lab.io/registration-error: `). A small companion `CatalogProcessor` + (`preProcessEntity` hook, added to the same backend module) inspects that annotation and throws + a `InputError` when present — Backstage's core catalog processing engine natively records + processor errors per-entity and surfaces them in the entity's own "Inspect entity" / + unprocessed-entities view, which is exactly the built-in "catalog processing errors UI" the + Clarifications session specified. One message, written once, drives both surfaces. + + Files that fail the **shape check in R3** (not parseable as an API spec at all — no + `openapi`/`asyncapi` field, no `info.title`) cannot produce even a minimal entity (no name to + register under) and are therefore logged only; this is called out explicitly in quickstart.md + as the one category of error that is log-only rather than log+UI, and is why FR-008 lists + "fails to parse" separately from the owner/collision cases. + +**Owner validation**: before emitting the full mutation, the provider fetches all existing +`User`/`Group` entities via the in-process `CatalogService` and checks each candidate's +`x-backstage-owner` value resolves to a known entity ref (defaulting kind to `group:default/` if +no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the +repo). Unresolvable owners produce the marker-entity + processor-error path above. + +**Name collisions**: if two discovered files slugify to the same candidate entity name, the first +(sorted by file path for determinism) is registered normally; the second is emitted as a marker +entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the +conflict is independently visible without ever silently overwriting the first. + +**Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a +candidate in its full mutation, the provider queries the catalog for an existing entity of the +same `kind: API` and `metadata.name` that did **not** originate from this provider (i.e. whose +`metadata.annotations['backstage.io/managed-by-origin-location']` differs from our provider's +entity source). If one exists, the auto-sourced candidate is skipped for that name — the +hand-authored entity wins, matching the documented Assumption. + +## R5: New sample API for the "previously unregistered" discovery demo + +**Decision**: Vendor a trimmed copy of the **Scalar Galaxy API** (MIT-licensed, +github.com/scalar/scalar, published build at +`https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as +`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with `x-backstage-owner`, +`x-backstage-lifecycle`, and `x-backstage-tags` fields added under `info` directly in the vendored +copy. The upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, +all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), +covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and +Streetlights examples already used in Labs 1–3, satisfying Constitution Principle VII. + +A second, small hand-crafted OpenAPI file (`precedence-demo-openapi.yaml`, ~2 paths) is added +alongside a hand-authored `precedence-demo-catalog-info.yaml` in +`labs/lab-04-auto-registration/apis/precedence-demo/`, specifically to exercise the FR-011 +precedence rule end-to-end without touching any Lab 1–3 registered API. + +**Alternatives considered**: +- Reusing an existing Lab 1–3 API for the discovery demo — rejected: it is already registered via + a hand-authored `catalog-info.yaml`, so it cannot authentically demonstrate "a previously + unregistered API gets discovered" (User Story 1's core scenario). +- Redocly's other example repos (TicTacToe, etc.) as backup — not needed; Scalar Galaxy confirmed + suitable on inspection. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index 4b273ca..b914754 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -8,6 +8,16 @@ **Input**: User description: "Create new feature and specification from feature 4 in GOAL.md" — Lab 4 (Auto registration): Setup a process for automatic registration of APIs and creation of catalog metadata. Assume API definition files exist within a Git mono-repo. Include sourcing of metadata from x-* fields, within the API specifications themselves, into catalog metadata, fields and annotations. +## Clarifications + +### Session 2026-07-02 + +- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) +- Q: How is the mono-repo discovery scope defined — fixed folder(s) or filename pattern? → A: No fixed folder; discovery matches a filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) anywhere in the repo +- Q: Where should discovery/registration errors be surfaced to the learner? → A: Both Backstage's built-in catalog processing errors UI (entity-level) and backend log output +- Q: How is the catalog entity's name derived from the API definition file? → A: From the spec's own `info.title` field, slugified (not the file path, not a separate `x-*` name field) +- Q: Where do recognized `x-*` metadata fields live within an AsyncAPI document, relative to OpenAPI? → A: Same placement as OpenAPI — under the document's top-level `info` object for both formats + ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) @@ -60,7 +70,7 @@ As a learner adapting this repository to my own organization, I want to understa ### Edge Cases - What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. -- What happens when two API definition files resolve to the same catalog entity name? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. +- What happens when two API definition files resolve to the same catalog entity name (i.e., the same slugified `info.title`)? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. - What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. - What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. - How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. @@ -69,14 +79,14 @@ As a learner adapting this repository to my own organization, I want to understa ### Functional Requirements -- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) located within a designated mono-repo folder structure, without requiring a hand-authored `catalog-info.yaml` per API. +- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) anywhere within the mono-repo by matching a documented filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) rather than requiring a fixed folder location, and without requiring a hand-authored `catalog-info.yaml` per API. - **FR-002**: The system MUST automatically create a catalog API entity for each newly discovered API definition file. - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields and map them into the corresponding catalog entity fields or annotations. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields — located under the document's top-level `info` object for both OpenAPI and AsyncAPI — and map them into the corresponding catalog entity fields or annotations. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. -- **FR-008**: The system MUST surface a visible, actionable error when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition — and MUST continue processing remaining, unaffected API definitions. +- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. - **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. - **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. @@ -86,8 +96,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities - **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. -- **Mono-repo Location**: The designated folder structure/convention within which API Definition Files are expected to live, used as the scope for discovery. +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name is derived from the spec's own `info.title` field (slugified), and it carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. - **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. ## Success Criteria *(mandatory)* @@ -96,15 +106,15 @@ As a learner adapting this repository to my own organization, I want to understa - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. - **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. -- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error, exactly which file and problem caused registration to fail, without needing to inspect system internals. +- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. ## Assumptions -- The "Git mono-repo" referenced in Lab 4 is simulated as a folder/convention within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. +- The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- Default `x-*` field names (e.g., for owning team, lifecycle, tags) will be defined by this lab as a documented convention; learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- Default `x-*` field names are defined by this lab as `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` (namespaced under `x-backstage-*` to avoid collision with other vendor extensions); learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 1d17a2b737e47c23619e29ea4e93d50bc8544390 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 07:48:54 +1000 Subject: [PATCH 03/12] Update plan to support scaling to real world portal API and scan volumes --- .../004-lab-4-auto-registration/data-model.md | 49 +++++-- specs/004-lab-4-auto-registration/plan.md | 47 +++++-- .../004-lab-4-auto-registration/quickstart.md | 16 ++- specs/004-lab-4-auto-registration/research.md | 132 ++++++++++++++++-- 4 files changed, 207 insertions(+), 37 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index abe48d4..f75bcc4 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -40,18 +40,45 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta |---|---|---|---| | `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | | `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | -| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | How often the EntityProvider re-scans. | +| `autoApiRegistration.ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | +| `autoApiRegistration.mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). | +| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | +| `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | +| `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | | `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | +## Scan-state cache (persisted, `coreServices.database`) + +A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart +and (b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2). +Not a catalog entity, not written back into the mono-repo — internal backend state only. + +| Column | Type | Purpose | +|---|---|---| +| `file_path` | string (PK) | Absolute path of the discovered file. | +| `mtime_ms` | number | Last-observed file modification time; cheap first-pass change check on restart/reconciliation. | +| `content_hash` | string | sha1 of file bytes; only computed/compared when `mtime_ms` differs, since hashing every file on every check is itself costly at 1000+ files. | +| `entity_name` | string | The catalog entity name this file currently maps to — doubles as the slug→path collision index (research.md R4). | +| `last_error` | string \| null | Last registration error for this file, if any (mirrors the `apiportal-lab.io/registration-error` annotation; lets a restart re-emit the same marker entity without re-deriving the error). | + ## State transitions (per discovery cycle) -1. Glob scan → candidate file list. -2. Parse + shape-check each file → valid candidates / log-only errors (unparseable files). -3. Slugify `info.title` → candidate entity name; detect duplicates among this cycle's candidates - → second-and-later duplicates become marker/error entities. -4. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners become - marker/error entities. -5. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored - `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). -6. Emit a `type: 'full'` mutation containing every entity produced by steps 2–5 (normal + - marker/error entities). Entities not included are retracted by the catalog (FR-004). +1. **First run only**: glob scan (streamed, ignore patterns applied) → parse + shape-check every + matched file → build the initial scan-state cache → emit one `type: 'full'` mutation + establishing the baseline with the catalog (research.md R6). +2. **Every subsequent cycle** (`poll` mode: full sweep on `schedule.frequencySeconds`; `watch` + mode: per-event, plus a full sweep on `reconciliation.frequencySeconds`): + a. Glob scan (or, in `watch` mode, the single changed path from the watcher event). + b. For each candidate, `stat()` and compare `mtime_ms` against the cache; only files that + differ (or are new) proceed to content-hash comparison and re-parse — unchanged files are + skipped entirely (research.md R6, Problem 2). + c. Parse + shape-check changed/new files → valid candidates / log-only errors. + d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index + (O(1) lookup, not a full re-sort) → collisions become marker/error entities. + e. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners + become marker/error entities. + f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored + `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). + g. Diff the resulting candidate set against the cache → update cache rows → emit a + `type: 'delta'` mutation (`added`/`removed`) for just what changed (FR-002/003/004), + **not** a fresh full mutation (research.md R6, Problem 1). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 72e0dbf..802bec2 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -10,7 +10,14 @@ Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replac per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend `EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / `*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s -schedule — no catalog-info.yaml required per API (FR-001–FR-004). Catalog metadata (owner, +poll schedule for the lab's small sample set — no catalog-info.yaml required per API +(FR-001–FR-004). The same mechanism is designed to hold up at real-world mono-repo scale +(1000+ spec files, 1GB+ repos): a persisted scan-state cache (`coreServices.database`) lets a +restart skip re-parsing unchanged files, delta (not full) mutations are emitted after the +first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the +steady-state discovery signal — see research.md R6 for the full scaling rationale; only the +config *values* differ between the lab demo and a production-scale deployment, not the +architecture. Catalog metadata (owner, lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) @@ -35,6 +42,9 @@ adaptable conventions per Constitution Principle VIII (FR-010, US3). in the workspace, promoted to an explicit dependency for the filesystem scan (research.md R2). - `js-yaml@4.2.0` — new direct dependency of `packages/backend`; already present transitively, promoted to explicit for parsing candidate spec files (research.md R3). +- `chokidar` — new direct dependency of `packages/backend`; already present transitively; used + only when `autoApiRegistration.mode: 'watch'` (not the lab default), to give the discovery + mechanism a steady-state path that scales past the lab's poll-based default (research.md R6). - `@backstage/plugin-catalog-node` (already installed, `2.2.1`) — supplies `EntityProvider`, `CatalogProcessor`, and `catalogProcessingExtensionPoint` used to register the new backend module, following the same `createBackendModule` pattern as @@ -42,9 +52,13 @@ adaptable conventions per Constitution Principle VIII (FR-010, US3). - No new frontend dependencies — this lab is backend-only; no new UI surfaces are added (errors ride on Backstage's existing built-in processing-error UI). -**Storage**: YAML files only — two new vendored API spec files, one new hand-authored -`catalog-info.yaml` (precedence demo), and one `app-config.yaml` addition (new -`autoApiRegistration` block). No database schema changes. +**Storage**: YAML files (two new vendored API spec files, one new hand-authored +`catalog-info.yaml` for the precedence demo, one `app-config.yaml` addition) plus one new +backend-owned database table (`autoApiRegistration` scan-state cache, via `coreServices.database` +— same pluggable SQLite/Postgres service the catalog already uses) that persists per-file +mtime/hash/entity-name/error state across restarts so a real-world-scale mono-repo doesn't +require a full re-parse on every backend start (research.md R6, data-model.md). Not a schema +change to any existing Backstage table. **Testing**: Manual browser + log verification per lab README (no automated test suite — consistent with Labs 1–3's tutorial-lab testing approach). @@ -55,17 +69,21 @@ consistent with Labs 1–3's tutorial-lab testing approach). TypeScript backend module (EntityProvider + CatalogProcessor). **Performance Goals**: New/changed API definitions appear in the catalog within one discovery -cycle (≤30s, SC-001); scan cost is bounded by mono-repo size, acceptable at lab scale (a handful -of spec files). +cycle (≤30s at lab scale, SC-001). Mechanism-level goal (not exercised by the lab itself, but +designed for per research.md R6): a real mono-repo of 1000+ spec files across 1GB+ should incur +parse cost proportional to *changed* files, not total repo size, on both restart (persisted +scan-state cache) and steady-state discovery (watch mode + delta mutations). **Constraints**: Zero cost; cross-platform; no external network access beyond the one-time vendor of the Scalar Galaxy API file at authoring time (the vendored copy itself requires no runtime network access, per Constitution Principle VII); builds on Labs 1–3 without disrupting existing registered APIs (museum, streetlights, train-travel) or the Lab 3 quality tooling. -**Scale/Scope**: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored -`catalog-info.yaml`, 1 new backend module (~2 files: EntityProvider+processor and its -registration), 1 `app-config.yaml` config block, 3 existing registered APIs left untouched. +**Scale/Scope**: Lab demo: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored +`catalog-info.yaml`, 1 new backend module (~3 files: EntityProvider+processor, DB migration for +the scan-state cache, and registration), 1 `app-config.yaml` config block, 3 existing registered +APIs left untouched. Designed-for scale (config change only, no code change, per research.md R6): +1000+ spec files, 1GB+ mono-repo size. ## Constitution Check @@ -127,13 +145,16 @@ but the module source itself IS committed since it is lab teaching content): # In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): app-config.yaml # Modified: add autoApiRegistration config block packages/backend/ -├── package.json # Modified: add fast-glob, js-yaml +├── package.json # Modified: add fast-glob, js-yaml, chokidar └── src/ ├── index.ts # Modified: backend.add(import('./extensions/autoApiRegistration')) └── extensions/ - └── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend - # module (discovery, parsing, mapping, owner - # validation, precedence check, error marking) + ├── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend + │ # module (discovery, parsing, mapping, owner + │ # validation, precedence check, error marking, + │ # poll/watch mode, delta mutations) + └── autoApiRegistrationMigrations/ # New: coreServices.database migration creating + └── 001_scan_state_cache.ts # the scan-state cache table (research.md R6) ``` **Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–3. All diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 053a942..320c2e0 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -42,4 +42,18 @@ Constitution Principle II. ## Out of scope for this quickstart - Full JSON-Schema validation of spec files (documented as a learner extension point, not built). -- Real-time file-watching (documented as a deliberate simplification — 30s poll cycle). +- Exercising `watch` mode or the reconciliation safety net end-to-end (the lab's 2–3 sample files + don't demonstrate anything a poll cycle wouldn't; `mode: 'watch'` is documented, not walked + through step-by-step). + +## Scaling beyond the lab (documented, not walked through) + +The README includes a "Scaling to a real mono-repo" note pointing learners at: +- Why the default is `mode: 'poll'` + `type: 'full'`-then-`delta'` mutations, and what changes + (`mode: 'watch'`, larger `reconciliation.frequencySeconds`, mandatory `ignore` patterns) at + 1000+ files / 1GB+ (research.md R6). +- Why scan state is persisted in the backend's own database rather than written back into the + mono-repo as generated catalog files (data-model.md, research.md R6 Problem 2). +- That restart behavior at scale relies on the persisted cache, not a full re-scan — this is + called out explicitly since it's the part most likely to surprise a learner who scales this + lab up and then wonders why a restart is instant instead of taking minutes. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 68b2515..ba26fd6 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -7,13 +7,12 @@ backend module via `catalogProcessingExtensionPoint.addEntityProvider(...)`, fol `createBackendModule` pattern already used for the permission policy (`labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/permissionPolicy.ts`). -**Rationale**: An `EntityProvider` owns a full, versioned mutation of "the set of entities this -provider currently knows about." On each scheduled run it can emit a `type: 'full'` mutation -containing exactly the API entities currently discoverable — anything omitted from that mutation -is automatically retracted by the catalog. This gives create (FR-002), update (FR-003), and -removal-on-delete (FR-004) for free from one code path, with no separate cleanup logic needed. -A `CatalogProcessor` only transforms entities that already exist via some `Location`; it cannot -originate new entities from an arbitrary filesystem scan on its own. +**Rationale**: An `EntityProvider` owns a versioned mutation of "the set of entities this provider +currently knows about." Anything it does not currently claim is retracted by the catalog. This +gives create (FR-002), update (FR-003), and removal-on-delete (FR-004) for free from one code +path, with no separate cleanup logic needed. A `CatalogProcessor` only transforms entities that +already exist via some `Location`; it cannot originate new entities from an arbitrary filesystem +scan on its own. **Alternatives considered**: - GitHub discovery processor (`catalog-backend-module-github`) — rejected: requires a real GitHub @@ -23,6 +22,15 @@ originate new entities from an arbitrary filesystem scan on its own. this still requires a hand-authored `catalog-info.yaml` per API, which is precisely what Lab 4 must eliminate (FR-001). +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: the lab defaults use a `type: 'full'` +mutation on every cycle because it is the simplest thing to explain and the entity count is tiny. +That does **not** scale: a `full` mutation re-transmits and re-diffs every known entity on every +cycle, so its cost grows with total catalog size, not with how much actually changed. The +mechanism as designed (an `EntityProvider` driving mutations) scales fine — what must change is +*which mutation type it sends*. See R6 for the delta-mutation and persisted-cache design that +keeps the same `EntityProvider` architecture viable at scale; the lab's own config simply doesn't +need to turn those knobs on for 2–3 sample files. + ## R2: File discovery within the mono-repo **Decision**: Use `fast-glob` (already present in the workspace `node_modules`, will be added as @@ -37,11 +45,28 @@ directory (`packages/backend` under `backstage-cli package start`) up to the rep exact relative segment count will be confirmed experimentally during implementation and recorded here as a Run update, consistent with how Lab 3's research.md captured verified findings. +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: two changes are load-bearing at scale +and are therefore built into the mechanism now, not deferred as a "someday" rewrite, even though +the lab's default config doesn't need them: +1. **Ignore patterns are mandatory, not optional.** `fast-glob` is called with an `ignore` list + (`**/node_modules/**`, `**/.git/**`, `**/dist/**`, `**/build/**`, plus a configurable + `autoApiRegistration.ignore` extension point) from day one. Without this, a glob walk over a + 1GB tree spends nearly all its time inside dependency/build directories that will never contain + a hand-authored spec file. This is a correctness-adjacent default, not a performance tune-up — + it is documented and defaulted in the lab even though the lab repo itself has few such + directories at the discovery root, so learners inherit a safe default rather than discovering + the need for it the hard way at 1000 files. +2. **Streamed, not buffered, matching.** Use `fastGlob.stream(...)` and process results as they + arrive rather than `fastGlob(...)`'s array form, so peak memory during a scan is bounded by + in-flight parse concurrency (see R6), not by total file count. + **Alternatives considered**: -- `chokidar` file-watcher for real-time push updates — rejected: adds a persistent watch process - and cross-platform edge cases (network drives, WSL) for marginal benefit over a 30s poll in a - local tutorial; the spec's Assumptions already state discovery is a "periodic/on-demand" cycle, - not real-time push. +- `chokidar` file-watcher for real-time push updates — rejected *as the lab default* (adds a + persistent watch process and cross-platform edge cases — network drives, WSL, inotify limits on + very large trees — for marginal benefit over a 30s poll against 2–3 files). Reconsidered and + **adopted as the scale-up path** in R6: at 1000+ files, repeated full-tree poll walks are the + wrong steady-state mechanism, and a watcher layered over the same ignore-pattern set is what + makes the "how often do we rescan" question stop scaling with repo size. ## R3: Parsing OpenAPI / AsyncAPI files and extracting `x-*` metadata @@ -102,7 +127,10 @@ repo). Unresolvable owners produce the marker-entity + processor-error path abov **Name collisions**: if two discovered files slugify to the same candidate entity name, the first (sorted by file path for determinism) is registered normally; the second is emitted as a marker entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the -conflict is independently visible without ever silently overwriting the first. +conflict is independently visible without ever silently overwriting the first. Collision detection +is done against the persisted slug→path index described in R6, not by re-deriving and sorting the +full candidate list on every check — at 1000+ files, an O(N log N) sort per cycle (or per watch +event, in watch mode) is avoidable work; an index lookup is O(1). **Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a candidate in its full mutation, the provider queries the catalog for an existing entity of the @@ -134,3 +162,83 @@ precedence rule end-to-end without touching any Lab 1–3 registered API. unregistered API gets discovered" (User Story 1's core scenario). - Redocly's other example repos (TicTacToe, etc.) as backup — not needed; Scalar Galaxy confirmed suitable on inspection. + +## R6: Scalability & restart behavior at real-world mono-repo scale + +The lab itself discovers 2–3 files in a small repo, and its default config values (30s full poll, +no persistence) are chosen for that scale — they are explicitly **not** what a 1000+ file, 1GB+ +mono-repo should run with. This section records the mechanism-level decisions that keep the same +architecture (EntityProvider + fast-glob) viable at that scale, so scaling up is a config change +plus turning on code paths that already exist, not a rewrite. + +**Problem 1 — full mutations don't scale with catalog size.** A `type: 'full'` `EntityProvider` +mutation re-sends and re-diffs every currently-known entity on every cycle. At 1000+ APIs, most of +which are unchanged between cycles, this wastes network/serialization cost and catalog-processing +time proportional to total catalog size rather than to actual change volume. + +**Decision**: after the provider's first `full` mutation (needed once, to establish the baseline +with the catalog), all subsequent cycles emit `type: 'delta'` mutations (`added`/`removed` only), +computed by diffing the current scan against the persisted scan-state cache (Problem 2). This is +the standard Backstage `EntityProvider` pattern for large, incrementally-changing entity sets (the +same pattern the built-in GitHub discovery provider uses at organization scale) — no custom +protocol is invented. + +**Problem 2 — a cold backend restart re-derives everything from zero.** Backstage's catalog +database already persists previously-ingested *entities* across restarts (a provider's entities +are not deleted until that provider explicitly retracts them), so the catalog itself doesn't go +empty on restart. What is **not** persisted anywhere by default is the provider's own *scan +state* — which files it has seen, their content hashes, and what they mapped to — so without +additional persistence, every restart forces a full re-glob and full re-parse of the entire +mono-repo (1000+ files) before the provider can even confirm nothing changed. + +**Decision**: persist scan state — `filePath`, `mtimeMs`, `contentHash` (sha1 of file bytes), +`entityName`, `lastError` — as rows in a small table owned by this backend module, using +`coreServices.database` (the same pluggable database service Backstage's own catalog uses; +SQLite in the lab, Postgres in a real deployment — zero new infrastructure, consistent with +Principle V). On startup, the provider loads this cache and can validate it against the +filesystem far more cheaply than re-parsing everything: a `stat()` per known path compares +`mtimeMs` (and falls back to a content-hash check only when `mtimeMs` differs, since hashing +1000+ files is itself non-trivial I/O at 1GB scale) before deciding a file needs re-parsing. +Unchanged files contribute zero parse work to a restart. This directly answers "should scanner +results be persisted so restarts restore the catalog faster" — yes, but as internal scan-state +cache rows in the backend's own database, **not** as generated `catalog-info.yaml`-style files +written back into the mono-repo. Writing discovery output back into the source tree would (a) +reintroduce the hand-maintained-duplicate-file problem Lab 4 exists to eliminate, and (b) require +the discovery tool to have write access to a repo it should only need to read — a materially +larger and riskier permission footprint at real-world scale. + +**Problem 3 — polling a 1GB tree every 30s doesn't scale as the primary discovery loop.** Even +with ignore patterns (R2) and streaming (R2), a full `fast-glob` walk of a very large tree still +costs real I/O, and running that walk every 30s regardless of whether anything changed is wasted +work at scale — the lab's 30s interval exists for interactive demo feedback on 2–3 files, not +because 30s is an appropriate steady-state cadence for 1000+ files. + +**Decision**: at scale, discovery should run in **watch mode**: a `chokidar` watcher (rejected for +the lab default in R2, adopted here) observes `add`/`change`/`unlink` events under the same +ignore-pattern set, and each event triggers a targeted single-file re-parse + delta mutation +(update the scan-state cache row, emit one delta) rather than a full-tree rescan. The periodic +`fast-glob` sweep is retained but demoted to an infrequent **reconciliation safety net** +(independently configurable interval, e.g. minutes not seconds) that catches anything a watcher +event could plausibly miss (coalesced events under a large batch change such as a big `git pull`, +or watch reliability limits on some network/CI filesystems) — it is a correctness backstop, not +the primary discovery signal. Parsing is concurrency-limited (bounded worker count, default small) +so a large batch of simultaneous file-change events — e.g. after checking out a branch that +touches hundreds of spec files — doesn't spike memory or block the Node event loop. + +**Config surface** (see data-model.md): `autoApiRegistration.mode: 'poll' | 'watch'` (lab default: +`poll`, no watcher, matching the simple demo story in quickstart.md); `ignore` (glob exclusions, +always applied regardless of mode); `reconciliation.frequencySeconds` (decoupled from watch-event +responsiveness); `parseConcurrency`. Switching a real deployment to scale is: set `mode: 'watch'`, +raise `reconciliation.frequencySeconds` from 30 to something like 900–1800, and rely on the +already-built delta-mutation + persisted-cache path — no architectural change. + +**Alternatives considered**: +- Always persisting a `full` mutation every cycle regardless of scale — rejected: the cost model + is wrong for 1000+ entities (see Problem 1); would work fine at lab scale but silently becomes a + performance cliff at real-world scale, which is exactly the trap this section exists to avoid. +- Writing discovered metadata back into the mono-repo as generated catalog files, to "cache" scan + results — rejected (see Problem 2 rationale): wrong persistence layer, reintroduces duplication, + expands write-access footprint unnecessarily. +- Only a filesystem watcher, no periodic reconciliation sweep — rejected: watchers can miss events + (coalescing, restart races, some CI/network filesystem limitations); a slow safety-net sweep is + cheap insurance at any scale and is what makes watch mode trustworthy enough to recommend. From 2957f1191ff65415becd0e7d7f8fb0a53151581c Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:09:30 +1000 Subject: [PATCH 04/12] Revise x-* naming scheme for best practices --- .../004-lab-4-auto-registration/data-model.md | 21 ++++++------ specs/004-lab-4-auto-registration/plan.md | 14 ++++---- .../004-lab-4-auto-registration/quickstart.md | 8 ++--- specs/004-lab-4-auto-registration/research.md | 34 ++++++++++++++----- specs/004-lab-4-auto-registration/spec.md | 22 +++++++----- 5 files changed, 62 insertions(+), 37 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index f75bcc4..1744f2d 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -11,11 +11,11 @@ A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncap | Field | Type | Notes | |---|---|---| | `openapi` / `asyncapi` | string | Version field; presence of either identifies the spec type. Exactly one is expected. | -| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. | -| `info.description` | string | Optional. Maps to `metadata.description` if present. | -| `info.x-backstage-owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration proceeds with no `spec.owner` override... actually **required by FR-005/FR-007's documented default** — see Mapping Rule 1 below. | -| `info.x-backstage-lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | -| `info.x-backstage-tags` | string[] | Optional. Default: `[]`. | +| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. Not duplicated into `x-examplecorp`. | +| `info.description` | string | Optional. Maps to `metadata.description` if present. Not duplicated into `x-examplecorp`. | +| `tags[].name` | string[] | Optional, native top-level OpenAPI/AsyncAPI field (not under `info`, not under `x-examplecorp`). Default `[]` if absent. | +| `info.x-examplecorp.owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration still succeeds using a documented default — see Mapping Rule 1 below. | +| `info.x-examplecorp.lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | ## Entity: Catalog API Entity (output) @@ -26,12 +26,12 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | | `metadata.title` | `info.title` | Verbatim. | | `metadata.description` | `info.description` | Verbatim if present, else omitted. | -| `metadata.tags` | `info.x-backstage-tags` | Default `[]` if absent. | +| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | -| `spec.lifecycle` | `info.x-backstage-lifecycle` | Default `experimental` if absent (FR-007). | -| `spec.owner` | `info.x-backstage-owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | +| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | | `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | ## Config: `autoApiRegistration` (app-config.yaml) @@ -45,7 +45,8 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | | `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | | `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | -| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-backstage-owner` is absent (FR-007). | +| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-examplecorp.owner` is absent (FR-007). | +| `autoApiRegistration.xNamespace` | string | `'examplecorp'` | The vendor namespace under `info.x-` that this discovery instance reads owner/lifecycle from. Adaptable per FR-010/US3 — this is the single setting a learner changes to rename the convention to their own company. | ## Scan-state cache (persisted, `coreServices.database`) @@ -75,7 +76,7 @@ Not a catalog entity, not written back into the mono-repo — internal backend s c. Parse + shape-check changed/new files → valid candidates / log-only errors. d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index (O(1) lookup, not a full re-sort) → collisions become marker/error entities. - e. Resolve `x-backstage-owner` against known `User`/`Group` entities → unresolvable owners + e. Resolve `x-examplecorp.owner` against known `User`/`Group` entities → unresolvable owners become marker/error entities. f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 802bec2..75c283d 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -17,10 +17,12 @@ restart skip re-parsing unchanged files, delta (not full) mutations are emitted first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the steady-state discovery signal — see research.md R6 for the full scaling rationale; only the config *values* differ between the lab demo and a production-scale deployment, not the -architecture. Catalog metadata (owner, -lifecycle, tags) is sourced from `x-backstage-owner`, `x-backstage-lifecycle`, and -`x-backstage-tags` fields under each spec's `info` object (FR-005–FR-006), with documented -defaults when absent (FR-007). Errors (malformed files, unresolvable owners, name collisions) +architecture. Catalog owner/lifecycle metadata is sourced from a single vendor-namespaced +`info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one +namespace learners rename to their own company (FR-005–FR-006), with documented defaults when +absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), +description (`info.description`), tags (native top-level `tags`) — is sourced from there instead +of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, name collisions) are surfaced via backend logs and, where an entity can be identified, via Backstage's native catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a @@ -98,7 +100,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | | VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | | VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | -| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `x-backstage-*` field names as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | | IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | No violations. Complexity Tracking section omitted. @@ -130,7 +132,7 @@ labs/ └── apis/ ├── galaxy/ │ └── galaxy-openapi.yaml # New: vendored Scalar Galaxy API (MIT), - │ # x-backstage-owner/lifecycle/tags added; + │ # info.x-examplecorp.owner/lifecycle added; │ # no catalog-info.yaml — pure auto-discovery └── precedence-demo/ ├── precedence-demo-openapi.yaml # New: minimal spec for the precedence demo diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 320c2e0..c814db5 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -21,14 +21,14 @@ Constitution Principle II. data-model.md for keys/defaults). 4. **Add the two new sample files**: - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API - with `x-backstage-*` fields added) — no pre-existing `catalog-info.yaml`. + with an `info.x-examplecorp` object added) — no pre-existing `catalog-info.yaml`. - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. 5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually written catalog file (US1, SC-001). - - Its `spec.owner`, `spec.lifecycle`, and tags match the `x-backstage-*` values in the spec - file (US2, SC-002). + - Its `spec.owner` and `spec.lifecycle` match the `info.x-examplecorp` values, and its tags + match the spec's native `tags` array — not duplicated into `x-examplecorp` (US2, SC-002). - The precedence-demo API's catalog entity reflects the hand-authored `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). 6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, @@ -36,7 +36,7 @@ Constitution Principle II. 7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. 8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line - identifies the file; temporarily set `x-backstage-owner` to a nonexistent team, confirm a + identifies the file; temporarily set `info.x-examplecorp.owner` to a nonexistent team, confirm a catalog processing error is visible on that entity (FR-008, SC-003). ## Out of scope for this quickstart diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index ba26fd6..9f05b38 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -76,11 +76,26 @@ YAML/JSON **and** contains either an `openapi` or `asyncapi` top-level version f `info.title` string. Anything else (parse failure, missing both version fields, missing title) is treated as malformed per the Edge Cases section. -Recognized `x-*` fields are read from the top-level `info` object for both OpenAPI and AsyncAPI -(per the Clarifications session): -- `info.x-backstage-owner` → `spec.owner` -- `info.x-backstage-lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) -- `info.x-backstage-tags` → `metadata.tags` (default: `[]`, if absent) +Recognized `x-*` metadata is read from a single vendor-namespaced object nested under the +top-level `info` object, for both OpenAPI and AsyncAPI (per the Clarifications session, +2026-07-03 update): +- `info.x-examplecorp.owner` → `spec.owner` +- `info.x-examplecorp.lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) + +`examplecorp` is this lab's fictional company namespace (documented as the one thing every +learner renames first, per FR-010/US3). The field names deliberately omit `backstage` — this +metadata (who owns an API, what lifecycle stage it's in) is meaningful to any consumer of the +spec, not just this tool, so the namespace names the company, not the tool. + +Metadata with a natural home elsewhere in the spec is read from there instead, never duplicated +into `x-examplecorp`: +- `info.title` → `metadata.title` / entity name (already established, unchanged) +- `info.description` → `metadata.description` (already established, unchanged) +- top-level `tags[].name` → `metadata.tags` (default: `[]`, if the spec has no `tags` array) — + **changed from the original `x-backstage-tags` design**: both OpenAPI and AsyncAPI already + define a native top-level `tags` field for exactly this purpose, so sourcing tags from `x-*` as + well would create two places that could drift apart. `tags[].name` (ignoring + `tags[].description`, which has no catalog equivalent) is the source of truth. **Rationale**: Full JSON-Schema validation against the OpenAPI/AsyncAPI meta-schemas (e.g. via `@apidevtools/swagger-parser`, also present in `node_modules`) is available but deliberately not @@ -120,7 +135,7 @@ no bespoke error-reporting system is built: **Owner validation**: before emitting the full mutation, the provider fetches all existing `User`/`Group` entities via the in-process `CatalogService` and checks each candidate's -`x-backstage-owner` value resolves to a known entity ref (defaulting kind to `group:default/` if +`x-examplecorp.owner` value resolves to a known entity ref (defaulting kind to `group:default/` if no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the repo). Unresolvable owners produce the marker-entity + processor-error path above. @@ -144,9 +159,10 @@ hand-authored entity wins, matching the documented Assumption. **Decision**: Vendor a trimmed copy of the **Scalar Galaxy API** (MIT-licensed, github.com/scalar/scalar, published build at `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as -`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with `x-backstage-owner`, -`x-backstage-lifecycle`, and `x-backstage-tags` fields added under `info` directly in the vendored -copy. The upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, +`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with an `info.x-examplecorp` +object (`owner`, `lifecycle`) added directly in the vendored copy; its existing native `tags` +array is used as-is for `metadata.tags`. The upstream file is a single self-contained OpenAPI 3.1 +document (confirmed: 1469 lines, all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and Streetlights examples already used in Labs 1–3, satisfying Constitution Principle VII. diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index b914754..60e98e2 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -12,12 +12,18 @@ ### Session 2026-07-02 -- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) +- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) — **superseded, see Session 2026-07-03 below** - Q: How is the mono-repo discovery scope defined — fixed folder(s) or filename pattern? → A: No fixed folder; discovery matches a filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) anywhere in the repo - Q: Where should discovery/registration errors be surfaced to the learner? → A: Both Backstage's built-in catalog processing errors UI (entity-level) and backend log output - Q: How is the catalog entity's name derived from the API definition file? → A: From the spec's own `info.title` field, slugified (not the file path, not a separate `x-*` name field) - Q: Where do recognized `x-*` metadata fields live within an AsyncAPI document, relative to OpenAPI? → A: Same placement as OpenAPI — under the document's top-level `info` object for both formats +### Session 2026-07-03 + +- Q: Should `x-*` fields sit directly in `info`, or nested under a single vendor-namespaced object? → A: Nested under a single `info.x-examplecorp` object (e.g. `info.x-examplecorp.owner`, `info.x-examplecorp.lifecycle`) — `examplecorp` is this lab's fictional company namespace, which learners are expected to rename to their own; a single grouping object avoids polluting `info` with multiple top-level `x-*` keys and makes "which fields are ours" unambiguous at a glance. +- Q: Should field names retain the `backstage` tool name (`x-backstage-owner`, etc.)? → A: No — the field names describe the metadata (owner, lifecycle), not the tool consuming it. Catalog metadata like ownership is tool-agnostic (equally meaningful to an API gateway or any other consumer), so `backstage` is dropped: `x-examplecorp.owner`, `x-examplecorp.lifecycle`. +- Q: Should `tags` remain a recognized `x-*` field? → A: No — removed. OpenAPI/AsyncAPI both already have a native top-level `tags` field; sourcing tags from `x-*` as well would duplicate data that already has a natural home in the spec, risking divergence. Catalog `metadata.tags` is now sourced from the spec's native `tags` array (`tags[].name`), not from any `x-*` field. The same reasoning already applied to `info.title` (entity name) and `info.description` (entity description), which were never duplicated into `x-*` fields. + ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) @@ -47,7 +53,7 @@ As an API owner, I want to declare metadata that Backstage needs (such as owning **Acceptance Scenarios**: 1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. -2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, tags, system grouping), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. 3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. 4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. @@ -84,7 +90,7 @@ As a learner adapting this repository to my own organization, I want to understa - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and tags/labels) from documented `x-*` fields — located under the document's top-level `info` object for both OpenAPI and AsyncAPI — and map them into the corresponding catalog entity fields or annotations. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. - **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. @@ -95,17 +101,17 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities -- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing both the technical API contract and `x-*` vendor extension fields carrying catalog metadata (owning team, lifecycle, tags, etc.). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name is derived from the spec's own `info.title` field (slugified), and it carries ownership, lifecycle, and tag metadata sourced from that file's `x-*` fields. +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle metadata sourced from `info.x-examplecorp`. - **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. -- **`x-*` Metadata Mapping**: The documented set of vendor extension field names and their corresponding catalog entity fields/annotations, including default values applied when a field is absent. +- **`x-*` Metadata Mapping**: The documented mapping from the `info.x-examplecorp` object's fields to catalog entity fields/annotations, including default values applied when a field is absent, and the explicit list of fields sourced natively rather than from `x-*` (name, description, tags). ## Success Criteria *(mandatory)* ### Measurable Outcomes - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. -- **SC-002**: 100% of the owning-team, lifecycle, and tag metadata visible on a registered API's catalog entity originates from `x-*` fields in its definition file, with zero manual duplication of that metadata elsewhere. +- **SC-002**: 100% of the owning-team and lifecycle metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. - **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. @@ -114,7 +120,7 @@ As a learner adapting this repository to my own organization, I want to understa - The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- Default `x-*` field names are defined by this lab as `x-backstage-owner`, `x-backstage-lifecycle`, and `x-backstage-tags` (namespaced under `x-backstage-*` to avoid collision with other vendor extensions); learners are expected to adapt these names to their own organization's practices, per Constitution Principle VIII. +- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner` and `lifecycle` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From 41b24620a25bc4732efb9685d2ab1f91f3ed9153 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:24:01 +1000 Subject: [PATCH 05/12] Added ability to support multiple repos --- .../004-lab-4-auto-registration/data-model.md | 112 ++++++++++++------ specs/004-lab-4-auto-registration/plan.md | 23 ++-- .../004-lab-4-auto-registration/quickstart.md | 15 +++ specs/004-lab-4-auto-registration/research.md | 99 +++++++++++++++- 4 files changed, 204 insertions(+), 45 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index 1744f2d..091ec77 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -31,55 +31,99 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | -| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using a documented default owner ref (`group:default/platform-team`, the same shared/fallback team introduced in Lab 3), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | | `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | ## Config: `autoApiRegistration` (app-config.yaml) -| Key | Type | Default | Purpose | +The canonical shape is a list of sources (research.md R7), each independently configured. A flat +`autoApiRegistration.{rootPath,patterns,...}` (no `sources` key) is shorthand for a single +implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so a +single-repo setup never has to write the list form. + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo # required, unique; becomes part of the EntityProvider name + # and the scan-state cache's source_id column + rootPath: ... + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + ignore: ['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**'] + mode: poll # or 'watch' — independent per source (research.md R6/R7) + schedule: { frequencySeconds: 30 } + reconciliation: { frequencySeconds: 900 } # watch mode only + parseConcurrency: 4 + defaultOwner: group:default/platform-team + xNamespace: examplecorp + - id: team-checkout-api # a second source: a different team's repo, a local sibling + rootPath: ../team-checkout-api # checkout for the lab; a real remote repo's + patterns: ['**/*-openapi.yaml'] # synced local worktree in production (R7 + defaultOwner: group:default/checkout-team # Problem 4) + xNamespace: checkoutteam +``` + +| Key (per source, under `sources[]`, or flat for the single-source shorthand) | Type | Default | Purpose | |---|---|---|---| -| `autoApiRegistration.rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. | -| `autoApiRegistration.patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010. | -| `autoApiRegistration.ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | -| `autoApiRegistration.mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). | -| `autoApiRegistration.schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | -| `autoApiRegistration.reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | -| `autoApiRegistration.parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | -| `autoApiRegistration.defaultOwner` | string | `group:default/platform-team` | Fallback owner ref applied when `x-examplecorp.owner` is absent (FR-007). | -| `autoApiRegistration.xNamespace` | string | `'examplecorp'` | The vendor namespace under `info.x-` that this discovery instance reads owner/lifecycle from. Adaptable per FR-010/US3 — this is the single setting a learner changes to rename the convention to their own company. | +| `id` | string | `'default'` (flat-config shorthand only) | Unique source identifier. Required when using the `sources[]` list form. Used as the `EntityProvider` name suffix and the scan-state cache's `source_id` (research.md R7). | +| `rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. In production, may point at a locally-synced worktree of a genuinely remote repo (research.md R7, Problem 4) — this schema does not change either way. | +| `patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010, per source. | +| `ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | +| `mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). Independent per source — e.g. a large mono-repo can run `watch` while a small team repo runs `poll` (research.md R7). | +| `schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | +| `reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | +| `parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | +| `defaultOwner` | string | *(no global default — each source must set its own, or inherit `autoApiRegistration.defaults.defaultOwner` if configured)* | Fallback owner ref applied when `x-.owner` is absent (FR-007). Deliberately per-source, not global — a team/pre-production repo plausibly has a different fallback owner than the platform mono-repo (research.md R7, Problem 2). | +| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | ## Scan-state cache (persisted, `coreServices.database`) -A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart -and (b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2). -Not a catalog entity, not written back into the mono-repo — internal backend state only. +A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart, +(b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2), and +(c) support collision/precedence checks across every configured source, not just the one currently +scanning (research.md R7, Problem 3). Not a catalog entity, not written back into any source +repo — internal backend state only. | Column | Type | Purpose | |---|---|---| -| `file_path` | string (PK) | Absolute path of the discovered file. | +| `source_id` | string (PK, part 1) | Which configured source (research.md R7) this row came from — `'default'` for the flat single-source config. Lets a collision/error message name *which repo*, not just which file. | +| `file_path` | string (PK, part 2) | Absolute path of the discovered file, within that source. | | `mtime_ms` | number | Last-observed file modification time; cheap first-pass change check on restart/reconciliation. | | `content_hash` | string | sha1 of file bytes; only computed/compared when `mtime_ms` differs, since hashing every file on every check is itself costly at 1000+ files. | -| `entity_name` | string | The catalog entity name this file currently maps to — doubles as the slug→path collision index (research.md R4). | +| `entity_name` | string, indexed | The catalog entity name this file currently maps to — doubles as the slug→(source, path) collision index (research.md R4), queried **without** filtering by `source_id` so collisions across sources are caught (research.md R7, Problem 3). | | `last_error` | string \| null | Last registration error for this file, if any (mirrors the `apiportal-lab.io/registration-error` annotation; lets a restart re-emit the same marker entity without re-deriving the error). | ## State transitions (per discovery cycle) -1. **First run only**: glob scan (streamed, ignore patterns applied) → parse + shape-check every - matched file → build the initial scan-state cache → emit one `type: 'full'` mutation - establishing the baseline with the catalog (research.md R6). -2. **Every subsequent cycle** (`poll` mode: full sweep on `schedule.frequencySeconds`; `watch` - mode: per-event, plus a full sweep on `reconciliation.frequencySeconds`): - a. Glob scan (or, in `watch` mode, the single changed path from the watcher event). - b. For each candidate, `stat()` and compare `mtime_ms` against the cache; only files that - differ (or are new) proceed to content-hash comparison and re-parse — unchanged files are - skipped entirely (research.md R6, Problem 2). - c. Parse + shape-check changed/new files → valid candidates / log-only errors. +Runs independently **per configured source** (research.md R7) — each source is its own +`EntityProvider` instance with its own schedule/mode, so one source's cycle does not block or +depend on another's. Steps (d)–(f) below query the scan-state cache and catalog **without** +filtering by `source_id`, since collision and precedence checks must be global (research.md R7, +Problem 3) even though discovery itself runs per source. + +1. **First run of a given source**: glob scan (streamed, ignore patterns applied) → parse + + shape-check every matched file → build that source's rows in the scan-state cache → emit one + `type: 'full'` mutation (from that source's `EntityProvider` instance) establishing the + baseline with the catalog (research.md R6). +2. **Every subsequent cycle for that source** (`poll` mode: full sweep on + `schedule.frequencySeconds`; `watch` mode: per-event, plus a full sweep on + `reconciliation.frequencySeconds`): + a. Glob scan of that source's `rootPath` (or, in `watch` mode, the single changed path from the + watcher event). + b. For each candidate, `stat()` and compare `mtime_ms` against that source's cache rows; only + files that differ (or are new) proceed to content-hash comparison and re-parse — unchanged + files are skipped entirely (research.md R6, Problem 2). + c. Parse + shape-check changed/new files, using that source's `xNamespace` → valid candidates / + log-only errors. d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index - (O(1) lookup, not a full re-sort) → collisions become marker/error entities. - e. Resolve `x-examplecorp.owner` against known `User`/`Group` entities → unresolvable owners - become marker/error entities. - f. Check for a pre-existing, non-auto-sourced entity at the same name (hand-authored - `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). - g. Diff the resulting candidate set against the cache → update cache rows → emit a - `type: 'delta'` mutation (`added`/`removed`) for just what changed (FR-002/003/004), - **not** a fresh full mutation (research.md R6, Problem 1). + **across all sources** (O(1) lookup, not a full re-sort) → collisions become marker/error + entities, whether the colliding file is in this source or another one (research.md R7, + Problem 3). + e. Resolve `x-.owner` (falling back to this source's `defaultOwner` if absent) + against known `User`/`Group` entities → unresolvable owners become marker/error entities. + f. Check for a pre-existing, non-auto-sourced entity at the same name anywhere in the catalog + (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name + (FR-011). + g. Diff the resulting candidate set against this source's cache rows → update cache rows → + emit a `type: 'delta'` mutation (`added`/`removed`) from this source's `EntityProvider` + instance for just what changed (FR-002/003/004), **not** a fresh full mutation (research.md + R6, Problem 1). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 75c283d..5f685bf 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -8,16 +8,21 @@ Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replaces the per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend -`EntityProvider` that scans the mono-repo for files matching `*-openapi.yaml` / +`EntityProvider` that scans a mono-repo for files matching `*-openapi.yaml` / `*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s poll schedule for the lab's small sample set — no catalog-info.yaml required per API -(FR-001–FR-004). The same mechanism is designed to hold up at real-world mono-repo scale -(1000+ spec files, 1GB+ repos): a persisted scan-state cache (`coreServices.database`) lets a -restart skip re-parsing unchanged files, delta (not full) mutations are emitted after the -first cycle, and an optional `chokidar` watch mode replaces repeated full-tree polling as the -steady-state discovery signal — see research.md R6 for the full scaling rationale; only the -config *values* differ between the lab demo and a production-scale deployment, not the -architecture. Catalog owner/lifecycle metadata is sourced from a single vendor-namespaced +(FR-001–FR-004). The same mechanism is designed to hold up at real-world scale in two +independent dimensions: **repo size** (1000+ spec files, 1GB+, research.md R6 — a persisted +scan-state cache lets a restart skip re-parsing unchanged files, delta-not-full mutations after +the first cycle, and an optional `chokidar` watch mode replacing repeated full-tree polling as +the steady-state discovery signal) and **repo count** (research.md R7 — config generalizes from +one `rootPath` to a list of independently-configured `sources[]`, each its own `EntityProvider` +instance with its own schedule, default owner, and `x-*` namespace, so a platform mono-repo and +one or more separate team/pre-production repos can each be discovered with their own settings +while collision and `catalog-info.yaml`-precedence checks stay global across all of them). Only +the config *values and entry count* differ between the lab demo (one small source) and a +production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle metadata +is sourced from a single vendor-namespaced `info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one namespace learners rename to their own company (FR-005–FR-006), with documented defaults when absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), @@ -100,7 +105,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | | VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | | VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | -| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; the `sources[]` config shape (research.md R7) is documented as the path to onboarding additional team/pre-production repos with their own defaults, without code changes; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | | IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | No violations. Complexity Tracking section omitted. diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index c814db5..11e29ab 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -57,3 +57,18 @@ The README includes a "Scaling to a real mono-repo" note pointing learners at: - That restart behavior at scale relies on the persisted cache, not a full re-scan — this is called out explicitly since it's the part most likely to surprise a learner who scales this lab up and then wonders why a restart is instant instead of taking minutes. + +A second "Scaling to multiple source repositories" note (research.md R7) covers: +- The `autoApiRegistration.sources[]` config list — the lab's own `app-config.yaml` uses the flat + single-source shorthand, but the README shows the equivalent explicit `sources: [{ id: default, + ... }]` form alongside a second example entry, so learners can see exactly what changes to add a + team or pre-production repo (a different `rootPath`, `defaultOwner`, and `xNamespace`). +- Why `defaultOwner` and `xNamespace` are per-source with no built-in global fallback (a repo + onboarding to auto-registration shouldn't have to adopt the platform mono-repo's team or vendor + namespace to participate). +- Why collision detection and `catalog-info.yaml` precedence checks are explicitly called out as + *global* across sources, not per-source — two teams' repos producing the same entity name must + still surface a visible conflict, not silently coexist. +- That onboarding a genuinely separate remote (not a local sibling checkout) requires a sync step + (e.g. scheduled shallow `git clone`/`pull`) ahead of the existing discovery pipeline, which the + lab does not build or require — `rootPath` always points at something already on local disk. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 9f05b38..9f201bc 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -44,6 +44,8 @@ as an adaptable convention (FR-010). Default value resolves from the backend pro directory (`packages/backend` under `backstage-cli package start`) up to the repository root — exact relative segment count will be confirmed experimentally during implementation and recorded here as a Run update, consistent with how Lab 3's research.md captured verified findings. +(This is the single-source shorthand; R7 generalizes `rootPath` into a per-entry field of an +`autoApiRegistration.sources[]` list for organizations with more than one source repository.) **Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: two changes are load-bearing at scale and are therefore built into the mechanism now, not deferred as a "someday" rewrite, even though @@ -145,14 +147,18 @@ entity (suffixed, e.g. `-collision`) carrying the registration-error annot conflict is independently visible without ever silently overwriting the first. Collision detection is done against the persisted slug→path index described in R6, not by re-deriving and sorting the full candidate list on every check — at 1000+ files, an O(N log N) sort per cycle (or per watch -event, in watch mode) is avoidable work; an index lookup is O(1). +event, in watch mode) is avoidable work; an index lookup is O(1). **This index and check are +global, across every configured source, not scoped to the source currently being processed** — see +R7 Problem 3 for why two different teams' repos colliding must not go undetected. **Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a candidate in its full mutation, the provider queries the catalog for an existing entity of the same `kind: API` and `metadata.name` that did **not** originate from this provider (i.e. whose `metadata.annotations['backstage.io/managed-by-origin-location']` differs from our provider's entity source). If one exists, the auto-sourced candidate is skipped for that name — the -hand-authored entity wins, matching the documented Assumption. +hand-authored entity wins, matching the documented Assumption. This check, like collision +detection, runs against catalog state as a whole and is unaffected by which source is being +processed — a hand-authored file anywhere wins over an auto-sourced candidate from any source. ## R5: New sample API for the "previously unregistered" discovery demo @@ -258,3 +264,92 @@ already-built delta-mutation + persisted-cache path — no architectural change. - Only a filesystem watcher, no periodic reconciliation sweep — rejected: watchers can miss events (coalescing, restart races, some CI/network filesystem limitations); a slow safety-net sweep is cheap insurance at any scale and is what makes watch mode trustworthy enough to recommend. + +## R7: Multiple source repositories (platform mono-repo + team/pre-production repos) + +The lab itself discovers files from one location. A real organization is unlikely to have every +API definition in the platform team's mono-repo — other teams, and pre-production/experimental +APIs, plausibly live in their own separate repositories, each wanting different discovery +settings (in particular, a different default owner — Problem 2 below). The mechanism must +generalize to N independently-configured sources without becoming N copies of the same code. + +**Problem 1 — a single `rootPath` config key can't express "scan these several, separately +configured locations."** The R1–R6 design (and the lab's `autoApiRegistration.*` config) assumes +exactly one root and one set of settings. + +**Decision**: restructure config around a list, `autoApiRegistration.sources: [...]`, where each +entry carries its own `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`/`reconciliation`, +`defaultOwner`, and `xNamespace` (data-model.md has the full per-source schema). At backend-module +init time, the module reads this list and instantiates **one `EntityProvider` instance per +source**, each with a distinct `getProviderName()` (`auto-api-registration:`) — not one +provider trying to juggle multiple roots internally. This is the same pattern Backstage's own +multi-target providers use (e.g. a GitHub discovery provider configured against several +org/repo targets each becomes its own provider instance): each source gets its own independent +scheduling, its own independent full-then-delta mutation lifecycle (R6), and a config or scanning +problem in one source (e.g. an unreadable path for a team repo that hasn't been checked out yet) +cannot stall or break discovery for any other source. For backward/simple compatibility, a flat +`autoApiRegistration.rootPath`/`.patterns`/etc. (no `sources` list) is treated as shorthand for a +single implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so +the single-repo lab config does not need to change shape. + +**Problem 2 — some settings are legitimately per-repo, not global.** The two called out +explicitly: `defaultOwner` (a pre-production or team repo plausibly has a different fallback team +than the platform mono-repo's `platform-team`) and, less obviously, `xNamespace` — a team repo +that already has its own established `x-*` vendor-extension convention before onboarding to +auto-registration shouldn't be forced to rename it to match the platform mono-repo's convention +just to participate. + +**Decision**: `defaultOwner` and `xNamespace` are per-source config keys with no global fallback +required (each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` +block if the learner wants shared values) — see data-model.md's per-source schema. `patterns`, +`ignore`, `mode`, and scheduling are also per-source for the same reason (a small pre-production +repo may reasonably use `mode: 'poll'` with a short interval while the platform mono-repo runs +`mode: 'watch'` at scale, per R6). + +**Problem 3 — collision detection and `catalog-info.yaml` precedence (R4) must stay global, not +per-source.** Two different teams' repos can trivially produce the same slugified entity name +(e.g. two teams each shipping a `payments-api`). If collision/precedence checks were scoped to a +single source's own scan-state rows, cross-source collisions would go undetected and both +entities would silently coexist as long as neither the collision index nor the "does a +non-auto-sourced entity already exist" precedence check restricts by source. The scan-state cache +table already keyed `entity_name` for O(1) collision lookups (R6) — the fix is simply that this +index (and the query behind the `catalog-info.yaml` precedence check) is queried **across all +sources**, not scoped to the source currently being processed. The `source_id` column added to +the cache table (data-model.md) exists precisely so a collision error message can say *which two +repos* collided — important for SC-003-style debuggability once "which file" becomes "which file, +in which repo." + +**Problem 4 — "other repos" may mean a genuinely separate Git remote, not a local sibling +directory.** The lab's own Assumptions section already anticipates a local sibling checkout ("a +local sibling repository the learner creates") for its own zero-network, zero-cost demo — that +covers the lab's own multi-source walkthrough (see quickstart.md). It does **not** cover a real +deployment where a team's repo is a separate remote the discovery backend has never had checked +out locally at all. + +**Decision (documented extension point, not built or demonstrated by the lab)**: each source's +`rootPath` config key is designed to remain a plain local filesystem path — the discovery, +parsing, and mapping logic (R1–R6) is entirely decoupled from *how bytes arrive on local disk*. +A production deployment onboarding a genuinely remote repo adds a lightweight sync step ahead of +the existing glob/parse pipeline — e.g. a shallow `git clone`/`git pull` into a local cache +directory on each scheduled cycle (or triggered by the same watch/reconciliation cadence as R6) — +and points that source's `rootPath` at the resulting local worktree. Every other mechanism (glob +patterns, ignore rules, delta mutations, scan-state cache, collision/precedence checks) is reused +unchanged; only the sync step differs per source. This keeps the zero-cost constraint intact for +the lab (no sync step is needed or built when `rootPath` already points at a local sibling +directory) while giving a real multi-repo deployment a documented, non-invasive path to genuinely +separate remotes — noted in quickstart.md as a "beyond the lab" extension point, consistent with +how R6's watch-mode scale-up is documented rather than exercised. + +**Alternatives considered**: +- One `EntityProvider` instance internally looping over multiple roots — rejected: conflates + scheduling, failure isolation, and mutation lifecycle across unrelated sources; a parse error or + slow filesystem in one team's repo could delay or corrupt discovery for every other source. Per + Backstage's own `EntityProvider` contract, provider identity is meant to be 1:1 with "one + coherent versioned view of a set of entities" — multiple sources are multiple such views. +- Making collision detection per-source (accept that two repos could produce the same name) — + rejected: silently produces whichever entity happens to process last as the "winner" with no + visible error, directly violating the Edge Cases requirement that name collisions must surface a + visible conflict rather than being silently resolved. +- Building real Git-remote syncing (clone/pull) as part of this lab — rejected: requires network + access and (for private repos) credentials, which conflicts with the lab's zero-network, + zero-cost constraint (Constitution V); documented as an extension point instead (Problem 4). From f90811de665f4f93ac3f3f33dd2fc07416d4bbc0 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:36:25 +1000 Subject: [PATCH 06/12] Added ability to configure API visibility via x-* fields and repo defaults --- .../004-lab-4-auto-registration/data-model.md | 14 +++- specs/004-lab-4-auto-registration/plan.md | 38 ++++++----- .../004-lab-4-auto-registration/quickstart.md | 25 +++++-- specs/004-lab-4-auto-registration/research.md | 67 ++++++++++++++----- specs/004-lab-4-auto-registration/spec.md | 23 ++++--- 5 files changed, 119 insertions(+), 48 deletions(-) diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index 091ec77..e8c3c08 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -16,6 +16,7 @@ A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncap | `tags[].name` | string[] | Optional, native top-level OpenAPI/AsyncAPI field (not under `info`, not under `x-examplecorp`). Default `[]` if absent. | | `info.x-examplecorp.owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration still succeeds using a documented default — see Mapping Rule 1 below. | | `info.x-examplecorp.lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | +| `info.x-examplecorp.visibility` | string | Optional. One of `private` \| `shared` — the same two values Lab 2 established. Absent → falls back to that file's source's configured `defaultVisibility` (source-level default is `'private'` if the source doesn't set one — research.md R3/R7). Present-but-unrecognized → marker/error entity, not silently defaulted (research.md R4). | ## Entity: Catalog API Entity (output) @@ -28,7 +29,8 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.description` | `info.description` | Verbatim if present, else omitted. | | `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | -| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref or name collision); absent on normally-registered entities. | +| `metadata.annotations['example.com/visibility']` | `info.x-examplecorp.visibility` | **The exact same annotation key the Lab 2/3 permission policy already reads** (`packages/backend/src/extensions/permissionPolicy.ts`) — no new annotation, no policy change. **Mapping Rule 2**: if absent, falls back to that file's source's configured `defaultVisibility` — itself optional per-source, falling back to the fixed `'private'` "default default" if the source doesn't set one either (research.md R3/R7, FR-006a). An unrecognized (non-`private`/`shared`) value is never silently defaulted — it's a marker/error entity instead (research.md R4). Reused verbatim rather than reinvented, per research.md R3. | +| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref, invalid visibility value, or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | | `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | @@ -54,12 +56,14 @@ autoApiRegistration: reconciliation: { frequencySeconds: 900 } # watch mode only parseConcurrency: 4 defaultOwner: group:default/platform-team + defaultVisibility: private # optional — falls back to the fixed 'private' default if unset xNamespace: examplecorp - id: team-checkout-api # a second source: a different team's repo, a local sibling rootPath: ../team-checkout-api # checkout for the lab; a real remote repo's patterns: ['**/*-openapi.yaml'] # synced local worktree in production (R7 defaultOwner: group:default/checkout-team # Problem 4) - xNamespace: checkoutteam + defaultVisibility: shared # e.g. a pre-production repo where every API should default + xNamespace: checkoutteam # to shared visibility unless a spec says otherwise ``` | Key (per source, under `sources[]`, or flat for the single-source shorthand) | Type | Default | Purpose | @@ -73,7 +77,8 @@ autoApiRegistration: | `reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | | `parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | | `defaultOwner` | string | *(no global default — each source must set its own, or inherit `autoApiRegistration.defaults.defaultOwner` if configured)* | Fallback owner ref applied when `x-.owner` is absent (FR-007). Deliberately per-source, not global — a team/pre-production repo plausibly has a different fallback owner than the platform mono-repo (research.md R7, Problem 2). | -| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | +| `defaultVisibility` | `'private' \| 'shared'` | `'private'` (the one fixed, hardcoded fallback in the whole config surface — the "default default") | Fallback visibility applied when `x-.visibility` is absent for a file in this source (research.md R3/R4). Per-source, like `defaultOwner` — e.g. a pre-production repo may reasonably default every undeclared API to `shared`, while the platform mono-repo keeps the conservative `private` fallback. Unlike `defaultOwner`/`xNamespace`, this key is optional even at the source level: a source that omits it gets `private`, so a learner adding a new source doesn't have to think about visibility defaults unless they want to change them. | +| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle/visibility from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | ## Scan-state cache (persisted, `coreServices.database`) @@ -120,6 +125,9 @@ Problem 3) even though discovery itself runs per source. Problem 3). e. Resolve `x-.owner` (falling back to this source's `defaultOwner` if absent) against known `User`/`Group` entities → unresolvable owners become marker/error entities. + Resolve `x-.visibility` (falling back to this source's `defaultVisibility`, or + `'private'` if the source doesn't set one, if absent) → validate against `{private, shared}` + → an unrecognized (present-but-invalid) value becomes a marker/error entity (research.md R4). f. Check for a pre-existing, non-auto-sourced entity at the same name anywhere in the catalog (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name (FR-011). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md index 5f685bf..64a5ba8 100644 --- a/specs/004-lab-4-auto-registration/plan.md +++ b/specs/004-lab-4-auto-registration/plan.md @@ -21,24 +21,30 @@ instance with its own schedule, default owner, and `x-*` namespace, so a platfor one or more separate team/pre-production repos can each be discovered with their own settings while collision and `catalog-info.yaml`-precedence checks stay global across all of them). Only the config *values and entry count* differ between the lab demo (one small source) and a -production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle metadata -is sourced from a single vendor-namespaced -`info.x-examplecorp` object (`owner`, `lifecycle`) — tool-agnostic field names, grouped under one -namespace learners rename to their own company (FR-005–FR-006), with documented defaults when -absent (FR-007). Metadata that already has a natural home in the spec — name (`info.title`), +production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle/visibility +metadata is sourced from a single vendor-namespaced +`info.x-examplecorp` object (`owner`, `lifecycle`, `visibility`) — tool-agnostic field names, +grouped under one namespace learners rename to their own company (FR-005–FR-006, FR-006a), with +documented defaults when absent (FR-007). The `visibility` value reuses the exact +`example.com/visibility` annotation and `private`/`shared` values the Lab 2/3 permission policy +already enforces — no new visibility mechanism is introduced, only a new way of setting the +existing annotation. Metadata that already has a natural home in the spec — name (`info.title`), description (`info.description`), tags (native top-level `tags`) — is sourced from there instead -of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, name collisions) -are surfaced via backend logs and, where an entity can be identified, via Backstage's native -catalog processing-error UI through a companion `CatalogProcessor` that throws on a marker -annotation (FR-008). A hand-authored `catalog-info.yaml`, where one exists alongside a -discovered file, takes precedence over auto-sourced metadata (FR-011). +of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, invalid +visibility values, name collisions) are surfaced via backend logs and, where an entity can be +identified, via Backstage's native catalog processing-error UI through a companion +`CatalogProcessor` that throws on a marker annotation (FR-008). A hand-authored +`catalog-info.yaml`, where one exists alongside a discovered file, takes precedence over +auto-sourced metadata (FR-011). Two new sample files support the lab: a vendored copy of the MIT-licensed Scalar Galaxy API -(`galaxy-openapi.yaml`) with no pre-existing catalog registration, demonstrating genuine -"previously unregistered API" discovery (US1); and a small `precedence-demo-openapi.yaml` + -hand-authored `precedence-demo-catalog-info.yaml` pair demonstrating the precedence rule. -Lab documentation calls out which parts (glob patterns, root path, `x-*` field names) are -adaptable conventions per Constitution Principle VIII (FR-010, US3). +(`galaxy-openapi.yaml`, `x-examplecorp.visibility: shared`) with no pre-existing catalog +registration, demonstrating genuine "previously unregistered API" discovery (US1) and, alongside +the precedence-demo API's differing visibility, a real private/shared contrast to verify (SC-006); +and a small `precedence-demo-openapi.yaml` + hand-authored `precedence-demo-catalog-info.yaml` +pair demonstrating the precedence rule. Lab documentation calls out which parts (glob patterns, +root path, `x-*` field names) are adaptable conventions per Constitution Principle VIII (FR-010, +US3). ## Technical Context @@ -99,7 +105,7 @@ APIs left untouched. Designed-for scale (config change only, no code change, per | Principle | Gate | Status | |-----------|------|--------| | I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: custom `EntityProvider` for filesystem-based catalog discovery, `x-*` vendor-extension metadata sourcing, native catalog processing-error surfacing via a custom `CatalogProcessor`, full-mutation-based create/update/retract lifecycle | ✅ Pass | -| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), and WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011) | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011), and WHY visibility reuses the Lab 2/3 `example.com/visibility` annotation/policy verbatim instead of introducing a parallel mechanism (FR-006a) | ✅ Pass | | III. Cross-Platform Compatibility | All new code is TypeScript/YAML; `fast-glob` and `js-yaml` are pure-JS, no native/platform-specific dependencies; `yarn add` and `yarn start` work identically on Windows and macOS; root-path resolution documented with platform-neutral relative paths | ✅ Pass | | IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–3's existing Node/Yarn setup — `fast-glob`/`js-yaml` are installed via the existing `yarn add` workflow; README documents the install step | ✅ Pass | | V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md index 11e29ab..799c3cd 100644 --- a/specs/004-lab-4-auto-registration/quickstart.md +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -21,23 +21,33 @@ Constitution Principle II. data-model.md for keys/defaults). 4. **Add the two new sample files**: - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API - with an `info.x-examplecorp` object added) — no pre-existing `catalog-info.yaml`. + with an `info.x-examplecorp` object added, `visibility: shared`) — no pre-existing + `catalog-info.yaml`. - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + - `precedence-demo-catalog-info.yaml` — demonstrates FR-011 precedence. + `precedence-demo-catalog-info.yaml` (hand-authored, `example.com/visibility: private`, + owner a non-platform team) — demonstrates FR-011 precedence and gives a real + private/shared contrast to check in step 5. 5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually written catalog file (US1, SC-001). - Its `spec.owner` and `spec.lifecycle` match the `info.x-examplecorp` values, and its tags match the spec's native `tags` array — not duplicated into `x-examplecorp` (US2, SC-002). + - The Galaxy API (`visibility: shared`) is visible to any authenticated user, while the + precedence-demo API (`visibility: private`, hand-authored) is visible only to its owning + team and platform-team — logging in as a non-owning, non-platform user confirms the + difference (SC-006). - The precedence-demo API's catalog entity reflects the hand-authored - `precedence-demo-catalog-info.yaml`, not the auto-sourced values (FR-011). + `precedence-demo-catalog-info.yaml` throughout (owner, visibility), not the auto-sourced + values (FR-011). 6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, confirm the existing entity updates rather than duplicating (FR-003). 7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. 8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line identifies the file; temporarily set `info.x-examplecorp.owner` to a nonexistent team, confirm a - catalog processing error is visible on that entity (FR-008, SC-003). + catalog processing error is visible on that entity (FR-008, SC-003); temporarily set + `info.x-examplecorp.visibility` to an unrecognized value (e.g. `public`), confirm the same kind + of visible error rather than a silent default. ## Out of scope for this quickstart @@ -62,10 +72,13 @@ A second "Scaling to multiple source repositories" note (research.md R7) covers: - The `autoApiRegistration.sources[]` config list — the lab's own `app-config.yaml` uses the flat single-source shorthand, but the README shows the equivalent explicit `sources: [{ id: default, ... }]` form alongside a second example entry, so learners can see exactly what changes to add a - team or pre-production repo (a different `rootPath`, `defaultOwner`, and `xNamespace`). + team or pre-production repo (a different `rootPath`, `defaultOwner`, `defaultVisibility`, and + `xNamespace`). - Why `defaultOwner` and `xNamespace` are per-source with no built-in global fallback (a repo onboarding to auto-registration shouldn't have to adopt the platform mono-repo's team or vendor - namespace to participate). + namespace to participate), while `defaultVisibility` is per-source but *does* have a built-in + global fallback (`private`) — visibility is the one setting where "unconfigured" must still be + safe by default, not just adaptable. - Why collision detection and `catalog-info.yaml` precedence checks are explicitly called out as *global* across sources, not per-source — two teams' repos producing the same entity name must still surface a visible conflict, not silently coexist. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 9f201bc..51057e4 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -83,6 +83,30 @@ top-level `info` object, for both OpenAPI and AsyncAPI (per the Clarifications s 2026-07-03 update): - `info.x-examplecorp.owner` → `spec.owner` - `info.x-examplecorp.lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) +- `info.x-examplecorp.visibility` → `metadata.annotations['example.com/visibility']` (default: that + file's source's configured `defaultVisibility`, itself defaulting to `private` if the source + doesn't set one — see rationale below and R7) + +**Visibility is deliberately reused, not reinvented.** Lab 2/3 already established +`example.com/visibility: private | shared` as the annotation the permission policy +(`packages/backend/src/extensions/permissionPolicy.ts`) reads to decide access: `shared` bypasses +ownership entirely (visible to all authenticated users); anything else (including the annotation +being absent) falls through to the ownership-only rule. This lab adds a **third way to set that +same annotation** — sourced from `x-examplecorp.visibility` instead of hand-authored in +`catalog-info.yaml` — it does not add a new visibility concept, a new annotation key, or touch the +permission policy at all. The absent-field default is **per-source**, not a single hardcoded +value: each source in `autoApiRegistration.sources[]` may set its own `defaultVisibility` +(`private` or `shared`) for files that don't declare one — e.g. a pre-production repo where most +APIs are intentionally shared for cross-team testing might reasonably default to `shared`, while +the platform mono-repo keeps the conservative default. The "default default" — what a source gets +if it doesn't configure `defaultVisibility` at all — is `private`, mirroring the permission +policy's own existing fallback behavior when the annotation is missing entirely (Rule 3, +ownership-only), so a freshly-added source with no visibility configuration behaves identically to +a hand-authored entity with no visibility annotation. An `x-examplecorp.visibility` value that is +present but not `private` or `shared` is treated as malformed input (Edge Cases) — surfaced via +the same marker-entity + processor-error path as an invalid owner reference (R4), not silently +coerced to a default (source-level or otherwise), since silently defaulting a security-relevant +setting is exactly the kind of mistake that must stay visible. `examplecorp` is this lab's fictional company namespace (documented as the one thing every learner renames first, per FR-010/US3). The field names deliberately omit `backstage` — this @@ -141,6 +165,13 @@ no bespoke error-reporting system is built: no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the repo). Unresolvable owners produce the marker-entity + processor-error path above. +**Visibility validation**: `x-examplecorp.visibility`, if present, is checked against the fixed +set of values the existing permission policy understands (`private`, `shared`) — this is a static +enum check, not a catalog lookup, so it's cheap regardless of scale. An unrecognized value follows +the same marker-entity + processor-error path as an invalid owner reference. If absent, the value +comes from that file's source's `defaultVisibility` (falling back to `private` if the source +doesn't configure one) — see R3 for the full rationale and R7 for why this is per-source. + **Name collisions**: if two discovered files slugify to the same candidate entity name, the first (sorted by file path for determinism) is registered normally; the second is emitted as a marker entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the @@ -166,8 +197,10 @@ processed — a hand-authored file anywhere wins over an auto-sourced candidate github.com/scalar/scalar, published build at `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with an `info.x-examplecorp` -object (`owner`, `lifecycle`) added directly in the vendored copy; its existing native `tags` -array is used as-is for `metadata.tags`. The upstream file is a single self-contained OpenAPI 3.1 +object (`owner`, `lifecycle`, `visibility: shared` — chosen so the quickstart's SC-006 visibility +check has a real difference to demonstrate against the precedence-demo API below) added directly +in the vendored copy; its existing native `tags` array is used as-is for `metadata.tags`. The +upstream file is a single self-contained OpenAPI 3.1 document (confirmed: 1469 lines, all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and @@ -292,19 +325,23 @@ cannot stall or break discovery for any other source. For backward/simple compat single implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so the single-repo lab config does not need to change shape. -**Problem 2 — some settings are legitimately per-repo, not global.** The two called out -explicitly: `defaultOwner` (a pre-production or team repo plausibly has a different fallback team -than the platform mono-repo's `platform-team`) and, less obviously, `xNamespace` — a team repo -that already has its own established `x-*` vendor-extension convention before onboarding to -auto-registration shouldn't be forced to rename it to match the platform mono-repo's convention -just to participate. - -**Decision**: `defaultOwner` and `xNamespace` are per-source config keys with no global fallback -required (each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` -block if the learner wants shared values) — see data-model.md's per-source schema. `patterns`, -`ignore`, `mode`, and scheduling are also per-source for the same reason (a small pre-production -repo may reasonably use `mode: 'poll'` with a short interval while the platform mono-repo runs -`mode: 'watch'` at scale, per R6). +**Problem 2 — some settings are legitimately per-repo, not global.** Called out explicitly: +`defaultOwner` (a pre-production or team repo plausibly has a different fallback team than the +platform mono-repo's `platform-team`); `defaultVisibility` (a pre-production repo might reasonably +default undeclared APIs to `shared` for cross-team testing, while the platform mono-repo keeps a +conservative `private` default — R3); and, less obviously, `xNamespace` — a team repo that already +has its own established `x-*` vendor-extension convention before onboarding to auto-registration +shouldn't be forced to rename it to match the platform mono-repo's convention just to participate. + +**Decision**: `defaultOwner`, `defaultVisibility`, and `xNamespace` are per-source config keys — +see data-model.md's per-source schema. `defaultOwner` and `xNamespace` have no global fallback +(each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` block if +the learner wants shared values); `defaultVisibility` is the one exception with a genuine global +fallback (`private`, R3) precisely because leaving visibility completely unconfigured must still +be safe by default — a source shouldn't have to opt into a safe default, only opt out of it. +`patterns`, `ignore`, `mode`, and scheduling are also per-source for the same reason (a small +pre-production repo may reasonably use `mode: 'poll'` with a short interval while the platform +mono-repo runs `mode: 'watch'` at scale, per R6). **Problem 3 — collision detection and `catalog-info.yaml` precedence (R4) must stay global, not per-source.** Two different teams' repos can trivially produce the same slugified entity name diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md index 60e98e2..cba7734 100644 --- a/specs/004-lab-4-auto-registration/spec.md +++ b/specs/004-lab-4-auto-registration/spec.md @@ -23,6 +23,7 @@ - Q: Should `x-*` fields sit directly in `info`, or nested under a single vendor-namespaced object? → A: Nested under a single `info.x-examplecorp` object (e.g. `info.x-examplecorp.owner`, `info.x-examplecorp.lifecycle`) — `examplecorp` is this lab's fictional company namespace, which learners are expected to rename to their own; a single grouping object avoids polluting `info` with multiple top-level `x-*` keys and makes "which fields are ours" unambiguous at a glance. - Q: Should field names retain the `backstage` tool name (`x-backstage-owner`, etc.)? → A: No — the field names describe the metadata (owner, lifecycle), not the tool consuming it. Catalog metadata like ownership is tool-agnostic (equally meaningful to an API gateway or any other consumer), so `backstage` is dropped: `x-examplecorp.owner`, `x-examplecorp.lifecycle`. - Q: Should `tags` remain a recognized `x-*` field? → A: No — removed. OpenAPI/AsyncAPI both already have a native top-level `tags` field; sourcing tags from `x-*` as well would duplicate data that already has a natural home in the spec, risking divergence. Catalog `metadata.tags` is now sourced from the spec's native `tags` array (`tags[].name`), not from any `x-*` field. The same reasoning already applied to `info.title` (entity name) and `info.description` (entity description), which were never duplicated into `x-*` fields. +- Q: Should Lab 2's API visibility concept (private/shared) be added to the `x-*`-managed metadata? → A: Yes — `info.x-examplecorp.visibility`, accepting the same `private` \| `shared` values Lab 2 already established, mapped to the same `example.com/visibility` annotation the Lab 2/3 permission policy already reads. Visibility has no natural home elsewhere in the spec (unlike name/description/tags), so it belongs in `x-examplecorp` alongside owner and lifecycle, not as a new mechanism. ## User Scenarios & Testing *(mandatory)* @@ -44,7 +45,7 @@ As a platform engineer, when a new API definition file (OpenAPI or AsyncAPI) is ### User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) -As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or tags) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. +As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or visibility) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. **Why this priority**: This is what makes registration genuinely "automatic" rather than just file discovery — without metadata sourcing, every API would still need a hand-maintained catalog file for ownership and lifecycle, which is exactly what Lab 2's ownership model depends on. @@ -53,9 +54,10 @@ As an API owner, I want to declare metadata that Backstage needs (such as owning **Acceptance Scenarios**: 1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. -2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, visibility), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. 3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. 4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. +5. **Given** an API definition containing an `x-*` visibility field, **When** the entity is registered, **Then** the catalog entity's visibility annotation matches that value and is enforced by the same permission policy introduced in Lab 2 (private → owning team + platform team only; shared → all authenticated users), with no separate visibility mechanism introduced for auto-registered APIs. --- @@ -78,6 +80,7 @@ As a learner adapting this repository to my own organization, I want to understa - What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. - What happens when two API definition files resolve to the same catalog entity name (i.e., the same slugified `info.title`)? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. - What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. +- What happens when the `x-*` visibility field is present but is not one of the recognized values (`private`, `shared`)? The system MUST surface a visible error for that entity rather than silently defaulting or registering it with an unenforceable visibility value. - What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. - How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. @@ -90,9 +93,10 @@ As a learner adapting this repository to my own organization, I want to understa - **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. - **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. - **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). -- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and visibility) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. +- **FR-006a**: The system MUST extract API visibility from the documented `x-*` object and set the same visibility annotation the Lab 2 permission policy already enforces (`example.com/visibility`, values `private` or `shared`), reusing that existing policy mechanism rather than introducing a separate visibility concept or enforcement path for auto-registered APIs. When a given API definition omits the visibility field, the system MUST apply a default that is independently configurable per mono-repo/source location (so different source repositories can have different default visibility), falling back to `private` for any source that does not configure its own default. - **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. -- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. +- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, specifies an unrecognized visibility value, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. - **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. - **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. - **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. @@ -101,8 +105,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Key Entities -- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle). -- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle metadata sourced from `info.x-examplecorp`. +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle, visibility). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle/visibility metadata sourced from `info.x-examplecorp`. - **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. - **`x-*` Metadata Mapping**: The documented mapping from the `info.x-examplecorp` object's fields to catalog entity fields/annotations, including default values applied when a field is absent, and the explicit list of fields sourced natively rather than from `x-*` (name, description, tags). @@ -111,7 +115,8 @@ As a learner adapting this repository to my own organization, I want to understa ### Measurable Outcomes - **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. -- **SC-002**: 100% of the owning-team and lifecycle metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. +- **SC-002**: 100% of the owning-team, lifecycle, and visibility metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. +- **SC-006**: A learner can set an API's visibility via its `x-*` field alone (no hand-authored catalog file) and observe the same access behavior Lab 2 documented for that visibility value (private vs. shared) when viewing the catalog as different users, confirming the auto-sourced value drives the existing permission policy rather than a separate mechanism. - **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. - **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. - **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. @@ -120,7 +125,9 @@ As a learner adapting this repository to my own organization, I want to understa - The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. - "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. -- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner` and `lifecycle` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. +- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner`, `lifecycle`, and `visibility` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. - The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. +- The visibility model established in Lab 2/3 (the `example.com/visibility` annotation and its `private`/`shared` values, enforced by the existing permission policy) is the source of truth referenced when mapping the `x-*` visibility field — Lab 4 does not introduce a new/parallel visibility mechanism, only a new way of setting the same annotation. +- The default visibility applied when a definition omits the `x-*` visibility field is configured per mono-repo/source location, not globally fixed — different source repositories (e.g. a pre-production repo vs. the platform mono-repo) may reasonably want different defaults. A source that does not configure its own default falls back to `private`, matching the permission policy's existing behavior when no visibility annotation is present at all. - When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. - Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. From e2f3d96299ed7f0cc0f991fd20b54196180678e9 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 08:41:56 +1000 Subject: [PATCH 07/12] Implementation tasks planned --- specs/004-lab-4-auto-registration/tasks.md | 207 +++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 specs/004-lab-4-auto-registration/tasks.md diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md new file mode 100644 index 0000000..f232d5c --- /dev/null +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -0,0 +1,207 @@ +--- + +description: "Task list for Lab 4 — Auto Registration" +--- + +# Tasks: Lab 4 — Auto Registration + +**Input**: Design documents from `/specs/004-lab-4-auto-registration/` + +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md (all present; no `contracts/` — this lab produces no external API contracts) + +**Tests**: Not requested. Per plan.md's Testing section, verification is manual (browser + backend logs) against quickstart.md's steps — consistent with Labs 1–3. No automated test tasks are generated; quickstart validation is folded into each story's checkpoint and the final Polish phase. + +**Organization**: Tasks are grouped by user story (spec.md priorities P1/P2/P3) to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (US1, US2, US3) + +## Path Conventions + +This is a tutorial-lab project (not a generic web/mobile app). Two path roots are used: + +- **Backstage instance** (modified in place, from Lab 1): `labs/lab-01-base-backstage/backstage/` +- **Lab 4 teaching content** (new, committed alongside README): `labs/lab-04-auto-registration/` +- **Speckit artifacts**: `specs/004-lab-4-auto-registration/` (this directory — no code changes here) + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Add dependencies and scaffold the directories every later task writes into. + +- [ ] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) +- [ ] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` +- [ ] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` + +**Checkpoint**: Dependencies installed, directories exist — ready for foundational work. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Build the shared scanning/parsing/persistence/config plumbing that both US1 (discovery/CRUD) and US2 (metadata sourcing) sit on top of. No user story is independently testable until this phase is done — the EntityProvider skeleton it produces is what US1 wires up to catalog mutations and US2 extends with `x-*` extraction. + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete. + +- [ ] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section +- [ ] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) +- [ ] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) +- [ ] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) +- [ ] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) +- [ ] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) +- [ ] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) +- [ ] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) + +**Checkpoint**: Config loads, files are discoverable and parseable, scan-state cache persists — foundation ready for US1. + +--- + +## Phase 3: User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) 🎯 MVP + +**Goal**: A new OpenAPI/AsyncAPI file dropped anywhere under the mono-repo location appears in the Backstage catalog with no hand-authored `catalog-info.yaml`; edits update the entity in place; removing the file retracts it. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file, trigger discovery, confirm a corresponding API entity appears — with default owner/lifecycle values (no `x-*` extraction required for this story) — with no manual `catalog-info.yaml` written for it; edit the file and confirm update-in-place; remove the file and confirm the entity is no longer active. + +### Implementation for User Story 1 + +- [ ] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) +- [ ] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) +- [ ] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) +- [ ] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) +- [ ] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) +- [ ] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) + +**Checkpoint**: User Story 1 fully functional and independently testable — discovery, create, update, and removal all work end-to-end with default metadata. + +--- + +## Phase 4: User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +**Goal**: `spec.owner`, `spec.lifecycle`, and the `example.com/visibility` annotation are sourced from each file's `info.x-examplecorp` object (with documented defaults when absent), invalid values/references surface visible errors on both backend logs and the catalog processing-errors UI, name collisions and hand-authored `catalog-info.yaml` precedence are respected. + +**Independent Test**: Author an API definition with `x-examplecorp` owner/lifecycle/visibility fields, run discovery, confirm the resulting entity's `spec.owner`, `spec.lifecycle`, and `example.com/visibility` annotation reflect those values (not defaults); omit fields and confirm documented defaults apply without blocking registration; set an invalid owner/visibility and confirm a visible error on that entity without affecting other entities; confirm a hand-authored `catalog-info.yaml` wins over auto-sourced metadata for the same API. + +### Implementation for User Story 2 + +- [ ] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) +- [ ] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) +- [ ] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) +- [ ] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) +- [ ] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) +- [ ] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) +- [ ] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) +- [ ] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) +- [ ] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) +- [ ] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) +- [ ] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) + +**Checkpoint**: User Stories 1 AND 2 both work independently — metadata sourcing, defaults, validation errors, collisions, and precedence are all verified. + +--- + +## Phase 5: User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +**Goal**: Lab documentation clearly separates adaptable conventions (file location pattern, `x-*` field names, multi-source config) from fixed discovery mechanics, so a learner can repoint the lab at their own layout. + +**Independent Test**: Follow the lab README's configuration section to identify which settings are adaptable vs. fixed, and confirm the documented `sources[]` multi-repo example is internally consistent with data-model.md's schema. + +### Implementation for User Story 3 + +- [ ] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) +- [ ] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [ ] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) + +**Checkpoint**: All three user stories independently functional; README fully documents adaptable vs. fixed conventions. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +**Purpose**: Final repo-wide consistency checks that span multiple stories. + +- [ ] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI +- [ ] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development +- [ ] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies — start immediately +- **Foundational (Phase 2)**: Depends on Setup completion — BLOCKS all user stories +- **User Story 1 (Phase 3)**: Depends on Foundational completion — no dependency on US2/US3 +- **User Story 2 (Phase 4)**: Depends on Foundational completion; extends the same file US1 built (T012), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency +- **User Story 3 (Phase 5)**: Depends on US1 (T015) and US2 (T024) being implemented, since the README documents both the discovery mechanism and the metadata-sourcing/error behavior +- **Polish (Phase 6)**: Depends on all user stories being complete + +### User Story Dependencies + +- **User Story 1 (P1)**: Can start after Foundational — no dependency on other stories +- **User Story 2 (P2)**: Can start after Foundational, but shares `autoApiRegistration.ts` with US1 (same file, sequential edits) — implement after US1 for a working MVP checkpoint first +- **User Story 3 (P3)**: Documentation-only; depends on US1 and US2 being functionally complete so the README describes real, working behavior + +### Within Each User Story + +- Mapping/extraction logic before mutation-emission logic before scheduling +- Sample files before manual verification steps +- Verification task last in each phase (checkpoint gate) + +### Parallel Opportunities + +- T002, T003 (Setup) in parallel +- T006, T007, T008 (Foundational: cache migration, discovery utility, parse utility) in parallel — distinct concerns, though ultimately land in the same file, so treat [P] as "logically independent," and serialize the actual edits if working solo +- T026, T027 (US2 sample files) in parallel with each other +- T031, T032 (US3 README sections) in parallel with each other +- T035, T037 (Polish) in parallel + +--- + +## Parallel Example: Foundational Phase + +```bash +# Logically independent foundational concerns (same target file — serialize edits if solo): +Task: "Implement scan-state cache DB migration in autoApiRegistrationMigrations/001_scan_state_cache.ts" +Task: "Implement streamed file discovery utility in autoApiRegistration.ts" +Task: "Implement YAML parse + shape-check utility in autoApiRegistration.ts" +``` + +## Parallel Example: User Story 2 Sample Files + +```bash +Task: "Create precedence-demo-openapi.yaml" +Task: "Create precedence-demo-catalog-info.yaml" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup +2. Complete Phase 2: Foundational (CRITICAL — blocks all stories) +3. Complete Phase 3: User Story 1 +4. **STOP and VALIDATE**: T017 confirms discovery/create/update/remove work end-to-end with default metadata +5. Demo: a new spec file appears in the catalog with zero hand-authored `catalog-info.yaml` + +### Incremental Delivery + +1. Setup + Foundational → scanning/parsing/persistence plumbing ready +2. Add User Story 1 → validate independently (T017) → MVP demo +3. Add User Story 2 → validate independently (T028) → owner/lifecycle/visibility sourcing, error surfacing, precedence all work +4. Add User Story 3 → README documents adaptability → lab is learner-ready +5. Polish → cold-start validation, constitution re-check + +### Notes + +- US1 and US2 share one implementation file (`autoApiRegistration.ts`) because the architecture (research.md R1) is a single `EntityProvider` — this is a deliberate single-module design, not an accidental cross-story coupling. Each story still has an independently verifiable acceptance test (T017 vs. T028) even though the code lands in the same file. +- No test tasks are included (Tests: Not requested — see header). Verification is manual, per quickstart.md, folded into each story's final task. +- `watch` mode (research.md R6) and the multi-source sync-step extension point (research.md R7 Problem 4) are documented (T031/T032) but not built or exercised end-to-end — consistent with quickstart.md's explicit "Out of scope for this quickstart" section. From fff3dd42107041eef7471f3dfe4b0b9a7bb78496 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 11:41:45 +1000 Subject: [PATCH 08/12] Implemented --- labs/lab-04-auto-registration/README.md | 1196 +++++++++++++ .../apis/galaxy/galaxy-openapi.yaml | 1476 +++++++++++++++++ .../precedence-demo-catalog-info.yaml | 21 + .../precedence-demo-openapi.yaml | 59 + .../004-lab-4-auto-registration/data-model.md | 4 +- specs/004-lab-4-auto-registration/research.md | 12 +- specs/004-lab-4-auto-registration/tasks.md | 74 +- 7 files changed, 2799 insertions(+), 43 deletions(-) create mode 100644 labs/lab-04-auto-registration/README.md create mode 100644 labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml create mode 100644 labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml create mode 100644 labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md new file mode 100644 index 0000000..955a8ed --- /dev/null +++ b/labs/lab-04-auto-registration/README.md @@ -0,0 +1,1196 @@ +# Lab 4 — Auto Registration + +## Overview + +Labs 1–3 registered every API by hand-authoring a `catalog-info.yaml` per API and adding it as a +`type: url` catalog location. That works for three APIs; it does not work for a mono-repo with +hundreds. This lab replaces the per-API hand-authored file with a custom Backstage backend module +that scans a directory tree for `*-openapi.yaml` / `*-asyncapi.yaml` files and turns each one +directly into a catalog `API` entity — no `catalog-info.yaml` required. + +By the end of this lab you will have: + +- A new backend module (`autoApiRegistration.ts` + a small database migration) that discovers, + parses, and registers API definitions on a poll schedule, using Backstage's `EntityProvider` + mechanism so create, update, and removal all come from the same code path. +- Owner, lifecycle, and visibility metadata sourced from each spec's own `info.x-examplecorp` + object, with documented defaults when a field is absent — no duplication of metadata that + already has a natural home in the spec (`info.title`, `info.description`, native `tags`). +- Errors (malformed files, unresolvable owners, invalid visibility values, name collisions) + surfaced through Backstage's own backend logs and its native catalog processing-error UI — no + bespoke error-reporting system. +- Two new sample APIs: a vendored copy of the Scalar Galaxy API (pure auto-discovery, no + `catalog-info.yaml`) and a small precedence-demo API pair that demonstrates a hand-authored + `catalog-info.yaml` winning over auto-sourced metadata for the same API. + +**What you will learn:** + +- Why a custom `EntityProvider` (not a `CatalogProcessor`) is the right mechanism for + filesystem-based catalog discovery, and how its `full`/`delta` mutation lifecycle gives you + create/update/remove for free +- How to source catalog metadata from a vendor-namespaced `x-*` extension instead of a parallel + hand-authored file, and how to fall back safely when a field is absent +- How to surface processing errors through Backstage's built-in mechanisms (backend logs + a + `CatalogProcessor` that throws on a marker annotation) instead of inventing a new one +- How a persisted scan-state cache and `sources[]` config generalize this mechanism from "one + small demo repo" to "1000+ files across multiple team repos" without an architecture change + +--- + +## Prerequisites + +- **Lab 3 completed** — Backstage running locally with Museum API, Streetlights API, and Train + Travel API registered; `eve` (platform team) able to sign in and see quality detail +- **Node.js 20 LTS** and **Yarn** — same versions as Labs 1–3 + +--- + +## Step 1 — Add Dependencies + +From `packages/backend`, add the three new direct dependencies (`fast-glob` for the filesystem +scan, `js-yaml` for parsing, `chokidar` for the optional `watch` mode used at real-world scale): + +``` +cd packages/backend +yarn add fast-glob@3.3.3 js-yaml@4.2.0 chokidar@3.6.0 +``` + +All three are already present transitively in the workspace — this promotes them to explicit +dependencies of the backend package, which is what actually imports them. + +--- + +## Step 2 — Add the Auto-Registration Backend Module + +This is the heart of the lab: a backend module that scans a directory for API definition files, +parses each one, and emits catalog `API` entities via a custom `EntityProvider`. A companion +`CatalogProcessor` surfaces registration errors (invalid owner, invalid visibility, name +collisions) through Backstage's native processing-error UI. + +**Why an `EntityProvider`, not a `CatalogProcessor`?** A `CatalogProcessor` only transforms +entities that already exist via some `Location` — it can't originate new entities from an +arbitrary filesystem scan on its own. An `EntityProvider` owns a versioned "set of entities I +currently know about"; anything it stops claiming is automatically retracted by the catalog. That +gives create, update, and removal-on-delete from one code path, with no separate cleanup logic. + +**Why is owner/lifecycle/visibility sourced from `info.x-examplecorp` instead of duplicated +fields?** Metadata that already has a natural home in the spec — `info.title` (name), +`info.description` (description), native top-level `tags` — is read from there, not duplicated +into a vendor extension. Only metadata with **no** natural OpenAPI/AsyncAPI home (who owns this +API, what lifecycle stage it's in, whether it's private or shared) goes under `x-examplecorp`. +`examplecorp` is a fictional company namespace — it's the first thing you should rename to your +own company when adapting this lab (see "Adaptable Conventions" below). + +**Why does visibility reuse the exact `example.com/visibility` annotation from Lab 2/3?** The +permission policy from Lab 3 (`packages/backend/src/extensions/permissionPolicy.ts`) already +reads that annotation to decide access: `shared` bypasses ownership entirely, anything else +(including absent) falls through to the ownership-only rule. This lab adds a **third way to set +that same annotation** — sourced from `x-examplecorp.visibility` — not a new visibility mechanism +or a change to the permission policy. + +**Why do errors ride on Backstage's own backend logs and processing-error UI, instead of a new +error-reporting system?** Every discovery cycle logs one line per skipped/errored file via the +standard backend logger. For errors on a file specific enough to have a candidate entity (invalid +owner, invalid visibility, name collision), the provider still emits a minimal entity tagged with +an `apiportal-lab.io/registration-error` annotation; a small companion `CatalogProcessor` throws an +`InputError` when it sees that annotation, and Backstage's catalog engine natively records and +surfaces per-entity processing errors — the same "Inspect entity" / processing-errors view you'd +see for any other catalog ingestion problem. One message, written once, drives both surfaces. +Files that fail even the basic shape check (not parseable as an API spec at all — no +`openapi`/`asyncapi` field, no `info.title`) can't produce a minimal entity (there's no name to +register under), so those are logged only, not surfaced in the UI. + +**Why does a hand-authored `catalog-info.yaml` win over auto-sourced metadata?** If a file already +has a hand-authored catalog entry (`kind: API`, matching `metadata.name`) that didn't come from +this provider, the provider skips its own auto-sourced candidate for that name entirely. This +lets you hand-author precise metadata for a specific API — pinning an owner, tightening +visibility — without the auto-registration mechanism fighting you for it every cycle. + +Create the database migration first, at +`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`: + +```typescript +// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts +// +// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal +// state private to this backend module, used to avoid re-parsing unchanged files on restart, to +// compute delta mutations, and to support collision/precedence checks across every configured +// source. +// +// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a +// filesystem `directory` migration source, since this file lives alongside the module's +// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. + +import type { Knex } from 'knex'; + +export const TABLE_NAME = 'auto_api_registration_scan_state'; + +export async function up(knex: Knex): Promise { + await knex.schema.createTable(TABLE_NAME, table => { + table.string('source_id').notNullable(); + table.string('file_path').notNullable(); + table.bigInteger('mtime_ms').notNullable(); + table.string('content_hash').notNullable(); + table.string('entity_name').notNullable(); + table.text('last_error').nullable(); + + table.primary(['source_id', 'file_path']); + table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); + }); +} + +export async function down(knex: Knex): Promise { + await knex.schema.dropTable(TABLE_NAME); +} +``` + +Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts`: + +```typescript +// packages/backend/src/extensions/autoApiRegistration.ts +// +// Lab 4: Auto Registration. +// +// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files +// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. + +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import fg from 'fast-glob'; +import chokidar from 'chokidar'; +import yaml from 'js-yaml'; +import { + coreServices, + createBackendModule, + resolvePackagePath, +} from '@backstage/backend-plugin-api'; +import { + catalogProcessingExtensionPoint, + catalogServiceRef, +} from '@backstage/plugin-catalog-node'; +import type { + CatalogProcessor, + CatalogProcessorEmit, + CatalogService, + DeferredEntity, + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-node'; +import type { + AuthService, + DatabaseService, + LoggerService, + RootConfigService, +} from '@backstage/backend-plugin-api'; +import { + ANNOTATION_LOCATION, + ANNOTATION_ORIGIN_LOCATION, + parseEntityRef, + stringifyEntityRef, +} from '@backstage/catalog-model'; +import type { Entity } from '@backstage/catalog-model'; +import { InputError } from '@backstage/errors'; +import type { Knex } from 'knex'; +import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; + +// --------------------------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------------------------- + +export const VISIBILITY_ANNOTATION = 'example.com/visibility'; +export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; +// Internal-only marker: identifies which provider instance produced an entity, so the +// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from +// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. +const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; + +const KNOWN_VISIBILITIES = new Set(['private', 'shared']); +const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; +const DEFAULT_IGNORE = [ + '**/node_modules/**', + '**/.git/**', + '**/dist/**', + '**/build/**', +]; + +export interface SourceConfig { + id: string; + rootPath: string; + patterns: string[]; + ignore: string[]; + mode: 'poll' | 'watch'; + scheduleFrequencySeconds: number; + reconciliationFrequencySeconds: number; + parseConcurrency: number; + defaultOwner: string; + defaultVisibility: 'private' | 'shared'; + xNamespace: string; +} + +function defaultRootPath(): string { + // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs + return path.resolve( + resolvePackagePath('backend'), + '../../../../lab-04-auto-registration/apis', + ); +} + +export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { + const raw = rootConfig.getOptionalConfig('autoApiRegistration'); + if (!raw) { + return []; + } + + const entries = raw.has('sources') + ? raw.getConfigArray('sources') + : [raw]; + + return entries.map(entry => { + const id = raw.has('sources') ? entry.getString('id') : 'default'; + const defaultOwner = entry.getOptionalString('defaultOwner'); + const xNamespace = entry.getOptionalString('xNamespace'); + if (!defaultOwner) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "defaultOwner"`, + ); + } + if (!xNamespace) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "xNamespace"`, + ); + } + const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; + if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { + throw new Error( + `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, + ); + } + + return { + id, + rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), + patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, + ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, + mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', + scheduleFrequencySeconds: + entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, + reconciliationFrequencySeconds: + entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, + parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, + defaultOwner, + defaultVisibility: defaultVisibility as 'private' | 'shared', + xNamespace, + }; + }); +} + +// --------------------------------------------------------------------------------------------- +// Discovery + parsing +// --------------------------------------------------------------------------------------------- + +async function* discoverFiles(source: SourceConfig): AsyncGenerator { + const stream = fg.stream(source.patterns, { + cwd: source.rootPath, + ignore: source.ignore, + absolute: true, + onlyFiles: true, + }); + for await (const entry of stream) { + yield entry.toString(); + } +} + +interface ParsedSpec { + kind: 'openapi' | 'asyncapi'; + title: string; + description?: string; + tags: string[]; + raw: Record; +} + +/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ +function parseSpecFile(contents: string): ParsedSpec | undefined { + let doc: unknown; + try { + doc = yaml.load(contents); + } catch { + return undefined; + } + if (!doc || typeof doc !== 'object') { + return undefined; + } + const obj = doc as Record; + const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; + if (!kind) { + return undefined; + } + const info = obj.info as Record | undefined; + const title = info?.title; + if (typeof title !== 'string' || title.length === 0) { + return undefined; + } + // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag + // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed + // through verbatim. + const tags = Array.isArray(obj.tags) + ? (obj.tags as Array>) + .map(t => t?.name) + .filter((t): t is string => typeof t === 'string') + .map(slugify) + .filter(t => t.length > 0) + : []; + + return { + kind, + title, + description: typeof info?.description === 'string' ? (info.description as string) : undefined, + tags, + raw: obj, + }; +} + +export function slugify(title: string): string { + return title + .toLowerCase() + .trim() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 63); +} + +// --------------------------------------------------------------------------------------------- +// Candidate mapping +// --------------------------------------------------------------------------------------------- + +interface MappingResult { + entity: Entity; + entityName: string; + error?: string; +} + +function xField(raw: Record, namespace: string, field: string): string | undefined { + const info = raw.info as Record | undefined; + const ns = info?.[`x-${namespace}`] as Record | undefined; + const value = ns?.[field]; + return typeof value === 'string' ? value : undefined; +} + +function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { + const value = rawOwner ?? defaultOwner; + return stringifyEntityRef( + parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), + ); +} + +function buildEntity(opts: { + providerName: string; + filePath: string; + contents: string; + parsed: ParsedSpec; + entityName: string; + owner: string; + lifecycle: string; + visibility: string; + registrationError?: string; +}): Entity { + const annotations: Record = { + // Identifies the source file for debugging and the catalog-info.yaml precedence check — not + // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no + // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider + // already holds the bytes; no fetch is needed). + [ANNOTATION_LOCATION]: `file:${opts.filePath}`, + [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, + [VISIBILITY_ANNOTATION]: opts.visibility, + [MANAGED_BY_ANNOTATION]: opts.providerName, + }; + if (opts.registrationError) { + annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; + } + + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'API', + metadata: { + name: opts.entityName, + title: opts.parsed.title, + description: opts.parsed.description, + tags: opts.parsed.tags, + annotations, + }, + spec: { + type: opts.parsed.kind, + lifecycle: opts.lifecycle, + owner: opts.owner, + definition: opts.contents, + }, + }; +} + +/** + * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility + * validation, and collision detection. Does not perform the catalog-info.yaml precedence check + * (that requires a catalog query and is done by the caller, once, per file). + */ +function mapCandidate(opts: { + providerName: string; + source: SourceConfig; + filePath: string; + contents: string; + parsed: ParsedSpec; + knownOwnerRefs: Set; + collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any +}): MappingResult { + const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; + const baseSlug = slugify(parsed.title); + + const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); + const owner = resolveOwnerRef(rawOwner, source.defaultOwner); + const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; + const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); + const visibility = rawVisibility ?? source.defaultVisibility; + + // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently + // defaulted. + if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility: source.defaultVisibility, + registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, + }), + error: 'invalid-visibility', + }; + } + + // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity + // instead of blocking the whole cycle. + if (!knownOwnerRefs.has(owner)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility, + registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, + }), + error: 'invalid-owner', + }; + } + + // Rule: a name collision with another file (in this source or another) is a suffixed + // marker/error entity — first-by-path wins, this one loses. + if (collidesWith) { + const collisionName = `${baseSlug}-collision`; + return { + entityName: collisionName, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: collisionName, + owner, + lifecycle, + visibility, + registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, + }), + error: 'name-collision', + }; + } + + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner, + lifecycle, + visibility, + }), + }; +} + +// --------------------------------------------------------------------------------------------- +// Scan-state cache +// --------------------------------------------------------------------------------------------- + +interface CacheRow { + source_id: string; + file_path: string; + mtime_ms: number; + content_hash: string; + entity_name: string; + last_error: string | null; +} + +class ScanStateCache { + constructor(private readonly knex: Knex) {} + + static async create(database: DatabaseService): Promise { + const knex = await database.getClient(); + // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered + // against the 'catalog' plugin shares that plugin's own database, so using knex's default + // `knex_migrations` table here would validate our one-migration `migrationSource` against the + // real catalog plugin's own (much longer) applied-migrations history and fail with + // "migration directory is corrupt". + await knex.migrate.latest({ + tableName: 'auto_api_registration_migrations', + migrationSource: { + async getMigrations() { + return ['001_scan_state_cache']; + }, + getMigrationName(migration: string) { + return migration; + }, + async getMigration() { + return scanStateCacheMigration; + }, + }, + }); + return new ScanStateCache(knex); + } + + async rowsForSource(sourceId: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); + } + + /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ + async rowsByEntityName(entityName: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); + } + + async upsert(row: CacheRow): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .insert(row) + .onConflict(['source_id', 'file_path']) + .merge(); + } + + async remove(sourceId: string, filePath: string): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .where({ source_id: sourceId, file_path: filePath }) + .delete(); + } +} + +function sha1(contents: string): string { + return crypto.createHash('sha1').update(contents).digest('hex'); +} + +// --------------------------------------------------------------------------------------------- +// EntityProvider (one instance per source) +// --------------------------------------------------------------------------------------------- + +async function mapLimit( + items: T[], + limit: number, + fn: (item: T) => Promise, +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + async function worker() { + while (next < items.length) { + const index = next++; + results[index] = await fn(items[index]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); + return results; +} + +class AutoApiRegistrationEntityProvider implements EntityProvider { + private connection?: EntityProviderConnection; + private hasRunOnce = false; + + constructor( + private readonly source: SourceConfig, + private readonly cache: ScanStateCache, + private readonly catalogApi: CatalogService, + private readonly auth: AuthService, + private readonly logger: LoggerService, + ) {} + + getProviderName(): string { + return `auto-api-registration:${this.source.id}`; + } + + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + async runCycle(): Promise { + if (!this.connection) { + this.logger.warn( + `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, + ); + return; + } + + const existingRows = await this.cache.rowsForSource(this.source.id); + const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); + + const currentPaths = new Set(); + for await (const filePath of discoverFiles(this.source)) { + currentPaths.add(filePath); + } + + const knownOwnerRefs = await this.fetchKnownOwnerRefs(); + + const changedPaths = [...currentPaths].filter(filePath => { + const cached = existingByPath.get(filePath); + if (!cached) return true; + const stat = fs.statSync(filePath); + return stat.mtimeMs !== cached.mtime_ms; + }); + + const added: DeferredEntity[] = []; + const removedRefs: string[] = []; + const upserts: CacheRow[] = []; + const deletes: string[] = []; + + await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { + const cached = existingByPath.get(filePath); + const stat = fs.statSync(filePath); + const contents = fs.readFileSync(filePath, 'utf8'); + const hash = sha1(contents); + + if (cached && cached.content_hash === hash) { + // mtime touched, content unchanged — refresh mtime only, no mutation needed. + upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); + return; + } + + const parsed = parseSpecFile(contents); + if (!parsed) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + const slug = slugify(parsed.title); + + // Precedence check: a hand-authored catalog-info.yaml for this name wins. + const precedenceWinner = await this.findNonAutoSourcedEntity(slug); + if (precedenceWinner) { + this.logger.info( + `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + // Global collision check: excludes this file's own prior row. + const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( + row => !(row.source_id === this.source.id && row.file_path === filePath), + ); + const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; + + const mapped = mapCandidate({ + providerName: this.getProviderName(), + source: this.source, + filePath, + contents, + parsed, + knownOwnerRefs, + collidesWith, + }); + + if (mapped.error) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, + ); + } + + added.push({ entity: mapped.entity }); + upserts.push({ + source_id: this.source.id, + file_path: filePath, + mtime_ms: stat.mtimeMs, + content_hash: hash, + entity_name: mapped.entityName, + last_error: mapped.error ?? null, + }); + }); + + for (const [filePath, row] of existingByPath) { + if (!currentPaths.has(filePath)) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); + deletes.push(filePath); + } + } + + if (!this.hasRunOnce) { + // First run: establish the baseline with one `full` mutation. + await this.connection.applyMutation({ type: 'full', entities: added }); + this.hasRunOnce = true; + } else if (added.length > 0 || removedRefs.length > 0) { + await this.connection.applyMutation({ + type: 'delta', + added, + removed: removedRefs.map(entityRef => ({ entityRef })), + }); + } + + for (const row of upserts) { + await this.cache.upsert(row); + } + for (const filePath of deletes) { + await this.cache.remove(this.source.id, filePath); + } + } + + private async fetchKnownOwnerRefs(): Promise> { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, + { credentials }, + ); + return new Set(response.items.map(entity => stringifyEntityRef(entity))); + } + + private async findNonAutoSourcedEntity(entityName: string): Promise { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: 'API', 'metadata.name': entityName } }, + { credentials }, + ); + return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); + } +} + +// --------------------------------------------------------------------------------------------- +// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI +// --------------------------------------------------------------------------------------------- + +class AutoApiRegistrationErrorProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'auto-api-registration-error-processor'; + } + + async preProcessEntity( + entity: Entity, + _location: unknown, + _emit: CatalogProcessorEmit, + ): Promise { + const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; + if (message) { + throw new InputError(message); + } + return entity; + } +} + +// --------------------------------------------------------------------------------------------- +// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) +// --------------------------------------------------------------------------------------------- + +export default createBackendModule({ + pluginId: 'catalog', + moduleId: 'auto-api-registration', + register(reg) { + reg.registerInit({ + deps: { + catalog: catalogProcessingExtensionPoint, + database: coreServices.database, + scheduler: coreServices.scheduler, + logger: coreServices.logger, + rootConfig: coreServices.rootConfig, + catalogApi: catalogServiceRef, + auth: coreServices.auth, + }, + async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { + const sources = normalizeConfig(rootConfig); + if (sources.length === 0) { + logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); + return; + } + + const cache = await ScanStateCache.create(database); + + const providers = sources.map( + source => + new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), + ); + + catalog.addEntityProvider(providers); + catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); + + for (const [index, source] of sources.entries()) { + const provider = providers[index]; + + if (source.mode === 'poll') { + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:poll`, + frequency: { seconds: source.scheduleFrequencySeconds }, + timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); + } + }, + }); + } else { + // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of + // just this source on add/change/unlink, coalesced by chokidar's own event batching. + // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer + // `reconciliation.frequencySeconds` safety-net sweep below. + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:reconciliation`, + frequency: { seconds: source.reconciliationFrequencySeconds }, + timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); + } + }, + }); + + const watcher = chokidar.watch(source.patterns, { + cwd: source.rootPath, + ignored: source.ignore, + ignoreInitial: true, + }); + let debounce: NodeJS.Timeout | undefined; + const triggerRescan = () => { + clearTimeout(debounce); + debounce = setTimeout(() => { + provider + .runCycle() + .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); + }, 500); + }; + watcher.on('add', triggerRescan); + watcher.on('change', triggerRescan); + watcher.on('unlink', triggerRescan); + } + } + }, + }); + }, +}); +``` + +--- + +## Step 3 — Register the Module + +Open `packages/backend/src/index.ts` and add one line after the existing catalog module +registrations: + +```typescript +// See https://backstage.io/docs/features/software-catalog/configuration#subscribing-to-catalog-errors +backend.add(import('@backstage/plugin-catalog-backend-module-logs')); + +// Lab 4: auto-discover *-openapi.yaml / *-asyncapi.yaml files and register them as API entities +// with no hand-authored catalog-info.yaml required — see labs/lab-04-auto-registration/README.md +backend.add(import('./extensions/autoApiRegistration')); +``` + +Like `permissionPolicy.ts` in Lab 3, the module self-registers and declares its own dependencies +via `createBackendModule` — no other wiring is needed. + +--- + +## Step 4 — Configure Discovery + +Add an `autoApiRegistration` block to `app-config.yaml`. This is the **single-source shorthand** +— equivalent to a one-entry `sources: [{ id: default, ... }]` list — which is all a single-repo +setup like this lab needs: + +```yaml +autoApiRegistration: + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp +``` + +`rootPath` is deliberately left unset here — the module resolves a default of +`labs/lab-04-auto-registration/apis` (relative to the backend package) on its own. `patterns`, +`ignore`, `mode`, and `schedule.frequencySeconds` all default sensibly too (see "Adaptable +Conventions vs. Fixed Mechanics" below for the full list). `defaultOwner` and `xNamespace` have no +built-in default and must be set explicitly — a source shouldn't silently inherit a fallback team +or vendor namespace it never opted into. + +--- + +## Step 5 — Add the Sample API Files + +Two sample files are already committed alongside this README (nothing to create in this step — +just take a look): + +- **`apis/galaxy/galaxy-openapi.yaml`** — a vendored copy of the MIT-licensed + [Scalar Galaxy API](https://github.com/scalar/scalar), with an `info.x-examplecorp` object + (`owner`, `lifecycle`, `visibility: shared`) added. It has **no** `catalog-info.yaml` — this is + the "previously unregistered API" that Step 6 will confirm gets discovered automatically. +- **`apis/precedence-demo/precedence-demo-openapi.yaml`** + **`precedence-demo-catalog-info.yaml`** + — a minimal API paired with a *hand-authored* catalog entry (owner `museum-team`, + `visibility: private`) that deliberately differs from the spec file's own `x-examplecorp` + values (owner `platform-team`, `visibility: shared`). This demonstrates the precedence rule: + the hand-authored file wins. + +The hand-authored `precedence-demo-catalog-info.yaml` needs its own catalog location, the same way +Lab 2/3's hand-authored files do — add it to `app-config.yaml`'s `catalog.locations`, replacing +`` and `` with your fork and the current branch name: + +```yaml +catalog: + locations: + # --- Lab 4: Precedence-demo API (hand-authored — must win over auto-sourced metadata) --- + - type: url + target: https://raw.githubusercontent.com//backstage-apiportal-lab//labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml + rules: + - allow: [API] +``` + +This only works once you've pushed your branch — the raw URL has to actually resolve. If you +haven't pushed yet, push now before continuing to Step 6. + +--- + +## Step 6 — Start Backstage and Verify Discovery + +``` +yarn start +``` + +Within one discovery cycle (default 30s), open the catalog and confirm: + +- **The Galaxy API appears with no hand-authored `catalog-info.yaml`.** Search for "Scalar + Galaxy" — it should show up as an `API` entity even though you never wrote a `catalog-info.yaml` + for it. +- **Its owner, lifecycle, and tags match the spec.** `spec.owner` and `spec.lifecycle` should + match the `info.x-examplecorp` values in `galaxy-openapi.yaml`; `metadata.tags` should match its + native `tags` array (slugified — kebab-case, not the original mixed-case names). +- **Visibility contrast.** Sign in as a non-owning, non-platform user (e.g. `charlie`, from Lab + 2 — see `app-config.local.yaml`). The Galaxy API (`visibility: shared`) should be visible; the + precedence-demo API (`visibility: private`, hand-authored, owned by `museum-team`) should not be. +- **Precedence.** The precedence-demo API's entity should reflect the hand-authored + `precedence-demo-catalog-info.yaml` (owner `museum-team`, `visibility: private`) — **not** the + auto-sourced `x-examplecorp` values in its own spec file (owner `platform-team`, + `visibility: shared`). + +--- + +## Step 7 — Verify Update-in-Place + +Edit `apis/galaxy/galaxy-openapi.yaml`'s `info.description` and wait one poll cycle. Reload the +Galaxy API's entity page — the description should update in place. The entity's `metadata.uid` +does not change; this is an update, not a duplicate registration. + +--- + +## Step 8 — Verify Removal + +Temporarily rename `galaxy-openapi.yaml` to something outside the discovery pattern (e.g. +`galaxy-openapi.yaml.bak`) and wait one poll cycle. The Galaxy API should disappear from the +catalog. Rename it back — it reappears within one more cycle, as a fresh registration (a new +`metadata.uid`, since from the provider's perspective this is indistinguishable from a new file). + +--- + +## Step 9 — Verify Error Handling + +Three failure modes, each temporary — revert after checking: + +1. **Malformed YAML.** Break the syntax in one sample file (e.g. an unclosed quote). Check the + backend logs — you should see a line identifying the specific file, and the entity for that + file should simply stop appearing in the catalog (not appear broken; log-only, since a file + that doesn't parse has no name to register an error entity under). +2. **Unresolvable owner.** Set `info.x-examplecorp.owner` to a nonexistent team (e.g. + `group:default/nonexistent-team`). Within one cycle, the entity's "Inspect entity" view should + show a catalog processing error naming the bad owner reference. Other entities are unaffected. +3. **Invalid visibility.** Set `info.x-examplecorp.visibility` to an unrecognized value (e.g. + `public`). You should see the same kind of processing error — not a silent fallback to + `private`. + +--- + +## Adaptable Conventions vs. Fixed Mechanics + +**Adaptable — change these to fit your own repo layout:** + +- `rootPath` — where the scan starts. Point it at your own mono-repo root. +- `patterns` — the filename glob(s) that identify an API definition file. The default + (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`) is this lab's convention, not Backstage's; use + whatever your team already follows. +- `xNamespace` — the vendor namespace under `info.x-` that owner/lifecycle/visibility + are read from. `examplecorp` is a fictional placeholder — **rename this to your own company or + team first** when adapting this lab. +- `defaultOwner` / `defaultVisibility` — the fallback values used when a spec doesn't declare its + own. Per-source, so different repos can have different fallbacks. +- `ignore` — glob exclusions, always applied. Extend (don't replace) the defaults + (`node_modules`, `.git`, `dist`, `build`) for your own build-output directories. + +**Fixed — this is how the mechanism works, not a config knob:** + +- The `EntityProvider` full-then-delta mutation lifecycle (Step 2's rationale above). +- The scan-state cache schema and its role in restart/delta behavior (see "Scaling to a Real + Mono-Repo" below). +- Errors riding on Backstage's native backend-log + processing-error-UI mechanisms, rather than a + bespoke error system. +- The catalog-info.yaml precedence rule: a hand-authored entity for a name always wins over an + auto-sourced candidate for that same name. + +--- + +## Scaling to a Real Mono-Repo + +This lab's defaults (`mode: 'poll'`, 30-second full sweep, no reconciliation) are tuned for 2–3 +sample files and interactive feedback — they are **not** what a 1000+ file, 1GB+ mono-repo should +run with. The architecture (an `EntityProvider` driving mutations) scales fine as-is; only the +config values need to change: + +- **`mode: 'watch'`** replaces the repeated full-tree poll with a `chokidar` watcher that reacts + to individual file add/change/unlink events, so "how often do we rescan" stops scaling with + repo size. A much longer `reconciliation.frequencySeconds` (minutes, not seconds) runs + alongside it as a safety-net full sweep, catching anything a watcher event could plausibly miss + (coalesced events under a big `git pull`, watcher reliability limits on some filesystems). +- **The scan-state cache is what makes restarts fast at scale.** It's a small table in the + backend's own database (`coreServices.database` — SQLite in this lab, Postgres in a real + deployment), storing each file's path, modification time, content hash, and mapped entity name. + On restart, the provider compares this cache against the filesystem instead of re-parsing + everything from zero — unchanged files cost nothing. This is deliberately **not** written back + into the mono-repo as generated `catalog-info.yaml` files: that would reintroduce the + hand-maintained-duplicate-file problem this lab exists to eliminate, and would require the + discovery tool to have *write* access to a repo it should only need to read. +- After the first `full` mutation establishes the baseline, every later cycle sends a `delta` + mutation (`added`/`removed` only) — cost proportional to what changed, not to total catalog + size. This is the same pattern Backstage's own GitHub discovery provider uses at organization + scale. +- `parseConcurrency` bounds how many files are parsed at once, so a large batch of simultaneous + changes (e.g. checking out a branch that touches hundreds of files) doesn't spike memory or + block the event loop. + +If you scale this lab up, expect restarts to be fast (cache-driven, not a full re-scan) — that's +the part most likely to surprise you the first time. + +--- + +## Scaling to Multiple Source Repositories + +The config schema generalizes from one `rootPath` to a list of independently-configured sources: + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo + rootPath: ../platform-monorepo/apis + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + mode: watch + reconciliation: + frequencySeconds: 900 + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp + - id: team-checkout-api + rootPath: ../team-checkout-api + patterns: ['**/*-openapi.yaml'] + mode: poll + schedule: + frequencySeconds: 60 + defaultOwner: group:default/checkout-team + defaultVisibility: shared + xNamespace: checkoutteam +``` + +Each entry becomes its own `EntityProvider` instance with its own schedule, own mode, own default +owner, and own `x-*` namespace — a slow filesystem or config problem in one source can't stall or +break discovery for any other source. + +`defaultOwner` and `xNamespace` are per-source with **no** built-in global fallback: a team +onboarding their own repo shouldn't have to adopt the platform mono-repo's team or vendor +namespace just to participate. `defaultVisibility` is the one exception — it's per-source but +*does* fall back to `private` if a source doesn't set it, because leaving visibility completely +unconfigured must still be safe by default, not just adaptable. + +Collision detection and the catalog-info.yaml precedence check are **global across every source**, +not scoped to whichever source is currently being scanned — two different teams' repos can easily +produce the same slugified entity name (two `payments-api`s), and that must surface a visible +conflict rather than silently coexisting or overwriting. + +Onboarding a genuinely separate Git remote (not a local sibling checkout, which is what this lab's +own multi-source walkthrough would use) needs one more piece this lab doesn't build: a sync step +(e.g. a scheduled shallow `git clone`/`pull` into a local cache directory) ahead of the existing +glob/parse pipeline, pointing that source's `rootPath` at the resulting local worktree. Every other +mechanism — glob patterns, ignore rules, delta mutations, scan-state cache, collision/precedence +checks — is reused unchanged; only how bytes arrive on local disk differs. + +--- + +## Verification Checklist + +- [ ] Galaxy API appears in the catalog within one poll cycle, with no hand-authored + `catalog-info.yaml` +- [ ] Galaxy API's owner/lifecycle match its `x-examplecorp` values; tags match its native `tags` + array +- [ ] Editing `galaxy-openapi.yaml`'s description updates the existing entity, not a duplicate +- [ ] Renaming `galaxy-openapi.yaml` out of the discovery pattern retracts the entity; renaming it + back re-registers it +- [ ] Galaxy (`shared`) is visible to a non-owning, non-platform user; precedence-demo + (`private`, hand-authored) is not +- [ ] The precedence-demo API reflects `precedence-demo-catalog-info.yaml`'s values, not its own + spec file's `x-examplecorp` values +- [ ] A broken-YAML file logs an identifying error and its entity does not appear +- [ ] A nonexistent owner reference surfaces a catalog processing error on that entity only +- [ ] An unrecognized visibility value surfaces a processing error rather than a silent default + +--- + +## Troubleshooting + +- **Nothing appears after 30+ seconds.** Check the backend logs for + `auto-api-registration:default:` lines. A "skipped a cycle — provider not yet connected" + warning on the very first tick is normal (the scheduler's first run can fire slightly before + the catalog engine finishes wiring up the provider); it should not repeat. +- **"Owner ... does not resolve to a known User or Group entity" for an owner you're sure + exists.** Confirm that group is actually loaded in the catalog (check its entity page directly) + — this error means the *catalog* doesn't know about the group yet, not that your YAML is wrong. + A common cause locally is a `catalog.locations` URL that hasn't been pushed yet (see Step 5). +- **An entity you expect to see is silently missing, with no error logged.** Check the backend + logs for a `Policy check failed for api:default/` warning — this means the entity was + built and emitted, but failed Backstage's own entity-schema validation (for example, + `metadata.tags` containing something that isn't a valid kebab-case slug) during processing, so + it never got "stitched" into the catalog you can query. This is different from a registration + error (which the error processor surfaces intentionally) — it's a schema mismatch the entity + never should have had in the first place. +- **A stale entity keeps reappearing after you thought you removed it.** The scan-state cache + persists across backend restarts (it's a real database table, not in-memory). If you manually + edited the database or copied it between environments, the cache may be out of sync with the + filesystem. Deleting the `auto_api_registration_scan_state` table (or just wiping the local + SQLite file, per this lab's `':memory:'` dev config, restarting achieves the same thing) forces + a fresh full rescan. +- **Glob pattern typos.** If a file you expect to be discovered isn't, double check `patterns` — + a mismatched suffix (`-openapi.yml` vs `-openapi.yaml`) is a common culprit, and the discovery + mechanism has no way to tell you "this almost matched." +- **Wrong `xNamespace`.** If owner/lifecycle/visibility all come back as defaults even though your + spec file has an `x-*` object, check that the object's key matches `xNamespace` exactly — + `x-examplecorp` in config but `x-example-corp` (extra hyphen) in the spec file will silently + fall through to defaults rather than error, since an absent `x-*` object is valid input. diff --git a/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml new file mode 100644 index 0000000..053975a --- /dev/null +++ b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml @@ -0,0 +1,1476 @@ +openapi: 3.1.1 +info: + title: Scalar Galaxy + description: | + The Scalar Galaxy is an example OpenAPI document to test OpenAPI tools and libraries. It's a fictional universe with fictional planets and fictional data. + + ## Resources + + * https://github.com/scalar/scalar + * https://github.com/OAI/OpenAPI-Specification + * https://scalar.com + + ## Markdown Support + + All descriptions *can* contain ~~tons of text~~ **Markdown**. [If GitHub supports the syntax](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax), chances are we're supporting it, too. + +
+ Examples + + **Blockquotes** + + > I love OpenAPI. <3 + + **Tables** + + | Feature | Availability | + | ---------------- | ------------ | + | Markdown Support | ✓ | + + **Accordion** + + ```html +
+ Using Details Tags +

HTML Example

+
+ ``` + + **Images** + + Yes, there's support for images, too! + + ![Empty placeholder image showing the width/height](https://images.placeholders.dev/?width=1280&height=720) + + **Alerts** + + > [!tip] + > You can use Markdown alerts in your descriptions. + +
+ version: 0.6.8 + contact: + name: Scalar Support + url: https://scalar.com + email: support@scalar.com + license: + name: MIT + url: https://opensource.org/license/MIT + # Lab 4: catalog metadata sourced by autoApiRegistration (owner/lifecycle/visibility) — + # see labs/lab-04-auto-registration/README.md. No catalog-info.yaml exists for this API; + # it is discovered and registered purely from this file. + x-examplecorp: + owner: group:default/platform-team + lifecycle: production + visibility: shared +externalDocs: + description: Documentation + url: https://github.com/scalar/scalar +servers: + - url: https://galaxy.scalar.com + - url: '{protocol}://void.scalar.com/{path}' + description: Responds with your request data + variables: + protocol: + enum: + - https + - http + default: https + path: + default: '' +security: + - bearerAuth: [] + - basicAuth: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + - apiKeyHeader: [] + apiKeyQuery: [] + - apiKeyCookie: [] + - oAuth2: [] + - openIdConnect: [] +x-speakeasy-webhooks: + security: + type: signature + headerName: x-signature + signatureTextEncoding: base64 + algorithm: hmac-sha256 +tags: + - name: Authentication + description: Some endpoints are public, but some require authentication. We + provide all the required endpoints to create an account and authorize + yourself. + - name: Planets + description: Everything about planets + - name: Celestial Bodies + description: Celestial bodies are the planets and satellites in the Scalar Galaxy. +paths: + /planets: + get: + tags: + - Planets + summary: Get all planets + description: + It's easy to say you know them all, but do you really? Retrieve all + the planets and check whether you missed one. + operationId: getAllData + security: [] + parameters: + - $ref: '#/components/parameters/limit' + - $ref: '#/components/parameters/offset' + responses: + '200': + description: OK + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Pagination-Total: + $ref: '#/components/headers/X-Pagination-Total' + X-Pagination-Page: + $ref: '#/components/headers/X-Pagination-Page' + X-Pagination-Per-Page: + $ref: '#/components/headers/X-Pagination-Per-Page' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanets' + # Bind the generic Paginated template's item type to Planet via a $dynamicAnchor. + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + application/xml: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanetsXml' + xml: + name: planets + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has data array and meta", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("data"); + pm.expect(jsonData.data).to.be.an("array"); + pm.expect(jsonData).to.have.property("meta"); + }); + post: + tags: + - Planets + summary: Create a planet + description: + Time to play god and create a new planet. What do you think? Ah, + don't think too much. What could go wrong anyway? + operationId: createPlanet + callbacks: + planetCreated: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly created planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the callback + '204': + description: + Your server should return this HTTP status code if no longer + interested in further updates + planetCreationFailed: + '{$request.body#/failureCallbackUrl}': + post: + security: [] + requestBody: + description: Information about which fields failed to validate + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the failure callback notification + planetExploded: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly exploded planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the planet explosion callback notification + requestBody: + description: Planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + responses: + '201': + description: Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /planets/{planetId}: + get: + tags: + - Planets + summary: Get a planet + description: + You'll better learn a little bit more about the planets. It might + come in handy once space travel is available for everyone. + operationId: getPlanet + security: [] + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet Found + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + put: + tags: + - Planets + summary: Update a planet + description: Sometimes you make mistakes, that's fine. No worries, you can + update all planets. + operationId: updatePlanet + requestBody: + description: New information about the planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet updated successfully + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + delete: + tags: + - Planets + summary: Delete a planet + operationId: deletePlanet + description: This endpoint was used to delete planets. Unfortunately, that + caused a lot of trouble for planets with life. So, this endpoint is now + deprecated and should not be used anymore. + x-scalar-stability: experimental + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '204': + description: No Content + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 204", () => { + pm.response.to.have.status(204); + }); + /planets/{planetId}/image: + post: + tags: + - Planets + summary: Upload an image to a planet + description: Got a crazy good photo of a planet? Share it with the world! + operationId: uploadImage + parameters: + - $ref: '#/components/parameters/planetId' + requestBody: + description: Image to upload + content: + multipart/form-data: + schema: + type: object + properties: + image: + type: string + format: binary + description: The image file to upload + examples: + - '@mars.jpg' + - '@jupiter.png' + responses: + '200': + $ref: '#/components/responses/ImageUploaded' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has message and imageUrl", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("message"); + pm.expect(jsonData).to.have.property("imageUrl"); + }); + /celestial-bodies: + post: + tags: + - Celestial Bodies + summary: Create a celestial body + description: Stars, moons, comets, the occasional rogue asteroid — if it + glows or drifts through the void, you can add it here. + operationId: createCelestialBody + requestBody: + description: Celestial body to create + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + responses: + '201': + description: Celestial body created + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a celestial body with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /user/signup: + post: + tags: + - Authentication + summary: Create a user + description: Time to create a user account, eh? + operationId: createUser + security: [] + requestBody: + description: User to create + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + application/xml: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + responses: + '201': + description: User account created successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '422': + $ref: '#/components/responses/UnprocessableEntity' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /auth/token: + post: + tags: + - Authentication + summary: Get a token + description: + Yeah, this is the boring security stuff. Just get your super secret + token and move on. + operationId: getToken + security: [] + requestBody: + description: Body for credentials to authenticate a user + content: + application/json: + schema: + $ref: '#/components/schemas/Credentials' + application/xml: + schema: + $ref: '#/components/schemas/Credentials' + responses: + '201': + description: Token Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + content: + application/json: + schema: + $ref: '#/components/schemas/Token' + application/xml: + schema: + $ref: '#/components/schemas/Token' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '429': + $ref: '#/components/responses/TooManyRequests' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response contains token", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("token"); + pm.expect(jsonData.token).to.be.an("string"); + }); + /me: + get: + tags: + - Authentication + summary: Get authenticated user + description: Find yourself they say. That's what you can do here. + operationId: getMe + security: + - basicAuth: [] + - oAuth2: + - read:account + - bearerAuth: [] + - apiKeyHeader: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + apiKeyQuery: [] + responses: + '200': + description: Authenticated user information retrieved successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); +webhooks: + newPlanet: + post: + tags: + - Planets + security: [] + requestBody: + description: Information about a new planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: + Return a 200 status to indicate that the data was received + successfully + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: JWT Bearer token authentication + basicAuth: + type: http + scheme: basic + description: Basic HTTP authentication + apiKeyHeader: + type: apiKey + in: header + name: X-API-Key + description: API key request header + apiKeyQuery: + type: apiKey + in: query + name: api_key + description: API key query parameter + apiKeyCookie: + type: apiKey + in: cookie + name: api_key + description: API key browser cookie + oAuth2: + type: oauth2 + description: OAuth 2.0 authentication + flows: + authorizationCode: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + clientCredentials: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + implicit: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + password: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + openIdConnect: + type: openIdConnect + openIdConnectUrl: https://galaxy.scalar.com/.well-known/openid-configuration + description: OpenID Connect Authentication + parameters: + planetId: + name: planetId + description: The ID of the planet to get + in: path + required: true + schema: + type: integer + format: int64 + examples: + - 1 + limit: + name: limit + description: The number of items to return + in: query + required: false + schema: + type: integer + format: int64 + default: 10 + offset: + name: offset + description: The number of items to skip before starting to collect the result set + in: query + required: false + schema: + type: integer + format: int64 + default: 0 + headers: + X-RateLimit-Limit: + description: The number of allowed requests in the current period + schema: + type: integer + example: 100 + X-RateLimit-Remaining: + description: The number of remaining requests in the current period + schema: + type: integer + example: 95 + X-RateLimit-Reset: + description: The number of seconds left in the current period + schema: + type: integer + example: 3600 + X-Request-ID: + description: Unique identifier for the request + schema: + type: string + format: uuid + example: 123e4567-e89b-12d3-a456-426614174000 + X-Pagination-Total: + description: Total number of items available + schema: + type: integer + example: 1000 + X-Pagination-Page: + description: Current page number + schema: + type: integer + example: 1 + X-Pagination-Per-Page: + description: Number of items per page + schema: + type: integer + example: 10 + X-Cache-Control: + description: Cache control directives + schema: + type: string + example: max-age=3600, public + ETag: + description: Entity tag for caching + schema: + type: string + example: '"33a64df551"' + Last-Modified: + description: The last modification date of the resource + schema: + type: string + format: date-time + example: 2023-12-01T12:00:00Z + Location: + description: The URL of the newly created resource + schema: + type: string + format: uri + example: https://galaxy.scalar.com/planets/123 + X-Processing-Time: + description: The time taken to process the request in milliseconds + schema: + type: integer + example: 150 + responses: + ImageUploaded: + description: Image uploaded + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + application/xml: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + BadRequest: + description: Bad Request + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + content: + application/json: + schema: + $ref: '#/components/schemas/BadRequestError' + application/xml: + schema: + $ref: '#/components/schemas/BadRequestError' + Forbidden: + description: Forbidden + content: + application/json: + schema: + $ref: '#/components/schemas/ForbiddenError' + application/xml: + schema: + $ref: '#/components/schemas/ForbiddenError' + NotFound: + description: Not Found + content: + application/json: + schema: + $ref: '#/components/schemas/NotFoundError' + application/xml: + schema: + $ref: '#/components/schemas/NotFoundError' + Unauthorized: + description: Unauthorized + content: + application/json: + schema: + $ref: '#/components/schemas/UnauthorizedError' + application/xml: + schema: + $ref: '#/components/schemas/UnauthorizedError' + Conflict: + description: Conflict + content: + application/json: + schema: + $ref: '#/components/schemas/Conflict' + application/xml: + schema: + $ref: '#/components/schemas/Conflict' + UnprocessableEntity: + description: Unprocessable Entity + content: + application/json: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + application/xml: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + TooManyRequests: + description: Too Many Requests + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + Retry-After: + description: The number of seconds to wait before retrying + schema: + type: integer + example: 60 + content: + application/json: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + application/xml: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + schemas: + User: + description: A user + type: object + xml: + name: user + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Marc + - Cam + - Hans + Credentials: + description: Credentials to authenticate a user + type: object + required: + - email + - password + properties: + email: + type: string + format: email + examples: + - marc@scalar.com + - cam@scalar.com + - hans@scalar.com + password: + type: string + writeOnly: true + examples: + - i-love-scalar + - i-love-oss + - qwerty123 + Token: + description: A token to authenticate a user + type: object + properties: + token: + type: string + examples: + - eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c + CelestialBody: + oneOf: + - $ref: '#/components/schemas/Planet' + - $ref: '#/components/schemas/Satellite' + discriminator: + propertyName: type + mapping: + terrestrial: '#/components/schemas/Planet' + gas_giant: '#/components/schemas/Planet' + ice_giant: '#/components/schemas/Planet' + dwarf: '#/components/schemas/Planet' + super_earth: '#/components/schemas/Planet' + moon: '#/components/schemas/Satellite' + asteroid: '#/components/schemas/Satellite' + comet: '#/components/schemas/Satellite' + description: A celestial body which can be either a planet or a satellite + Planet: + description: A planet in the Scalar Galaxy + type: object + required: + - id + - name + additionalProperties: false + xml: + name: planet + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + x-variable: planetId + name: + type: string + examples: + - Mars + - Jupiter + - HD 40307g + description: + type: + - string + - 'null' + examples: + - The red planet + - A gas giant with a great red spot + type: + type: string + enum: + - terrestrial + - gas_giant + - ice_giant + - dwarf + - super_earth + x-enum-varnames: + - Terrestrial + - GasGiant + - IceGiant + - Dwarf + - SuperEarth + x-enum-descriptions: + terrestrial: Rocky planets with solid surfaces, like Earth and Mars + gas_giant: + Large planets composed mainly of hydrogen and helium, like Jupiter + and Saturn + ice_giant: + Planets composed of water, ammonia, and methane ices, like Uranus and + Neptune + dwarf: Small planetary bodies that don't meet full planet criteria, like Pluto + super_earth: Rocky planets larger than Earth but smaller than gas giants + examples: + - terrestrial + habitabilityIndex: + type: number + format: float + minimum: 0 + maximum: 1 + description: A score from 0 to 1 indicating potential habitability + examples: + - 0.68 + physicalProperties: + type: object + additionalProperties: + x-additionalPropertiesName: measurement + type: number + format: float + description: Additional physical measurements for the planet + properties: + mass: + type: number + format: float + exclusiveMinimum: 0 + description: Mass in Earth masses (must be greater than 0) + examples: + - 0.107 + radius: + type: number + format: float + exclusiveMinimum: 0 + description: Radius in Earth radii (must be greater than 0) + examples: + - 0.532 + gravity: + type: number + format: float + description: Surface gravity in Earth g + examples: + - 0.378 + temperature: + type: object + additionalProperties: + x-additionalPropertiesName: temperatureMetric + type: number + format: float + description: Additional temperature-related measurements in Kelvin + properties: + min: + type: number + format: float + description: Minimum temperature in Kelvin + examples: + - 130 + max: + type: number + format: float + description: Maximum temperature in Kelvin + examples: + - 308 + average: + type: number + format: float + description: Average temperature in Kelvin + examples: + - 210 + atmosphere: + type: array + description: Atmospheric composition + items: + type: object + additionalProperties: + x-additionalPropertiesName: atmosphericData + type: string + description: Additional atmospheric composition data + properties: + compound: + type: string + examples: + - CO2 + - N2 + percentage: + type: number + format: float + exclusiveMaximum: 100 + examples: + - 95.3 + discoveredAt: + type: string + format: date-time + examples: + - 1610-01-07T00:00:00Z + image: + type: + - string + - 'null' + examples: + - https://cdn.scalar.com/photos/mars.jpg + satellites: + type: array + items: + $ref: '#/components/schemas/Satellite' + creator: + $ref: '#/components/schemas/User' + tags: + type: array + items: + type: string + examples: + - - solar-system + - rocky + - explored + lastUpdated: + type: string + format: date-time + readOnly: true + examples: + - 2024-01-15T14:30:00Z + successCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a successful operation + examples: + - https://example.com/webhook + failureCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a failed operation + examples: + - https://example.com/webhook + Satellite: + description: Every satellite in the Scalar Galaxy + type: object + required: + - name + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Phobos + description: + type: + - string + - 'null' + examples: + - Phobos is the larger and innermost of the two moons of Mars. + diameter: + type: number + format: float + description: Diameter in kilometers + examples: + - 22.2 + type: + type: string + enum: + - moon + - asteroid + - comet + x-enum-varnames: + - Moon + - Asteroid + - Comet + x-enum-descriptions: + moon: Natural satellites that orbit planets + asteroid: Rocky objects that orbit the sun, typically found in the asteroid belt + comet: + Icy bodies that release gas when approaching the sun, creating visible + tails + examples: + - moon + orbit: + type: object + properties: + planet: + $ref: '#/components/schemas/Planet' + orbitalPeriod: + type: number + format: float + description: Orbital period in Earth days + examples: + - 0.319 + distance: + type: number + format: float + description: Average distance from the planet in kilometers + examples: + - 9376 + Paginated: + description: + A generic paginated response. Specializing schemas bind the item type by declaring a + `$dynamicAnchor` named `itemType`, the JSON Schema 2020-12 way to express `Paginated`. + $id: 'https://galaxy.scalar.com/schemas/Paginated' + $defs: + itemType: + $dynamicAnchor: itemType + not: {} + type: object + properties: + data: + type: array + items: + $dynamicRef: '#itemType' + allOf: + - $ref: '#/components/schemas/PaginatedResource' + PaginatedResource: + description: A paginated resource + type: object + properties: + meta: + type: object + properties: + limit: + type: integer + format: int64 + examples: + - 10 + offset: + type: integer + format: int64 + examples: + - 0 + total: + type: integer + format: int64 + examples: + - 100 + next: + type: + - string + - 'null' + examples: + - /planets?limit=10&offset=10 + ImageUploadedMessage: + x-scalar-ignore: true + description: Message about an image upload + type: object + properties: + message: + type: string + examples: + - Image uploaded successfully + imageUrl: + type: string + description: The URL where the uploaded image can be accessed + examples: + - https://cdn.scalar.com/images/8f47c132-9d1f-4f83-b5a4-91db5ee757ab.jpg + uploadedAt: + type: string + format: date-time + description: Timestamp when the image was uploaded + examples: + - 2024-01-15T14:30:00Z + fileSize: + type: integer + description: Size of the uploaded image in bytes + examples: + - 1048576 + mimeType: + type: string + description: The content type of the uploaded image + examples: + - image/jpeg + - image/png + BadRequestError: + x-scalar-ignore: true + description: RFC 7807 (https://datatracker.ietf.org/doc/html/rfc7807) + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/bad-request + title: + type: string + examples: + - Bad Request + status: + type: integer + format: int64 + examples: + - 400 + detail: + type: string + examples: + - The request was invalid. + ForbiddenError: + x-scalar-ignore: true + description: Error response for forbidden access (RFC 7807). Returned when the user does not have permission to access the requested resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/forbidden + title: + type: string + examples: + - Forbidden + status: + type: integer + format: int64 + examples: + - 403 + detail: + type: string + examples: + - You are not authorized to access this resource. + NotFoundError: + x-scalar-ignore: true + description: Error response for resource not found (RFC 7807). Returned when the requested resource does not exist. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Not Found + status: + type: integer + format: int64 + examples: + - 404 + detail: + type: string + examples: + - The resource you are trying to access does not exist. + UnauthorizedError: + x-scalar-ignore: true + description: Error response for unauthorized access (RFC 7807). Returned when authentication is required or has failed. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Unauthorized + status: + type: integer + format: int64 + examples: + - 401 + detail: + type: string + examples: + - You are not authorized to access this resource. + Conflict: + x-scalar-ignore: true + description: Error response for resource conflicts (RFC 7807). Returned when the request conflicts with the current state of the resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/conflict + title: + type: string + examples: + - Conflict + status: + type: integer + format: int64 + examples: + - 409 + detail: + type: string + examples: + - The resource you are trying to access is in conflict. + UnprocessableEntity: + x-scalar-ignore: true + description: Error response for unprocessable entity (RFC 7807). Returned when the request is well-formed but contains semantic errors. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/unprocessable-entity + title: + type: string + examples: + - Unprocessable Entity + status: + type: integer + format: int64 + examples: + - 422 + detail: + type: string + examples: + - The request was invalid. + TooManyRequestsError: + x-scalar-ignore: true + description: Error response for rate limiting (RFC 7807). Returned when the client has exceeded the rate limit for requests. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/too-many-requests + title: + type: string + examples: + - Too Many Requests + status: + type: integer + format: int64 + examples: + - 429 + detail: + type: string + examples: + - Rate limit exceeded. Please try again later. diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml new file mode 100644 index 0000000..6e79bf3 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml @@ -0,0 +1,21 @@ +# Hand-authored entity for the precedence-demo API — deliberately NOT auto-discovered from +# precedence-demo-openapi.yaml's own x-examplecorp values. This file demonstrates FR-011: a +# hand-authored catalog-info.yaml takes precedence over auto-sourced metadata for the same API. +# See labs/lab-04-auto-registration/README.md. +apiVersion: backstage.io/v1alpha1 +kind: API +metadata: + name: precedence-demo-api + title: Precedence Demo API + description: > + Hand-authored registration for the precedence-demo API — this entity, not the auto-sourced + candidate from precedence-demo-openapi.yaml's x-examplecorp object, is what appears in the + catalog (FR-011). + annotations: + example.com/visibility: private +spec: + type: openapi + lifecycle: production + owner: group:default/museum-team + definition: + $text: https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/004-lab-4-auto-registration/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml new file mode 100644 index 0000000..0650e37 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml @@ -0,0 +1,59 @@ +openapi: 3.1.0 +info: + title: Precedence Demo API + description: > + A minimal API used only to demonstrate FR-011: a hand-authored catalog-info.yaml + (precedence-demo-catalog-info.yaml, alongside this file) takes precedence over the metadata + auto-sourced from this file's own x-examplecorp object. See labs/lab-04-auto-registration/README.md. + version: 1.0.0 + # Deliberately different from the hand-authored catalog-info.yaml's owner/visibility, so the + # precedence check has something real to demonstrate (FR-011). + x-examplecorp: + owner: group:default/platform-team + lifecycle: experimental + visibility: shared +tags: + - name: Demo + description: Precedence demonstration endpoints +paths: + /widgets: + get: + tags: [Demo] + summary: List widgets + operationId: listWidgets + responses: + '200': + description: A list of widgets + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/Widget' + /widgets/{widgetId}: + get: + tags: [Demo] + summary: Get a widget + operationId: getWidget + parameters: + - name: widgetId + in: path + required: true + schema: + type: string + responses: + '200': + description: A single widget + content: + application/json: + schema: + $ref: '#/components/schemas/Widget' +components: + schemas: + Widget: + type: object + properties: + id: + type: string + name: + type: string diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md index e8c3c08..6070187 100644 --- a/specs/004-lab-4-auto-registration/data-model.md +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -27,14 +27,14 @@ Standard Backstage `API` kind entity, produced by the EntityProvider's full muta | `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | | `metadata.title` | `info.title` | Verbatim. | | `metadata.description` | `info.description` | Verbatim if present, else omitted. | -| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. | +| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. **Confirmed during implementation**: Backstage's `metadata.tags` format requires kebab-case slugs (`[a-z0-9+#]` separated by `-`); native spec tag names (e.g. `"Celestial Bodies"`) are free text, so each is slugified with the same slug function used for `metadata.name` before being written — otherwise the entity fails catalog policy validation and is silently dropped during processing (never reaches the catalog, no user-visible error). | | `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | | `metadata.annotations['example.com/visibility']` | `info.x-examplecorp.visibility` | **The exact same annotation key the Lab 2/3 permission policy already reads** (`packages/backend/src/extensions/permissionPolicy.ts`) — no new annotation, no policy change. **Mapping Rule 2**: if absent, falls back to that file's source's configured `defaultVisibility` — itself optional per-source, falling back to the fixed `'private'` "default default" if the source doesn't set one either (research.md R3/R7, FR-006a). An unrecognized (non-`private`/`shared`) value is never silently defaulted — it's a marker/error entity instead (research.md R4). Reused verbatim rather than reinvented, per research.md R3. | | `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref, invalid visibility value, or name collision); absent on normally-registered entities. | | `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | | `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | | `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | -| `spec.definition.$text` | discovery | The source file's own path/URL — same `$text` mechanism used by hand-authored entities in Labs 1–3 (per the existing project convention noted in memory: `$text` requires `type: url`-style resolution, not inline embedding). | +| `spec.definition` | discovery | **Revised during implementation**: `spec.definition` is a plain `string` field (not an object), and Backstage's default `UrlReaderService` stack has no `file:`-scheme reader — a `{ $text: 'file:...' }` placeholder (the originally-planned approach, mirroring Labs 1–3's hand-authored `$text` + `type: url` pattern) fails with `NotAllowedError` for a local filesystem path with no matching `reading.allow` host entry. Since the provider already holds the file's bytes in memory at parse time, `spec.definition` is populated with that raw file content directly — no read-back, no placeholder resolution, no duplication (the file itself isn't a second copy; it's the copy). The `$text`/`type: url` pattern remains correct and unchanged for genuinely hand-authored `catalog-info.yaml` files (e.g. `precedence-demo-catalog-info.yaml`), which reference a *different* file over a real network-reachable URL — this revision only applies to entities this provider constructs programmatically from a file it already read. | ## Config: `autoApiRegistration` (app-config.yaml) diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md index 51057e4..c950573 100644 --- a/specs/004-lab-4-auto-registration/research.md +++ b/specs/004-lab-4-auto-registration/research.md @@ -40,10 +40,14 @@ path for `**/*-openapi.yaml` and `**/*-asyncapi.yaml`, on a scheduled interval v hammering the filesystem). **Root path**: configurable via `autoApiRegistration.rootPath` in `app-config.yaml`; documented -as an adaptable convention (FR-010). Default value resolves from the backend process's working -directory (`packages/backend` under `backstage-cli package start`) up to the repository root — -exact relative segment count will be confirmed experimentally during implementation and recorded -here as a Run update, consistent with how Lab 3's research.md captured verified findings. +as an adaptable convention (FR-010). Default value resolves via `resolvePackagePath('backend')` +(from `@backstage/backend-plugin-api`) rather than `process.cwd()` directly — `resolvePackagePath` +locates the `packages/backend` directory via Node module resolution of its own `package.json` +(`name: "backend"`), which is robust regardless of which working directory the backend process was +actually launched from, unlike a `process.cwd()`-relative assumption. From there, 4 `..` segments +(`packages/backend` → `packages` → `backstage` → `lab-01-base-backstage` → `labs`) reach `labs/`, +then into `lab-04-auto-registration/apis` — confirmed experimentally during implementation +(`packages/backend/src/extensions/autoApiRegistration.ts`'s `defaultRootPath()`). (This is the single-source shorthand; R7 generalizes `rootPath` into a per-entry field of an `autoApiRegistration.sources[]` list for organizations with more than one source repository.) diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md index f232d5c..ac48f3f 100644 --- a/specs/004-lab-4-auto-registration/tasks.md +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -32,9 +32,9 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **Purpose**: Add dependencies and scaffold the directories every later task writes into. -- [ ] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) -- [ ] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` -- [ ] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` +- [X] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) +- [X] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` +- [X] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` **Checkpoint**: Dependencies installed, directories exist — ready for foundational work. @@ -46,14 +46,14 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **⚠️ CRITICAL**: No user story work can begin until this phase is complete. -- [ ] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section -- [ ] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) -- [ ] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) -- [ ] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) -- [ ] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) -- [ ] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) -- [ ] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) -- [ ] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) +- [X] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section +- [X] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) +- [X] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) +- [X] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) +- [X] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) +- [X] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) +- [X] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) +- [X] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) **Checkpoint**: Config loads, files are discoverable and parseable, scan-state cache persists — foundation ready for US1. @@ -67,12 +67,12 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 1 -- [ ] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) -- [ ] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) -- [ ] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) -- [ ] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) -- [ ] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) -- [ ] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) +- [X] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) +- [X] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) +- [X] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) +- [X] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) +- [X] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) +- [X] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) **Checkpoint**: User Story 1 fully functional and independently testable — discovery, create, update, and removal all work end-to-end with default metadata. @@ -86,17 +86,17 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 2 -- [ ] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) -- [ ] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) -- [ ] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) -- [ ] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) -- [ ] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) -- [ ] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) -- [ ] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) -- [ ] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) -- [ ] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) -- [ ] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) -- [ ] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) +- [X] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) +- [X] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) +- [X] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) +- [X] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) +- [X] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) +- [X] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) +- [X] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) +- [X] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) +- [X] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) +- [X] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) +- [X] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) **Checkpoint**: User Stories 1 AND 2 both work independently — metadata sourcing, defaults, validation errors, collisions, and precedence are all verified. @@ -110,12 +110,12 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar ### Implementation for User Story 3 -- [ ] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) -- [ ] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) -- [ ] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) +- [X] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) +- [X] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) **Checkpoint**: All three user stories independently functional; README fully documents adaptable vs. fixed conventions. @@ -125,9 +125,9 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar **Purpose**: Final repo-wide consistency checks that span multiple stories. -- [ ] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI -- [ ] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development -- [ ] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built +- [X] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI +- [X] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development +- [X] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built --- From db35fb821709cccc94e10af81a410e2201de4289 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 12:17:06 +1000 Subject: [PATCH 09/12] Extracted code from documentation, to make implementation and docs more usable --- labs/lab-02-users-roles/README.md | 155 +--- .../apiVisibility/ApiVisibilityCard.tsx | 41 + .../app/src/modules/apiVisibility/index.ts | 18 + .../src/extensions/permissionPolicy.ts | 66 ++ labs/lab-03-api-quality/README.md | 133 +-- .../app/src/modules/apiGrade/index.ts | 18 + .../spectralLinter/SpectralLinterContent.tsx | 41 + .../app/src/modules/spectralLinter/index.ts | 29 + .../spectral-linter-content.d.ts | 4 + labs/lab-04-auto-registration/README.md | 806 +----------------- .../src/extensions/autoApiRegistration.ts | 749 ++++++++++++++++ .../001_scan_state_cache.ts | 32 + 12 files changed, 1055 insertions(+), 1037 deletions(-) create mode 100644 labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx create mode 100644 labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts create mode 100644 labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts create mode 100644 labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts create mode 100644 labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts create mode 100644 labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts diff --git a/labs/lab-02-users-roles/README.md b/labs/lab-02-users-roles/README.md index d13a808..c1ea96a 100644 --- a/labs/lab-02-users-roles/README.md +++ b/labs/lab-02-users-roles/README.md @@ -319,77 +319,19 @@ Run this command from inside your Backstage root directory #### Create `ApiVisibilityCard.tsx` -Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` with the following -content: - -```typescript -import React from 'react'; -import { useEntity } from '@backstage/plugin-catalog-react'; -import { InfoCard } from '@backstage/core-components'; -import { Typography, Box, Chip } from '@material-ui/core'; - -const VISIBILITY_ANNOTATION = 'example.com/visibility'; - -export function ApiVisibilityCard() { - const { entity } = useEntity(); - const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; - - if (!visibility) { - return ( - - - No visibility designation set. Under the permission policy, this API is visible - only to members of the owning team (treated as private by default). - - - ); - } - - const isShared = visibility === 'shared'; - - return ( - - - - - {isShared - ? 'Visible to all authenticated users regardless of team membership.' - : 'Visible only to members of the owning team.'} - - - - ); -} -``` +Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx`](./code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx); +copy it in as-is. It reads the `example.com/visibility` annotation off the entity and renders a +"Shared"/"Private" chip, with a fallback message when the annotation is absent (treated as +private by the permission policy). #### Create `index.ts` -Create `packages/app/src/modules/apiVisibility/index.ts` with the following content: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiVisibilityCard } from './ApiVisibilityCard'; - -const apiVisibilityCard = EntityCardBlueprint.make({ - name: 'api-visibility', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiVisibilityCard), - }, -}); - -export const apiVisibilityModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiVisibilityCard], -}); -``` +Create `packages/app/src/modules/apiVisibility/index.ts` — its full content is committed at +[`code/packages/app/src/modules/apiVisibility/index.ts`](./code/packages/app/src/modules/apiVisibility/index.ts); +copy it in as-is. It wraps `ApiVisibilityCard` in an `EntityCardBlueprint` extension and bundles +it into a `createFrontendModule`. > **Note on Alpha imports**: `EntityCardBlueprint` is imported from > `@backstage/plugin-catalog-react/alpha`. This is the officially supported extension @@ -599,76 +541,13 @@ For all non-catalog permissions (scaffolding, TechDocs, search, etc.), the polic Create the directory `packages/backend/src/extensions/` inside your Backstage root (`labs/lab-01-base-backstage/backstage/`), then create a new file called -`permissionPolicy.ts` inside it with the following content: - -```typescript -// packages/backend/src/extensions/permissionPolicy.ts -import { createBackendModule } from '@backstage/backend-plugin-api'; -import { - PolicyDecision, - AuthorizeResult, - isResourcePermission, -} from '@backstage/plugin-permission-common'; -import { - PermissionPolicy, - PolicyQuery, - PolicyQueryUser, -} from '@backstage/plugin-permission-node'; -import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; -import { - catalogConditions, - createCatalogConditionalDecision, -} from '@backstage/plugin-catalog-backend/alpha'; - -class CatalogOwnershipPolicy implements PermissionPolicy { - async handle( - request: PolicyQuery, - user?: PolicyQueryUser, - ): Promise { - if (isResourcePermission(request.permission, 'catalog-entity')) { - return createCatalogConditionalDecision( - request.permission, - { - anyOf: [ - // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. - // This keeps the org chart and team pages open for everyone. - { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, - - // Rule 2: APIs annotated as shared are visible to all authenticated users, - // regardless of which team they belong to. - catalogConditions.hasAnnotation({ - annotation: 'example.com/visibility', - value: 'shared', - }), - - // Rule 3: Private APIs are visible only to members of the owning team. - // The user's ownershipEntityRefs contains their user ref plus all their - // group refs, resolved from the catalog User entity's memberOf list. - catalogConditions.isEntityOwner({ - claims: user?.info.ownershipEntityRefs ?? [], - }), - ], - }, - ); - } - // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. - return { result: AuthorizeResult.ALLOW }; - } -} - -export default createBackendModule({ - pluginId: 'permission', - moduleId: 'permission-policy', - register(reg) { - reg.registerInit({ - deps: { policy: policyExtensionPoint }, - async init({ policy }) { - policy.setPolicy(new CatalogOwnershipPolicy()); - }, - }); - }, -}); -``` +`permissionPolicy.ts` inside it. Its full content is committed alongside this README at +[`code/packages/backend/src/extensions/permissionPolicy.ts`](./code/packages/backend/src/extensions/permissionPolicy.ts); +copy it in as-is. In short: it's a `CatalogOwnershipPolicy` that, for the `catalog-entity` +resource permission, allows non-API entities unconditionally, allows APIs annotated +`example.com/visibility: shared` for everyone, and otherwise falls back to an ownership check +via `catalogConditions.isEntityOwner`; every other permission is unconditionally allowed. It's +registered as a `permission-policy` backend module via `createBackendModule`. **Import sources** (all packages are already installed in Lab 1 — no `npm install` needed): diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx new file mode 100644 index 0000000..0fcff5c --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx @@ -0,0 +1,41 @@ +import React from 'react'; +import { useEntity } from '@backstage/plugin-catalog-react'; +import { InfoCard } from '@backstage/core-components'; +import { Typography, Box, Chip } from '@material-ui/core'; + +const VISIBILITY_ANNOTATION = 'example.com/visibility'; + +export function ApiVisibilityCard() { + const { entity } = useEntity(); + const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; + + if (!visibility) { + return ( + + + No visibility designation set. Under the permission policy, this API is visible + only to members of the owning team (treated as private by default). + + + ); + } + + const isShared = visibility === 'shared'; + + return ( + + + + + {isShared + ? 'Visible to all authenticated users regardless of team membership.' + : 'Visible only to members of the owning team.'} + + + + ); +} diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts new file mode 100644 index 0000000..e21dd81 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiVisibilityCard } from './ApiVisibilityCard'; + +const apiVisibilityCard = EntityCardBlueprint.make({ + name: 'api-visibility', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiVisibilityCard), + }, +}); + +export const apiVisibilityModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiVisibilityCard], +}); diff --git a/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts new file mode 100644 index 0000000..e5b52e8 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts @@ -0,0 +1,66 @@ +// packages/backend/src/extensions/permissionPolicy.ts +import { createBackendModule } from '@backstage/backend-plugin-api'; +import { + PolicyDecision, + AuthorizeResult, + isResourcePermission, +} from '@backstage/plugin-permission-common'; +import { + PermissionPolicy, + PolicyQuery, + PolicyQueryUser, +} from '@backstage/plugin-permission-node'; +import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; +import { + catalogConditions, + createCatalogConditionalDecision, +} from '@backstage/plugin-catalog-backend/alpha'; + +class CatalogOwnershipPolicy implements PermissionPolicy { + async handle( + request: PolicyQuery, + user?: PolicyQueryUser, + ): Promise { + if (isResourcePermission(request.permission, 'catalog-entity')) { + return createCatalogConditionalDecision( + request.permission, + { + anyOf: [ + // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. + // This keeps the org chart and team pages open for everyone. + { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, + + // Rule 2: APIs annotated as shared are visible to all authenticated users, + // regardless of which team they belong to. + catalogConditions.hasAnnotation({ + annotation: 'example.com/visibility', + value: 'shared', + }), + + // Rule 3: Private APIs are visible only to members of the owning team. + // The user's ownershipEntityRefs contains their user ref plus all their + // group refs, resolved from the catalog User entity's memberOf list. + catalogConditions.isEntityOwner({ + claims: user?.info.ownershipEntityRefs ?? [], + }), + ], + }, + ); + } + // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. + return { result: AuthorizeResult.ALLOW }; + } +} + +export default createBackendModule({ + pluginId: 'permission', + moduleId: 'permission-policy', + register(reg) { + reg.registerInit({ + deps: { policy: policyExtensionPoint }, + async init({ policy }) { + policy.setPolicy(new CatalogOwnershipPolicy()); + }, + }); + }, +}); diff --git a/labs/lab-03-api-quality/README.md b/labs/lab-03-api-quality/README.md index fe4447b..f5d59d8 100644 --- a/labs/lab-03-api-quality/README.md +++ b/labs/lab-03-api-quality/README.md @@ -134,28 +134,11 @@ value `'info'` places the card in the right-hand Info column (the same column as card). Without it, the card defaults to the main content area at the bottom of the page — far less visible. -Create the file `packages/app/src/modules/apiGrade/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; - -const apiGradeCard = EntityCardBlueprint.make({ - name: 'api-grade', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiGradeCard), - }, -}); - -export const apiGradeModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiGradeCard], -}); -``` +Create the file `packages/app/src/modules/apiGrade/index.ts` — its full content is committed +alongside this README at +[`code/packages/app/src/modules/apiGrade/index.ts`](./code/packages/app/src/modules/apiGrade/index.ts); +copy it in as-is. It wraps the plugin's `ApiGradeCard` in an `EntityCardBlueprint` extension and +bundles it into a `createFrontendModule`, the same pattern as Lab 2's `apiVisibility` module. `filter: 'kind:API'` ensures the card only appears on API entity pages — not on Component, System, or other entity pages. @@ -354,60 +337,17 @@ mode. Importing it directly sidesteps the incompatibility entirely instead of wo it. Because this subpath has no TypeScript declaration file, add a small ambient module -declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`: - -```typescript -declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { - import { ComponentType } from 'react'; - export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; -} -``` - -Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`: - -```typescript -import React, { useEffect, useState } from 'react'; -import { useApi, identityApiRef } from '@backstage/core-plugin-api'; -import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; -import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; -import { InfoCard } from '@backstage/core-components'; -import { Typography } from '@material-ui/core'; - -export function SpectralLinterContent() { - const { entity } = useEntity(); - const identityApi = useApi(identityApiRef); - const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); - const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); - const [identityLoading, setIdentityLoading] = useState(true); - - useEffect(() => { - identityApi.getBackstageIdentity().then(identity => { - setIsPlatformTeamMember( - identity.ownershipEntityRefs.includes('group:default/platform-team'), - ); - setIdentityLoading(false); - }); - }, [identityApi]); - - if (ownershipLoading || identityLoading) return null; - - // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate - // FUNCTION, not a boolean — it must be called with the current entity. - if (!isOwnedEntity(entity) && !isPlatformTeamMember) { - return ( - - - Detailed quality information is restricted to members of the API's owning team - and the platform team. Sign in as a member of the owning team or the platform - team to view linting results. - - - ); - } - - return ; -} -``` +declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts` — its +full content is committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`](./code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts); +copy it in as-is. + +Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx` — its full +content is committed at +[`code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`](./code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx); +copy it in as-is. In short: it renders the plugin's inner `EntityApiDocsSpectralLinterContent` +component, gated behind an ownership/platform-team check — the owning team or a platform-team +member sees the lint results; everyone else sees an access-restricted message. **Why is `isOwnedEntity` called as a function?** `useEntityOwnership()` (from `@backstage/plugin-catalog-react`) returns `{ loading, isOwnedEntity }` where `isOwnedEntity` @@ -428,39 +368,14 @@ permission policy. ## Step 9 — Create the spectralLinter Frontend Module -Create the file `packages/app/src/modules/spectralLinter/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import { convertLegacyPlugin } from '@backstage/core-compat-api'; -import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; -import React from 'react'; -import { SpectralLinterContent } from './SpectralLinterContent'; - -const spectralLinterContent = EntityContentBlueprint.make({ - name: 'spectral-linter', - params: { - defaultPath: '/spectral', - defaultTitle: 'Spectral', - filter: 'kind:API', - loader: async () => React.createElement(SpectralLinterContent), - }, -}); - -export const spectralLinterModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [spectralLinterContent], -}); - -// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend -// system's API registry. `extensions: []` means we deliberately do NOT bring in the -// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped -// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. -export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { - extensions: [], -}); -``` +Create the file `packages/app/src/modules/spectralLinter/index.ts` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/index.ts`](./code/packages/app/src/modules/spectralLinter/index.ts); +copy it in as-is. In short: it wraps `SpectralLinterContent` in an `EntityContentBlueprint` +extension (the "Spectral" tab) and, separately, uses `convertLegacyPlugin` to register the +plugin's `linterApiRef` (backed by its `LinterClient`) with the new frontend system's API +registry, with `extensions: []` since only the API — not the plugin's own page/route +extensions — is needed. `defaultPath: '/spectral'` and `defaultTitle: 'Spectral'` define the tab's URL path and display label. `filter: 'kind:API'` ensures the tab appears only on API entity pages. diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts new file mode 100644 index 0000000..826d22c --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; + +const apiGradeCard = EntityCardBlueprint.make({ + name: 'api-grade', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiGradeCard), + }, +}); + +export const apiGradeModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiGradeCard], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx new file mode 100644 index 0000000..90760ac --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx @@ -0,0 +1,41 @@ +import React, { useEffect, useState } from 'react'; +import { useApi, identityApiRef } from '@backstage/core-plugin-api'; +import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; +import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; +import { InfoCard } from '@backstage/core-components'; +import { Typography } from '@material-ui/core'; + +export function SpectralLinterContent() { + const { entity } = useEntity(); + const identityApi = useApi(identityApiRef); + const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); + const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); + const [identityLoading, setIdentityLoading] = useState(true); + + useEffect(() => { + identityApi.getBackstageIdentity().then(identity => { + setIsPlatformTeamMember( + identity.ownershipEntityRefs.includes('group:default/platform-team'), + ); + setIdentityLoading(false); + }); + }, [identityApi]); + + if (ownershipLoading || identityLoading) return null; + + // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate + // FUNCTION, not a boolean — it must be called with the current entity. + if (!isOwnedEntity(entity) && !isPlatformTeamMember) { + return ( + + + Detailed quality information is restricted to members of the API's owning team + and the platform team. Sign in as a member of the owning team or the platform + team to view linting results. + + + ); + } + + return ; +} diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts new file mode 100644 index 0000000..aa32826 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts @@ -0,0 +1,29 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import { convertLegacyPlugin } from '@backstage/core-compat-api'; +import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; +import React from 'react'; +import { SpectralLinterContent } from './SpectralLinterContent'; + +const spectralLinterContent = EntityContentBlueprint.make({ + name: 'spectral-linter', + params: { + defaultPath: '/spectral', + defaultTitle: 'Spectral', + filter: 'kind:API', + loader: async () => React.createElement(SpectralLinterContent), + }, +}); + +export const spectralLinterModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [spectralLinterContent], +}); + +// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend +// system's API registry. `extensions: []` means we deliberately do NOT bring in the +// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped +// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. +export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { + extensions: [], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts new file mode 100644 index 0000000..c71e3a1 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts @@ -0,0 +1,4 @@ +declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { + import { ComponentType } from 'react'; + export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; +} diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md index 955a8ed..11786bd 100644 --- a/labs/lab-04-auto-registration/README.md +++ b/labs/lab-04-auto-registration/README.md @@ -107,796 +107,22 @@ lets you hand-author precise metadata for a specific API — pinning an owner, t visibility — without the auto-registration mechanism fighting you for it every cycle. Create the database migration first, at -`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`: - -```typescript -// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts -// -// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal -// state private to this backend module, used to avoid re-parsing unchanged files on restart, to -// compute delta mutations, and to support collision/precedence checks across every configured -// source. -// -// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a -// filesystem `directory` migration source, since this file lives alongside the module's -// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. - -import type { Knex } from 'knex'; - -export const TABLE_NAME = 'auto_api_registration_scan_state'; - -export async function up(knex: Knex): Promise { - await knex.schema.createTable(TABLE_NAME, table => { - table.string('source_id').notNullable(); - table.string('file_path').notNullable(); - table.bigInteger('mtime_ms').notNullable(); - table.string('content_hash').notNullable(); - table.string('entity_name').notNullable(); - table.text('last_error').nullable(); - - table.primary(['source_id', 'file_path']); - table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); - }); -} - -export async function down(knex: Knex): Promise { - await knex.schema.dropTable(TABLE_NAME); -} -``` - -Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts`: - -```typescript -// packages/backend/src/extensions/autoApiRegistration.ts -// -// Lab 4: Auto Registration. -// -// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files -// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. - -import crypto from 'crypto'; -import fs from 'fs'; -import path from 'path'; -import fg from 'fast-glob'; -import chokidar from 'chokidar'; -import yaml from 'js-yaml'; -import { - coreServices, - createBackendModule, - resolvePackagePath, -} from '@backstage/backend-plugin-api'; -import { - catalogProcessingExtensionPoint, - catalogServiceRef, -} from '@backstage/plugin-catalog-node'; -import type { - CatalogProcessor, - CatalogProcessorEmit, - CatalogService, - DeferredEntity, - EntityProvider, - EntityProviderConnection, -} from '@backstage/plugin-catalog-node'; -import type { - AuthService, - DatabaseService, - LoggerService, - RootConfigService, -} from '@backstage/backend-plugin-api'; -import { - ANNOTATION_LOCATION, - ANNOTATION_ORIGIN_LOCATION, - parseEntityRef, - stringifyEntityRef, -} from '@backstage/catalog-model'; -import type { Entity } from '@backstage/catalog-model'; -import { InputError } from '@backstage/errors'; -import type { Knex } from 'knex'; -import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; - -// --------------------------------------------------------------------------------------------- -// Config -// --------------------------------------------------------------------------------------------- - -export const VISIBILITY_ANNOTATION = 'example.com/visibility'; -export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; -// Internal-only marker: identifies which provider instance produced an entity, so the -// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from -// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. -const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; - -const KNOWN_VISIBILITIES = new Set(['private', 'shared']); -const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; -const DEFAULT_IGNORE = [ - '**/node_modules/**', - '**/.git/**', - '**/dist/**', - '**/build/**', -]; - -export interface SourceConfig { - id: string; - rootPath: string; - patterns: string[]; - ignore: string[]; - mode: 'poll' | 'watch'; - scheduleFrequencySeconds: number; - reconciliationFrequencySeconds: number; - parseConcurrency: number; - defaultOwner: string; - defaultVisibility: 'private' | 'shared'; - xNamespace: string; -} - -function defaultRootPath(): string { - // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs - return path.resolve( - resolvePackagePath('backend'), - '../../../../lab-04-auto-registration/apis', - ); -} - -export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { - const raw = rootConfig.getOptionalConfig('autoApiRegistration'); - if (!raw) { - return []; - } - - const entries = raw.has('sources') - ? raw.getConfigArray('sources') - : [raw]; - - return entries.map(entry => { - const id = raw.has('sources') ? entry.getString('id') : 'default'; - const defaultOwner = entry.getOptionalString('defaultOwner'); - const xNamespace = entry.getOptionalString('xNamespace'); - if (!defaultOwner) { - throw new Error( - `autoApiRegistration source "${id}" is missing required "defaultOwner"`, - ); - } - if (!xNamespace) { - throw new Error( - `autoApiRegistration source "${id}" is missing required "xNamespace"`, - ); - } - const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; - if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { - throw new Error( - `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, - ); - } - - return { - id, - rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), - patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, - ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, - mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', - scheduleFrequencySeconds: - entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, - reconciliationFrequencySeconds: - entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, - parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, - defaultOwner, - defaultVisibility: defaultVisibility as 'private' | 'shared', - xNamespace, - }; - }); -} - -// --------------------------------------------------------------------------------------------- -// Discovery + parsing -// --------------------------------------------------------------------------------------------- - -async function* discoverFiles(source: SourceConfig): AsyncGenerator { - const stream = fg.stream(source.patterns, { - cwd: source.rootPath, - ignore: source.ignore, - absolute: true, - onlyFiles: true, - }); - for await (const entry of stream) { - yield entry.toString(); - } -} - -interface ParsedSpec { - kind: 'openapi' | 'asyncapi'; - title: string; - description?: string; - tags: string[]; - raw: Record; -} - -/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ -function parseSpecFile(contents: string): ParsedSpec | undefined { - let doc: unknown; - try { - doc = yaml.load(contents); - } catch { - return undefined; - } - if (!doc || typeof doc !== 'object') { - return undefined; - } - const obj = doc as Record; - const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; - if (!kind) { - return undefined; - } - const info = obj.info as Record | undefined; - const title = info?.title; - if (typeof title !== 'string' || title.length === 0) { - return undefined; - } - // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag - // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed - // through verbatim. - const tags = Array.isArray(obj.tags) - ? (obj.tags as Array>) - .map(t => t?.name) - .filter((t): t is string => typeof t === 'string') - .map(slugify) - .filter(t => t.length > 0) - : []; - - return { - kind, - title, - description: typeof info?.description === 'string' ? (info.description as string) : undefined, - tags, - raw: obj, - }; -} - -export function slugify(title: string): string { - return title - .toLowerCase() - .trim() - .replace(/[^a-z0-9]+/g, '-') - .replace(/^-+|-+$/g, '') - .slice(0, 63); -} - -// --------------------------------------------------------------------------------------------- -// Candidate mapping -// --------------------------------------------------------------------------------------------- - -interface MappingResult { - entity: Entity; - entityName: string; - error?: string; -} - -function xField(raw: Record, namespace: string, field: string): string | undefined { - const info = raw.info as Record | undefined; - const ns = info?.[`x-${namespace}`] as Record | undefined; - const value = ns?.[field]; - return typeof value === 'string' ? value : undefined; -} - -function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { - const value = rawOwner ?? defaultOwner; - return stringifyEntityRef( - parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), - ); -} - -function buildEntity(opts: { - providerName: string; - filePath: string; - contents: string; - parsed: ParsedSpec; - entityName: string; - owner: string; - lifecycle: string; - visibility: string; - registrationError?: string; -}): Entity { - const annotations: Record = { - // Identifies the source file for debugging and the catalog-info.yaml precedence check — not - // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no - // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider - // already holds the bytes; no fetch is needed). - [ANNOTATION_LOCATION]: `file:${opts.filePath}`, - [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, - [VISIBILITY_ANNOTATION]: opts.visibility, - [MANAGED_BY_ANNOTATION]: opts.providerName, - }; - if (opts.registrationError) { - annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; - } - - return { - apiVersion: 'backstage.io/v1alpha1', - kind: 'API', - metadata: { - name: opts.entityName, - title: opts.parsed.title, - description: opts.parsed.description, - tags: opts.parsed.tags, - annotations, - }, - spec: { - type: opts.parsed.kind, - lifecycle: opts.lifecycle, - owner: opts.owner, - definition: opts.contents, - }, - }; -} - -/** - * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility - * validation, and collision detection. Does not perform the catalog-info.yaml precedence check - * (that requires a catalog query and is done by the caller, once, per file). - */ -function mapCandidate(opts: { - providerName: string; - source: SourceConfig; - filePath: string; - contents: string; - parsed: ParsedSpec; - knownOwnerRefs: Set; - collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any -}): MappingResult { - const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; - const baseSlug = slugify(parsed.title); - - const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); - const owner = resolveOwnerRef(rawOwner, source.defaultOwner); - const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; - const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); - const visibility = rawVisibility ?? source.defaultVisibility; - - // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently - // defaulted. - if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner: source.defaultOwner, - lifecycle, - visibility: source.defaultVisibility, - registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, - }), - error: 'invalid-visibility', - }; - } - - // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity - // instead of blocking the whole cycle. - if (!knownOwnerRefs.has(owner)) { - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner: source.defaultOwner, - lifecycle, - visibility, - registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, - }), - error: 'invalid-owner', - }; - } - - // Rule: a name collision with another file (in this source or another) is a suffixed - // marker/error entity — first-by-path wins, this one loses. - if (collidesWith) { - const collisionName = `${baseSlug}-collision`; - return { - entityName: collisionName, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: collisionName, - owner, - lifecycle, - visibility, - registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, - }), - error: 'name-collision', - }; - } - - return { - entityName: baseSlug, - entity: buildEntity({ - providerName, - filePath, - contents, - parsed, - entityName: baseSlug, - owner, - lifecycle, - visibility, - }), - }; -} - -// --------------------------------------------------------------------------------------------- -// Scan-state cache -// --------------------------------------------------------------------------------------------- - -interface CacheRow { - source_id: string; - file_path: string; - mtime_ms: number; - content_hash: string; - entity_name: string; - last_error: string | null; -} - -class ScanStateCache { - constructor(private readonly knex: Knex) {} - - static async create(database: DatabaseService): Promise { - const knex = await database.getClient(); - // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered - // against the 'catalog' plugin shares that plugin's own database, so using knex's default - // `knex_migrations` table here would validate our one-migration `migrationSource` against the - // real catalog plugin's own (much longer) applied-migrations history and fail with - // "migration directory is corrupt". - await knex.migrate.latest({ - tableName: 'auto_api_registration_migrations', - migrationSource: { - async getMigrations() { - return ['001_scan_state_cache']; - }, - getMigrationName(migration: string) { - return migration; - }, - async getMigration() { - return scanStateCacheMigration; - }, - }, - }); - return new ScanStateCache(knex); - } - - async rowsForSource(sourceId: string): Promise { - return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); - } - - /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ - async rowsByEntityName(entityName: string): Promise { - return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); - } - - async upsert(row: CacheRow): Promise { - await this.knex(scanStateCacheMigration.TABLE_NAME) - .insert(row) - .onConflict(['source_id', 'file_path']) - .merge(); - } - - async remove(sourceId: string, filePath: string): Promise { - await this.knex(scanStateCacheMigration.TABLE_NAME) - .where({ source_id: sourceId, file_path: filePath }) - .delete(); - } -} - -function sha1(contents: string): string { - return crypto.createHash('sha1').update(contents).digest('hex'); -} - -// --------------------------------------------------------------------------------------------- -// EntityProvider (one instance per source) -// --------------------------------------------------------------------------------------------- - -async function mapLimit( - items: T[], - limit: number, - fn: (item: T) => Promise, -): Promise { - const results: R[] = new Array(items.length); - let next = 0; - async function worker() { - while (next < items.length) { - const index = next++; - results[index] = await fn(items[index]); - } - } - await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); - return results; -} - -class AutoApiRegistrationEntityProvider implements EntityProvider { - private connection?: EntityProviderConnection; - private hasRunOnce = false; - - constructor( - private readonly source: SourceConfig, - private readonly cache: ScanStateCache, - private readonly catalogApi: CatalogService, - private readonly auth: AuthService, - private readonly logger: LoggerService, - ) {} - - getProviderName(): string { - return `auto-api-registration:${this.source.id}`; - } - - async connect(connection: EntityProviderConnection): Promise { - this.connection = connection; - } - - async runCycle(): Promise { - if (!this.connection) { - this.logger.warn( - `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, - ); - return; - } - - const existingRows = await this.cache.rowsForSource(this.source.id); - const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); - - const currentPaths = new Set(); - for await (const filePath of discoverFiles(this.source)) { - currentPaths.add(filePath); - } - - const knownOwnerRefs = await this.fetchKnownOwnerRefs(); - - const changedPaths = [...currentPaths].filter(filePath => { - const cached = existingByPath.get(filePath); - if (!cached) return true; - const stat = fs.statSync(filePath); - return stat.mtimeMs !== cached.mtime_ms; - }); - - const added: DeferredEntity[] = []; - const removedRefs: string[] = []; - const upserts: CacheRow[] = []; - const deletes: string[] = []; - - await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { - const cached = existingByPath.get(filePath); - const stat = fs.statSync(filePath); - const contents = fs.readFileSync(filePath, 'utf8'); - const hash = sha1(contents); - - if (cached && cached.content_hash === hash) { - // mtime touched, content unchanged — refresh mtime only, no mutation needed. - upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); - return; - } - - const parsed = parseSpecFile(contents); - if (!parsed) { - this.logger.error( - `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, - ); - if (cached) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); - deletes.push(filePath); - } - return; - } - - const slug = slugify(parsed.title); - - // Precedence check: a hand-authored catalog-info.yaml for this name wins. - const precedenceWinner = await this.findNonAutoSourcedEntity(slug); - if (precedenceWinner) { - this.logger.info( - `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, - ); - if (cached) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); - deletes.push(filePath); - } - return; - } - - // Global collision check: excludes this file's own prior row. - const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( - row => !(row.source_id === this.source.id && row.file_path === filePath), - ); - const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; - - const mapped = mapCandidate({ - providerName: this.getProviderName(), - source: this.source, - filePath, - contents, - parsed, - knownOwnerRefs, - collidesWith, - }); - - if (mapped.error) { - this.logger.error( - `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, - ); - } - - added.push({ entity: mapped.entity }); - upserts.push({ - source_id: this.source.id, - file_path: filePath, - mtime_ms: stat.mtimeMs, - content_hash: hash, - entity_name: mapped.entityName, - last_error: mapped.error ?? null, - }); - }); - - for (const [filePath, row] of existingByPath) { - if (!currentPaths.has(filePath)) { - removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); - deletes.push(filePath); - } - } - - if (!this.hasRunOnce) { - // First run: establish the baseline with one `full` mutation. - await this.connection.applyMutation({ type: 'full', entities: added }); - this.hasRunOnce = true; - } else if (added.length > 0 || removedRefs.length > 0) { - await this.connection.applyMutation({ - type: 'delta', - added, - removed: removedRefs.map(entityRef => ({ entityRef })), - }); - } - - for (const row of upserts) { - await this.cache.upsert(row); - } - for (const filePath of deletes) { - await this.cache.remove(this.source.id, filePath); - } - } - - private async fetchKnownOwnerRefs(): Promise> { - const credentials = await this.auth.getOwnServiceCredentials(); - const response = await this.catalogApi.getEntities( - { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, - { credentials }, - ); - return new Set(response.items.map(entity => stringifyEntityRef(entity))); - } - - private async findNonAutoSourcedEntity(entityName: string): Promise { - const credentials = await this.auth.getOwnServiceCredentials(); - const response = await this.catalogApi.getEntities( - { filter: { kind: 'API', 'metadata.name': entityName } }, - { credentials }, - ); - return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); - } -} - -// --------------------------------------------------------------------------------------------- -// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI -// --------------------------------------------------------------------------------------------- - -class AutoApiRegistrationErrorProcessor implements CatalogProcessor { - getProcessorName(): string { - return 'auto-api-registration-error-processor'; - } - - async preProcessEntity( - entity: Entity, - _location: unknown, - _emit: CatalogProcessorEmit, - ): Promise { - const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; - if (message) { - throw new InputError(message); - } - return entity; - } -} - -// --------------------------------------------------------------------------------------------- -// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) -// --------------------------------------------------------------------------------------------- - -export default createBackendModule({ - pluginId: 'catalog', - moduleId: 'auto-api-registration', - register(reg) { - reg.registerInit({ - deps: { - catalog: catalogProcessingExtensionPoint, - database: coreServices.database, - scheduler: coreServices.scheduler, - logger: coreServices.logger, - rootConfig: coreServices.rootConfig, - catalogApi: catalogServiceRef, - auth: coreServices.auth, - }, - async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { - const sources = normalizeConfig(rootConfig); - if (sources.length === 0) { - logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); - return; - } - - const cache = await ScanStateCache.create(database); - - const providers = sources.map( - source => - new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), - ); - - catalog.addEntityProvider(providers); - catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); - - for (const [index, source] of sources.entries()) { - const provider = providers[index]; - - if (source.mode === 'poll') { - await scheduler.scheduleTask({ - id: `auto-api-registration:${source.id}:poll`, - frequency: { seconds: source.scheduleFrequencySeconds }, - timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, - fn: async () => { - try { - await provider.runCycle(); - } catch (error) { - logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); - } - }, - }); - } else { - // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of - // just this source on add/change/unlink, coalesced by chokidar's own event batching. - // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer - // `reconciliation.frequencySeconds` safety-net sweep below. - await scheduler.scheduleTask({ - id: `auto-api-registration:${source.id}:reconciliation`, - frequency: { seconds: source.reconciliationFrequencySeconds }, - timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, - fn: async () => { - try { - await provider.runCycle(); - } catch (error) { - logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); - } - }, - }); - - const watcher = chokidar.watch(source.patterns, { - cwd: source.rootPath, - ignored: source.ignore, - ignoreInitial: true, - }); - let debounce: NodeJS.Timeout | undefined; - const triggerRescan = () => { - clearTimeout(debounce); - debounce = setTimeout(() => { - provider - .runCycle() - .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); - }, 500); - }; - watcher.on('add', triggerRescan); - watcher.on('change', triggerRescan); - watcher.on('unlink', triggerRescan); - } - } - }, - }); - }, -}); -``` +`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`. Its full +content is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`](./code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts) +— copy it in as-is. It creates the `auto_api_registration_scan_state` table (`source_id`, +`file_path`, `mtime_ms`, `content_hash`, `entity_name`, `last_error`), applied via a knex custom +`migrationSource` (see the module below) rather than a filesystem `directory` migration source, +since this file lives alongside the module's TypeScript source rather than in a package-level +`migrations/` folder resolved at runtime. + +Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts` — this is the +heart of the lab. Its full content (~750 lines: config normalization, filesystem discovery and +YAML parsing, `x-*` candidate mapping with owner/visibility/collision validation, the scan-state +cache, the `EntityProvider` itself with its poll/watch scheduling, and the companion +`CatalogProcessor`) is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistration.ts`](./code/packages/backend/src/extensions/autoApiRegistration.ts) +— copy it in as-is. --- diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts new file mode 100644 index 0000000..45529f2 --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts @@ -0,0 +1,749 @@ +// packages/backend/src/extensions/autoApiRegistration.ts +// +// Lab 4: Auto Registration. +// +// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files +// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. + +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import fg from 'fast-glob'; +import chokidar from 'chokidar'; +import yaml from 'js-yaml'; +import { + coreServices, + createBackendModule, + resolvePackagePath, +} from '@backstage/backend-plugin-api'; +import { + catalogProcessingExtensionPoint, + catalogServiceRef, +} from '@backstage/plugin-catalog-node'; +import type { + CatalogProcessor, + CatalogProcessorEmit, + CatalogService, + DeferredEntity, + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-node'; +import type { + AuthService, + DatabaseService, + LoggerService, + RootConfigService, +} from '@backstage/backend-plugin-api'; +import { + ANNOTATION_LOCATION, + ANNOTATION_ORIGIN_LOCATION, + parseEntityRef, + stringifyEntityRef, +} from '@backstage/catalog-model'; +import type { Entity } from '@backstage/catalog-model'; +import { InputError } from '@backstage/errors'; +import type { Knex } from 'knex'; +import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; + +// --------------------------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------------------------- + +export const VISIBILITY_ANNOTATION = 'example.com/visibility'; +export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; +// Internal-only marker: identifies which provider instance produced an entity, so the +// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from +// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. +const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; + +const KNOWN_VISIBILITIES = new Set(['private', 'shared']); +const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; +const DEFAULT_IGNORE = [ + '**/node_modules/**', + '**/.git/**', + '**/dist/**', + '**/build/**', +]; + +export interface SourceConfig { + id: string; + rootPath: string; + patterns: string[]; + ignore: string[]; + mode: 'poll' | 'watch'; + scheduleFrequencySeconds: number; + reconciliationFrequencySeconds: number; + parseConcurrency: number; + defaultOwner: string; + defaultVisibility: 'private' | 'shared'; + xNamespace: string; +} + +function defaultRootPath(): string { + // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs + return path.resolve( + resolvePackagePath('backend'), + '../../../../lab-04-auto-registration/apis', + ); +} + +export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { + const raw = rootConfig.getOptionalConfig('autoApiRegistration'); + if (!raw) { + return []; + } + + const entries = raw.has('sources') + ? raw.getConfigArray('sources') + : [raw]; + + return entries.map(entry => { + const id = raw.has('sources') ? entry.getString('id') : 'default'; + const defaultOwner = entry.getOptionalString('defaultOwner'); + const xNamespace = entry.getOptionalString('xNamespace'); + if (!defaultOwner) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "defaultOwner"`, + ); + } + if (!xNamespace) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "xNamespace"`, + ); + } + const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; + if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { + throw new Error( + `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, + ); + } + + return { + id, + rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), + patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, + ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, + mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', + scheduleFrequencySeconds: + entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, + reconciliationFrequencySeconds: + entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, + parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, + defaultOwner, + defaultVisibility: defaultVisibility as 'private' | 'shared', + xNamespace, + }; + }); +} + +// --------------------------------------------------------------------------------------------- +// Discovery + parsing +// --------------------------------------------------------------------------------------------- + +async function* discoverFiles(source: SourceConfig): AsyncGenerator { + const stream = fg.stream(source.patterns, { + cwd: source.rootPath, + ignore: source.ignore, + absolute: true, + onlyFiles: true, + }); + for await (const entry of stream) { + yield entry.toString(); + } +} + +interface ParsedSpec { + kind: 'openapi' | 'asyncapi'; + title: string; + description?: string; + tags: string[]; + raw: Record; +} + +/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ +function parseSpecFile(contents: string): ParsedSpec | undefined { + let doc: unknown; + try { + doc = yaml.load(contents); + } catch { + return undefined; + } + if (!doc || typeof doc !== 'object') { + return undefined; + } + const obj = doc as Record; + const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; + if (!kind) { + return undefined; + } + const info = obj.info as Record | undefined; + const title = info?.title; + if (typeof title !== 'string' || title.length === 0) { + return undefined; + } + // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag + // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed + // through verbatim. + const tags = Array.isArray(obj.tags) + ? (obj.tags as Array>) + .map(t => t?.name) + .filter((t): t is string => typeof t === 'string') + .map(slugify) + .filter(t => t.length > 0) + : []; + + return { + kind, + title, + description: typeof info?.description === 'string' ? (info.description as string) : undefined, + tags, + raw: obj, + }; +} + +export function slugify(title: string): string { + return title + .toLowerCase() + .trim() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 63); +} + +// --------------------------------------------------------------------------------------------- +// Candidate mapping +// --------------------------------------------------------------------------------------------- + +interface MappingResult { + entity: Entity; + entityName: string; + error?: string; +} + +function xField(raw: Record, namespace: string, field: string): string | undefined { + const info = raw.info as Record | undefined; + const ns = info?.[`x-${namespace}`] as Record | undefined; + const value = ns?.[field]; + return typeof value === 'string' ? value : undefined; +} + +function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { + const value = rawOwner ?? defaultOwner; + return stringifyEntityRef( + parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), + ); +} + +function buildEntity(opts: { + providerName: string; + filePath: string; + contents: string; + parsed: ParsedSpec; + entityName: string; + owner: string; + lifecycle: string; + visibility: string; + registrationError?: string; +}): Entity { + const annotations: Record = { + // Identifies the source file for debugging and the catalog-info.yaml precedence check — not + // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no + // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider + // already holds the bytes; no fetch is needed). + [ANNOTATION_LOCATION]: `file:${opts.filePath}`, + [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, + [VISIBILITY_ANNOTATION]: opts.visibility, + [MANAGED_BY_ANNOTATION]: opts.providerName, + }; + if (opts.registrationError) { + annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; + } + + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'API', + metadata: { + name: opts.entityName, + title: opts.parsed.title, + description: opts.parsed.description, + tags: opts.parsed.tags, + annotations, + }, + spec: { + type: opts.parsed.kind, + lifecycle: opts.lifecycle, + owner: opts.owner, + definition: opts.contents, + }, + }; +} + +/** + * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility + * validation, and collision detection. Does not perform the catalog-info.yaml precedence check + * (that requires a catalog query and is done by the caller, once, per file). + */ +function mapCandidate(opts: { + providerName: string; + source: SourceConfig; + filePath: string; + contents: string; + parsed: ParsedSpec; + knownOwnerRefs: Set; + collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any +}): MappingResult { + const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; + const baseSlug = slugify(parsed.title); + + const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); + const owner = resolveOwnerRef(rawOwner, source.defaultOwner); + const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; + const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); + const visibility = rawVisibility ?? source.defaultVisibility; + + // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently + // defaulted. + if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility: source.defaultVisibility, + registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, + }), + error: 'invalid-visibility', + }; + } + + // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity + // instead of blocking the whole cycle. + if (!knownOwnerRefs.has(owner)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility, + registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, + }), + error: 'invalid-owner', + }; + } + + // Rule: a name collision with another file (in this source or another) is a suffixed + // marker/error entity — first-by-path wins, this one loses. + if (collidesWith) { + const collisionName = `${baseSlug}-collision`; + return { + entityName: collisionName, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: collisionName, + owner, + lifecycle, + visibility, + registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, + }), + error: 'name-collision', + }; + } + + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner, + lifecycle, + visibility, + }), + }; +} + +// --------------------------------------------------------------------------------------------- +// Scan-state cache +// --------------------------------------------------------------------------------------------- + +interface CacheRow { + source_id: string; + file_path: string; + mtime_ms: number; + content_hash: string; + entity_name: string; + last_error: string | null; +} + +class ScanStateCache { + constructor(private readonly knex: Knex) {} + + static async create(database: DatabaseService): Promise { + const knex = await database.getClient(); + // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered + // against the 'catalog' plugin shares that plugin's own database, so using knex's default + // `knex_migrations` table here would validate our one-migration `migrationSource` against the + // real catalog plugin's own (much longer) applied-migrations history and fail with + // "migration directory is corrupt". + await knex.migrate.latest({ + tableName: 'auto_api_registration_migrations', + migrationSource: { + async getMigrations() { + return ['001_scan_state_cache']; + }, + getMigrationName(migration: string) { + return migration; + }, + async getMigration() { + return scanStateCacheMigration; + }, + }, + }); + return new ScanStateCache(knex); + } + + async rowsForSource(sourceId: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); + } + + /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ + async rowsByEntityName(entityName: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); + } + + async upsert(row: CacheRow): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .insert(row) + .onConflict(['source_id', 'file_path']) + .merge(); + } + + async remove(sourceId: string, filePath: string): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .where({ source_id: sourceId, file_path: filePath }) + .delete(); + } +} + +function sha1(contents: string): string { + return crypto.createHash('sha1').update(contents).digest('hex'); +} + +// --------------------------------------------------------------------------------------------- +// EntityProvider (one instance per source) +// --------------------------------------------------------------------------------------------- + +async function mapLimit( + items: T[], + limit: number, + fn: (item: T) => Promise, +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + async function worker() { + while (next < items.length) { + const index = next++; + results[index] = await fn(items[index]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); + return results; +} + +class AutoApiRegistrationEntityProvider implements EntityProvider { + private connection?: EntityProviderConnection; + private hasRunOnce = false; + + constructor( + private readonly source: SourceConfig, + private readonly cache: ScanStateCache, + private readonly catalogApi: CatalogService, + private readonly auth: AuthService, + private readonly logger: LoggerService, + ) {} + + getProviderName(): string { + return `auto-api-registration:${this.source.id}`; + } + + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + async runCycle(): Promise { + if (!this.connection) { + this.logger.warn( + `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, + ); + return; + } + + const existingRows = await this.cache.rowsForSource(this.source.id); + const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); + + const currentPaths = new Set(); + for await (const filePath of discoverFiles(this.source)) { + currentPaths.add(filePath); + } + + const knownOwnerRefs = await this.fetchKnownOwnerRefs(); + + const changedPaths = [...currentPaths].filter(filePath => { + const cached = existingByPath.get(filePath); + if (!cached) return true; + const stat = fs.statSync(filePath); + return stat.mtimeMs !== cached.mtime_ms; + }); + + const added: DeferredEntity[] = []; + const removedRefs: string[] = []; + const upserts: CacheRow[] = []; + const deletes: string[] = []; + + await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { + const cached = existingByPath.get(filePath); + const stat = fs.statSync(filePath); + const contents = fs.readFileSync(filePath, 'utf8'); + const hash = sha1(contents); + + if (cached && cached.content_hash === hash) { + // mtime touched, content unchanged — refresh mtime only, no mutation needed. + upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); + return; + } + + const parsed = parseSpecFile(contents); + if (!parsed) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + const slug = slugify(parsed.title); + + // Precedence check: a hand-authored catalog-info.yaml for this name wins. + const precedenceWinner = await this.findNonAutoSourcedEntity(slug); + if (precedenceWinner) { + this.logger.info( + `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + // Global collision check: excludes this file's own prior row. + const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( + row => !(row.source_id === this.source.id && row.file_path === filePath), + ); + const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; + + const mapped = mapCandidate({ + providerName: this.getProviderName(), + source: this.source, + filePath, + contents, + parsed, + knownOwnerRefs, + collidesWith, + }); + + if (mapped.error) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, + ); + } + + added.push({ entity: mapped.entity }); + upserts.push({ + source_id: this.source.id, + file_path: filePath, + mtime_ms: stat.mtimeMs, + content_hash: hash, + entity_name: mapped.entityName, + last_error: mapped.error ?? null, + }); + }); + + for (const [filePath, row] of existingByPath) { + if (!currentPaths.has(filePath)) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); + deletes.push(filePath); + } + } + + if (!this.hasRunOnce) { + // First run: establish the baseline with one `full` mutation. + await this.connection.applyMutation({ type: 'full', entities: added }); + this.hasRunOnce = true; + } else if (added.length > 0 || removedRefs.length > 0) { + await this.connection.applyMutation({ + type: 'delta', + added, + removed: removedRefs.map(entityRef => ({ entityRef })), + }); + } + + for (const row of upserts) { + await this.cache.upsert(row); + } + for (const filePath of deletes) { + await this.cache.remove(this.source.id, filePath); + } + } + + private async fetchKnownOwnerRefs(): Promise> { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, + { credentials }, + ); + return new Set(response.items.map(entity => stringifyEntityRef(entity))); + } + + private async findNonAutoSourcedEntity(entityName: string): Promise { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: 'API', 'metadata.name': entityName } }, + { credentials }, + ); + return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); + } +} + +// --------------------------------------------------------------------------------------------- +// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI +// --------------------------------------------------------------------------------------------- + +class AutoApiRegistrationErrorProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'auto-api-registration-error-processor'; + } + + async preProcessEntity( + entity: Entity, + _location: unknown, + _emit: CatalogProcessorEmit, + ): Promise { + const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; + if (message) { + throw new InputError(message); + } + return entity; + } +} + +// --------------------------------------------------------------------------------------------- +// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) +// --------------------------------------------------------------------------------------------- + +export default createBackendModule({ + pluginId: 'catalog', + moduleId: 'auto-api-registration', + register(reg) { + reg.registerInit({ + deps: { + catalog: catalogProcessingExtensionPoint, + database: coreServices.database, + scheduler: coreServices.scheduler, + logger: coreServices.logger, + rootConfig: coreServices.rootConfig, + catalogApi: catalogServiceRef, + auth: coreServices.auth, + }, + async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { + const sources = normalizeConfig(rootConfig); + if (sources.length === 0) { + logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); + return; + } + + const cache = await ScanStateCache.create(database); + + const providers = sources.map( + source => + new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), + ); + + catalog.addEntityProvider(providers); + catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); + + for (const [index, source] of sources.entries()) { + const provider = providers[index]; + + if (source.mode === 'poll') { + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:poll`, + frequency: { seconds: source.scheduleFrequencySeconds }, + timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); + } + }, + }); + } else { + // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of + // just this source on add/change/unlink, coalesced by chokidar's own event batching. + // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer + // `reconciliation.frequencySeconds` safety-net sweep below. + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:reconciliation`, + frequency: { seconds: source.reconciliationFrequencySeconds }, + timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); + } + }, + }); + + const watcher = chokidar.watch(source.patterns, { + cwd: source.rootPath, + ignored: source.ignore, + ignoreInitial: true, + }); + let debounce: NodeJS.Timeout | undefined; + const triggerRescan = () => { + clearTimeout(debounce); + debounce = setTimeout(() => { + provider + .runCycle() + .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); + }, 500); + }; + watcher.on('add', triggerRescan); + watcher.on('change', triggerRescan); + watcher.on('unlink', triggerRescan); + } + } + }, + }); + }, +}); diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts new file mode 100644 index 0000000..78f6a7b --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts @@ -0,0 +1,32 @@ +// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts +// +// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal +// state private to this backend module, used to avoid re-parsing unchanged files on restart, to +// compute delta mutations, and to support collision/precedence checks across every configured +// source. +// +// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a +// filesystem `directory` migration source, since this file lives alongside the module's +// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. + +import type { Knex } from 'knex'; + +export const TABLE_NAME = 'auto_api_registration_scan_state'; + +export async function up(knex: Knex): Promise { + await knex.schema.createTable(TABLE_NAME, table => { + table.string('source_id').notNullable(); + table.string('file_path').notNullable(); + table.bigInteger('mtime_ms').notNullable(); + table.string('content_hash').notNullable(); + table.string('entity_name').notNullable(); + table.text('last_error').nullable(); + + table.primary(['source_id', 'file_path']); + table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); + }); +} + +export async function down(knex: Knex): Promise { + await knex.schema.dropTable(TABLE_NAME); +} From 7198868523d180f1ec2a708fed47c79f6c0cda54 Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 12:24:17 +1000 Subject: [PATCH 10/12] Prevent large source files being inlined within lab documentation --- .specify/memory/constitution.md | 41 +++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 843f761..ce7ff1f 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,9 +1,12 @@ @@ -35,8 +48,22 @@ Documentation MUST explain the *why* and *how* of each step, not just the *what* The journey of setting up Backstage is as important as the final running instance. Steps that are non-obvious, error-prone, or surprising MUST include explanatory context. +When a lab step instructs the user to create a file, and that file's full content either +(a) cannot reasonably fit on one screen (as a rough guide, more than ~40–50 lines) or +(b) is a complete, reusable source file rather than a short edit/diff snippet — regardless of +length — the file's full content MUST be committed to the repository under the lab's own +directory (convention: a `code/` subdirectory mirroring the target relative path, e.g. +`labs/lab-0N-*/code/packages/app/src/modules/foo/Bar.tsx`), and the README MUST link to that +committed file instead of embedding the whole content inline in a fenced code block. Short +edit-in-place snippets that show surrounding context for a targeted diff against a file created +in an earlier step (not a full new file) are exempt and may remain inline. + **Rationale**: A user who follows steps blindly cannot troubleshoot or adapt. Understanding -the process produces durable knowledge; following a script does not. +the process produces durable knowledge; following a script does not. Embedding entire source +files inline inflates page length without adding explanatory value — the prose around the file +(the *why*) is what teaches; the file content itself is better consulted, copied, or diffed as +a real file. A committed, linked file is also directly reusable by a learner adapting the lab +to their own repo, which an inline fence is not. ### III. Cross-Platform Compatibility @@ -162,6 +189,10 @@ Labs that use simplified security practices (Principle IX) MUST also include: Labs MUST NOT require external network access beyond downloading freely available software and dependencies. All API samples and test data MUST be included in the repository. +Labs that instruct the user to create a large or reusable source file MUST commit that file's +content under a `code/` subdirectory within the lab's own directory and link to it from the +README, per Principle II, rather than embedding it inline. + ## Development Workflow - Each lab corresponds to one speckit feature branch following the `###-lab-name` convention. @@ -193,4 +224,4 @@ All plan `Constitution Check` gates MUST reference the principles by Roman numer - `yarn dev` does not exist. Use `yarn start` instead. -**Version**: 1.3.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-06-08 +**Version**: 1.4.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-07-03 From cb2a283c71a5017582ca340bcacaa2f946f6768f Mon Sep 17 00:00:00 2001 From: DawMatt Date: Fri, 3 Jul 2026 23:20:27 +1000 Subject: [PATCH 11/12] Auto-registration lab functional --- labs/lab-04-auto-registration/README.md | 8 +++++ .../checklists/issues.md | 33 +++++++++++++++++++ specs/004-lab-4-auto-registration/tasks.md | 19 +++++++++++ 3 files changed, 60 insertions(+) create mode 100644 specs/004-lab-4-auto-registration/checklists/issues.md diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md index 11786bd..8ca2198 100644 --- a/labs/lab-04-auto-registration/README.md +++ b/labs/lab-04-auto-registration/README.md @@ -400,6 +400,14 @@ checks — is reused unchanged; only how bytes arrive on local disk differs. exists.** Confirm that group is actually loaded in the catalog (check its entity page directly) — this error means the *catalog* doesn't know about the group yet, not that your YAML is wrong. A common cause locally is a `catalog.locations` URL that hasn't been pushed yet (see Step 5). + This can also show up as "no APIs appear in the catalog at all" rather than a single owner + error: if the `catalog.locations` entry for your org data (`teams.yaml`/`users.yaml`) points at + a branch name that's since been merged and deleted (a stale reference from an earlier lab, + rather than your current branch), that location 404s, the group it would have defined never + registers, and *every* auto-sourced API whose owner references that group fails validation — + which looks like the discovery mechanism itself is broken. Check the backend logs for "Unable + to read url, no matching files found" lines against your `catalog.locations` URLs first; if any + point at a branch other than `main` or your current branch, that's almost always the real cause. - **An entity you expect to see is silently missing, with no error logged.** Check the backend logs for a `Policy check failed for api:default/` warning — this means the entity was built and emitted, but failed Backstage's own entity-schema validation (for example, diff --git a/specs/004-lab-4-auto-registration/checklists/issues.md b/specs/004-lab-4-auto-registration/checklists/issues.md new file mode 100644 index 0000000..d6cfebe --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/issues.md @@ -0,0 +1,33 @@ +## Run 1 - 2026/07/03 + +- [X] No APIs were available in the catalog after starting backstage. Relevant log entries are included below. + + **Resolved** (specs/004-lab-4-auto-registration/tasks.md T038–T040): root cause was six + `catalog.locations` URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` still + pinned to the stale, merged-and-gone `003-api-quality` branch. `teams.yaml` (defining + `platform-team`) 404'd as a result, so the Lab 4 owner validator correctly rejected + `galaxy-openapi.yaml`'s `owner: group:default/platform-team` — the actual cause of the empty + catalog. Fixed by repointing those six URLs at `main`, where the content is already merged. + Verified via a clean `yarn start`: museum/streetlights/train-travel/scalar-galaxy/ + precedence-demo-api all register with no `AutoApiRegistrationErrorProcessor` errors and no + "Unable to read url" warnings. + +``` +2026-07-03T12:38:51.027Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:00.820Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml entity="location:default/generated-4c23d1d635d435e80cc8b698fc86956f9fffff3c" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml" +2026-07-03T12:39:00.895Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml entity="location:default/generated-2cde689609df331e9267009ea6c02850d6b1603e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml" +2026-07-03T12:39:00.896Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml entity="location:default/generated-cc3b19c3e787fc46d27f583f1d53dd2e8ec1a456" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml" +2026-07-03T12:39:00.899Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml entity="location:default/generated-ae8dd75945a327a0201813834829ec035913de8e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml" +2026-07-03T12:39:00.907Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml entity="location:default/generated-1390492c77b8f266a9f6608c0595f96efa060611" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml" +2026-07-03T12:39:00.915Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml entity="location:default/generated-9ecb0d8da3f46f025b2e5e23de21f385b1a49388" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml" +2026-07-03T12:39:07.085Z catalog warn Processor AutoApiRegistrationErrorProcessor threw an error while preprocessing; caused by InputError: Owner "group:default/platform-team" does not resolve to a known User or Group entity entity="api:default/scalar-galaxy" location="file:/Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml" +2026-07-03T12:39:21.056Z rootHttpRouter info [2026-07-03T12:39:21.056Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.056Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.060Z rootHttpRouter info [2026-07-03T12:39:21.060Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.060Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.061Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:51.087Z rootHttpRouter info [2026-07-03T12:39:51.087Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.087Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.092Z rootHttpRouter info [2026-07-03T12:39:51.092Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.092Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.093Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:40:21.123Z rootHttpRouter info [2026-07-03T12:40:21.123Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.123Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.127Z rootHttpRouter info [2026-07-03T12:40:21.127Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.127Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.128Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +``` \ No newline at end of file diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md index ac48f3f..2ae6c85 100644 --- a/specs/004-lab-4-auto-registration/tasks.md +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -131,6 +131,24 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar --- +## Phase 7: Bug Fixes (from checklists/issues.md — Run 1, 2026/07/03) + +**Purpose**: Fix the reported "No APIs were available in the catalog after starting backstage" issue. Root cause analysis of the pasted log: + +- The `group:default/platform-team` owner-resolution failure (T020's validator, `InputError` on `galaxy-openapi.yaml`) is a *symptom*: `teams.yaml` — which defines `platform-team` — never loads, because its catalog location URL is 404ing. +- That 404, and the four other "Unable to read url" warnings (`teams.yaml`, `users.yaml`, `museum-api.yaml`, `streetlights-api.yaml`, `train-travel-api.yaml`... actually `platform-user.yaml` too), all point at `raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/...` — a stale feature-branch name left over from when Lab 3's instructions were followed. That content has since been merged to `main` (verified: present at `main:labs/lab-02-users-roles/catalog/teams.yaml` and `main:labs/lab-03-api-quality/catalog/platform-user.yaml`), so the branch-pinned URLs should point at `main` instead. +- With `teams.yaml` unreachable, `platform-team` never registers as a `Group`, so the Lab 4 owner validator (correctly) rejects `galaxy-openapi.yaml`'s `owner: group:default/platform-team` and emits a marker/error entity instead of a real one — leaving the catalog with no visible APIs. +- The one remaining branch-pinned location (`precedence-demo-catalog-info.yaml`, pointing at `004-lab-4-auto-registration`) is *not* a bug — it is new-on-this-branch content that genuinely requires the branch to be pushed to GitHub first, exactly as documented in `labs/lab-04-auto-registration/README.md` (the "This only works once you've pushed your branch" note). No code or config change fixes that one; it's a sequencing step the learner has to perform. + +- [X] T038 Fix the five stale `003-api-quality`-branch catalog location URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` (lines ~130, 136, 142, 148, 154, 160, plus the two `spectralLinter` ruleset URLs at ~206–207) to point at `main` instead, since that content is already merged to `main` +- [X] T039 [P] Add a troubleshooting entry to `labs/lab-04-auto-registration/README.md`'s Troubleshooting section explaining the observed failure chain (stale/unpushed branch in an org-data or API catalog location → that entity never registers → any auto-sourced API whose `x-examplecorp.owner` references a group defined only in that unreachable location fails owner validation → looks like "no APIs in the catalog" even though the real cause is upstream of Lab 4's own code) so learners can self-diagnose instead of assuming the `EntityProvider` is broken +- [X] T040 Restart `labs/lab-01-base-backstage/backstage/` from a clean `yarn start` after T038 and confirm via backend logs and the catalog UI: `platform-team` resolves (no more `InputError` on `galaxy-openapi.yaml`), museum/streetlights/train-travel APIs and the `platform-team`/`platform-user` org entities all load without "Unable to read url" warnings, and the Galaxy API entity appears with `spec.owner: group:default/platform-team` (SC-001/SC-002); note that `precedence-demo-api` will still be absent unless the current branch has been pushed to GitHub, per T039's documented caveat +- [X] T041 Update `specs/004-lab-4-auto-registration/checklists/issues.md`: mark the Run 1 item resolved with a one-line pointer to T038–T040 once T040's re-verification passes + +**Checkpoint**: Catalog populates on a clean start; the only remaining "gap" (precedence-demo-api before the branch is pushed) is expected and documented, not a bug. + +--- + ## Dependencies & Execution Order ### Phase Dependencies @@ -141,6 +159,7 @@ This is a tutorial-lab project (not a generic web/mobile app). Two path roots ar - **User Story 2 (Phase 4)**: Depends on Foundational completion; extends the same file US1 built (T012), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency - **User Story 3 (Phase 5)**: Depends on US1 (T015) and US2 (T024) being implemented, since the README documents both the discovery mechanism and the metadata-sourcing/error behavior - **Polish (Phase 6)**: Depends on all user stories being complete +- **Bug Fixes (Phase 7)**: Independent of Phases 3–6's code (config/docs-only fix) but logically follows Polish since it was found during Polish-phase (T036) cold-start validation ### User Story Dependencies From e7edc032d1aea85078c26a69da19ddd06aaae19e Mon Sep 17 00:00:00 2001 From: DawMatt Date: Sat, 4 Jul 2026 05:00:38 +1000 Subject: [PATCH 12/12] Add lab to readme. Ensure readme updated automatically in future. --- .github/workflows/readme-sync.yml | 14 ++++++++ .specify/memory/constitution.md | 48 ++++++++++++++++++---------- .specify/templates/tasks-template.md | 3 ++ CONTRIBUTING.md | 9 +++++- README.md | 20 ++++++++---- scripts/check-readme.sh | 31 ++++++++++++++++++ 6 files changed, 100 insertions(+), 25 deletions(-) create mode 100644 .github/workflows/readme-sync.yml create mode 100755 scripts/check-readme.sh diff --git a/.github/workflows/readme-sync.yml b/.github/workflows/readme-sync.yml new file mode 100644 index 0000000..32e9c73 --- /dev/null +++ b/.github/workflows/readme-sync.yml @@ -0,0 +1,14 @@ +name: README sync check + +on: + push: + branches: [main] + pull_request: + +jobs: + check-readme: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Verify root README references every labs/ and specs/ directory + run: bash scripts/check-readme.sh diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index ce7ff1f..5b4f204 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,27 +1,27 @@