diff --git a/.github/workflows/readme-sync.yml b/.github/workflows/readme-sync.yml new file mode 100644 index 0000000..32e9c73 --- /dev/null +++ b/.github/workflows/readme-sync.yml @@ -0,0 +1,14 @@ +name: README sync check + +on: + push: + branches: [main] + pull_request: + +jobs: + check-readme: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Verify root README references every labs/ and specs/ directory + run: bash scripts/check-readme.sh diff --git a/.specify/feature.json b/.specify/feature.json index 1476533..aee6f97 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1,3 +1,3 @@ { - "feature_directory": "specs/003-lab-3-api-quality" + "feature_directory": "specs/004-lab-4-auto-registration" } diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 843f761..5b4f204 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,18 +1,31 @@ @@ -35,8 +48,22 @@ Documentation MUST explain the *why* and *how* of each step, not just the *what* The journey of setting up Backstage is as important as the final running instance. Steps that are non-obvious, error-prone, or surprising MUST include explanatory context. +When a lab step instructs the user to create a file, and that file's full content either +(a) cannot reasonably fit on one screen (as a rough guide, more than ~40–50 lines) or +(b) is a complete, reusable source file rather than a short edit/diff snippet — regardless of +length — the file's full content MUST be committed to the repository under the lab's own +directory (convention: a `code/` subdirectory mirroring the target relative path, e.g. +`labs/lab-0N-*/code/packages/app/src/modules/foo/Bar.tsx`), and the README MUST link to that +committed file instead of embedding the whole content inline in a fenced code block. Short +edit-in-place snippets that show surrounding context for a targeted diff against a file created +in an earlier step (not a full new file) are exempt and may remain inline. + **Rationale**: A user who follows steps blindly cannot troubleshoot or adapt. Understanding -the process produces durable knowledge; following a script does not. +the process produces durable knowledge; following a script does not. Embedding entire source +files inline inflates page length without adding explanatory value — the prose around the file +(the *why*) is what teaches; the file content itself is better consulted, copied, or diffed as +a real file. A committed, linked file is also directly reusable by a learner adapting the lab +to their own repo, which an inline fence is not. ### III. Cross-Platform Compatibility @@ -162,6 +189,24 @@ Labs that use simplified security practices (Principle IX) MUST also include: Labs MUST NOT require external network access beyond downloading freely available software and dependencies. All API samples and test data MUST be included in the repository. +Labs that instruct the user to create a large or reusable source file MUST commit that file's +content under a `code/` subdirectory within the lab's own directory and link to it from the +README, per Principle II, rather than embedding it inline. + +A lab is NOT complete until the root `README.md` is updated to reference it: a Lab Series +table row (linked, no longer marked "coming soon"), and an entry in both the Getting Started +tree and the Repository Structure tree for the new `labs/` and `specs/` directories. This is +part of the lab's definition of done, not a follow-up chore — `/speckit-plan` and +`/speckit-tasks` MUST account for it (an explicit task, not a generic "documentation updates" +placeholder), so it is done during `/speckit-implement` rather than caught later at PR review +or in CI. + +**Rationale**: The root README previously drifted out of date — Lab 4 shipped fully functional +while the README still listed it as "coming soon" — because README maintenance wasn't part of +any lab's definition of done. It was only caught after the fact, via an ad hoc CI check, which +is too late: by then the fix is a follow-up interruption rather than something completed as +part of the original work. + ## Development Workflow - Each lab corresponds to one speckit feature branch following the `###-lab-name` convention. @@ -193,4 +238,4 @@ All plan `Constitution Check` gates MUST reference the principles by Roman numer - `yarn dev` does not exist. Use `yarn start` instead. -**Version**: 1.3.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-06-08 +**Version**: 1.5.0 | **Ratified**: 2026-06-07 | **Last Amended**: 2026-07-04 diff --git a/.specify/templates/tasks-template.md b/.specify/templates/tasks-template.md index d46a1f1..92967ba 100644 --- a/.specify/templates/tasks-template.md +++ b/.specify/templates/tasks-template.md @@ -152,6 +152,9 @@ Examples of foundational tasks (adjust based on your project): **Purpose**: Improvements that affect multiple user stories - [ ] TXXX [P] Documentation updates in docs/ +- [ ] TXXX Update root README.md (Lab Series table entry, Getting Started tree, Repository + Structure tree) to reference this lab's `labs/` and `specs/` directories, per the + Constitution's Lab Structure Standards — required for this lab to be considered complete - [ ] TXXX Code cleanup and refactoring - [ ] TXXX Performance optimization across all stories - [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/ diff --git a/CLAUDE.md b/CLAUDE.md index adb4816..e465dfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,5 +1,5 @@ For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan -at specs/003-lab-3-api-quality/plan.md +at specs/004-lab-4-auto-registration/plan.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7e16cc5..ebd8b5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -252,7 +252,11 @@ Step 7 is complete and the PR checklist below is satisfied. 6. Confirm the lab starts from the state left by the previous lab and documents any dependencies on prior lab outputs. 7. Validate manually (Step 7 above) on at least one platform. -8. Open a PR and merge following Step 8 above. +8. Update the root `README.md` — add the lab to the Lab Series table and to the Getting + Started / Repository Structure trees. Run `bash scripts/check-readme.sh` to confirm + nothing is missing; CI runs the same check on every push and PR and will fail the build + if the root README falls out of sync with `labs/` or `specs/`. +9. Open a PR and merge following Step 8 above. --- @@ -300,6 +304,9 @@ Before submitting a PR for a new or changed lab, confirm: - [ ] No Petstore or similarly dated API examples are used (Constitution Principle VII) - [ ] Pre-committed API spec files are self-contained (no external `$ref`s) - [ ] The lab builds correctly on top of the previous lab's end state +- [ ] Root `README.md` Lab Series table and Repository Structure sections mention the new + `labs/` and `specs/` directories (run `bash scripts/check-readme.sh` to verify; this is + also enforced by CI on every push and PR) --- diff --git a/README.md b/README.md index 03723f2..79cbc6f 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Each lab is self-contained, cross-platform (Windows and macOS), and costs nothin | [Lab 1](labs/lab-01-base-backstage/) | Base Backstage | Install Backstage locally; register a REST API (OpenAPI) and an event API (AsyncAPI); verify both are visible and searchable | | [Lab 2](labs/lab-02-users-roles/) | Users, Roles & API Visibility | Add users, teams, and a custom permission policy; private APIs are visible only to their owning team, shared APIs are visible to everyone | | [Lab 3](labs/lab-03-api-quality/) | API Quality | Add a shared Spectral ruleset, the api-grade quality plugin, and the Spectral linter plugin; API owners and a platform team see detailed quality/lint results, everyone else sees a summary grade | -| Lab 4 *(coming soon)* | Auto Registration | Auto-discover and register APIs from a Git mono-repo; pull catalog metadata from `x-*` fields in the spec itself | +| [Lab 4](labs/lab-04-auto-registration/) | Auto Registration | Auto-discover and register APIs from a Git mono-repo via a custom `EntityProvider`; source owner, lifecycle, and visibility metadata from `x-*` fields in the spec itself | | Lab 5 *(coming soon)* | Mocking & Testing | Dynamically mock or exercise a test implementation of any registered API, with support for user-supplied non-production credentials | | Lab 6 *(coming soon)* | API Lifecycle Management | Register multiple major versions of an API in parallel; track lifecycle state (development/test/production) and deprecation/retirement per version | | Lab 7 *(coming soon)* | Other Documentation | Add the Thoughtworks Tech Radar plugin; register blips to plot your API landscape | @@ -66,10 +66,14 @@ labs/ ├── lab-02-users-roles/ │ ├── README.md ← continue here after Lab 1 │ └── catalog/ ← teams.yaml, users, and updated API owners/visibility -└── lab-03-api-quality/ - ├── README.md ← continue here after Lab 2 - ├── .spectral.yaml ← shared Spectral ruleset used by both quality plugins - └── catalog/ ← platform team member (eve) added in this lab +├── lab-03-api-quality/ +│ ├── README.md ← continue here after Lab 2 +│ ├── .spectral.yaml ← shared Spectral ruleset used by both quality plugins +│ └── catalog/ ← platform team member (eve) added in this lab +└── lab-04-auto-registration/ + ├── README.md ← continue here after Lab 3 + ├── autoApiRegistration.ts ← backend EntityProvider that scans and registers APIs + └── apis/ ← auto-discovered specs, incl. the Scalar Galaxy vendor copy ``` --- @@ -81,11 +85,13 @@ backstage-apiportal-lab/ ├── labs/ ← one directory per lab │ ├── lab-01-base-backstage/ │ ├── lab-02-users-roles/ -│ └── lab-03-api-quality/ +│ ├── lab-03-api-quality/ +│ └── lab-04-auto-registration/ ├── specs/ ← SDD artifacts (spec, plan, tasks per lab) │ ├── 001-lab-1-base-backstage/ │ ├── 002-lab-2-users-roles/ -│ └── 003-lab-3-api-quality/ +│ ├── 003-lab-3-api-quality/ +│ └── 004-lab-4-auto-registration/ ├── .specify/ ← Speckit configuration and templates ├── GOAL.md ← high-level goals for the full lab series ├── CONTRIBUTING.md ← how to contribute new labs diff --git a/labs/lab-02-users-roles/README.md b/labs/lab-02-users-roles/README.md index d13a808..c1ea96a 100644 --- a/labs/lab-02-users-roles/README.md +++ b/labs/lab-02-users-roles/README.md @@ -319,77 +319,19 @@ Run this command from inside your Backstage root directory #### Create `ApiVisibilityCard.tsx` -Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` with the following -content: - -```typescript -import React from 'react'; -import { useEntity } from '@backstage/plugin-catalog-react'; -import { InfoCard } from '@backstage/core-components'; -import { Typography, Box, Chip } from '@material-ui/core'; - -const VISIBILITY_ANNOTATION = 'example.com/visibility'; - -export function ApiVisibilityCard() { - const { entity } = useEntity(); - const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; - - if (!visibility) { - return ( - - - No visibility designation set. Under the permission policy, this API is visible - only to members of the owning team (treated as private by default). - - - ); - } - - const isShared = visibility === 'shared'; - - return ( - - - - - {isShared - ? 'Visible to all authenticated users regardless of team membership.' - : 'Visible only to members of the owning team.'} - - - - ); -} -``` +Create `packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx`](./code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx); +copy it in as-is. It reads the `example.com/visibility` annotation off the entity and renders a +"Shared"/"Private" chip, with a fallback message when the annotation is absent (treated as +private by the permission policy). #### Create `index.ts` -Create `packages/app/src/modules/apiVisibility/index.ts` with the following content: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiVisibilityCard } from './ApiVisibilityCard'; - -const apiVisibilityCard = EntityCardBlueprint.make({ - name: 'api-visibility', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiVisibilityCard), - }, -}); - -export const apiVisibilityModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiVisibilityCard], -}); -``` +Create `packages/app/src/modules/apiVisibility/index.ts` — its full content is committed at +[`code/packages/app/src/modules/apiVisibility/index.ts`](./code/packages/app/src/modules/apiVisibility/index.ts); +copy it in as-is. It wraps `ApiVisibilityCard` in an `EntityCardBlueprint` extension and bundles +it into a `createFrontendModule`. > **Note on Alpha imports**: `EntityCardBlueprint` is imported from > `@backstage/plugin-catalog-react/alpha`. This is the officially supported extension @@ -599,76 +541,13 @@ For all non-catalog permissions (scaffolding, TechDocs, search, etc.), the polic Create the directory `packages/backend/src/extensions/` inside your Backstage root (`labs/lab-01-base-backstage/backstage/`), then create a new file called -`permissionPolicy.ts` inside it with the following content: - -```typescript -// packages/backend/src/extensions/permissionPolicy.ts -import { createBackendModule } from '@backstage/backend-plugin-api'; -import { - PolicyDecision, - AuthorizeResult, - isResourcePermission, -} from '@backstage/plugin-permission-common'; -import { - PermissionPolicy, - PolicyQuery, - PolicyQueryUser, -} from '@backstage/plugin-permission-node'; -import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; -import { - catalogConditions, - createCatalogConditionalDecision, -} from '@backstage/plugin-catalog-backend/alpha'; - -class CatalogOwnershipPolicy implements PermissionPolicy { - async handle( - request: PolicyQuery, - user?: PolicyQueryUser, - ): Promise { - if (isResourcePermission(request.permission, 'catalog-entity')) { - return createCatalogConditionalDecision( - request.permission, - { - anyOf: [ - // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. - // This keeps the org chart and team pages open for everyone. - { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, - - // Rule 2: APIs annotated as shared are visible to all authenticated users, - // regardless of which team they belong to. - catalogConditions.hasAnnotation({ - annotation: 'example.com/visibility', - value: 'shared', - }), - - // Rule 3: Private APIs are visible only to members of the owning team. - // The user's ownershipEntityRefs contains their user ref plus all their - // group refs, resolved from the catalog User entity's memberOf list. - catalogConditions.isEntityOwner({ - claims: user?.info.ownershipEntityRefs ?? [], - }), - ], - }, - ); - } - // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. - return { result: AuthorizeResult.ALLOW }; - } -} - -export default createBackendModule({ - pluginId: 'permission', - moduleId: 'permission-policy', - register(reg) { - reg.registerInit({ - deps: { policy: policyExtensionPoint }, - async init({ policy }) { - policy.setPolicy(new CatalogOwnershipPolicy()); - }, - }); - }, -}); -``` +`permissionPolicy.ts` inside it. Its full content is committed alongside this README at +[`code/packages/backend/src/extensions/permissionPolicy.ts`](./code/packages/backend/src/extensions/permissionPolicy.ts); +copy it in as-is. In short: it's a `CatalogOwnershipPolicy` that, for the `catalog-entity` +resource permission, allows non-API entities unconditionally, allows APIs annotated +`example.com/visibility: shared` for everyone, and otherwise falls back to an ownership check +via `catalogConditions.isEntityOwner`; every other permission is unconditionally allowed. It's +registered as a `permission-policy` backend module via `createBackendModule`. **Import sources** (all packages are already installed in Lab 1 — no `npm install` needed): diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx new file mode 100644 index 0000000..0fcff5c --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/ApiVisibilityCard.tsx @@ -0,0 +1,41 @@ +import React from 'react'; +import { useEntity } from '@backstage/plugin-catalog-react'; +import { InfoCard } from '@backstage/core-components'; +import { Typography, Box, Chip } from '@material-ui/core'; + +const VISIBILITY_ANNOTATION = 'example.com/visibility'; + +export function ApiVisibilityCard() { + const { entity } = useEntity(); + const visibility = entity.metadata.annotations?.[VISIBILITY_ANNOTATION]; + + if (!visibility) { + return ( + + + No visibility designation set. Under the permission policy, this API is visible + only to members of the owning team (treated as private by default). + + + ); + } + + const isShared = visibility === 'shared'; + + return ( + + + + + {isShared + ? 'Visible to all authenticated users regardless of team membership.' + : 'Visible only to members of the owning team.'} + + + + ); +} diff --git a/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts new file mode 100644 index 0000000..e21dd81 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/app/src/modules/apiVisibility/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiVisibilityCard } from './ApiVisibilityCard'; + +const apiVisibilityCard = EntityCardBlueprint.make({ + name: 'api-visibility', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiVisibilityCard), + }, +}); + +export const apiVisibilityModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiVisibilityCard], +}); diff --git a/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts new file mode 100644 index 0000000..e5b52e8 --- /dev/null +++ b/labs/lab-02-users-roles/code/packages/backend/src/extensions/permissionPolicy.ts @@ -0,0 +1,66 @@ +// packages/backend/src/extensions/permissionPolicy.ts +import { createBackendModule } from '@backstage/backend-plugin-api'; +import { + PolicyDecision, + AuthorizeResult, + isResourcePermission, +} from '@backstage/plugin-permission-common'; +import { + PermissionPolicy, + PolicyQuery, + PolicyQueryUser, +} from '@backstage/plugin-permission-node'; +import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha'; +import { + catalogConditions, + createCatalogConditionalDecision, +} from '@backstage/plugin-catalog-backend/alpha'; + +class CatalogOwnershipPolicy implements PermissionPolicy { + async handle( + request: PolicyQuery, + user?: PolicyQueryUser, + ): Promise { + if (isResourcePermission(request.permission, 'catalog-entity')) { + return createCatalogConditionalDecision( + request.permission, + { + anyOf: [ + // Rule 1: Non-API entities (User, Group, etc.) are always visible to all users. + // This keeps the org chart and team pages open for everyone. + { not: catalogConditions.isEntityKind({ kinds: ['API'] }) }, + + // Rule 2: APIs annotated as shared are visible to all authenticated users, + // regardless of which team they belong to. + catalogConditions.hasAnnotation({ + annotation: 'example.com/visibility', + value: 'shared', + }), + + // Rule 3: Private APIs are visible only to members of the owning team. + // The user's ownershipEntityRefs contains their user ref plus all their + // group refs, resolved from the catalog User entity's memberOf list. + catalogConditions.isEntityOwner({ + claims: user?.info.ownershipEntityRefs ?? [], + }), + ], + }, + ); + } + // All non-catalog permissions (scaffolding, TechDocs, search) are unconditionally allowed. + return { result: AuthorizeResult.ALLOW }; + } +} + +export default createBackendModule({ + pluginId: 'permission', + moduleId: 'permission-policy', + register(reg) { + reg.registerInit({ + deps: { policy: policyExtensionPoint }, + async init({ policy }) { + policy.setPolicy(new CatalogOwnershipPolicy()); + }, + }); + }, +}); diff --git a/labs/lab-03-api-quality/README.md b/labs/lab-03-api-quality/README.md index fe4447b..f5d59d8 100644 --- a/labs/lab-03-api-quality/README.md +++ b/labs/lab-03-api-quality/README.md @@ -134,28 +134,11 @@ value `'info'` places the card in the right-hand Info column (the same column as card). Without it, the card defaults to the main content area at the bottom of the page — far less visible. -Create the file `packages/app/src/modules/apiGrade/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import React from 'react'; -import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; - -const apiGradeCard = EntityCardBlueprint.make({ - name: 'api-grade', - params: { - filter: 'kind:API', - type: 'info', - loader: async () => React.createElement(ApiGradeCard), - }, -}); - -export const apiGradeModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [apiGradeCard], -}); -``` +Create the file `packages/app/src/modules/apiGrade/index.ts` — its full content is committed +alongside this README at +[`code/packages/app/src/modules/apiGrade/index.ts`](./code/packages/app/src/modules/apiGrade/index.ts); +copy it in as-is. It wraps the plugin's `ApiGradeCard` in an `EntityCardBlueprint` extension and +bundles it into a `createFrontendModule`, the same pattern as Lab 2's `apiVisibility` module. `filter: 'kind:API'` ensures the card only appears on API entity pages — not on Component, System, or other entity pages. @@ -354,60 +337,17 @@ mode. Importing it directly sidesteps the incompatibility entirely instead of wo it. Because this subpath has no TypeScript declaration file, add a small ambient module -declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`: - -```typescript -declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { - import { ComponentType } from 'react'; - export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; -} -``` - -Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`: - -```typescript -import React, { useEffect, useState } from 'react'; -import { useApi, identityApiRef } from '@backstage/core-plugin-api'; -import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; -import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; -import { InfoCard } from '@backstage/core-components'; -import { Typography } from '@material-ui/core'; - -export function SpectralLinterContent() { - const { entity } = useEntity(); - const identityApi = useApi(identityApiRef); - const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); - const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); - const [identityLoading, setIdentityLoading] = useState(true); - - useEffect(() => { - identityApi.getBackstageIdentity().then(identity => { - setIsPlatformTeamMember( - identity.ownershipEntityRefs.includes('group:default/platform-team'), - ); - setIdentityLoading(false); - }); - }, [identityApi]); - - if (ownershipLoading || identityLoading) return null; - - // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate - // FUNCTION, not a boolean — it must be called with the current entity. - if (!isOwnedEntity(entity) && !isPlatformTeamMember) { - return ( - - - Detailed quality information is restricted to members of the API's owning team - and the platform team. Sign in as a member of the owning team or the platform - team to view linting results. - - - ); - } - - return ; -} -``` +declaration. Create `packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts` — its +full content is committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts`](./code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts); +copy it in as-is. + +Create the file `packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx` — its full +content is committed at +[`code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx`](./code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx); +copy it in as-is. In short: it renders the plugin's inner `EntityApiDocsSpectralLinterContent` +component, gated behind an ownership/platform-team check — the owning team or a platform-team +member sees the lint results; everyone else sees an access-restricted message. **Why is `isOwnedEntity` called as a function?** `useEntityOwnership()` (from `@backstage/plugin-catalog-react`) returns `{ loading, isOwnedEntity }` where `isOwnedEntity` @@ -428,39 +368,14 @@ permission policy. ## Step 9 — Create the spectralLinter Frontend Module -Create the file `packages/app/src/modules/spectralLinter/index.ts`: - -```typescript -import { createFrontendModule } from '@backstage/frontend-plugin-api'; -import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; -import { convertLegacyPlugin } from '@backstage/core-compat-api'; -import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; -import React from 'react'; -import { SpectralLinterContent } from './SpectralLinterContent'; - -const spectralLinterContent = EntityContentBlueprint.make({ - name: 'spectral-linter', - params: { - defaultPath: '/spectral', - defaultTitle: 'Spectral', - filter: 'kind:API', - loader: async () => React.createElement(SpectralLinterContent), - }, -}); - -export const spectralLinterModule = createFrontendModule({ - pluginId: 'catalog', - extensions: [spectralLinterContent], -}); - -// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend -// system's API registry. `extensions: []` means we deliberately do NOT bring in the -// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped -// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. -export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { - extensions: [], -}); -``` +Create the file `packages/app/src/modules/spectralLinter/index.ts` — its full content is +committed alongside this README at +[`code/packages/app/src/modules/spectralLinter/index.ts`](./code/packages/app/src/modules/spectralLinter/index.ts); +copy it in as-is. In short: it wraps `SpectralLinterContent` in an `EntityContentBlueprint` +extension (the "Spectral" tab) and, separately, uses `convertLegacyPlugin` to register the +plugin's `linterApiRef` (backed by its `LinterClient`) with the new frontend system's API +registry, with `extensions: []` since only the API — not the plugin's own page/route +extensions — is needed. `defaultPath: '/spectral'` and `defaultTitle: 'Spectral'` define the tab's URL path and display label. `filter: 'kind:API'` ensures the tab appears only on API entity pages. diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts new file mode 100644 index 0000000..826d22c --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/apiGrade/index.ts @@ -0,0 +1,18 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityCardBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import React from 'react'; +import { ApiGradeCard } from '@dawmatt/backstage-plugin-api-grade'; + +const apiGradeCard = EntityCardBlueprint.make({ + name: 'api-grade', + params: { + filter: 'kind:API', + type: 'info', + loader: async () => React.createElement(ApiGradeCard), + }, +}); + +export const apiGradeModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [apiGradeCard], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx new file mode 100644 index 0000000..90760ac --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/SpectralLinterContent.tsx @@ -0,0 +1,41 @@ +import React, { useEffect, useState } from 'react'; +import { useApi, identityApiRef } from '@backstage/core-plugin-api'; +import { useEntity, useEntityOwnership } from '@backstage/plugin-catalog-react'; +import { EntityApiDocsSpectralLinterContent } from '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js'; +import { InfoCard } from '@backstage/core-components'; +import { Typography } from '@material-ui/core'; + +export function SpectralLinterContent() { + const { entity } = useEntity(); + const identityApi = useApi(identityApiRef); + const { isOwnedEntity, loading: ownershipLoading } = useEntityOwnership(); + const [isPlatformTeamMember, setIsPlatformTeamMember] = useState(false); + const [identityLoading, setIdentityLoading] = useState(true); + + useEffect(() => { + identityApi.getBackstageIdentity().then(identity => { + setIsPlatformTeamMember( + identity.ownershipEntityRefs.includes('group:default/platform-team'), + ); + setIdentityLoading(false); + }); + }, [identityApi]); + + if (ownershipLoading || identityLoading) return null; + + // `isOwnedEntity` from useEntityOwnership() is a per-entity predicate + // FUNCTION, not a boolean — it must be called with the current entity. + if (!isOwnedEntity(entity) && !isPlatformTeamMember) { + return ( + + + Detailed quality information is restricted to members of the API's owning team + and the platform team. Sign in as a member of the owning team or the platform + team to view linting results. + + + ); + } + + return ; +} diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts new file mode 100644 index 0000000..aa32826 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/index.ts @@ -0,0 +1,29 @@ +import { createFrontendModule } from '@backstage/frontend-plugin-api'; +import { EntityContentBlueprint } from '@backstage/plugin-catalog-react/alpha'; +import { convertLegacyPlugin } from '@backstage/core-compat-api'; +import { apiDocsSpectralLinterPlugin } from '@dweber019/backstage-plugin-api-docs-spectral-linter'; +import React from 'react'; +import { SpectralLinterContent } from './SpectralLinterContent'; + +const spectralLinterContent = EntityContentBlueprint.make({ + name: 'spectral-linter', + params: { + defaultPath: '/spectral', + defaultTitle: 'Spectral', + filter: 'kind:API', + loader: async () => React.createElement(SpectralLinterContent), + }, +}); + +export const spectralLinterModule = createFrontendModule({ + pluginId: 'catalog', + extensions: [spectralLinterContent], +}); + +// Registers the plugin's linterApiRef (backed by its LinterClient) with the new frontend +// system's API registry. `extensions: []` means we deliberately do NOT bring in the +// plugin's own page/route extensions — we only need its API, since Step 8's unwrapped +// component looks up `linterApiRef` via the ambient `useApi` context, not via a prop. +export const spectralLinterApiPlugin = convertLegacyPlugin(apiDocsSpectralLinterPlugin, { + extensions: [], +}); diff --git a/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts new file mode 100644 index 0000000..c71e3a1 --- /dev/null +++ b/labs/lab-03-api-quality/code/packages/app/src/modules/spectralLinter/spectral-linter-content.d.ts @@ -0,0 +1,4 @@ +declare module '@dweber019/backstage-plugin-api-docs-spectral-linter/dist/components/EntityApiDocsSpectralLinterContent/index.esm.js' { + import { ComponentType } from 'react'; + export const EntityApiDocsSpectralLinterContent: ComponentType<{}>; +} diff --git a/labs/lab-04-auto-registration/README.md b/labs/lab-04-auto-registration/README.md new file mode 100644 index 0000000..8ca2198 --- /dev/null +++ b/labs/lab-04-auto-registration/README.md @@ -0,0 +1,430 @@ +# Lab 4 — Auto Registration + +## Overview + +Labs 1–3 registered every API by hand-authoring a `catalog-info.yaml` per API and adding it as a +`type: url` catalog location. That works for three APIs; it does not work for a mono-repo with +hundreds. This lab replaces the per-API hand-authored file with a custom Backstage backend module +that scans a directory tree for `*-openapi.yaml` / `*-asyncapi.yaml` files and turns each one +directly into a catalog `API` entity — no `catalog-info.yaml` required. + +By the end of this lab you will have: + +- A new backend module (`autoApiRegistration.ts` + a small database migration) that discovers, + parses, and registers API definitions on a poll schedule, using Backstage's `EntityProvider` + mechanism so create, update, and removal all come from the same code path. +- Owner, lifecycle, and visibility metadata sourced from each spec's own `info.x-examplecorp` + object, with documented defaults when a field is absent — no duplication of metadata that + already has a natural home in the spec (`info.title`, `info.description`, native `tags`). +- Errors (malformed files, unresolvable owners, invalid visibility values, name collisions) + surfaced through Backstage's own backend logs and its native catalog processing-error UI — no + bespoke error-reporting system. +- Two new sample APIs: a vendored copy of the Scalar Galaxy API (pure auto-discovery, no + `catalog-info.yaml`) and a small precedence-demo API pair that demonstrates a hand-authored + `catalog-info.yaml` winning over auto-sourced metadata for the same API. + +**What you will learn:** + +- Why a custom `EntityProvider` (not a `CatalogProcessor`) is the right mechanism for + filesystem-based catalog discovery, and how its `full`/`delta` mutation lifecycle gives you + create/update/remove for free +- How to source catalog metadata from a vendor-namespaced `x-*` extension instead of a parallel + hand-authored file, and how to fall back safely when a field is absent +- How to surface processing errors through Backstage's built-in mechanisms (backend logs + a + `CatalogProcessor` that throws on a marker annotation) instead of inventing a new one +- How a persisted scan-state cache and `sources[]` config generalize this mechanism from "one + small demo repo" to "1000+ files across multiple team repos" without an architecture change + +--- + +## Prerequisites + +- **Lab 3 completed** — Backstage running locally with Museum API, Streetlights API, and Train + Travel API registered; `eve` (platform team) able to sign in and see quality detail +- **Node.js 20 LTS** and **Yarn** — same versions as Labs 1–3 + +--- + +## Step 1 — Add Dependencies + +From `packages/backend`, add the three new direct dependencies (`fast-glob` for the filesystem +scan, `js-yaml` for parsing, `chokidar` for the optional `watch` mode used at real-world scale): + +``` +cd packages/backend +yarn add fast-glob@3.3.3 js-yaml@4.2.0 chokidar@3.6.0 +``` + +All three are already present transitively in the workspace — this promotes them to explicit +dependencies of the backend package, which is what actually imports them. + +--- + +## Step 2 — Add the Auto-Registration Backend Module + +This is the heart of the lab: a backend module that scans a directory for API definition files, +parses each one, and emits catalog `API` entities via a custom `EntityProvider`. A companion +`CatalogProcessor` surfaces registration errors (invalid owner, invalid visibility, name +collisions) through Backstage's native processing-error UI. + +**Why an `EntityProvider`, not a `CatalogProcessor`?** A `CatalogProcessor` only transforms +entities that already exist via some `Location` — it can't originate new entities from an +arbitrary filesystem scan on its own. An `EntityProvider` owns a versioned "set of entities I +currently know about"; anything it stops claiming is automatically retracted by the catalog. That +gives create, update, and removal-on-delete from one code path, with no separate cleanup logic. + +**Why is owner/lifecycle/visibility sourced from `info.x-examplecorp` instead of duplicated +fields?** Metadata that already has a natural home in the spec — `info.title` (name), +`info.description` (description), native top-level `tags` — is read from there, not duplicated +into a vendor extension. Only metadata with **no** natural OpenAPI/AsyncAPI home (who owns this +API, what lifecycle stage it's in, whether it's private or shared) goes under `x-examplecorp`. +`examplecorp` is a fictional company namespace — it's the first thing you should rename to your +own company when adapting this lab (see "Adaptable Conventions" below). + +**Why does visibility reuse the exact `example.com/visibility` annotation from Lab 2/3?** The +permission policy from Lab 3 (`packages/backend/src/extensions/permissionPolicy.ts`) already +reads that annotation to decide access: `shared` bypasses ownership entirely, anything else +(including absent) falls through to the ownership-only rule. This lab adds a **third way to set +that same annotation** — sourced from `x-examplecorp.visibility` — not a new visibility mechanism +or a change to the permission policy. + +**Why do errors ride on Backstage's own backend logs and processing-error UI, instead of a new +error-reporting system?** Every discovery cycle logs one line per skipped/errored file via the +standard backend logger. For errors on a file specific enough to have a candidate entity (invalid +owner, invalid visibility, name collision), the provider still emits a minimal entity tagged with +an `apiportal-lab.io/registration-error` annotation; a small companion `CatalogProcessor` throws an +`InputError` when it sees that annotation, and Backstage's catalog engine natively records and +surfaces per-entity processing errors — the same "Inspect entity" / processing-errors view you'd +see for any other catalog ingestion problem. One message, written once, drives both surfaces. +Files that fail even the basic shape check (not parseable as an API spec at all — no +`openapi`/`asyncapi` field, no `info.title`) can't produce a minimal entity (there's no name to +register under), so those are logged only, not surfaced in the UI. + +**Why does a hand-authored `catalog-info.yaml` win over auto-sourced metadata?** If a file already +has a hand-authored catalog entry (`kind: API`, matching `metadata.name`) that didn't come from +this provider, the provider skips its own auto-sourced candidate for that name entirely. This +lets you hand-author precise metadata for a specific API — pinning an owner, tightening +visibility — without the auto-registration mechanism fighting you for it every cycle. + +Create the database migration first, at +`packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`. Its full +content is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts`](./code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts) +— copy it in as-is. It creates the `auto_api_registration_scan_state` table (`source_id`, +`file_path`, `mtime_ms`, `content_hash`, `entity_name`, `last_error`), applied via a knex custom +`migrationSource` (see the module below) rather than a filesystem `directory` migration source, +since this file lives alongside the module's TypeScript source rather than in a package-level +`migrations/` folder resolved at runtime. + +Now the module itself, at `packages/backend/src/extensions/autoApiRegistration.ts` — this is the +heart of the lab. Its full content (~750 lines: config normalization, filesystem discovery and +YAML parsing, `x-*` candidate mapping with owner/visibility/collision validation, the scan-state +cache, the `EntityProvider` itself with its poll/watch scheduling, and the companion +`CatalogProcessor`) is committed alongside this README — +[`code/packages/backend/src/extensions/autoApiRegistration.ts`](./code/packages/backend/src/extensions/autoApiRegistration.ts) +— copy it in as-is. + +--- + +## Step 3 — Register the Module + +Open `packages/backend/src/index.ts` and add one line after the existing catalog module +registrations: + +```typescript +// See https://backstage.io/docs/features/software-catalog/configuration#subscribing-to-catalog-errors +backend.add(import('@backstage/plugin-catalog-backend-module-logs')); + +// Lab 4: auto-discover *-openapi.yaml / *-asyncapi.yaml files and register them as API entities +// with no hand-authored catalog-info.yaml required — see labs/lab-04-auto-registration/README.md +backend.add(import('./extensions/autoApiRegistration')); +``` + +Like `permissionPolicy.ts` in Lab 3, the module self-registers and declares its own dependencies +via `createBackendModule` — no other wiring is needed. + +--- + +## Step 4 — Configure Discovery + +Add an `autoApiRegistration` block to `app-config.yaml`. This is the **single-source shorthand** +— equivalent to a one-entry `sources: [{ id: default, ... }]` list — which is all a single-repo +setup like this lab needs: + +```yaml +autoApiRegistration: + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp +``` + +`rootPath` is deliberately left unset here — the module resolves a default of +`labs/lab-04-auto-registration/apis` (relative to the backend package) on its own. `patterns`, +`ignore`, `mode`, and `schedule.frequencySeconds` all default sensibly too (see "Adaptable +Conventions vs. Fixed Mechanics" below for the full list). `defaultOwner` and `xNamespace` have no +built-in default and must be set explicitly — a source shouldn't silently inherit a fallback team +or vendor namespace it never opted into. + +--- + +## Step 5 — Add the Sample API Files + +Two sample files are already committed alongside this README (nothing to create in this step — +just take a look): + +- **`apis/galaxy/galaxy-openapi.yaml`** — a vendored copy of the MIT-licensed + [Scalar Galaxy API](https://github.com/scalar/scalar), with an `info.x-examplecorp` object + (`owner`, `lifecycle`, `visibility: shared`) added. It has **no** `catalog-info.yaml` — this is + the "previously unregistered API" that Step 6 will confirm gets discovered automatically. +- **`apis/precedence-demo/precedence-demo-openapi.yaml`** + **`precedence-demo-catalog-info.yaml`** + — a minimal API paired with a *hand-authored* catalog entry (owner `museum-team`, + `visibility: private`) that deliberately differs from the spec file's own `x-examplecorp` + values (owner `platform-team`, `visibility: shared`). This demonstrates the precedence rule: + the hand-authored file wins. + +The hand-authored `precedence-demo-catalog-info.yaml` needs its own catalog location, the same way +Lab 2/3's hand-authored files do — add it to `app-config.yaml`'s `catalog.locations`, replacing +`` and `` with your fork and the current branch name: + +```yaml +catalog: + locations: + # --- Lab 4: Precedence-demo API (hand-authored — must win over auto-sourced metadata) --- + - type: url + target: https://raw.githubusercontent.com//backstage-apiportal-lab//labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml + rules: + - allow: [API] +``` + +This only works once you've pushed your branch — the raw URL has to actually resolve. If you +haven't pushed yet, push now before continuing to Step 6. + +--- + +## Step 6 — Start Backstage and Verify Discovery + +``` +yarn start +``` + +Within one discovery cycle (default 30s), open the catalog and confirm: + +- **The Galaxy API appears with no hand-authored `catalog-info.yaml`.** Search for "Scalar + Galaxy" — it should show up as an `API` entity even though you never wrote a `catalog-info.yaml` + for it. +- **Its owner, lifecycle, and tags match the spec.** `spec.owner` and `spec.lifecycle` should + match the `info.x-examplecorp` values in `galaxy-openapi.yaml`; `metadata.tags` should match its + native `tags` array (slugified — kebab-case, not the original mixed-case names). +- **Visibility contrast.** Sign in as a non-owning, non-platform user (e.g. `charlie`, from Lab + 2 — see `app-config.local.yaml`). The Galaxy API (`visibility: shared`) should be visible; the + precedence-demo API (`visibility: private`, hand-authored, owned by `museum-team`) should not be. +- **Precedence.** The precedence-demo API's entity should reflect the hand-authored + `precedence-demo-catalog-info.yaml` (owner `museum-team`, `visibility: private`) — **not** the + auto-sourced `x-examplecorp` values in its own spec file (owner `platform-team`, + `visibility: shared`). + +--- + +## Step 7 — Verify Update-in-Place + +Edit `apis/galaxy/galaxy-openapi.yaml`'s `info.description` and wait one poll cycle. Reload the +Galaxy API's entity page — the description should update in place. The entity's `metadata.uid` +does not change; this is an update, not a duplicate registration. + +--- + +## Step 8 — Verify Removal + +Temporarily rename `galaxy-openapi.yaml` to something outside the discovery pattern (e.g. +`galaxy-openapi.yaml.bak`) and wait one poll cycle. The Galaxy API should disappear from the +catalog. Rename it back — it reappears within one more cycle, as a fresh registration (a new +`metadata.uid`, since from the provider's perspective this is indistinguishable from a new file). + +--- + +## Step 9 — Verify Error Handling + +Three failure modes, each temporary — revert after checking: + +1. **Malformed YAML.** Break the syntax in one sample file (e.g. an unclosed quote). Check the + backend logs — you should see a line identifying the specific file, and the entity for that + file should simply stop appearing in the catalog (not appear broken; log-only, since a file + that doesn't parse has no name to register an error entity under). +2. **Unresolvable owner.** Set `info.x-examplecorp.owner` to a nonexistent team (e.g. + `group:default/nonexistent-team`). Within one cycle, the entity's "Inspect entity" view should + show a catalog processing error naming the bad owner reference. Other entities are unaffected. +3. **Invalid visibility.** Set `info.x-examplecorp.visibility` to an unrecognized value (e.g. + `public`). You should see the same kind of processing error — not a silent fallback to + `private`. + +--- + +## Adaptable Conventions vs. Fixed Mechanics + +**Adaptable — change these to fit your own repo layout:** + +- `rootPath` — where the scan starts. Point it at your own mono-repo root. +- `patterns` — the filename glob(s) that identify an API definition file. The default + (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`) is this lab's convention, not Backstage's; use + whatever your team already follows. +- `xNamespace` — the vendor namespace under `info.x-` that owner/lifecycle/visibility + are read from. `examplecorp` is a fictional placeholder — **rename this to your own company or + team first** when adapting this lab. +- `defaultOwner` / `defaultVisibility` — the fallback values used when a spec doesn't declare its + own. Per-source, so different repos can have different fallbacks. +- `ignore` — glob exclusions, always applied. Extend (don't replace) the defaults + (`node_modules`, `.git`, `dist`, `build`) for your own build-output directories. + +**Fixed — this is how the mechanism works, not a config knob:** + +- The `EntityProvider` full-then-delta mutation lifecycle (Step 2's rationale above). +- The scan-state cache schema and its role in restart/delta behavior (see "Scaling to a Real + Mono-Repo" below). +- Errors riding on Backstage's native backend-log + processing-error-UI mechanisms, rather than a + bespoke error system. +- The catalog-info.yaml precedence rule: a hand-authored entity for a name always wins over an + auto-sourced candidate for that same name. + +--- + +## Scaling to a Real Mono-Repo + +This lab's defaults (`mode: 'poll'`, 30-second full sweep, no reconciliation) are tuned for 2–3 +sample files and interactive feedback — they are **not** what a 1000+ file, 1GB+ mono-repo should +run with. The architecture (an `EntityProvider` driving mutations) scales fine as-is; only the +config values need to change: + +- **`mode: 'watch'`** replaces the repeated full-tree poll with a `chokidar` watcher that reacts + to individual file add/change/unlink events, so "how often do we rescan" stops scaling with + repo size. A much longer `reconciliation.frequencySeconds` (minutes, not seconds) runs + alongside it as a safety-net full sweep, catching anything a watcher event could plausibly miss + (coalesced events under a big `git pull`, watcher reliability limits on some filesystems). +- **The scan-state cache is what makes restarts fast at scale.** It's a small table in the + backend's own database (`coreServices.database` — SQLite in this lab, Postgres in a real + deployment), storing each file's path, modification time, content hash, and mapped entity name. + On restart, the provider compares this cache against the filesystem instead of re-parsing + everything from zero — unchanged files cost nothing. This is deliberately **not** written back + into the mono-repo as generated `catalog-info.yaml` files: that would reintroduce the + hand-maintained-duplicate-file problem this lab exists to eliminate, and would require the + discovery tool to have *write* access to a repo it should only need to read. +- After the first `full` mutation establishes the baseline, every later cycle sends a `delta` + mutation (`added`/`removed` only) — cost proportional to what changed, not to total catalog + size. This is the same pattern Backstage's own GitHub discovery provider uses at organization + scale. +- `parseConcurrency` bounds how many files are parsed at once, so a large batch of simultaneous + changes (e.g. checking out a branch that touches hundreds of files) doesn't spike memory or + block the event loop. + +If you scale this lab up, expect restarts to be fast (cache-driven, not a full re-scan) — that's +the part most likely to surprise you the first time. + +--- + +## Scaling to Multiple Source Repositories + +The config schema generalizes from one `rootPath` to a list of independently-configured sources: + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo + rootPath: ../platform-monorepo/apis + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + mode: watch + reconciliation: + frequencySeconds: 900 + defaultOwner: group:default/platform-team + defaultVisibility: private + xNamespace: examplecorp + - id: team-checkout-api + rootPath: ../team-checkout-api + patterns: ['**/*-openapi.yaml'] + mode: poll + schedule: + frequencySeconds: 60 + defaultOwner: group:default/checkout-team + defaultVisibility: shared + xNamespace: checkoutteam +``` + +Each entry becomes its own `EntityProvider` instance with its own schedule, own mode, own default +owner, and own `x-*` namespace — a slow filesystem or config problem in one source can't stall or +break discovery for any other source. + +`defaultOwner` and `xNamespace` are per-source with **no** built-in global fallback: a team +onboarding their own repo shouldn't have to adopt the platform mono-repo's team or vendor +namespace just to participate. `defaultVisibility` is the one exception — it's per-source but +*does* fall back to `private` if a source doesn't set it, because leaving visibility completely +unconfigured must still be safe by default, not just adaptable. + +Collision detection and the catalog-info.yaml precedence check are **global across every source**, +not scoped to whichever source is currently being scanned — two different teams' repos can easily +produce the same slugified entity name (two `payments-api`s), and that must surface a visible +conflict rather than silently coexisting or overwriting. + +Onboarding a genuinely separate Git remote (not a local sibling checkout, which is what this lab's +own multi-source walkthrough would use) needs one more piece this lab doesn't build: a sync step +(e.g. a scheduled shallow `git clone`/`pull` into a local cache directory) ahead of the existing +glob/parse pipeline, pointing that source's `rootPath` at the resulting local worktree. Every other +mechanism — glob patterns, ignore rules, delta mutations, scan-state cache, collision/precedence +checks — is reused unchanged; only how bytes arrive on local disk differs. + +--- + +## Verification Checklist + +- [ ] Galaxy API appears in the catalog within one poll cycle, with no hand-authored + `catalog-info.yaml` +- [ ] Galaxy API's owner/lifecycle match its `x-examplecorp` values; tags match its native `tags` + array +- [ ] Editing `galaxy-openapi.yaml`'s description updates the existing entity, not a duplicate +- [ ] Renaming `galaxy-openapi.yaml` out of the discovery pattern retracts the entity; renaming it + back re-registers it +- [ ] Galaxy (`shared`) is visible to a non-owning, non-platform user; precedence-demo + (`private`, hand-authored) is not +- [ ] The precedence-demo API reflects `precedence-demo-catalog-info.yaml`'s values, not its own + spec file's `x-examplecorp` values +- [ ] A broken-YAML file logs an identifying error and its entity does not appear +- [ ] A nonexistent owner reference surfaces a catalog processing error on that entity only +- [ ] An unrecognized visibility value surfaces a processing error rather than a silent default + +--- + +## Troubleshooting + +- **Nothing appears after 30+ seconds.** Check the backend logs for + `auto-api-registration:default:` lines. A "skipped a cycle — provider not yet connected" + warning on the very first tick is normal (the scheduler's first run can fire slightly before + the catalog engine finishes wiring up the provider); it should not repeat. +- **"Owner ... does not resolve to a known User or Group entity" for an owner you're sure + exists.** Confirm that group is actually loaded in the catalog (check its entity page directly) + — this error means the *catalog* doesn't know about the group yet, not that your YAML is wrong. + A common cause locally is a `catalog.locations` URL that hasn't been pushed yet (see Step 5). + This can also show up as "no APIs appear in the catalog at all" rather than a single owner + error: if the `catalog.locations` entry for your org data (`teams.yaml`/`users.yaml`) points at + a branch name that's since been merged and deleted (a stale reference from an earlier lab, + rather than your current branch), that location 404s, the group it would have defined never + registers, and *every* auto-sourced API whose owner references that group fails validation — + which looks like the discovery mechanism itself is broken. Check the backend logs for "Unable + to read url, no matching files found" lines against your `catalog.locations` URLs first; if any + point at a branch other than `main` or your current branch, that's almost always the real cause. +- **An entity you expect to see is silently missing, with no error logged.** Check the backend + logs for a `Policy check failed for api:default/` warning — this means the entity was + built and emitted, but failed Backstage's own entity-schema validation (for example, + `metadata.tags` containing something that isn't a valid kebab-case slug) during processing, so + it never got "stitched" into the catalog you can query. This is different from a registration + error (which the error processor surfaces intentionally) — it's a schema mismatch the entity + never should have had in the first place. +- **A stale entity keeps reappearing after you thought you removed it.** The scan-state cache + persists across backend restarts (it's a real database table, not in-memory). If you manually + edited the database or copied it between environments, the cache may be out of sync with the + filesystem. Deleting the `auto_api_registration_scan_state` table (or just wiping the local + SQLite file, per this lab's `':memory:'` dev config, restarting achieves the same thing) forces + a fresh full rescan. +- **Glob pattern typos.** If a file you expect to be discovered isn't, double check `patterns` — + a mismatched suffix (`-openapi.yml` vs `-openapi.yaml`) is a common culprit, and the discovery + mechanism has no way to tell you "this almost matched." +- **Wrong `xNamespace`.** If owner/lifecycle/visibility all come back as defaults even though your + spec file has an `x-*` object, check that the object's key matches `xNamespace` exactly — + `x-examplecorp` in config but `x-example-corp` (extra hyphen) in the spec file will silently + fall through to defaults rather than error, since an absent `x-*` object is valid input. diff --git a/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml new file mode 100644 index 0000000..053975a --- /dev/null +++ b/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml @@ -0,0 +1,1476 @@ +openapi: 3.1.1 +info: + title: Scalar Galaxy + description: | + The Scalar Galaxy is an example OpenAPI document to test OpenAPI tools and libraries. It's a fictional universe with fictional planets and fictional data. + + ## Resources + + * https://github.com/scalar/scalar + * https://github.com/OAI/OpenAPI-Specification + * https://scalar.com + + ## Markdown Support + + All descriptions *can* contain ~~tons of text~~ **Markdown**. [If GitHub supports the syntax](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax), chances are we're supporting it, too. + +
+ Examples + + **Blockquotes** + + > I love OpenAPI. <3 + + **Tables** + + | Feature | Availability | + | ---------------- | ------------ | + | Markdown Support | ✓ | + + **Accordion** + + ```html +
+ Using Details Tags +

HTML Example

+
+ ``` + + **Images** + + Yes, there's support for images, too! + + ![Empty placeholder image showing the width/height](https://images.placeholders.dev/?width=1280&height=720) + + **Alerts** + + > [!tip] + > You can use Markdown alerts in your descriptions. + +
+ version: 0.6.8 + contact: + name: Scalar Support + url: https://scalar.com + email: support@scalar.com + license: + name: MIT + url: https://opensource.org/license/MIT + # Lab 4: catalog metadata sourced by autoApiRegistration (owner/lifecycle/visibility) — + # see labs/lab-04-auto-registration/README.md. No catalog-info.yaml exists for this API; + # it is discovered and registered purely from this file. + x-examplecorp: + owner: group:default/platform-team + lifecycle: production + visibility: shared +externalDocs: + description: Documentation + url: https://github.com/scalar/scalar +servers: + - url: https://galaxy.scalar.com + - url: '{protocol}://void.scalar.com/{path}' + description: Responds with your request data + variables: + protocol: + enum: + - https + - http + default: https + path: + default: '' +security: + - bearerAuth: [] + - basicAuth: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + - apiKeyHeader: [] + apiKeyQuery: [] + - apiKeyCookie: [] + - oAuth2: [] + - openIdConnect: [] +x-speakeasy-webhooks: + security: + type: signature + headerName: x-signature + signatureTextEncoding: base64 + algorithm: hmac-sha256 +tags: + - name: Authentication + description: Some endpoints are public, but some require authentication. We + provide all the required endpoints to create an account and authorize + yourself. + - name: Planets + description: Everything about planets + - name: Celestial Bodies + description: Celestial bodies are the planets and satellites in the Scalar Galaxy. +paths: + /planets: + get: + tags: + - Planets + summary: Get all planets + description: + It's easy to say you know them all, but do you really? Retrieve all + the planets and check whether you missed one. + operationId: getAllData + security: [] + parameters: + - $ref: '#/components/parameters/limit' + - $ref: '#/components/parameters/offset' + responses: + '200': + description: OK + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Pagination-Total: + $ref: '#/components/headers/X-Pagination-Total' + X-Pagination-Page: + $ref: '#/components/headers/X-Pagination-Page' + X-Pagination-Per-Page: + $ref: '#/components/headers/X-Pagination-Per-Page' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanets' + # Bind the generic Paginated template's item type to Planet via a $dynamicAnchor. + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + application/xml: + schema: + $id: 'https://galaxy.scalar.com/schemas/PaginatedPlanetsXml' + xml: + name: planets + $defs: + itemType: + $dynamicAnchor: itemType + $ref: '#/components/schemas/Planet' + $ref: '#/components/schemas/Paginated' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has data array and meta", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("data"); + pm.expect(jsonData.data).to.be.an("array"); + pm.expect(jsonData).to.have.property("meta"); + }); + post: + tags: + - Planets + summary: Create a planet + description: + Time to play god and create a new planet. What do you think? Ah, + don't think too much. What could go wrong anyway? + operationId: createPlanet + callbacks: + planetCreated: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly created planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the callback + '204': + description: + Your server should return this HTTP status code if no longer + interested in further updates + planetCreationFailed: + '{$request.body#/failureCallbackUrl}': + post: + security: [] + requestBody: + description: Information about which fields failed to validate + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the failure callback notification + planetExploded: + '{$request.body#/successCallbackUrl}': + post: + security: [] + requestBody: + description: Information about the newly exploded planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: Your server returns this code if it accepts the planet explosion callback notification + requestBody: + description: Planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + responses: + '201': + description: Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /planets/{planetId}: + get: + tags: + - Planets + summary: Get a planet + description: + You'll better learn a little bit more about the planets. It might + come in handy once space travel is available for everyone. + operationId: getPlanet + security: [] + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet Found + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + put: + tags: + - Planets + summary: Update a planet + description: Sometimes you make mistakes, that's fine. No worries, you can + update all planets. + operationId: updatePlanet + requestBody: + description: New information about the planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '200': + description: Planet updated successfully + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + application/xml: + schema: + $ref: '#/components/schemas/Planet' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a planet with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + delete: + tags: + - Planets + summary: Delete a planet + operationId: deletePlanet + description: This endpoint was used to delete planets. Unfortunately, that + caused a lot of trouble for planets with life. So, this endpoint is now + deprecated and should not be used anymore. + x-scalar-stability: experimental + parameters: + - $ref: '#/components/parameters/planetId' + responses: + '204': + description: No Content + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 204", () => { + pm.response.to.have.status(204); + }); + /planets/{planetId}/image: + post: + tags: + - Planets + summary: Upload an image to a planet + description: Got a crazy good photo of a planet? Share it with the world! + operationId: uploadImage + parameters: + - $ref: '#/components/parameters/planetId' + requestBody: + description: Image to upload + content: + multipart/form-data: + schema: + type: object + properties: + image: + type: string + format: binary + description: The image file to upload + examples: + - '@mars.jpg' + - '@jupiter.png' + responses: + '200': + $ref: '#/components/responses/ImageUploaded' + '400': + $ref: '#/components/responses/BadRequest' + '403': + $ref: '#/components/responses/Forbidden' + '404': + $ref: '#/components/responses/NotFound' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response has message and imageUrl", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("message"); + pm.expect(jsonData).to.have.property("imageUrl"); + }); + /celestial-bodies: + post: + tags: + - Celestial Bodies + summary: Create a celestial body + description: Stars, moons, comets, the occasional rogue asteroid — if it + glows or drifts through the void, you can add it here. + operationId: createCelestialBody + requestBody: + description: Celestial body to create + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + responses: + '201': + description: Celestial body created + content: + application/json: + schema: + $ref: '#/components/schemas/CelestialBody' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a celestial body with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /user/signup: + post: + tags: + - Authentication + summary: Create a user + description: Time to create a user account, eh? + operationId: createUser + security: [] + requestBody: + description: User to create + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + application/xml: + schema: + allOf: + - $ref: '#/components/schemas/User' + - $ref: '#/components/schemas/Credentials' + examples: + Marc: + value: + name: Marc + email: marc@scalar.com + password: i-love-scalar + Cam: + value: + name: Cam + email: cam@scalar.com + password: scalar-is-cool + Hans: + value: + name: Hans + email: hans@scalar.com + password: 5c4l4r + responses: + '201': + description: User account created successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + Location: + $ref: '#/components/headers/Location' + X-Processing-Time: + $ref: '#/components/headers/X-Processing-Time' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '409': + $ref: '#/components/responses/Conflict' + '422': + $ref: '#/components/responses/UnprocessableEntity' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); + /auth/token: + post: + tags: + - Authentication + summary: Get a token + description: + Yeah, this is the boring security stuff. Just get your super secret + token and move on. + operationId: getToken + security: [] + requestBody: + description: Body for credentials to authenticate a user + content: + application/json: + schema: + $ref: '#/components/schemas/Credentials' + application/xml: + schema: + $ref: '#/components/schemas/Credentials' + responses: + '201': + description: Token Created + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + content: + application/json: + schema: + $ref: '#/components/schemas/Token' + application/xml: + schema: + $ref: '#/components/schemas/Token' + '400': + $ref: '#/components/responses/BadRequest' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + '429': + $ref: '#/components/responses/TooManyRequests' + x-post-response: | + pm.test("Status code is 201", () => { + pm.response.to.have.status(201); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response contains token", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("token"); + pm.expect(jsonData.token).to.be.an("string"); + }); + /me: + get: + tags: + - Authentication + summary: Get authenticated user + description: Find yourself they say. That's what you can do here. + operationId: getMe + security: + - basicAuth: [] + - oAuth2: + - read:account + - bearerAuth: [] + - apiKeyHeader: [] + - apiKeyQuery: [] + - apiKeyHeader: [] + apiKeyQuery: [] + responses: + '200': + description: Authenticated user information retrieved successfully + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + ETag: + $ref: '#/components/headers/ETag' + Last-Modified: + $ref: '#/components/headers/Last-Modified' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/User' + application/xml: + schema: + $ref: '#/components/schemas/User' + '401': + $ref: '#/components/responses/Unauthorized' + '403': + $ref: '#/components/responses/Forbidden' + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); + + pm.test("Content-Type header is present", () => { + pm.response.to.have.header("Content-Type"); + }); + + pm.test("Response is a user with id and name", () => { + const jsonData = pm.response.json(); + pm.expect(jsonData).to.have.property("id"); + pm.expect(jsonData).to.have.property("name"); + }); +webhooks: + newPlanet: + post: + tags: + - Planets + security: [] + requestBody: + description: Information about a new planet + content: + application/json: + schema: + $ref: '#/components/schemas/Planet' + responses: + '200': + description: + Return a 200 status to indicate that the data was received + successfully + x-post-response: | + pm.test("Status code is 200", () => { + pm.response.to.have.status(200); + }); +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: JWT Bearer token authentication + basicAuth: + type: http + scheme: basic + description: Basic HTTP authentication + apiKeyHeader: + type: apiKey + in: header + name: X-API-Key + description: API key request header + apiKeyQuery: + type: apiKey + in: query + name: api_key + description: API key query parameter + apiKeyCookie: + type: apiKey + in: cookie + name: api_key + description: API key browser cookie + oAuth2: + type: oauth2 + description: OAuth 2.0 authentication + flows: + authorizationCode: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + clientCredentials: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + implicit: + authorizationUrl: https://galaxy.scalar.com/oauth/authorize + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + # Legacy + password: + tokenUrl: https://galaxy.scalar.com/oauth/token + scopes: + read:account: read your account information + write:planets: modify planets in your account + read:planets: read your planets + openIdConnect: + type: openIdConnect + openIdConnectUrl: https://galaxy.scalar.com/.well-known/openid-configuration + description: OpenID Connect Authentication + parameters: + planetId: + name: planetId + description: The ID of the planet to get + in: path + required: true + schema: + type: integer + format: int64 + examples: + - 1 + limit: + name: limit + description: The number of items to return + in: query + required: false + schema: + type: integer + format: int64 + default: 10 + offset: + name: offset + description: The number of items to skip before starting to collect the result set + in: query + required: false + schema: + type: integer + format: int64 + default: 0 + headers: + X-RateLimit-Limit: + description: The number of allowed requests in the current period + schema: + type: integer + example: 100 + X-RateLimit-Remaining: + description: The number of remaining requests in the current period + schema: + type: integer + example: 95 + X-RateLimit-Reset: + description: The number of seconds left in the current period + schema: + type: integer + example: 3600 + X-Request-ID: + description: Unique identifier for the request + schema: + type: string + format: uuid + example: 123e4567-e89b-12d3-a456-426614174000 + X-Pagination-Total: + description: Total number of items available + schema: + type: integer + example: 1000 + X-Pagination-Page: + description: Current page number + schema: + type: integer + example: 1 + X-Pagination-Per-Page: + description: Number of items per page + schema: + type: integer + example: 10 + X-Cache-Control: + description: Cache control directives + schema: + type: string + example: max-age=3600, public + ETag: + description: Entity tag for caching + schema: + type: string + example: '"33a64df551"' + Last-Modified: + description: The last modification date of the resource + schema: + type: string + format: date-time + example: 2023-12-01T12:00:00Z + Location: + description: The URL of the newly created resource + schema: + type: string + format: uri + example: https://galaxy.scalar.com/planets/123 + X-Processing-Time: + description: The time taken to process the request in milliseconds + schema: + type: integer + example: 150 + responses: + ImageUploaded: + description: Image uploaded + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-Cache-Control: + $ref: '#/components/headers/X-Cache-Control' + content: + application/json: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + application/xml: + schema: + $ref: '#/components/schemas/ImageUploadedMessage' + BadRequest: + description: Bad Request + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + content: + application/json: + schema: + $ref: '#/components/schemas/BadRequestError' + application/xml: + schema: + $ref: '#/components/schemas/BadRequestError' + Forbidden: + description: Forbidden + content: + application/json: + schema: + $ref: '#/components/schemas/ForbiddenError' + application/xml: + schema: + $ref: '#/components/schemas/ForbiddenError' + NotFound: + description: Not Found + content: + application/json: + schema: + $ref: '#/components/schemas/NotFoundError' + application/xml: + schema: + $ref: '#/components/schemas/NotFoundError' + Unauthorized: + description: Unauthorized + content: + application/json: + schema: + $ref: '#/components/schemas/UnauthorizedError' + application/xml: + schema: + $ref: '#/components/schemas/UnauthorizedError' + Conflict: + description: Conflict + content: + application/json: + schema: + $ref: '#/components/schemas/Conflict' + application/xml: + schema: + $ref: '#/components/schemas/Conflict' + UnprocessableEntity: + description: Unprocessable Entity + content: + application/json: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + application/xml: + schema: + $ref: '#/components/schemas/UnprocessableEntity' + TooManyRequests: + description: Too Many Requests + headers: + X-Request-ID: + $ref: '#/components/headers/X-Request-ID' + X-RateLimit-Limit: + $ref: '#/components/headers/X-RateLimit-Limit' + X-RateLimit-Remaining: + $ref: '#/components/headers/X-RateLimit-Remaining' + X-RateLimit-Reset: + $ref: '#/components/headers/X-RateLimit-Reset' + Retry-After: + description: The number of seconds to wait before retrying + schema: + type: integer + example: 60 + content: + application/json: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + application/xml: + schema: + $ref: '#/components/schemas/TooManyRequestsError' + schemas: + User: + description: A user + type: object + xml: + name: user + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Marc + - Cam + - Hans + Credentials: + description: Credentials to authenticate a user + type: object + required: + - email + - password + properties: + email: + type: string + format: email + examples: + - marc@scalar.com + - cam@scalar.com + - hans@scalar.com + password: + type: string + writeOnly: true + examples: + - i-love-scalar + - i-love-oss + - qwerty123 + Token: + description: A token to authenticate a user + type: object + properties: + token: + type: string + examples: + - eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c + CelestialBody: + oneOf: + - $ref: '#/components/schemas/Planet' + - $ref: '#/components/schemas/Satellite' + discriminator: + propertyName: type + mapping: + terrestrial: '#/components/schemas/Planet' + gas_giant: '#/components/schemas/Planet' + ice_giant: '#/components/schemas/Planet' + dwarf: '#/components/schemas/Planet' + super_earth: '#/components/schemas/Planet' + moon: '#/components/schemas/Satellite' + asteroid: '#/components/schemas/Satellite' + comet: '#/components/schemas/Satellite' + description: A celestial body which can be either a planet or a satellite + Planet: + description: A planet in the Scalar Galaxy + type: object + required: + - id + - name + additionalProperties: false + xml: + name: planet + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + x-variable: planetId + name: + type: string + examples: + - Mars + - Jupiter + - HD 40307g + description: + type: + - string + - 'null' + examples: + - The red planet + - A gas giant with a great red spot + type: + type: string + enum: + - terrestrial + - gas_giant + - ice_giant + - dwarf + - super_earth + x-enum-varnames: + - Terrestrial + - GasGiant + - IceGiant + - Dwarf + - SuperEarth + x-enum-descriptions: + terrestrial: Rocky planets with solid surfaces, like Earth and Mars + gas_giant: + Large planets composed mainly of hydrogen and helium, like Jupiter + and Saturn + ice_giant: + Planets composed of water, ammonia, and methane ices, like Uranus and + Neptune + dwarf: Small planetary bodies that don't meet full planet criteria, like Pluto + super_earth: Rocky planets larger than Earth but smaller than gas giants + examples: + - terrestrial + habitabilityIndex: + type: number + format: float + minimum: 0 + maximum: 1 + description: A score from 0 to 1 indicating potential habitability + examples: + - 0.68 + physicalProperties: + type: object + additionalProperties: + x-additionalPropertiesName: measurement + type: number + format: float + description: Additional physical measurements for the planet + properties: + mass: + type: number + format: float + exclusiveMinimum: 0 + description: Mass in Earth masses (must be greater than 0) + examples: + - 0.107 + radius: + type: number + format: float + exclusiveMinimum: 0 + description: Radius in Earth radii (must be greater than 0) + examples: + - 0.532 + gravity: + type: number + format: float + description: Surface gravity in Earth g + examples: + - 0.378 + temperature: + type: object + additionalProperties: + x-additionalPropertiesName: temperatureMetric + type: number + format: float + description: Additional temperature-related measurements in Kelvin + properties: + min: + type: number + format: float + description: Minimum temperature in Kelvin + examples: + - 130 + max: + type: number + format: float + description: Maximum temperature in Kelvin + examples: + - 308 + average: + type: number + format: float + description: Average temperature in Kelvin + examples: + - 210 + atmosphere: + type: array + description: Atmospheric composition + items: + type: object + additionalProperties: + x-additionalPropertiesName: atmosphericData + type: string + description: Additional atmospheric composition data + properties: + compound: + type: string + examples: + - CO2 + - N2 + percentage: + type: number + format: float + exclusiveMaximum: 100 + examples: + - 95.3 + discoveredAt: + type: string + format: date-time + examples: + - 1610-01-07T00:00:00Z + image: + type: + - string + - 'null' + examples: + - https://cdn.scalar.com/photos/mars.jpg + satellites: + type: array + items: + $ref: '#/components/schemas/Satellite' + creator: + $ref: '#/components/schemas/User' + tags: + type: array + items: + type: string + examples: + - - solar-system + - rocky + - explored + lastUpdated: + type: string + format: date-time + readOnly: true + examples: + - 2024-01-15T14:30:00Z + successCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a successful operation + examples: + - https://example.com/webhook + failureCallbackUrl: + type: string + format: uri + description: URL which gets invoked upon a failed operation + examples: + - https://example.com/webhook + Satellite: + description: Every satellite in the Scalar Galaxy + type: object + required: + - name + properties: + id: + type: integer + format: int64 + readOnly: true + examples: + - 1 + name: + type: string + examples: + - Phobos + description: + type: + - string + - 'null' + examples: + - Phobos is the larger and innermost of the two moons of Mars. + diameter: + type: number + format: float + description: Diameter in kilometers + examples: + - 22.2 + type: + type: string + enum: + - moon + - asteroid + - comet + x-enum-varnames: + - Moon + - Asteroid + - Comet + x-enum-descriptions: + moon: Natural satellites that orbit planets + asteroid: Rocky objects that orbit the sun, typically found in the asteroid belt + comet: + Icy bodies that release gas when approaching the sun, creating visible + tails + examples: + - moon + orbit: + type: object + properties: + planet: + $ref: '#/components/schemas/Planet' + orbitalPeriod: + type: number + format: float + description: Orbital period in Earth days + examples: + - 0.319 + distance: + type: number + format: float + description: Average distance from the planet in kilometers + examples: + - 9376 + Paginated: + description: + A generic paginated response. Specializing schemas bind the item type by declaring a + `$dynamicAnchor` named `itemType`, the JSON Schema 2020-12 way to express `Paginated`. + $id: 'https://galaxy.scalar.com/schemas/Paginated' + $defs: + itemType: + $dynamicAnchor: itemType + not: {} + type: object + properties: + data: + type: array + items: + $dynamicRef: '#itemType' + allOf: + - $ref: '#/components/schemas/PaginatedResource' + PaginatedResource: + description: A paginated resource + type: object + properties: + meta: + type: object + properties: + limit: + type: integer + format: int64 + examples: + - 10 + offset: + type: integer + format: int64 + examples: + - 0 + total: + type: integer + format: int64 + examples: + - 100 + next: + type: + - string + - 'null' + examples: + - /planets?limit=10&offset=10 + ImageUploadedMessage: + x-scalar-ignore: true + description: Message about an image upload + type: object + properties: + message: + type: string + examples: + - Image uploaded successfully + imageUrl: + type: string + description: The URL where the uploaded image can be accessed + examples: + - https://cdn.scalar.com/images/8f47c132-9d1f-4f83-b5a4-91db5ee757ab.jpg + uploadedAt: + type: string + format: date-time + description: Timestamp when the image was uploaded + examples: + - 2024-01-15T14:30:00Z + fileSize: + type: integer + description: Size of the uploaded image in bytes + examples: + - 1048576 + mimeType: + type: string + description: The content type of the uploaded image + examples: + - image/jpeg + - image/png + BadRequestError: + x-scalar-ignore: true + description: RFC 7807 (https://datatracker.ietf.org/doc/html/rfc7807) + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/bad-request + title: + type: string + examples: + - Bad Request + status: + type: integer + format: int64 + examples: + - 400 + detail: + type: string + examples: + - The request was invalid. + ForbiddenError: + x-scalar-ignore: true + description: Error response for forbidden access (RFC 7807). Returned when the user does not have permission to access the requested resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/forbidden + title: + type: string + examples: + - Forbidden + status: + type: integer + format: int64 + examples: + - 403 + detail: + type: string + examples: + - You are not authorized to access this resource. + NotFoundError: + x-scalar-ignore: true + description: Error response for resource not found (RFC 7807). Returned when the requested resource does not exist. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Not Found + status: + type: integer + format: int64 + examples: + - 404 + detail: + type: string + examples: + - The resource you are trying to access does not exist. + UnauthorizedError: + x-scalar-ignore: true + description: Error response for unauthorized access (RFC 7807). Returned when authentication is required or has failed. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/not-found + title: + type: string + examples: + - Unauthorized + status: + type: integer + format: int64 + examples: + - 401 + detail: + type: string + examples: + - You are not authorized to access this resource. + Conflict: + x-scalar-ignore: true + description: Error response for resource conflicts (RFC 7807). Returned when the request conflicts with the current state of the resource. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/conflict + title: + type: string + examples: + - Conflict + status: + type: integer + format: int64 + examples: + - 409 + detail: + type: string + examples: + - The resource you are trying to access is in conflict. + UnprocessableEntity: + x-scalar-ignore: true + description: Error response for unprocessable entity (RFC 7807). Returned when the request is well-formed but contains semantic errors. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/unprocessable-entity + title: + type: string + examples: + - Unprocessable Entity + status: + type: integer + format: int64 + examples: + - 422 + detail: + type: string + examples: + - The request was invalid. + TooManyRequestsError: + x-scalar-ignore: true + description: Error response for rate limiting (RFC 7807). Returned when the client has exceeded the rate limit for requests. + type: object + properties: + type: + type: string + examples: + - https://example.com/errors/too-many-requests + title: + type: string + examples: + - Too Many Requests + status: + type: integer + format: int64 + examples: + - 429 + detail: + type: string + examples: + - Rate limit exceeded. Please try again later. diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml new file mode 100644 index 0000000..6e79bf3 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml @@ -0,0 +1,21 @@ +# Hand-authored entity for the precedence-demo API — deliberately NOT auto-discovered from +# precedence-demo-openapi.yaml's own x-examplecorp values. This file demonstrates FR-011: a +# hand-authored catalog-info.yaml takes precedence over auto-sourced metadata for the same API. +# See labs/lab-04-auto-registration/README.md. +apiVersion: backstage.io/v1alpha1 +kind: API +metadata: + name: precedence-demo-api + title: Precedence Demo API + description: > + Hand-authored registration for the precedence-demo API — this entity, not the auto-sourced + candidate from precedence-demo-openapi.yaml's x-examplecorp object, is what appears in the + catalog (FR-011). + annotations: + example.com/visibility: private +spec: + type: openapi + lifecycle: production + owner: group:default/museum-team + definition: + $text: https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/004-lab-4-auto-registration/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml diff --git a/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml new file mode 100644 index 0000000..0650e37 --- /dev/null +++ b/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml @@ -0,0 +1,59 @@ +openapi: 3.1.0 +info: + title: Precedence Demo API + description: > + A minimal API used only to demonstrate FR-011: a hand-authored catalog-info.yaml + (precedence-demo-catalog-info.yaml, alongside this file) takes precedence over the metadata + auto-sourced from this file's own x-examplecorp object. See labs/lab-04-auto-registration/README.md. + version: 1.0.0 + # Deliberately different from the hand-authored catalog-info.yaml's owner/visibility, so the + # precedence check has something real to demonstrate (FR-011). + x-examplecorp: + owner: group:default/platform-team + lifecycle: experimental + visibility: shared +tags: + - name: Demo + description: Precedence demonstration endpoints +paths: + /widgets: + get: + tags: [Demo] + summary: List widgets + operationId: listWidgets + responses: + '200': + description: A list of widgets + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/Widget' + /widgets/{widgetId}: + get: + tags: [Demo] + summary: Get a widget + operationId: getWidget + parameters: + - name: widgetId + in: path + required: true + schema: + type: string + responses: + '200': + description: A single widget + content: + application/json: + schema: + $ref: '#/components/schemas/Widget' +components: + schemas: + Widget: + type: object + properties: + id: + type: string + name: + type: string diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts new file mode 100644 index 0000000..45529f2 --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistration.ts @@ -0,0 +1,749 @@ +// packages/backend/src/extensions/autoApiRegistration.ts +// +// Lab 4: Auto Registration. +// +// A custom EntityProvider that scans a mono-repo for `*-openapi.yaml` / `*-asyncapi.yaml` files +// and turns them into catalog `API` entities with no hand-authored `catalog-info.yaml` required. + +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import fg from 'fast-glob'; +import chokidar from 'chokidar'; +import yaml from 'js-yaml'; +import { + coreServices, + createBackendModule, + resolvePackagePath, +} from '@backstage/backend-plugin-api'; +import { + catalogProcessingExtensionPoint, + catalogServiceRef, +} from '@backstage/plugin-catalog-node'; +import type { + CatalogProcessor, + CatalogProcessorEmit, + CatalogService, + DeferredEntity, + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-node'; +import type { + AuthService, + DatabaseService, + LoggerService, + RootConfigService, +} from '@backstage/backend-plugin-api'; +import { + ANNOTATION_LOCATION, + ANNOTATION_ORIGIN_LOCATION, + parseEntityRef, + stringifyEntityRef, +} from '@backstage/catalog-model'; +import type { Entity } from '@backstage/catalog-model'; +import { InputError } from '@backstage/errors'; +import type { Knex } from 'knex'; +import * as scanStateCacheMigration from './autoApiRegistrationMigrations/001_scan_state_cache'; + +// --------------------------------------------------------------------------------------------- +// Config +// --------------------------------------------------------------------------------------------- + +export const VISIBILITY_ANNOTATION = 'example.com/visibility'; +export const REGISTRATION_ERROR_ANNOTATION = 'apiportal-lab.io/registration-error'; +// Internal-only marker: identifies which provider instance produced an entity, so the +// catalog-info.yaml precedence check can tell "already ours, safe to update" apart from +// "hand-authored, must win" without guessing based on the shape of ANNOTATION_LOCATION values. +const MANAGED_BY_ANNOTATION = 'apiportal-lab.io/managed-by'; + +const KNOWN_VISIBILITIES = new Set(['private', 'shared']); +const DEFAULT_PATTERNS = ['**/*-openapi.yaml', '**/*-asyncapi.yaml']; +const DEFAULT_IGNORE = [ + '**/node_modules/**', + '**/.git/**', + '**/dist/**', + '**/build/**', +]; + +export interface SourceConfig { + id: string; + rootPath: string; + patterns: string[]; + ignore: string[]; + mode: 'poll' | 'watch'; + scheduleFrequencySeconds: number; + reconciliationFrequencySeconds: number; + parseConcurrency: number; + defaultOwner: string; + defaultVisibility: 'private' | 'shared'; + xNamespace: string; +} + +function defaultRootPath(): string { + // packages/backend -> packages -> backstage -> lab-01-base-backstage -> labs + return path.resolve( + resolvePackagePath('backend'), + '../../../../lab-04-auto-registration/apis', + ); +} + +export function normalizeConfig(rootConfig: RootConfigService): SourceConfig[] { + const raw = rootConfig.getOptionalConfig('autoApiRegistration'); + if (!raw) { + return []; + } + + const entries = raw.has('sources') + ? raw.getConfigArray('sources') + : [raw]; + + return entries.map(entry => { + const id = raw.has('sources') ? entry.getString('id') : 'default'; + const defaultOwner = entry.getOptionalString('defaultOwner'); + const xNamespace = entry.getOptionalString('xNamespace'); + if (!defaultOwner) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "defaultOwner"`, + ); + } + if (!xNamespace) { + throw new Error( + `autoApiRegistration source "${id}" is missing required "xNamespace"`, + ); + } + const defaultVisibility = entry.getOptionalString('defaultVisibility') ?? 'private'; + if (!KNOWN_VISIBILITIES.has(defaultVisibility)) { + throw new Error( + `autoApiRegistration source "${id}" has an invalid "defaultVisibility": ${defaultVisibility}`, + ); + } + + return { + id, + rootPath: entry.getOptionalString('rootPath') ?? defaultRootPath(), + patterns: entry.getOptionalStringArray('patterns') ?? DEFAULT_PATTERNS, + ignore: entry.getOptionalStringArray('ignore') ?? DEFAULT_IGNORE, + mode: (entry.getOptionalString('mode') as 'poll' | 'watch') ?? 'poll', + scheduleFrequencySeconds: + entry.getOptionalNumber('schedule.frequencySeconds') ?? 30, + reconciliationFrequencySeconds: + entry.getOptionalNumber('reconciliation.frequencySeconds') ?? 900, + parseConcurrency: entry.getOptionalNumber('parseConcurrency') ?? 4, + defaultOwner, + defaultVisibility: defaultVisibility as 'private' | 'shared', + xNamespace, + }; + }); +} + +// --------------------------------------------------------------------------------------------- +// Discovery + parsing +// --------------------------------------------------------------------------------------------- + +async function* discoverFiles(source: SourceConfig): AsyncGenerator { + const stream = fg.stream(source.patterns, { + cwd: source.rootPath, + ignore: source.ignore, + absolute: true, + onlyFiles: true, + }); + for await (const entry of stream) { + yield entry.toString(); + } +} + +interface ParsedSpec { + kind: 'openapi' | 'asyncapi'; + title: string; + description?: string; + tags: string[]; + raw: Record; +} + +/** Valid input: parses as YAML *and* has an `openapi`/`asyncapi` field *and* `info.title`. */ +function parseSpecFile(contents: string): ParsedSpec | undefined { + let doc: unknown; + try { + doc = yaml.load(contents); + } catch { + return undefined; + } + if (!doc || typeof doc !== 'object') { + return undefined; + } + const obj = doc as Record; + const kind = obj.openapi ? 'openapi' : obj.asyncapi ? 'asyncapi' : undefined; + if (!kind) { + return undefined; + } + const info = obj.info as Record | undefined; + const title = info?.title; + if (typeof title !== 'string' || title.length === 0) { + return undefined; + } + // Backstage's metadata.tags format requires kebab-case slugs — native OpenAPI/AsyncAPI tag + // names (e.g. "Celestial Bodies") are free text, so they're slugified here rather than passed + // through verbatim. + const tags = Array.isArray(obj.tags) + ? (obj.tags as Array>) + .map(t => t?.name) + .filter((t): t is string => typeof t === 'string') + .map(slugify) + .filter(t => t.length > 0) + : []; + + return { + kind, + title, + description: typeof info?.description === 'string' ? (info.description as string) : undefined, + tags, + raw: obj, + }; +} + +export function slugify(title: string): string { + return title + .toLowerCase() + .trim() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 63); +} + +// --------------------------------------------------------------------------------------------- +// Candidate mapping +// --------------------------------------------------------------------------------------------- + +interface MappingResult { + entity: Entity; + entityName: string; + error?: string; +} + +function xField(raw: Record, namespace: string, field: string): string | undefined { + const info = raw.info as Record | undefined; + const ns = info?.[`x-${namespace}`] as Record | undefined; + const value = ns?.[field]; + return typeof value === 'string' ? value : undefined; +} + +function resolveOwnerRef(rawOwner: string | undefined, defaultOwner: string): string { + const value = rawOwner ?? defaultOwner; + return stringifyEntityRef( + parseEntityRef(value, { defaultKind: 'group', defaultNamespace: 'default' }), + ); +} + +function buildEntity(opts: { + providerName: string; + filePath: string; + contents: string; + parsed: ParsedSpec; + entityName: string; + owner: string; + lifecycle: string; + visibility: string; + registrationError?: string; +}): Entity { + const annotations: Record = { + // Identifies the source file for debugging and the catalog-info.yaml precedence check — not + // used as a live, re-readable `$text` target: Backstage's default UrlReader stack has no + // `file:`-scheme reader, so spec.definition is populated inline below instead (the provider + // already holds the bytes; no fetch is needed). + [ANNOTATION_LOCATION]: `file:${opts.filePath}`, + [ANNOTATION_ORIGIN_LOCATION]: `file:${opts.filePath}`, + [VISIBILITY_ANNOTATION]: opts.visibility, + [MANAGED_BY_ANNOTATION]: opts.providerName, + }; + if (opts.registrationError) { + annotations[REGISTRATION_ERROR_ANNOTATION] = opts.registrationError; + } + + return { + apiVersion: 'backstage.io/v1alpha1', + kind: 'API', + metadata: { + name: opts.entityName, + title: opts.parsed.title, + description: opts.parsed.description, + tags: opts.parsed.tags, + annotations, + }, + spec: { + type: opts.parsed.kind, + lifecycle: opts.lifecycle, + owner: opts.owner, + definition: opts.contents, + }, + }; +} + +/** + * Maps a parsed spec file to a candidate catalog entity, applying x-* extraction, owner/visibility + * validation, and collision detection. Does not perform the catalog-info.yaml precedence check + * (that requires a catalog query and is done by the caller, once, per file). + */ +function mapCandidate(opts: { + providerName: string; + source: SourceConfig; + filePath: string; + contents: string; + parsed: ParsedSpec; + knownOwnerRefs: Set; + collidesWith: string | undefined; // file_path of an existing different-name-slug claimant, if any +}): MappingResult { + const { providerName, source, filePath, contents, parsed, knownOwnerRefs, collidesWith } = opts; + const baseSlug = slugify(parsed.title); + + const rawOwner = xField(parsed.raw, source.xNamespace, 'owner'); + const owner = resolveOwnerRef(rawOwner, source.defaultOwner); + const lifecycle = xField(parsed.raw, source.xNamespace, 'lifecycle') ?? 'experimental'; + const rawVisibility = xField(parsed.raw, source.xNamespace, 'visibility'); + const visibility = rawVisibility ?? source.defaultVisibility; + + // Rule: an unrecognized-but-present visibility value is a marker/error entity, never silently + // defaulted. + if (rawVisibility !== undefined && !KNOWN_VISIBILITIES.has(rawVisibility)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility: source.defaultVisibility, + registrationError: `Unrecognized x-${source.xNamespace}.visibility value: "${rawVisibility}" (expected "private" or "shared")`, + }), + error: 'invalid-visibility', + }; + } + + // Rule: an owner that doesn't resolve to a known User/Group entity is a marker/error entity + // instead of blocking the whole cycle. + if (!knownOwnerRefs.has(owner)) { + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner: source.defaultOwner, + lifecycle, + visibility, + registrationError: `Owner "${owner}" does not resolve to a known User or Group entity`, + }), + error: 'invalid-owner', + }; + } + + // Rule: a name collision with another file (in this source or another) is a suffixed + // marker/error entity — first-by-path wins, this one loses. + if (collidesWith) { + const collisionName = `${baseSlug}-collision`; + return { + entityName: collisionName, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: collisionName, + owner, + lifecycle, + visibility, + registrationError: `Entity name "${baseSlug}" collides with an existing entity from ${collidesWith}`, + }), + error: 'name-collision', + }; + } + + return { + entityName: baseSlug, + entity: buildEntity({ + providerName, + filePath, + contents, + parsed, + entityName: baseSlug, + owner, + lifecycle, + visibility, + }), + }; +} + +// --------------------------------------------------------------------------------------------- +// Scan-state cache +// --------------------------------------------------------------------------------------------- + +interface CacheRow { + source_id: string; + file_path: string; + mtime_ms: number; + content_hash: string; + entity_name: string; + last_error: string | null; +} + +class ScanStateCache { + constructor(private readonly knex: Knex) {} + + static async create(database: DatabaseService): Promise { + const knex = await database.getClient(); + // A dedicated migration-bookkeeping table: `coreServices.database` for a module registered + // against the 'catalog' plugin shares that plugin's own database, so using knex's default + // `knex_migrations` table here would validate our one-migration `migrationSource` against the + // real catalog plugin's own (much longer) applied-migrations history and fail with + // "migration directory is corrupt". + await knex.migrate.latest({ + tableName: 'auto_api_registration_migrations', + migrationSource: { + async getMigrations() { + return ['001_scan_state_cache']; + }, + getMigrationName(migration: string) { + return migration; + }, + async getMigration() { + return scanStateCacheMigration; + }, + }, + }); + return new ScanStateCache(knex); + } + + async rowsForSource(sourceId: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ source_id: sourceId }); + } + + /** Global lookup (no source_id filter) — collision/precedence checks span every source. */ + async rowsByEntityName(entityName: string): Promise { + return this.knex(scanStateCacheMigration.TABLE_NAME).where({ entity_name: entityName }); + } + + async upsert(row: CacheRow): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .insert(row) + .onConflict(['source_id', 'file_path']) + .merge(); + } + + async remove(sourceId: string, filePath: string): Promise { + await this.knex(scanStateCacheMigration.TABLE_NAME) + .where({ source_id: sourceId, file_path: filePath }) + .delete(); + } +} + +function sha1(contents: string): string { + return crypto.createHash('sha1').update(contents).digest('hex'); +} + +// --------------------------------------------------------------------------------------------- +// EntityProvider (one instance per source) +// --------------------------------------------------------------------------------------------- + +async function mapLimit( + items: T[], + limit: number, + fn: (item: T) => Promise, +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + async function worker() { + while (next < items.length) { + const index = next++; + results[index] = await fn(items[index]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker)); + return results; +} + +class AutoApiRegistrationEntityProvider implements EntityProvider { + private connection?: EntityProviderConnection; + private hasRunOnce = false; + + constructor( + private readonly source: SourceConfig, + private readonly cache: ScanStateCache, + private readonly catalogApi: CatalogService, + private readonly auth: AuthService, + private readonly logger: LoggerService, + ) {} + + getProviderName(): string { + return `auto-api-registration:${this.source.id}`; + } + + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + async runCycle(): Promise { + if (!this.connection) { + this.logger.warn( + `auto-api-registration:${this.source.id}: skipped a cycle — provider not yet connected`, + ); + return; + } + + const existingRows = await this.cache.rowsForSource(this.source.id); + const existingByPath = new Map(existingRows.map(row => [row.file_path, row])); + + const currentPaths = new Set(); + for await (const filePath of discoverFiles(this.source)) { + currentPaths.add(filePath); + } + + const knownOwnerRefs = await this.fetchKnownOwnerRefs(); + + const changedPaths = [...currentPaths].filter(filePath => { + const cached = existingByPath.get(filePath); + if (!cached) return true; + const stat = fs.statSync(filePath); + return stat.mtimeMs !== cached.mtime_ms; + }); + + const added: DeferredEntity[] = []; + const removedRefs: string[] = []; + const upserts: CacheRow[] = []; + const deletes: string[] = []; + + await mapLimit(changedPaths, this.source.parseConcurrency, async filePath => { + const cached = existingByPath.get(filePath); + const stat = fs.statSync(filePath); + const contents = fs.readFileSync(filePath, 'utf8'); + const hash = sha1(contents); + + if (cached && cached.content_hash === hash) { + // mtime touched, content unchanged — refresh mtime only, no mutation needed. + upserts.push({ ...cached, mtime_ms: stat.mtimeMs }); + return; + } + + const parsed = parseSpecFile(contents); + if (!parsed) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} is not a valid OpenAPI/AsyncAPI file (malformed YAML, or missing openapi/asyncapi + info.title) — skipped`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + const slug = slugify(parsed.title); + + // Precedence check: a hand-authored catalog-info.yaml for this name wins. + const precedenceWinner = await this.findNonAutoSourcedEntity(slug); + if (precedenceWinner) { + this.logger.info( + `auto-api-registration:${this.source.id}: ${filePath} maps to "${slug}", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration`, + ); + if (cached) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: cached.entity_name })); + deletes.push(filePath); + } + return; + } + + // Global collision check: excludes this file's own prior row. + const collisionRows = (await this.cache.rowsByEntityName(slug)).filter( + row => !(row.source_id === this.source.id && row.file_path === filePath), + ); + const collidesWith = collisionRows.length > 0 ? collisionRows[0].file_path : undefined; + + const mapped = mapCandidate({ + providerName: this.getProviderName(), + source: this.source, + filePath, + contents, + parsed, + knownOwnerRefs, + collidesWith, + }); + + if (mapped.error) { + this.logger.error( + `auto-api-registration:${this.source.id}: ${filePath} registered as an error entity "${mapped.entityName}" — ${mapped.entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]}`, + ); + } + + added.push({ entity: mapped.entity }); + upserts.push({ + source_id: this.source.id, + file_path: filePath, + mtime_ms: stat.mtimeMs, + content_hash: hash, + entity_name: mapped.entityName, + last_error: mapped.error ?? null, + }); + }); + + for (const [filePath, row] of existingByPath) { + if (!currentPaths.has(filePath)) { + removedRefs.push(stringifyEntityRef({ kind: 'API', namespace: 'default', name: row.entity_name })); + deletes.push(filePath); + } + } + + if (!this.hasRunOnce) { + // First run: establish the baseline with one `full` mutation. + await this.connection.applyMutation({ type: 'full', entities: added }); + this.hasRunOnce = true; + } else if (added.length > 0 || removedRefs.length > 0) { + await this.connection.applyMutation({ + type: 'delta', + added, + removed: removedRefs.map(entityRef => ({ entityRef })), + }); + } + + for (const row of upserts) { + await this.cache.upsert(row); + } + for (const filePath of deletes) { + await this.cache.remove(this.source.id, filePath); + } + } + + private async fetchKnownOwnerRefs(): Promise> { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: ['User', 'Group'] }, fields: ['kind', 'metadata.name', 'metadata.namespace'] }, + { credentials }, + ); + return new Set(response.items.map(entity => stringifyEntityRef(entity))); + } + + private async findNonAutoSourcedEntity(entityName: string): Promise { + const credentials = await this.auth.getOwnServiceCredentials(); + const response = await this.catalogApi.getEntities( + { filter: { kind: 'API', 'metadata.name': entityName } }, + { credentials }, + ); + return response.items.find(entity => !entity.metadata.annotations?.[MANAGED_BY_ANNOTATION]); + } +} + +// --------------------------------------------------------------------------------------------- +// CatalogProcessor: surfaces registration errors via Backstage's native processing-error UI +// --------------------------------------------------------------------------------------------- + +class AutoApiRegistrationErrorProcessor implements CatalogProcessor { + getProcessorName(): string { + return 'auto-api-registration-error-processor'; + } + + async preProcessEntity( + entity: Entity, + _location: unknown, + _emit: CatalogProcessorEmit, + ): Promise { + const message = entity.metadata.annotations?.[REGISTRATION_ERROR_ANNOTATION]; + if (message) { + throw new InputError(message); + } + return entity; + } +} + +// --------------------------------------------------------------------------------------------- +// Backend module registration (following packages/backend/src/extensions/permissionPolicy.ts) +// --------------------------------------------------------------------------------------------- + +export default createBackendModule({ + pluginId: 'catalog', + moduleId: 'auto-api-registration', + register(reg) { + reg.registerInit({ + deps: { + catalog: catalogProcessingExtensionPoint, + database: coreServices.database, + scheduler: coreServices.scheduler, + logger: coreServices.logger, + rootConfig: coreServices.rootConfig, + catalogApi: catalogServiceRef, + auth: coreServices.auth, + }, + async init({ catalog, database, scheduler, logger, rootConfig, catalogApi, auth }) { + const sources = normalizeConfig(rootConfig); + if (sources.length === 0) { + logger.info('auto-api-registration: no autoApiRegistration config found — module inactive'); + return; + } + + const cache = await ScanStateCache.create(database); + + const providers = sources.map( + source => + new AutoApiRegistrationEntityProvider(source, cache, catalogApi, auth, logger), + ); + + catalog.addEntityProvider(providers); + catalog.addProcessor(new AutoApiRegistrationErrorProcessor()); + + for (const [index, source] of sources.entries()) { + const provider = providers[index]; + + if (source.mode === 'poll') { + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:poll`, + frequency: { seconds: source.scheduleFrequencySeconds }, + timeout: { seconds: Math.max(source.scheduleFrequencySeconds * 2, 30) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: cycle failed`, error as Error); + } + }, + }); + } else { + // Steady-state discovery signal at real-world scale: a watcher triggers a rescan of + // just this source on add/change/unlink, coalesced by chokidar's own event batching. + // `schedule.frequencySeconds` is ignored in watch mode in favor of the much longer + // `reconciliation.frequencySeconds` safety-net sweep below. + await scheduler.scheduleTask({ + id: `auto-api-registration:${source.id}:reconciliation`, + frequency: { seconds: source.reconciliationFrequencySeconds }, + timeout: { seconds: Math.max(source.reconciliationFrequencySeconds / 2, 60) }, + fn: async () => { + try { + await provider.runCycle(); + } catch (error) { + logger.error(`auto-api-registration:${source.id}: reconciliation sweep failed`, error as Error); + } + }, + }); + + const watcher = chokidar.watch(source.patterns, { + cwd: source.rootPath, + ignored: source.ignore, + ignoreInitial: true, + }); + let debounce: NodeJS.Timeout | undefined; + const triggerRescan = () => { + clearTimeout(debounce); + debounce = setTimeout(() => { + provider + .runCycle() + .catch(error => logger.error(`auto-api-registration:${source.id}: watch-triggered cycle failed`, error as Error)); + }, 500); + }; + watcher.on('add', triggerRescan); + watcher.on('change', triggerRescan); + watcher.on('unlink', triggerRescan); + } + } + }, + }); + }, +}); diff --git a/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts new file mode 100644 index 0000000..78f6a7b --- /dev/null +++ b/labs/lab-04-auto-registration/code/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts @@ -0,0 +1,32 @@ +// packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts +// +// Scan-state cache for the autoApiRegistration EntityProvider. Not a catalog table — internal +// state private to this backend module, used to avoid re-parsing unchanged files on restart, to +// compute delta mutations, and to support collision/precedence checks across every configured +// source. +// +// Applied via a knex custom `migrationSource` (see autoApiRegistration.ts) rather than a +// filesystem `directory` migration source, since this file lives alongside the module's +// TypeScript source rather than in a package-level `migrations/` folder resolved at runtime. + +import type { Knex } from 'knex'; + +export const TABLE_NAME = 'auto_api_registration_scan_state'; + +export async function up(knex: Knex): Promise { + await knex.schema.createTable(TABLE_NAME, table => { + table.string('source_id').notNullable(); + table.string('file_path').notNullable(); + table.bigInteger('mtime_ms').notNullable(); + table.string('content_hash').notNullable(); + table.string('entity_name').notNullable(); + table.text('last_error').nullable(); + + table.primary(['source_id', 'file_path']); + table.index(['entity_name'], 'auto_api_registration_scan_state_entity_name_idx'); + }); +} + +export async function down(knex: Knex): Promise { + await knex.schema.dropTable(TABLE_NAME); +} diff --git a/scripts/check-readme.sh b/scripts/check-readme.sh new file mode 100755 index 0000000..dac1ec1 --- /dev/null +++ b/scripts/check-readme.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Fails if root README.md hasn't been updated to reference every labs/ and specs/ directory. +# Run manually with `bash scripts/check-readme.sh`; also enforced in CI on every push/PR. +set -euo pipefail +cd "$(dirname "$0")/.." + +fail=0 + +for dir in labs/lab-*/; do + name=$(basename "$dir") + if ! grep -q "labs/${name}/" README.md; then + echo "README.md does not link to labs/${name}/ — add it to the Lab Series table and Repository Structure section." + fail=1 + fi +done + +for dir in specs/*/; do + name=$(basename "$dir") + if ! grep -q "$name" README.md; then + echo "README.md does not mention specs/${name}/ — add it to the Repository Structure section." + fail=1 + fi +done + +if [ "$fail" -ne 0 ]; then + echo + echo "Root README.md is out of sync with labs/ and specs/. Update it, then re-run this check." + exit 1 +fi + +echo "README.md references every labs/ and specs/ directory." diff --git a/specs/004-lab-4-auto-registration/checklists/issues.md b/specs/004-lab-4-auto-registration/checklists/issues.md new file mode 100644 index 0000000..d6cfebe --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/issues.md @@ -0,0 +1,33 @@ +## Run 1 - 2026/07/03 + +- [X] No APIs were available in the catalog after starting backstage. Relevant log entries are included below. + + **Resolved** (specs/004-lab-4-auto-registration/tasks.md T038–T040): root cause was six + `catalog.locations` URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` still + pinned to the stale, merged-and-gone `003-api-quality` branch. `teams.yaml` (defining + `platform-team`) 404'd as a result, so the Lab 4 owner validator correctly rejected + `galaxy-openapi.yaml`'s `owner: group:default/platform-team` — the actual cause of the empty + catalog. Fixed by repointing those six URLs at `main`, where the content is already merged. + Verified via a clean `yarn start`: museum/streetlights/train-travel/scalar-galaxy/ + precedence-demo-api all register with no `AutoApiRegistrationErrorProcessor` errors and no + "Unable to read url" warnings. + +``` +2026-07-03T12:38:51.027Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:00.820Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml entity="location:default/generated-4c23d1d635d435e80cc8b698fc86956f9fffff3c" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/streetlights-api.yaml" +2026-07-03T12:39:00.895Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml entity="location:default/generated-2cde689609df331e9267009ea6c02850d6b1603e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-03-api-quality/catalog/platform-user.yaml" +2026-07-03T12:39:00.896Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml entity="location:default/generated-cc3b19c3e787fc46d27f583f1d53dd2e8ec1a456" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/teams.yaml" +2026-07-03T12:39:00.899Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml entity="location:default/generated-ae8dd75945a327a0201813834829ec035913de8e" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/users.yaml" +2026-07-03T12:39:00.907Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml entity="location:default/generated-1390492c77b8f266a9f6608c0595f96efa060611" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/museum-api.yaml" +2026-07-03T12:39:00.915Z catalog warn Unable to read url, no matching files found for https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml entity="location:default/generated-9ecb0d8da3f46f025b2e5e23de21f385b1a49388" location="url:https://raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/labs/lab-02-users-roles/catalog/apis/train-travel-api.yaml" +2026-07-03T12:39:07.085Z catalog warn Processor AutoApiRegistrationErrorProcessor threw an error while preprocessing; caused by InputError: Owner "group:default/platform-team" does not resolve to a known User or Group entity entity="api:default/scalar-galaxy" location="file:/Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml" +2026-07-03T12:39:21.056Z rootHttpRouter info [2026-07-03T12:39:21.056Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.056Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.060Z rootHttpRouter info [2026-07-03T12:39:21.060Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:21.060Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:21.061Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:39:51.087Z rootHttpRouter info [2026-07-03T12:39:51.087Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.087Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.092Z rootHttpRouter info [2026-07-03T12:39:51.092Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:39:51.092Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:39:51.093Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +2026-07-03T12:40:21.123Z rootHttpRouter info [2026-07-03T12:40:21.123Z] "GET /api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.123Z" method="GET" url="/api/catalog/entities?fields=kind,metadata.name,metadata.namespace&filter=kind%3DUser%2Ckind%3DGroup" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.127Z rootHttpRouter info [2026-07-03T12:40:21.127Z] "GET /api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api HTTP/1.1" 200 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type="incomingRequest" date="2026-07-03T12:40:21.127Z" method="GET" url="/api/catalog/entities?filter=kind%3DAPI%2Cmetadata.name%3Dprecedence-demo-api" status=200 httpVersion="1.1" userAgent="node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" +2026-07-03T12:40:21.128Z catalog info auto-api-registration:default: /Users/matt/Code/DawMatt/backstage-apiportal-lab/labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml maps to "precedence-demo-api", which already has a hand-authored catalog-info.yaml — skipping auto-sourced registration +``` \ No newline at end of file diff --git a/specs/004-lab-4-auto-registration/checklists/requirements.md b/specs/004-lab-4-auto-registration/checklists/requirements.md new file mode 100644 index 0000000..35d33a7 --- /dev/null +++ b/specs/004-lab-4-auto-registration/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Lab 4 - Auto Registration + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-02 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Items marked incomplete require spec updates before `/speckit-clarify` or `/speckit-plan`. +- No [NEEDS CLARIFICATION] markers were needed: reasonable, documented defaults were used for + mono-repo simulation, discovery cadence, and catalog-info.yaml vs. x-* precedence — all + recorded in the spec's Assumptions section per Constitution Principle VIII. diff --git a/specs/004-lab-4-auto-registration/data-model.md b/specs/004-lab-4-auto-registration/data-model.md new file mode 100644 index 0000000..6070187 --- /dev/null +++ b/specs/004-lab-4-auto-registration/data-model.md @@ -0,0 +1,137 @@ +# Phase 1 Data Model: Lab 4 — Auto Registration + +This lab has no application database. "Data model" here means the shape of the catalog entities +produced, the config schema that drives discovery, and the mapping rules between an API +definition file and the resulting Backstage `API` entity. + +## Entity: API Definition File (input, not a catalog entity) + +A file on disk matched by the discovery glob (`**/*-openapi.yaml`, `**/*-asyncapi.yaml`). + +| Field | Type | Notes | +|---|---|---| +| `openapi` / `asyncapi` | string | Version field; presence of either identifies the spec type. Exactly one is expected. | +| `info.title` | string | **Required.** Source of the catalog entity name (slugified) and `metadata.title`. Not duplicated into `x-examplecorp`. | +| `info.description` | string | Optional. Maps to `metadata.description` if present. Not duplicated into `x-examplecorp`. | +| `tags[].name` | string[] | Optional, native top-level OpenAPI/AsyncAPI field (not under `info`, not under `x-examplecorp`). Default `[]` if absent. | +| `info.x-examplecorp.owner` | string | Optional. Owner entity ref (kind defaults to `group:default/` if unprefixed). Absent → registration still succeeds using a documented default — see Mapping Rule 1 below. | +| `info.x-examplecorp.lifecycle` | string | Optional. One of `experimental` \| `production` \| `deprecated` (Backstage's standard lifecycle values). Default: `experimental`. | +| `info.x-examplecorp.visibility` | string | Optional. One of `private` \| `shared` — the same two values Lab 2 established. Absent → falls back to that file's source's configured `defaultVisibility` (source-level default is `'private'` if the source doesn't set one — research.md R3/R7). Present-but-unrecognized → marker/error entity, not silently defaulted (research.md R4). | + +## Entity: Catalog API Entity (output) + +Standard Backstage `API` kind entity, produced by the EntityProvider's full mutation. + +| Field | Source | Mapping Rule | +|---|---|---| +| `metadata.name` | `info.title` | Slugified (lowercase, spaces/punctuation → `-`, per Backstage entity-name character rules). This is the identity used for update-in-place (FR-003) and collision detection. | +| `metadata.title` | `info.title` | Verbatim. | +| `metadata.description` | `info.description` | Verbatim if present, else omitted. | +| `metadata.tags` | `tags[].name` | Native top-level spec field, not `x-examplecorp`. Default `[]` if absent. **Confirmed during implementation**: Backstage's `metadata.tags` format requires kebab-case slugs (`[a-z0-9+#]` separated by `-`); native spec tag names (e.g. `"Celestial Bodies"`) are free text, so each is slugified with the same slug function used for `metadata.name` before being written — otherwise the entity fails catalog policy validation and is silently dropped during processing (never reaches the catalog, no user-visible error). | +| `metadata.annotations['backstage.io/managed-by-location']` | discovery | Set to the source file's resolved path/URL, identifying this entity as auto-sourced (used for the FR-011 precedence check and for the "which file caused this" debugging path in SC-003). | +| `metadata.annotations['example.com/visibility']` | `info.x-examplecorp.visibility` | **The exact same annotation key the Lab 2/3 permission policy already reads** (`packages/backend/src/extensions/permissionPolicy.ts`) — no new annotation, no policy change. **Mapping Rule 2**: if absent, falls back to that file's source's configured `defaultVisibility` — itself optional per-source, falling back to the fixed `'private'` "default default" if the source doesn't set one either (research.md R3/R7, FR-006a). An unrecognized (non-`private`/`shared`) value is never silently defaulted — it's a marker/error entity instead (research.md R4). Reused verbatim rather than reinvented, per research.md R3. | +| `metadata.annotations['apiportal-lab.io/registration-error']` | discovery | Present **only** on marker/error entities (invalid owner ref, invalid visibility value, or name collision); absent on normally-registered entities. | +| `spec.type` | discovery | `openapi` if the file has a top-level `openapi` field, `asyncapi` if it has a top-level `asyncapi` field. | +| `spec.lifecycle` | `info.x-examplecorp.lifecycle` | Default `experimental` if absent (FR-007). | +| `spec.owner` | `info.x-examplecorp.owner` | **Mapping Rule 1**: if absent, registration still succeeds (FR-007) using that file's source's configured `defaultOwner` (the lab's single source defaults this to `group:default/platform-team`, the same shared/fallback team introduced in Lab 3 — but this is per-source, not global, per research.md R7), rather than blocking. If present but unresolvable against the catalog's known `User`/`Group` entities, the file is registered as a marker/error entity instead (see research.md R4). | +| `spec.definition` | discovery | **Revised during implementation**: `spec.definition` is a plain `string` field (not an object), and Backstage's default `UrlReaderService` stack has no `file:`-scheme reader — a `{ $text: 'file:...' }` placeholder (the originally-planned approach, mirroring Labs 1–3's hand-authored `$text` + `type: url` pattern) fails with `NotAllowedError` for a local filesystem path with no matching `reading.allow` host entry. Since the provider already holds the file's bytes in memory at parse time, `spec.definition` is populated with that raw file content directly — no read-back, no placeholder resolution, no duplication (the file itself isn't a second copy; it's the copy). The `$text`/`type: url` pattern remains correct and unchanged for genuinely hand-authored `catalog-info.yaml` files (e.g. `precedence-demo-catalog-info.yaml`), which reference a *different* file over a real network-reachable URL — this revision only applies to entities this provider constructs programmatically from a file it already read. | + +## Config: `autoApiRegistration` (app-config.yaml) + +The canonical shape is a list of sources (research.md R7), each independently configured. A flat +`autoApiRegistration.{rootPath,patterns,...}` (no `sources` key) is shorthand for a single +implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so a +single-repo setup never has to write the list form. + +```yaml +autoApiRegistration: + sources: + - id: platform-monorepo # required, unique; becomes part of the EntityProvider name + # and the scan-state cache's source_id column + rootPath: ... + patterns: ['**/*-openapi.yaml', '**/*-asyncapi.yaml'] + ignore: ['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**'] + mode: poll # or 'watch' — independent per source (research.md R6/R7) + schedule: { frequencySeconds: 30 } + reconciliation: { frequencySeconds: 900 } # watch mode only + parseConcurrency: 4 + defaultOwner: group:default/platform-team + defaultVisibility: private # optional — falls back to the fixed 'private' default if unset + xNamespace: examplecorp + - id: team-checkout-api # a second source: a different team's repo, a local sibling + rootPath: ../team-checkout-api # checkout for the lab; a real remote repo's + patterns: ['**/*-openapi.yaml'] # synced local worktree in production (R7 + defaultOwner: group:default/checkout-team # Problem 4) + defaultVisibility: shared # e.g. a pre-production repo where every API should default + xNamespace: checkoutteam # to shared visibility unless a spec says otherwise +``` + +| Key (per source, under `sources[]`, or flat for the single-source shorthand) | Type | Default | Purpose | +|---|---|---|---| +| `id` | string | `'default'` (flat-config shorthand only) | Unique source identifier. Required when using the `sources[]` list form. Used as the `EntityProvider` name suffix and the scan-state cache's `source_id` (research.md R7). | +| `rootPath` | string | resolved repo root (see research.md R2) | Root directory the glob scan starts from. Adaptable per FR-010. In production, may point at a locally-synced worktree of a genuinely remote repo (research.md R7, Problem 4) — this schema does not change either way. | +| `patterns` | string[] | `['**/*-openapi.yaml', '**/*-asyncapi.yaml']` | Filename glob patterns. Adaptable per FR-010, per source. | +| `ignore` | string[] | `['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**']` | Glob exclusions, always applied regardless of `mode` (research.md R2/R6). Extendable, not replaceable, by learners. | +| `mode` | `'poll' \| 'watch'` | `'poll'` | `poll`: full `fast-glob` sweep every `schedule.frequencySeconds` (lab default — simple, predictable for 2–3 files). `watch`: `chokidar`-driven incremental discovery with `reconciliation.frequencySeconds` as a safety-net sweep interval (research.md R6 — the real-world-scale mode). Independent per source — e.g. a large mono-repo can run `watch` while a small team repo runs `poll` (research.md R7). | +| `schedule.frequencySeconds` | number | `30` | `poll` mode: full-sweep interval. `watch` mode: ignored in favor of `reconciliation.frequencySeconds`. | +| `reconciliation.frequencySeconds` | number | `900` | `watch` mode only: interval for the periodic full-sweep safety net that catches missed watcher events (research.md R6, Problem 3). | +| `parseConcurrency` | number | `4` | Max files parsed concurrently per cycle/event batch — bounds memory/event-loop impact when many files change at once (research.md R6). | +| `defaultOwner` | string | *(no global default — each source must set its own, or inherit `autoApiRegistration.defaults.defaultOwner` if configured)* | Fallback owner ref applied when `x-.owner` is absent (FR-007). Deliberately per-source, not global — a team/pre-production repo plausibly has a different fallback owner than the platform mono-repo (research.md R7, Problem 2). | +| `defaultVisibility` | `'private' \| 'shared'` | `'private'` (the one fixed, hardcoded fallback in the whole config surface — the "default default") | Fallback visibility applied when `x-.visibility` is absent for a file in this source (research.md R3/R4). Per-source, like `defaultOwner` — e.g. a pre-production repo may reasonably default every undeclared API to `shared`, while the platform mono-repo keeps the conservative `private` fallback. Unlike `defaultOwner`/`xNamespace`, this key is optional even at the source level: a source that omits it gets `private`, so a learner adding a new source doesn't have to think about visibility defaults unless they want to change them. | +| `xNamespace` | string | *(no global default — see `defaultOwner`)* | The vendor namespace under `info.x-` this source reads owner/lifecycle/visibility from. Per-source so a repo that already has its own established `x-*` convention isn't forced to rename it to onboard (research.md R7, Problem 2). The lab's single source sets this to `'examplecorp'`. | + +## Scan-state cache (persisted, `coreServices.database`) + +A table private to this backend module, used to (a) avoid re-parsing unchanged files on restart, +(b) compute delta mutations after the initial full mutation (research.md R6, Problem 1 & 2), and +(c) support collision/precedence checks across every configured source, not just the one currently +scanning (research.md R7, Problem 3). Not a catalog entity, not written back into any source +repo — internal backend state only. + +| Column | Type | Purpose | +|---|---|---| +| `source_id` | string (PK, part 1) | Which configured source (research.md R7) this row came from — `'default'` for the flat single-source config. Lets a collision/error message name *which repo*, not just which file. | +| `file_path` | string (PK, part 2) | Absolute path of the discovered file, within that source. | +| `mtime_ms` | number | Last-observed file modification time; cheap first-pass change check on restart/reconciliation. | +| `content_hash` | string | sha1 of file bytes; only computed/compared when `mtime_ms` differs, since hashing every file on every check is itself costly at 1000+ files. | +| `entity_name` | string, indexed | The catalog entity name this file currently maps to — doubles as the slug→(source, path) collision index (research.md R4), queried **without** filtering by `source_id` so collisions across sources are caught (research.md R7, Problem 3). | +| `last_error` | string \| null | Last registration error for this file, if any (mirrors the `apiportal-lab.io/registration-error` annotation; lets a restart re-emit the same marker entity without re-deriving the error). | + +## State transitions (per discovery cycle) + +Runs independently **per configured source** (research.md R7) — each source is its own +`EntityProvider` instance with its own schedule/mode, so one source's cycle does not block or +depend on another's. Steps (d)–(f) below query the scan-state cache and catalog **without** +filtering by `source_id`, since collision and precedence checks must be global (research.md R7, +Problem 3) even though discovery itself runs per source. + +1. **First run of a given source**: glob scan (streamed, ignore patterns applied) → parse + + shape-check every matched file → build that source's rows in the scan-state cache → emit one + `type: 'full'` mutation (from that source's `EntityProvider` instance) establishing the + baseline with the catalog (research.md R6). +2. **Every subsequent cycle for that source** (`poll` mode: full sweep on + `schedule.frequencySeconds`; `watch` mode: per-event, plus a full sweep on + `reconciliation.frequencySeconds`): + a. Glob scan of that source's `rootPath` (or, in `watch` mode, the single changed path from the + watcher event). + b. For each candidate, `stat()` and compare `mtime_ms` against that source's cache rows; only + files that differ (or are new) proceed to content-hash comparison and re-parse — unchanged + files are skipped entirely (research.md R6, Problem 2). + c. Parse + shape-check changed/new files, using that source's `xNamespace` → valid candidates / + log-only errors. + d. Slugify `info.title` → candidate entity name; check against the cache's `entity_name` index + **across all sources** (O(1) lookup, not a full re-sort) → collisions become marker/error + entities, whether the colliding file is in this source or another one (research.md R7, + Problem 3). + e. Resolve `x-.owner` (falling back to this source's `defaultOwner` if absent) + against known `User`/`Group` entities → unresolvable owners become marker/error entities. + Resolve `x-.visibility` (falling back to this source's `defaultVisibility`, or + `'private'` if the source doesn't set one, if absent) → validate against `{private, shared}` + → an unrecognized (present-but-invalid) value becomes a marker/error entity (research.md R4). + f. Check for a pre-existing, non-auto-sourced entity at the same name anywhere in the catalog + (hand-authored `catalog-info.yaml`) → if found, skip the auto-sourced candidate for this name + (FR-011). + g. Diff the resulting candidate set against this source's cache rows → update cache rows → + emit a `type: 'delta'` mutation (`added`/`removed`) from this source's `EntityProvider` + instance for just what changed (FR-002/003/004), **not** a fresh full mutation (research.md + R6, Problem 1). diff --git a/specs/004-lab-4-auto-registration/plan.md b/specs/004-lab-4-auto-registration/plan.md new file mode 100644 index 0000000..64a5ba8 --- /dev/null +++ b/specs/004-lab-4-auto-registration/plan.md @@ -0,0 +1,182 @@ +# Implementation Plan: Lab 4 — Auto Registration + +**Branch**: `004-lab-4-auto-registration` | **Date**: 2026-07-03 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `specs/004-lab-4-auto-registration/spec.md` + +## Summary + +Lab 4 builds on the running Backstage 1.51.0 instance from Labs 1–3. It replaces the +per-API hand-authored `catalog-info.yaml` pattern used in Labs 1–3 with a custom backend +`EntityProvider` that scans a mono-repo for files matching `*-openapi.yaml` / +`*-asyncapi.yaml`, parses them, and emits/updates/retracts catalog `API` entities on a 30s +poll schedule for the lab's small sample set — no catalog-info.yaml required per API +(FR-001–FR-004). The same mechanism is designed to hold up at real-world scale in two +independent dimensions: **repo size** (1000+ spec files, 1GB+, research.md R6 — a persisted +scan-state cache lets a restart skip re-parsing unchanged files, delta-not-full mutations after +the first cycle, and an optional `chokidar` watch mode replacing repeated full-tree polling as +the steady-state discovery signal) and **repo count** (research.md R7 — config generalizes from +one `rootPath` to a list of independently-configured `sources[]`, each its own `EntityProvider` +instance with its own schedule, default owner, and `x-*` namespace, so a platform mono-repo and +one or more separate team/pre-production repos can each be discovered with their own settings +while collision and `catalog-info.yaml`-precedence checks stay global across all of them). Only +the config *values and entry count* differ between the lab demo (one small source) and a +production-scale, multi-repo deployment, not the architecture. Catalog owner/lifecycle/visibility +metadata is sourced from a single vendor-namespaced +`info.x-examplecorp` object (`owner`, `lifecycle`, `visibility`) — tool-agnostic field names, +grouped under one namespace learners rename to their own company (FR-005–FR-006, FR-006a), with +documented defaults when absent (FR-007). The `visibility` value reuses the exact +`example.com/visibility` annotation and `private`/`shared` values the Lab 2/3 permission policy +already enforces — no new visibility mechanism is introduced, only a new way of setting the +existing annotation. Metadata that already has a natural home in the spec — name (`info.title`), +description (`info.description`), tags (native top-level `tags`) — is sourced from there instead +of being duplicated into `x-examplecorp`. Errors (malformed files, unresolvable owners, invalid +visibility values, name collisions) are surfaced via backend logs and, where an entity can be +identified, via Backstage's native catalog processing-error UI through a companion +`CatalogProcessor` that throws on a marker annotation (FR-008). A hand-authored +`catalog-info.yaml`, where one exists alongside a discovered file, takes precedence over +auto-sourced metadata (FR-011). + +Two new sample files support the lab: a vendored copy of the MIT-licensed Scalar Galaxy API +(`galaxy-openapi.yaml`, `x-examplecorp.visibility: shared`) with no pre-existing catalog +registration, demonstrating genuine "previously unregistered API" discovery (US1) and, alongside +the precedence-demo API's differing visibility, a real private/shared contrast to verify (SC-006); +and a small `precedence-demo-openapi.yaml` + hand-authored `precedence-demo-catalog-info.yaml` +pair demonstrating the precedence rule. Lab documentation calls out which parts (glob patterns, +root path, `x-*` field names) are adaptable conventions per Constitution Principle VIII (FR-010, +US3). + +## Technical Context + +**Language/Version**: TypeScript / Node.js 20 LTS (Backstage 1.51.0, pinned from Lab 1) + +**Primary Dependencies**: +- `fast-glob@3.3.3` — new direct dependency of `packages/backend`; already present transitively + in the workspace, promoted to an explicit dependency for the filesystem scan (research.md R2). +- `js-yaml@4.2.0` — new direct dependency of `packages/backend`; already present transitively, + promoted to explicit for parsing candidate spec files (research.md R3). +- `chokidar` — new direct dependency of `packages/backend`; already present transitively; used + only when `autoApiRegistration.mode: 'watch'` (not the lab default), to give the discovery + mechanism a steady-state path that scales past the lab's poll-based default (research.md R6). +- `@backstage/plugin-catalog-node` (already installed, `2.2.1`) — supplies `EntityProvider`, + `CatalogProcessor`, and `catalogProcessingExtensionPoint` used to register the new backend + module, following the same `createBackendModule` pattern as + `packages/backend/src/extensions/permissionPolicy.ts`. +- No new frontend dependencies — this lab is backend-only; no new UI surfaces are added (errors + ride on Backstage's existing built-in processing-error UI). + +**Storage**: YAML files (two new vendored API spec files, one new hand-authored +`catalog-info.yaml` for the precedence demo, one `app-config.yaml` addition) plus one new +backend-owned database table (`autoApiRegistration` scan-state cache, via `coreServices.database` +— same pluggable SQLite/Postgres service the catalog already uses) that persists per-file +mtime/hash/entity-name/error state across restarts so a real-world-scale mono-repo doesn't +require a full re-parse on every backend start (research.md R6, data-model.md). Not a schema +change to any existing Backstage table. + +**Testing**: Manual browser + log verification per lab README (no automated test suite — +consistent with Labs 1–3's tutorial-lab testing approach). + +**Target Platform**: Local development machine — Windows 10/11 and macOS 12+ (same as Labs 1–3). + +**Project Type**: Tutorial lab — Markdown documentation + YAML config/catalog files + one new +TypeScript backend module (EntityProvider + CatalogProcessor). + +**Performance Goals**: New/changed API definitions appear in the catalog within one discovery +cycle (≤30s at lab scale, SC-001). Mechanism-level goal (not exercised by the lab itself, but +designed for per research.md R6): a real mono-repo of 1000+ spec files across 1GB+ should incur +parse cost proportional to *changed* files, not total repo size, on both restart (persisted +scan-state cache) and steady-state discovery (watch mode + delta mutations). + +**Constraints**: Zero cost; cross-platform; no external network access beyond the one-time vendor +of the Scalar Galaxy API file at authoring time (the vendored copy itself requires no runtime +network access, per Constitution Principle VII); builds on Labs 1–3 without disrupting existing +registered APIs (museum, streetlights, train-travel) or the Lab 3 quality tooling. + +**Scale/Scope**: Lab demo: 2 new sample API files (Galaxy, precedence-demo), 1 new hand-authored +`catalog-info.yaml`, 1 new backend module (~3 files: EntityProvider+processor, DB migration for +the scan-state cache, and registration), 1 `app-config.yaml` config block, 3 existing registered +APIs left untouched. Designed-for scale (config change only, no code change, per research.md R6): +1000+ spec files, 1GB+ mono-repo size. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design — all gates hold.* + +| Principle | Gate | Status | +|-----------|------|--------| +| I. Feature-Focused Learning | Lab demonstrates concrete Backstage features: custom `EntityProvider` for filesystem-based catalog discovery, `x-*` vendor-extension metadata sourcing, native catalog processing-error surfacing via a custom `CatalogProcessor`, full-mutation-based create/update/retract lifecycle | ✅ Pass | +| II. Process-Oriented Documentation | README must explain WHY an `EntityProvider` (not a processor or GitHub discovery) was chosen (research.md R1), WHY errors ride on Backstage's native processing-error UI instead of a bespoke system (R4), WHY hand-authored `catalog-info.yaml` takes precedence over auto-sourced metadata (FR-011), and WHY visibility reuses the Lab 2/3 `example.com/visibility` annotation/policy verbatim instead of introducing a parallel mechanism (FR-006a) | ✅ Pass | +| III. Cross-Platform Compatibility | All new code is TypeScript/YAML; `fast-glob` and `js-yaml` are pure-JS, no native/platform-specific dependencies; `yarn add` and `yarn start` work identically on Windows and macOS; root-path resolution documented with platform-neutral relative paths | ✅ Pass | +| IV. Self-Contained Prerequisites | No new prerequisites beyond Labs 1–3's existing Node/Yarn setup — `fast-glob`/`js-yaml` are installed via the existing `yarn add` workflow; README documents the install step | ✅ Pass | +| V. Zero-Cost Operation | `fast-glob` and `js-yaml` are MIT-licensed OSS; Scalar Galaxy API is MIT-licensed; no cloud services, GitHub Apps, or webhooks required — mono-repo discovery runs entirely against the local filesystem | ✅ Pass | +| VI. Progressive Lab Structure | Lab 4 requires Labs 1–3 completion; the new `EntityProvider` runs alongside (does not replace) the existing manual `type: url` catalog locations from Labs 2–3, so museum/streetlights/train-travel remain registered exactly as before; no prior lab file is modified | ✅ Pass | +| VII. Modern & Purposeful API Examples | Scalar Galaxy API (MIT, self-contained, no external `$ref`s, realistic multi-resource domain) is new and distinct from Museum/Train Travel/Streetlights; the small precedence-demo API is a minimal-but-well-formed example used specifically to exercise precedence, not a primary teaching artifact | ✅ Pass | +| VIII. Support Experimentation | README documents `autoApiRegistration.rootPath`, `.patterns`, and the `xNamespace` (`x-examplecorp` → the learner's own company) as adaptable conventions (FR-010); User Story 3 is dedicated to this; the `sources[]` config shape (research.md R7) is documented as the path to onboarding additional team/pre-production repos with their own defaults, without code changes; verification tests outcomes (entity appears/updates/is retracted), not exact file layout | ✅ Pass | +| IX. Pragmatic Security for Learning Environments | No new authentication or credential configuration is introduced in Lab 4; the existing guest-provider auth from Lab 2 is unchanged. No Security Note section is required (no insecure practice introduced) | ✅ Pass | + +No violations. Complexity Tracking section omitted. + +## Project Structure + +### Documentation (this feature) + +```text +specs/004-lab-4-auto-registration/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +└── tasks.md # Phase 2 output (/speckit-tasks — NOT created here) +``` + +No `contracts/` directory — this lab produces no external API contracts. The `autoApiRegistration` +config shape and the entity mapping rules are documented in data-model.md instead. + +### Source Code (repository root) + +```text +labs/ +└── lab-04-auto-registration/ + ├── README.md # Lab documentation (overview, prereqs, steps, + │ # verification, troubleshooting, adaptable + │ # conventions per US3/FR-010) + └── apis/ + ├── galaxy/ + │ └── galaxy-openapi.yaml # New: vendored Scalar Galaxy API (MIT), + │ # info.x-examplecorp.owner/lifecycle added; + │ # no catalog-info.yaml — pure auto-discovery + └── precedence-demo/ + ├── precedence-demo-openapi.yaml # New: minimal spec for the precedence demo + └── precedence-demo-catalog-info.yaml # New: hand-authored entity that must win + # over the auto-sourced one (FR-011) +``` + +Changes to the student's Backstage instance (guided by README, not committed as generated output, +but the module source itself IS committed since it is lab teaching content): + +```text +# In the student's Backstage instance (labs/lab-01-base-backstage/backstage/): +app-config.yaml # Modified: add autoApiRegistration config block +packages/backend/ +├── package.json # Modified: add fast-glob, js-yaml, chokidar +└── src/ + ├── index.ts # Modified: backend.add(import('./extensions/autoApiRegistration')) + └── extensions/ + ├── autoApiRegistration.ts # New: EntityProvider + CatalogProcessor backend + │ # module (discovery, parsing, mapping, owner + │ # validation, precedence check, error marking, + │ # poll/watch mode, delta mutations) + └── autoApiRegistrationMigrations/ # New: coreServices.database migration creating + └── 001_scan_state_cache.ts # the scan-state cache table (research.md R6) +``` + +**Structure Decision**: Tutorial-documentation layout, consistent with Labs 1–3. All +student-facing content lives under `labs/lab-04-auto-registration/`. The two new sample API +files (and the one hand-authored precedence-demo catalog file) are committed alongside the lab +README. The student's Backstage instance (from Lab 1) is modified in-place per the README +instructions, same as Lab 3's pattern. Speckit artefacts live under +`specs/004-lab-4-auto-registration/` per convention. + +## Complexity Tracking + +No violations recorded — table omitted per template instructions. diff --git a/specs/004-lab-4-auto-registration/quickstart.md b/specs/004-lab-4-auto-registration/quickstart.md new file mode 100644 index 0000000..799c3cd --- /dev/null +++ b/specs/004-lab-4-auto-registration/quickstart.md @@ -0,0 +1,87 @@ +# Quickstart: Lab 4 — Auto Registration + +This is the design-time quickstart used to validate the plan; the learner-facing README (built in +`/speckit-tasks` + `/speckit-implement`) will follow the same flow with full explanatory prose per +Constitution Principle II. + +## Prerequisites + +- Labs 1–3 completed (running Backstage 1.51.0 instance, Lab 2 teams/users, Lab 3 quality tooling). +- No new external accounts or services. + +## Steps + +1. **Install dependencies**: add `fast-glob` and `js-yaml` as direct dependencies of + `packages/backend`. +2. **Add the auto-registration backend module** + (`packages/backend/src/extensions/autoApiRegistration.ts`): implements the `EntityProvider` + + companion `CatalogProcessor` described in research.md R1–R4. Registered in + `packages/backend/src/index.ts` via `backend.add(import('./extensions/autoApiRegistration'))`. +3. **Configure discovery** in `app-config.yaml` under a new `autoApiRegistration` block (see + data-model.md for keys/defaults). +4. **Add the two new sample files**: + - `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (vendored Scalar Galaxy API + with an `info.x-examplecorp` object added, `visibility: shared`) — no pre-existing + `catalog-info.yaml`. + - `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` + + `precedence-demo-catalog-info.yaml` (hand-authored, `example.com/visibility: private`, + owner a non-platform team) — demonstrates FR-011 precedence and gives a real + private/shared contrast to check in step 5. +5. **Start Backstage** (`yarn start` from the Lab 1 backstage workspace) and confirm: + - The Galaxy API appears in the catalog within one discovery cycle (~30s), with no manually + written catalog file (US1, SC-001). + - Its `spec.owner` and `spec.lifecycle` match the `info.x-examplecorp` values, and its tags + match the spec's native `tags` array — not duplicated into `x-examplecorp` (US2, SC-002). + - The Galaxy API (`visibility: shared`) is visible to any authenticated user, while the + precedence-demo API (`visibility: private`, hand-authored) is visible only to its owning + team and platform-team — logging in as a non-owning, non-platform user confirms the + difference (SC-006). + - The precedence-demo API's catalog entity reflects the hand-authored + `precedence-demo-catalog-info.yaml` throughout (owner, visibility), not the auto-sourced + values (FR-011). +6. **Verify update-in-place**: edit `galaxy-openapi.yaml`'s `info.description`, wait one cycle, + confirm the existing entity updates rather than duplicating (FR-003). +7. **Verify removal**: temporarily rename `galaxy-openapi.yaml` out of the discovery pattern, wait + one cycle, confirm the entity is no longer listed as active (FR-004, SC-005); rename it back. +8. **Verify error handling**: temporarily break the YAML syntax in one file, confirm a log line + identifies the file; temporarily set `info.x-examplecorp.owner` to a nonexistent team, confirm a + catalog processing error is visible on that entity (FR-008, SC-003); temporarily set + `info.x-examplecorp.visibility` to an unrecognized value (e.g. `public`), confirm the same kind + of visible error rather than a silent default. + +## Out of scope for this quickstart + +- Full JSON-Schema validation of spec files (documented as a learner extension point, not built). +- Exercising `watch` mode or the reconciliation safety net end-to-end (the lab's 2–3 sample files + don't demonstrate anything a poll cycle wouldn't; `mode: 'watch'` is documented, not walked + through step-by-step). + +## Scaling beyond the lab (documented, not walked through) + +The README includes a "Scaling to a real mono-repo" note pointing learners at: +- Why the default is `mode: 'poll'` + `type: 'full'`-then-`delta'` mutations, and what changes + (`mode: 'watch'`, larger `reconciliation.frequencySeconds`, mandatory `ignore` patterns) at + 1000+ files / 1GB+ (research.md R6). +- Why scan state is persisted in the backend's own database rather than written back into the + mono-repo as generated catalog files (data-model.md, research.md R6 Problem 2). +- That restart behavior at scale relies on the persisted cache, not a full re-scan — this is + called out explicitly since it's the part most likely to surprise a learner who scales this + lab up and then wonders why a restart is instant instead of taking minutes. + +A second "Scaling to multiple source repositories" note (research.md R7) covers: +- The `autoApiRegistration.sources[]` config list — the lab's own `app-config.yaml` uses the flat + single-source shorthand, but the README shows the equivalent explicit `sources: [{ id: default, + ... }]` form alongside a second example entry, so learners can see exactly what changes to add a + team or pre-production repo (a different `rootPath`, `defaultOwner`, `defaultVisibility`, and + `xNamespace`). +- Why `defaultOwner` and `xNamespace` are per-source with no built-in global fallback (a repo + onboarding to auto-registration shouldn't have to adopt the platform mono-repo's team or vendor + namespace to participate), while `defaultVisibility` is per-source but *does* have a built-in + global fallback (`private`) — visibility is the one setting where "unconfigured" must still be + safe by default, not just adaptable. +- Why collision detection and `catalog-info.yaml` precedence checks are explicitly called out as + *global* across sources, not per-source — two teams' repos producing the same entity name must + still surface a visible conflict, not silently coexist. +- That onboarding a genuinely separate remote (not a local sibling checkout) requires a sync step + (e.g. scheduled shallow `git clone`/`pull`) ahead of the existing discovery pipeline, which the + lab does not build or require — `rootPath` always points at something already on local disk. diff --git a/specs/004-lab-4-auto-registration/research.md b/specs/004-lab-4-auto-registration/research.md new file mode 100644 index 0000000..c950573 --- /dev/null +++ b/specs/004-lab-4-auto-registration/research.md @@ -0,0 +1,396 @@ +# Phase 0 Research: Lab 4 — Auto Registration + +## R1: Discovery mechanism + +**Decision**: A custom Backstage **EntityProvider** (not a `CatalogProcessor`), registered as a +backend module via `catalogProcessingExtensionPoint.addEntityProvider(...)`, following the same +`createBackendModule` pattern already used for the permission policy +(`labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/permissionPolicy.ts`). + +**Rationale**: An `EntityProvider` owns a versioned mutation of "the set of entities this provider +currently knows about." Anything it does not currently claim is retracted by the catalog. This +gives create (FR-002), update (FR-003), and removal-on-delete (FR-004) for free from one code +path, with no separate cleanup logic needed. A `CatalogProcessor` only transforms entities that +already exist via some `Location`; it cannot originate new entities from an arbitrary filesystem +scan on its own. + +**Alternatives considered**: +- GitHub discovery processor (`catalog-backend-module-github`) — rejected: requires a real GitHub + App/token and a reachable remote repo, violating the zero-cost/no-external-network assumption + already recorded in the spec (Assumptions: mono-repo is simulated locally). +- `type: file` catalog locations with glob targets pointing at `catalog-info.yaml` — rejected: + this still requires a hand-authored `catalog-info.yaml` per API, which is precisely what Lab 4 + must eliminate (FR-001). + +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: the lab defaults use a `type: 'full'` +mutation on every cycle because it is the simplest thing to explain and the entity count is tiny. +That does **not** scale: a `full` mutation re-transmits and re-diffs every known entity on every +cycle, so its cost grows with total catalog size, not with how much actually changed. The +mechanism as designed (an `EntityProvider` driving mutations) scales fine — what must change is +*which mutation type it sends*. See R6 for the delta-mutation and persisted-cache design that +keeps the same `EntityProvider` architecture viable at scale; the lab's own config simply doesn't +need to turn those knobs on for 2–3 sample files. + +## R2: File discovery within the mono-repo + +**Decision**: Use `fast-glob` (already present in the workspace `node_modules`, will be added as +an explicit `dependencies` entry in `packages/backend/package.json`) to scan a configured root +path for `**/*-openapi.yaml` and `**/*-asyncapi.yaml`, on a scheduled interval via +`coreServices.scheduler` (default every 30s — fast enough for interactive lab feedback without +hammering the filesystem). + +**Root path**: configurable via `autoApiRegistration.rootPath` in `app-config.yaml`; documented +as an adaptable convention (FR-010). Default value resolves via `resolvePackagePath('backend')` +(from `@backstage/backend-plugin-api`) rather than `process.cwd()` directly — `resolvePackagePath` +locates the `packages/backend` directory via Node module resolution of its own `package.json` +(`name: "backend"`), which is robust regardless of which working directory the backend process was +actually launched from, unlike a `process.cwd()`-relative assumption. From there, 4 `..` segments +(`packages/backend` → `packages` → `backstage` → `lab-01-base-backstage` → `labs`) reach `labs/`, +then into `lab-04-auto-registration/apis` — confirmed experimentally during implementation +(`packages/backend/src/extensions/autoApiRegistration.ts`'s `defaultRootPath()`). +(This is the single-source shorthand; R7 generalizes `rootPath` into a per-entry field of an +`autoApiRegistration.sources[]` list for organizations with more than one source repository.) + +**Scaling to a real mono-repo (1000+ files, 1GB+, see R6)**: two changes are load-bearing at scale +and are therefore built into the mechanism now, not deferred as a "someday" rewrite, even though +the lab's default config doesn't need them: +1. **Ignore patterns are mandatory, not optional.** `fast-glob` is called with an `ignore` list + (`**/node_modules/**`, `**/.git/**`, `**/dist/**`, `**/build/**`, plus a configurable + `autoApiRegistration.ignore` extension point) from day one. Without this, a glob walk over a + 1GB tree spends nearly all its time inside dependency/build directories that will never contain + a hand-authored spec file. This is a correctness-adjacent default, not a performance tune-up — + it is documented and defaulted in the lab even though the lab repo itself has few such + directories at the discovery root, so learners inherit a safe default rather than discovering + the need for it the hard way at 1000 files. +2. **Streamed, not buffered, matching.** Use `fastGlob.stream(...)` and process results as they + arrive rather than `fastGlob(...)`'s array form, so peak memory during a scan is bounded by + in-flight parse concurrency (see R6), not by total file count. + +**Alternatives considered**: +- `chokidar` file-watcher for real-time push updates — rejected *as the lab default* (adds a + persistent watch process and cross-platform edge cases — network drives, WSL, inotify limits on + very large trees — for marginal benefit over a 30s poll against 2–3 files). Reconsidered and + **adopted as the scale-up path** in R6: at 1000+ files, repeated full-tree poll walks are the + wrong steady-state mechanism, and a watcher layered over the same ignore-pattern set is what + makes the "how often do we rescan" question stop scaling with repo size. + +## R3: Parsing OpenAPI / AsyncAPI files and extracting `x-*` metadata + +**Decision**: Use `js-yaml` (already present, will be added as an explicit dependency) to parse +each matched file into a plain object. A file is treated as valid input if it parses as +YAML/JSON **and** contains either an `openapi` or `asyncapi` top-level version field **and** an +`info.title` string. Anything else (parse failure, missing both version fields, missing title) is +treated as malformed per the Edge Cases section. + +Recognized `x-*` metadata is read from a single vendor-namespaced object nested under the +top-level `info` object, for both OpenAPI and AsyncAPI (per the Clarifications session, +2026-07-03 update): +- `info.x-examplecorp.owner` → `spec.owner` +- `info.x-examplecorp.lifecycle` → `spec.lifecycle` (default: `experimental`, if absent) +- `info.x-examplecorp.visibility` → `metadata.annotations['example.com/visibility']` (default: that + file's source's configured `defaultVisibility`, itself defaulting to `private` if the source + doesn't set one — see rationale below and R7) + +**Visibility is deliberately reused, not reinvented.** Lab 2/3 already established +`example.com/visibility: private | shared` as the annotation the permission policy +(`packages/backend/src/extensions/permissionPolicy.ts`) reads to decide access: `shared` bypasses +ownership entirely (visible to all authenticated users); anything else (including the annotation +being absent) falls through to the ownership-only rule. This lab adds a **third way to set that +same annotation** — sourced from `x-examplecorp.visibility` instead of hand-authored in +`catalog-info.yaml` — it does not add a new visibility concept, a new annotation key, or touch the +permission policy at all. The absent-field default is **per-source**, not a single hardcoded +value: each source in `autoApiRegistration.sources[]` may set its own `defaultVisibility` +(`private` or `shared`) for files that don't declare one — e.g. a pre-production repo where most +APIs are intentionally shared for cross-team testing might reasonably default to `shared`, while +the platform mono-repo keeps the conservative default. The "default default" — what a source gets +if it doesn't configure `defaultVisibility` at all — is `private`, mirroring the permission +policy's own existing fallback behavior when the annotation is missing entirely (Rule 3, +ownership-only), so a freshly-added source with no visibility configuration behaves identically to +a hand-authored entity with no visibility annotation. An `x-examplecorp.visibility` value that is +present but not `private` or `shared` is treated as malformed input (Edge Cases) — surfaced via +the same marker-entity + processor-error path as an invalid owner reference (R4), not silently +coerced to a default (source-level or otherwise), since silently defaulting a security-relevant +setting is exactly the kind of mistake that must stay visible. + +`examplecorp` is this lab's fictional company namespace (documented as the one thing every +learner renames first, per FR-010/US3). The field names deliberately omit `backstage` — this +metadata (who owns an API, what lifecycle stage it's in) is meaningful to any consumer of the +spec, not just this tool, so the namespace names the company, not the tool. + +Metadata with a natural home elsewhere in the spec is read from there instead, never duplicated +into `x-examplecorp`: +- `info.title` → `metadata.title` / entity name (already established, unchanged) +- `info.description` → `metadata.description` (already established, unchanged) +- top-level `tags[].name` → `metadata.tags` (default: `[]`, if the spec has no `tags` array) — + **changed from the original `x-backstage-tags` design**: both OpenAPI and AsyncAPI already + define a native top-level `tags` field for exactly this purpose, so sourcing tags from `x-*` as + well would create two places that could drift apart. `tags[].name` (ignoring + `tags[].description`, which has no catalog equivalent) is the source of truth. + +**Rationale**: Full JSON-Schema validation against the OpenAPI/AsyncAPI meta-schemas (e.g. via +`@apidevtools/swagger-parser`, also present in `node_modules`) is available but deliberately not +used for the primary discovery path — it would reject specs with minor structural issues that are +otherwise perfectly usable for catalog registration, and doing full schema validation is out of +scope for what this lab teaches (catalog metadata sourcing, not spec authoring correctness). The +lightweight shape check above is sufficient to distinguish "not an API spec at all" from "a real +spec with metadata problems," which is the distinction FR-008's error handling cares about. + +**Alternatives considered**: +- `@apidevtools/swagger-parser` full validation — rejected as primary path (see rationale above); + noted as a documented extension point for learners in quickstart.md. + +## R4: Owner validation and "visible, actionable error" surfacing + +**Decision**: Errors are surfaced through **two independent, existing Backstage mechanisms** — +no bespoke error-reporting system is built: + +1. **Backend logs**: every discovery cycle logs one line per skipped/errored file (parse failure, + invalid owner reference, name collision) via the standard backend `logger`, already made + visible through the installed `@backstage/plugin-catalog-backend-module-logs`. +2. **Backstage's native processing-error UI**: for errors that occur *after* a candidate entity + has enough shape to be identified (invalid owner ref, name collision), the provider still emits + a minimal entity for that file, tagged with an annotation + (`apiportal-lab.io/registration-error: `). A small companion `CatalogProcessor` + (`preProcessEntity` hook, added to the same backend module) inspects that annotation and throws + a `InputError` when present — Backstage's core catalog processing engine natively records + processor errors per-entity and surfaces them in the entity's own "Inspect entity" / + unprocessed-entities view, which is exactly the built-in "catalog processing errors UI" the + Clarifications session specified. One message, written once, drives both surfaces. + + Files that fail the **shape check in R3** (not parseable as an API spec at all — no + `openapi`/`asyncapi` field, no `info.title`) cannot produce even a minimal entity (no name to + register under) and are therefore logged only; this is called out explicitly in quickstart.md + as the one category of error that is log-only rather than log+UI, and is why FR-008 lists + "fails to parse" separately from the owner/collision cases. + +**Owner validation**: before emitting the full mutation, the provider fetches all existing +`User`/`Group` entities via the in-process `CatalogService` and checks each candidate's +`x-examplecorp.owner` value resolves to a known entity ref (defaulting kind to `group:default/` if +no kind prefix is given, consistent with how `teams.yaml` entities are referenced elsewhere in the +repo). Unresolvable owners produce the marker-entity + processor-error path above. + +**Visibility validation**: `x-examplecorp.visibility`, if present, is checked against the fixed +set of values the existing permission policy understands (`private`, `shared`) — this is a static +enum check, not a catalog lookup, so it's cheap regardless of scale. An unrecognized value follows +the same marker-entity + processor-error path as an invalid owner reference. If absent, the value +comes from that file's source's `defaultVisibility` (falling back to `private` if the source +doesn't configure one) — see R3 for the full rationale and R7 for why this is per-source. + +**Name collisions**: if two discovered files slugify to the same candidate entity name, the first +(sorted by file path for determinism) is registered normally; the second is emitted as a marker +entity (suffixed, e.g. `-collision`) carrying the registration-error annotation so the +conflict is independently visible without ever silently overwriting the first. Collision detection +is done against the persisted slug→path index described in R6, not by re-deriving and sorting the +full candidate list on every check — at 1000+ files, an O(N log N) sort per cycle (or per watch +event, in watch mode) is avoidable work; an index lookup is O(1). **This index and check are +global, across every configured source, not scoped to the source currently being processed** — see +R7 Problem 3 for why two different teams' repos colliding must not go undetected. + +**Precedence with hand-authored `catalog-info.yaml`** (FR-011, Edge Case): before including a +candidate in its full mutation, the provider queries the catalog for an existing entity of the +same `kind: API` and `metadata.name` that did **not** originate from this provider (i.e. whose +`metadata.annotations['backstage.io/managed-by-origin-location']` differs from our provider's +entity source). If one exists, the auto-sourced candidate is skipped for that name — the +hand-authored entity wins, matching the documented Assumption. This check, like collision +detection, runs against catalog state as a whole and is unaffected by which source is being +processed — a hand-authored file anywhere wins over an auto-sourced candidate from any source. + +## R5: New sample API for the "previously unregistered" discovery demo + +**Decision**: Vendor a trimmed copy of the **Scalar Galaxy API** (MIT-licensed, +github.com/scalar/scalar, published build at +`https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`) as +`labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml`, with an `info.x-examplecorp` +object (`owner`, `lifecycle`, `visibility: shared` — chosen so the quickstart's SC-006 visibility +check has a real difference to demonstrate against the precedence-demo API below) added directly +in the vendored copy; its existing native `tags` array is used as-is for `metadata.tags`. The +upstream file is a single self-contained OpenAPI 3.1 +document (confirmed: 1469 lines, +all `$ref` usages are internal `#/components/...` references, no network-reachable `$ref`s), +covering a "planets/spaceships/users" domain distinct from the Museum, Train Travel, and +Streetlights examples already used in Labs 1–3, satisfying Constitution Principle VII. + +A second, small hand-crafted OpenAPI file (`precedence-demo-openapi.yaml`, ~2 paths) is added +alongside a hand-authored `precedence-demo-catalog-info.yaml` in +`labs/lab-04-auto-registration/apis/precedence-demo/`, specifically to exercise the FR-011 +precedence rule end-to-end without touching any Lab 1–3 registered API. + +**Alternatives considered**: +- Reusing an existing Lab 1–3 API for the discovery demo — rejected: it is already registered via + a hand-authored `catalog-info.yaml`, so it cannot authentically demonstrate "a previously + unregistered API gets discovered" (User Story 1's core scenario). +- Redocly's other example repos (TicTacToe, etc.) as backup — not needed; Scalar Galaxy confirmed + suitable on inspection. + +## R6: Scalability & restart behavior at real-world mono-repo scale + +The lab itself discovers 2–3 files in a small repo, and its default config values (30s full poll, +no persistence) are chosen for that scale — they are explicitly **not** what a 1000+ file, 1GB+ +mono-repo should run with. This section records the mechanism-level decisions that keep the same +architecture (EntityProvider + fast-glob) viable at that scale, so scaling up is a config change +plus turning on code paths that already exist, not a rewrite. + +**Problem 1 — full mutations don't scale with catalog size.** A `type: 'full'` `EntityProvider` +mutation re-sends and re-diffs every currently-known entity on every cycle. At 1000+ APIs, most of +which are unchanged between cycles, this wastes network/serialization cost and catalog-processing +time proportional to total catalog size rather than to actual change volume. + +**Decision**: after the provider's first `full` mutation (needed once, to establish the baseline +with the catalog), all subsequent cycles emit `type: 'delta'` mutations (`added`/`removed` only), +computed by diffing the current scan against the persisted scan-state cache (Problem 2). This is +the standard Backstage `EntityProvider` pattern for large, incrementally-changing entity sets (the +same pattern the built-in GitHub discovery provider uses at organization scale) — no custom +protocol is invented. + +**Problem 2 — a cold backend restart re-derives everything from zero.** Backstage's catalog +database already persists previously-ingested *entities* across restarts (a provider's entities +are not deleted until that provider explicitly retracts them), so the catalog itself doesn't go +empty on restart. What is **not** persisted anywhere by default is the provider's own *scan +state* — which files it has seen, their content hashes, and what they mapped to — so without +additional persistence, every restart forces a full re-glob and full re-parse of the entire +mono-repo (1000+ files) before the provider can even confirm nothing changed. + +**Decision**: persist scan state — `filePath`, `mtimeMs`, `contentHash` (sha1 of file bytes), +`entityName`, `lastError` — as rows in a small table owned by this backend module, using +`coreServices.database` (the same pluggable database service Backstage's own catalog uses; +SQLite in the lab, Postgres in a real deployment — zero new infrastructure, consistent with +Principle V). On startup, the provider loads this cache and can validate it against the +filesystem far more cheaply than re-parsing everything: a `stat()` per known path compares +`mtimeMs` (and falls back to a content-hash check only when `mtimeMs` differs, since hashing +1000+ files is itself non-trivial I/O at 1GB scale) before deciding a file needs re-parsing. +Unchanged files contribute zero parse work to a restart. This directly answers "should scanner +results be persisted so restarts restore the catalog faster" — yes, but as internal scan-state +cache rows in the backend's own database, **not** as generated `catalog-info.yaml`-style files +written back into the mono-repo. Writing discovery output back into the source tree would (a) +reintroduce the hand-maintained-duplicate-file problem Lab 4 exists to eliminate, and (b) require +the discovery tool to have write access to a repo it should only need to read — a materially +larger and riskier permission footprint at real-world scale. + +**Problem 3 — polling a 1GB tree every 30s doesn't scale as the primary discovery loop.** Even +with ignore patterns (R2) and streaming (R2), a full `fast-glob` walk of a very large tree still +costs real I/O, and running that walk every 30s regardless of whether anything changed is wasted +work at scale — the lab's 30s interval exists for interactive demo feedback on 2–3 files, not +because 30s is an appropriate steady-state cadence for 1000+ files. + +**Decision**: at scale, discovery should run in **watch mode**: a `chokidar` watcher (rejected for +the lab default in R2, adopted here) observes `add`/`change`/`unlink` events under the same +ignore-pattern set, and each event triggers a targeted single-file re-parse + delta mutation +(update the scan-state cache row, emit one delta) rather than a full-tree rescan. The periodic +`fast-glob` sweep is retained but demoted to an infrequent **reconciliation safety net** +(independently configurable interval, e.g. minutes not seconds) that catches anything a watcher +event could plausibly miss (coalesced events under a large batch change such as a big `git pull`, +or watch reliability limits on some network/CI filesystems) — it is a correctness backstop, not +the primary discovery signal. Parsing is concurrency-limited (bounded worker count, default small) +so a large batch of simultaneous file-change events — e.g. after checking out a branch that +touches hundreds of spec files — doesn't spike memory or block the Node event loop. + +**Config surface** (see data-model.md): `autoApiRegistration.mode: 'poll' | 'watch'` (lab default: +`poll`, no watcher, matching the simple demo story in quickstart.md); `ignore` (glob exclusions, +always applied regardless of mode); `reconciliation.frequencySeconds` (decoupled from watch-event +responsiveness); `parseConcurrency`. Switching a real deployment to scale is: set `mode: 'watch'`, +raise `reconciliation.frequencySeconds` from 30 to something like 900–1800, and rely on the +already-built delta-mutation + persisted-cache path — no architectural change. + +**Alternatives considered**: +- Always persisting a `full` mutation every cycle regardless of scale — rejected: the cost model + is wrong for 1000+ entities (see Problem 1); would work fine at lab scale but silently becomes a + performance cliff at real-world scale, which is exactly the trap this section exists to avoid. +- Writing discovered metadata back into the mono-repo as generated catalog files, to "cache" scan + results — rejected (see Problem 2 rationale): wrong persistence layer, reintroduces duplication, + expands write-access footprint unnecessarily. +- Only a filesystem watcher, no periodic reconciliation sweep — rejected: watchers can miss events + (coalescing, restart races, some CI/network filesystem limitations); a slow safety-net sweep is + cheap insurance at any scale and is what makes watch mode trustworthy enough to recommend. + +## R7: Multiple source repositories (platform mono-repo + team/pre-production repos) + +The lab itself discovers files from one location. A real organization is unlikely to have every +API definition in the platform team's mono-repo — other teams, and pre-production/experimental +APIs, plausibly live in their own separate repositories, each wanting different discovery +settings (in particular, a different default owner — Problem 2 below). The mechanism must +generalize to N independently-configured sources without becoming N copies of the same code. + +**Problem 1 — a single `rootPath` config key can't express "scan these several, separately +configured locations."** The R1–R6 design (and the lab's `autoApiRegistration.*` config) assumes +exactly one root and one set of settings. + +**Decision**: restructure config around a list, `autoApiRegistration.sources: [...]`, where each +entry carries its own `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`/`reconciliation`, +`defaultOwner`, and `xNamespace` (data-model.md has the full per-source schema). At backend-module +init time, the module reads this list and instantiates **one `EntityProvider` instance per +source**, each with a distinct `getProviderName()` (`auto-api-registration:`) — not one +provider trying to juggle multiple roots internally. This is the same pattern Backstage's own +multi-target providers use (e.g. a GitHub discovery provider configured against several +org/repo targets each becomes its own provider instance): each source gets its own independent +scheduling, its own independent full-then-delta mutation lifecycle (R6), and a config or scanning +problem in one source (e.g. an unreadable path for a team repo that hasn't been checked out yet) +cannot stall or break discovery for any other source. For backward/simple compatibility, a flat +`autoApiRegistration.rootPath`/`.patterns`/etc. (no `sources` list) is treated as shorthand for a +single implicit source named `default` — this is what the lab's own `app-config.yaml` uses, so +the single-repo lab config does not need to change shape. + +**Problem 2 — some settings are legitimately per-repo, not global.** Called out explicitly: +`defaultOwner` (a pre-production or team repo plausibly has a different fallback team than the +platform mono-repo's `platform-team`); `defaultVisibility` (a pre-production repo might reasonably +default undeclared APIs to `shared` for cross-team testing, while the platform mono-repo keeps a +conservative `private` default — R3); and, less obviously, `xNamespace` — a team repo that already +has its own established `x-*` vendor-extension convention before onboarding to auto-registration +shouldn't be forced to rename it to match the platform mono-repo's convention just to participate. + +**Decision**: `defaultOwner`, `defaultVisibility`, and `xNamespace` are per-source config keys — +see data-model.md's per-source schema. `defaultOwner` and `xNamespace` have no global fallback +(each source must set its own, or inherit an explicit `autoApiRegistration.defaults.*` block if +the learner wants shared values); `defaultVisibility` is the one exception with a genuine global +fallback (`private`, R3) precisely because leaving visibility completely unconfigured must still +be safe by default — a source shouldn't have to opt into a safe default, only opt out of it. +`patterns`, `ignore`, `mode`, and scheduling are also per-source for the same reason (a small +pre-production repo may reasonably use `mode: 'poll'` with a short interval while the platform +mono-repo runs `mode: 'watch'` at scale, per R6). + +**Problem 3 — collision detection and `catalog-info.yaml` precedence (R4) must stay global, not +per-source.** Two different teams' repos can trivially produce the same slugified entity name +(e.g. two teams each shipping a `payments-api`). If collision/precedence checks were scoped to a +single source's own scan-state rows, cross-source collisions would go undetected and both +entities would silently coexist as long as neither the collision index nor the "does a +non-auto-sourced entity already exist" precedence check restricts by source. The scan-state cache +table already keyed `entity_name` for O(1) collision lookups (R6) — the fix is simply that this +index (and the query behind the `catalog-info.yaml` precedence check) is queried **across all +sources**, not scoped to the source currently being processed. The `source_id` column added to +the cache table (data-model.md) exists precisely so a collision error message can say *which two +repos* collided — important for SC-003-style debuggability once "which file" becomes "which file, +in which repo." + +**Problem 4 — "other repos" may mean a genuinely separate Git remote, not a local sibling +directory.** The lab's own Assumptions section already anticipates a local sibling checkout ("a +local sibling repository the learner creates") for its own zero-network, zero-cost demo — that +covers the lab's own multi-source walkthrough (see quickstart.md). It does **not** cover a real +deployment where a team's repo is a separate remote the discovery backend has never had checked +out locally at all. + +**Decision (documented extension point, not built or demonstrated by the lab)**: each source's +`rootPath` config key is designed to remain a plain local filesystem path — the discovery, +parsing, and mapping logic (R1–R6) is entirely decoupled from *how bytes arrive on local disk*. +A production deployment onboarding a genuinely remote repo adds a lightweight sync step ahead of +the existing glob/parse pipeline — e.g. a shallow `git clone`/`git pull` into a local cache +directory on each scheduled cycle (or triggered by the same watch/reconciliation cadence as R6) — +and points that source's `rootPath` at the resulting local worktree. Every other mechanism (glob +patterns, ignore rules, delta mutations, scan-state cache, collision/precedence checks) is reused +unchanged; only the sync step differs per source. This keeps the zero-cost constraint intact for +the lab (no sync step is needed or built when `rootPath` already points at a local sibling +directory) while giving a real multi-repo deployment a documented, non-invasive path to genuinely +separate remotes — noted in quickstart.md as a "beyond the lab" extension point, consistent with +how R6's watch-mode scale-up is documented rather than exercised. + +**Alternatives considered**: +- One `EntityProvider` instance internally looping over multiple roots — rejected: conflates + scheduling, failure isolation, and mutation lifecycle across unrelated sources; a parse error or + slow filesystem in one team's repo could delay or corrupt discovery for every other source. Per + Backstage's own `EntityProvider` contract, provider identity is meant to be 1:1 with "one + coherent versioned view of a set of entities" — multiple sources are multiple such views. +- Making collision detection per-source (accept that two repos could produce the same name) — + rejected: silently produces whichever entity happens to process last as the "winner" with no + visible error, directly violating the Edge Cases requirement that name collisions must surface a + visible conflict rather than being silently resolved. +- Building real Git-remote syncing (clone/pull) as part of this lab — rejected: requires network + access and (for private repos) credentials, which conflicts with the lab's zero-network, + zero-cost constraint (Constitution V); documented as an extension point instead (Problem 4). diff --git a/specs/004-lab-4-auto-registration/spec.md b/specs/004-lab-4-auto-registration/spec.md new file mode 100644 index 0000000..cba7734 --- /dev/null +++ b/specs/004-lab-4-auto-registration/spec.md @@ -0,0 +1,133 @@ +# Feature Specification: Lab 4 - Auto Registration + +**Feature Branch**: `004-lab-4-auto-registration` + +**Created**: 2026-07-02 + +**Status**: Draft + +**Input**: User description: "Create new feature and specification from feature 4 in GOAL.md" — Lab 4 (Auto registration): Setup a process for automatic registration of APIs and creation of catalog metadata. Assume API definition files exist within a Git mono-repo. Include sourcing of metadata from x-* fields, within the API specifications themselves, into catalog metadata, fields and annotations. + +## Clarifications + +### Session 2026-07-02 + +- Q: What naming convention should the recognized `x-*` metadata fields use? → A: `x-backstage-owner`, `x-backstage-lifecycle`, `x-backstage-tags` (namespaced under `x-backstage-*`) — **superseded, see Session 2026-07-03 below** +- Q: How is the mono-repo discovery scope defined — fixed folder(s) or filename pattern? → A: No fixed folder; discovery matches a filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) anywhere in the repo +- Q: Where should discovery/registration errors be surfaced to the learner? → A: Both Backstage's built-in catalog processing errors UI (entity-level) and backend log output +- Q: How is the catalog entity's name derived from the API definition file? → A: From the spec's own `info.title` field, slugified (not the file path, not a separate `x-*` name field) +- Q: Where do recognized `x-*` metadata fields live within an AsyncAPI document, relative to OpenAPI? → A: Same placement as OpenAPI — under the document's top-level `info` object for both formats + +### Session 2026-07-03 + +- Q: Should `x-*` fields sit directly in `info`, or nested under a single vendor-namespaced object? → A: Nested under a single `info.x-examplecorp` object (e.g. `info.x-examplecorp.owner`, `info.x-examplecorp.lifecycle`) — `examplecorp` is this lab's fictional company namespace, which learners are expected to rename to their own; a single grouping object avoids polluting `info` with multiple top-level `x-*` keys and makes "which fields are ours" unambiguous at a glance. +- Q: Should field names retain the `backstage` tool name (`x-backstage-owner`, etc.)? → A: No — the field names describe the metadata (owner, lifecycle), not the tool consuming it. Catalog metadata like ownership is tool-agnostic (equally meaningful to an API gateway or any other consumer), so `backstage` is dropped: `x-examplecorp.owner`, `x-examplecorp.lifecycle`. +- Q: Should `tags` remain a recognized `x-*` field? → A: No — removed. OpenAPI/AsyncAPI both already have a native top-level `tags` field; sourcing tags from `x-*` as well would duplicate data that already has a natural home in the spec, risking divergence. Catalog `metadata.tags` is now sourced from the spec's native `tags` array (`tags[].name`), not from any `x-*` field. The same reasoning already applied to `info.title` (entity name) and `info.description` (entity description), which were never duplicated into `x-*` fields. +- Q: Should Lab 2's API visibility concept (private/shared) be added to the `x-*`-managed metadata? → A: Yes — `info.x-examplecorp.visibility`, accepting the same `private` \| `shared` values Lab 2 already established, mapped to the same `example.com/visibility` annotation the Lab 2/3 permission policy already reads. Visibility has no natural home elsewhere in the spec (unlike name/description/tags), so it belongs in `x-examplecorp` alongside owner and lifecycle, not as a new mechanism. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) + +As a platform engineer, when a new API definition file (OpenAPI or AsyncAPI) is added anywhere within the designated mono-repo location, I want it to appear in the Backstage catalog as a registered API entity without anyone hand-authoring a `catalog-info.yaml` for it. + +**Why this priority**: This is the core value of the lab — removing manual catalog registration is the entire premise of "auto registration." Without this, there is no lab. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file to the mono-repo location, trigger a catalog refresh, and confirm a corresponding API entity appears in the Backstage catalog with no manual `catalog-info.yaml` written for that API. + +**Acceptance Scenarios**: + +1. **Given** a mono-repo containing one or more OpenAPI/AsyncAPI definition files with no matching catalog entity, **When** the catalog discovery process runs, **Then** a catalog API entity is created for each definition file found. +2. **Given** an API definition file that was previously registered, **When** its contents change (e.g., description or version updated) and discovery runs again, **Then** the existing catalog entity is updated in place rather than duplicated. +3. **Given** an API definition file is removed from the mono-repo, **When** discovery runs again, **Then** the corresponding catalog entity is no longer present as an active, discoverable entity. + +--- + +### User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +As an API owner, I want to declare metadata that Backstage needs (such as owning team, lifecycle stage, or visibility) directly inside my OpenAPI/AsyncAPI specification using `x-*` vendor extension fields, so that this information lives in one place — the spec itself — instead of being duplicated and maintained separately in a catalog file. + +**Why this priority**: This is what makes registration genuinely "automatic" rather than just file discovery — without metadata sourcing, every API would still need a hand-maintained catalog file for ownership and lifecycle, which is exactly what Lab 2's ownership model depends on. + +**Independent Test**: Author an API definition containing `x-*` fields for owning team and lifecycle, run discovery, and confirm the resulting catalog entity's `spec.owner`, `spec.lifecycle`, and equivalent fields/annotations reflect the values from the `x-*` fields — not placeholder or default values. + +**Acceptance Scenarios**: + +1. **Given** an API definition containing an `x-*` field identifying the owning team, **When** the entity is registered, **Then** the catalog entity's owner reference matches that team, consistent with the ownership model established in Lab 2. +2. **Given** an API definition containing `x-*` fields for additional metadata (e.g., lifecycle stage, visibility), **When** the entity is registered, **Then** those values populate the corresponding catalog entity fields or annotations rather than requiring manual entry, while metadata with a natural home elsewhere in the spec (name, description, tags) is sourced from there instead of from `x-*`. +3. **Given** an API definition that omits one or more recognized `x-*` metadata fields, **When** the entity is registered, **Then** the entity is still created successfully using documented default values, and the missing metadata does not block registration. +4. **Given** an API definition with an `x-*` field referencing a team that does not exist in the catalog, **When** the entity is registered, **Then** the system surfaces a clear, visible error or warning identifying the invalid reference rather than failing silently or crashing discovery for other APIs. +5. **Given** an API definition containing an `x-*` visibility field, **When** the entity is registered, **Then** the catalog entity's visibility annotation matches that value and is enforced by the same permission policy introduced in Lab 2 (private → owning team + platform team only; shared → all authenticated users), with no separate visibility mechanism introduced for auto-registered APIs. + +--- + +### User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +As a learner adapting this repository to my own organization, I want to understand which parts of the discovery and metadata-sourcing configuration are conventions I can change (e.g., file naming patterns, folder layout, `x-*` field names) versus fixed requirements, so I can apply this pattern to my own mono-repo layout. + +**Why this priority**: Reinforces Constitution Principle VIII (Support Experimentation) — the lab must remain valuable to learners whose mono-repo doesn't match the example layout exactly. Lower priority than P1/P2 because the mechanism must exist and work correctly before its adaptability can be documented. + +**Independent Test**: Follow the lab documentation's guidance to point discovery at a differently-structured example folder (different glob pattern, different `x-*` field names) and confirm registration still works as described. + +**Acceptance Scenarios**: + +1. **Given** the lab documentation, **When** a learner reads the configuration section, **Then** they can identify which settings (file location pattern, recognized `x-*` field names) are conventions they are expected to adapt, and which are fixed mechanics of the discovery process. + +--- + +### Edge Cases + +- What happens when an API definition file is malformed or fails to parse (invalid YAML/JSON, not valid OpenAPI/AsyncAPI)? The system MUST skip that file, continue processing other files, and surface a visible error identifying the offending file. +- What happens when two API definition files resolve to the same catalog entity name (i.e., the same slugified `info.title`)? The system MUST surface a visible conflict/error rather than silently overwriting one entity with the other. +- What happens when an `x-*` field contains an unexpected type or shape (e.g., owner field is a list instead of a string)? The system MUST surface a visible error for that entity rather than registering it with corrupted metadata. +- What happens when the `x-*` visibility field is present but is not one of the recognized values (`private`, `shared`)? The system MUST surface a visible error for that entity rather than silently defaulting or registering it with an unenforceable visibility value. +- What happens when discovery runs against a mono-repo location containing zero API definitions? The system MUST complete without error and register zero entities. +- How does the system behave when the same API definition also contains a hand-authored `catalog-info.yaml` alongside it? The lab documentation MUST clarify precedence (i.e., which source of metadata wins) to avoid ambiguity for learners who mix approaches during Lab 4. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST discover API definition files (OpenAPI and AsyncAPI) anywhere within the mono-repo by matching a documented filename pattern (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) rather than requiring a fixed folder location, and without requiring a hand-authored `catalog-info.yaml` per API. +- **FR-002**: The system MUST automatically create a catalog API entity for each newly discovered API definition file. +- **FR-003**: The system MUST automatically update the corresponding catalog API entity when a previously-registered API definition file's contents change. +- **FR-004**: The system MUST stop presenting a catalog API entity as active once its source definition file is removed from the mono-repo. +- **FR-005**: The system MUST extract the owning team from a documented `x-*` field within the API specification and set it as the catalog entity's owner, using the same underlying ownership model relied upon by Lab 2's visibility policy (not a disconnected copy). +- **FR-006**: The system MUST extract additional catalog metadata (at minimum: lifecycle stage and visibility) from a documented, vendor-namespaced `x-*` object (e.g. `info.x-examplecorp`) located under the document's top-level `info` object for both OpenAPI and AsyncAPI, and map it into the corresponding catalog entity fields or annotations. Metadata that already has a natural home elsewhere in the spec (API name via `info.title`, description via `info.description`, tags via the spec's native top-level `tags` field) MUST be sourced from that native location, not duplicated into an `x-*` field. +- **FR-006a**: The system MUST extract API visibility from the documented `x-*` object and set the same visibility annotation the Lab 2 permission policy already enforces (`example.com/visibility`, values `private` or `shared`), reusing that existing policy mechanism rather than introducing a separate visibility concept or enforcement path for auto-registered APIs. When a given API definition omits the visibility field, the system MUST apply a default that is independently configurable per mono-repo/source location (so different source repositories can have different default visibility), falling back to `private` for any source that does not configure its own default. +- **FR-007**: The system MUST apply documented default values for any recognized `x-*` metadata field that is absent from a given API definition, rather than failing registration. +- **FR-008**: The system MUST surface a visible, actionable error — in both Backstage's built-in catalog processing errors UI (entity-level) and backend log output — when an API definition fails to parse, references a non-existent owning team, specifies an unrecognized visibility value, or produces an entity-name collision with another API definition, and MUST continue processing remaining, unaffected API definitions. +- **FR-009**: The system MUST support both OpenAPI and AsyncAPI definition files, consistent with the API types introduced in prior labs. +- **FR-010**: Lab documentation MUST identify which parts of the discovery and metadata-sourcing configuration (file location patterns, recognized `x-*` field names) are adaptable conventions versus fixed mechanics, per Constitution Principle VIII. +- **FR-011**: Lab documentation MUST state the precedence rule when both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API. +- **FR-012**: The lab MUST run entirely on local, freely available tooling with $0 cost, requiring no paid service, hosted CI/CD, or webhook-reachable public endpoint (Constitution Principle V). +- **FR-013**: The lab MUST build upon the environment established by Labs 1-3 (base Backstage setup, users/roles/teams and ownership-based visibility, API quality tooling) without requiring those labs to be redone. + +### Key Entities + +- **API Definition File**: An OpenAPI or AsyncAPI specification file stored within the mono-repo, containing the technical API contract, its native `info.title`/`info.description`/`tags` fields, and a single vendor-namespaced `info.x-examplecorp` object carrying catalog metadata that has no natural home elsewhere in the spec (owning team, lifecycle, visibility). +- **Catalog API Entity**: The Backstage catalog representation automatically created and kept in sync from an API Definition File; its entity name and description are derived from the spec's own `info.title` (slugified) and `info.description`, its tags from the spec's native `tags` field, and its ownership/lifecycle/visibility metadata sourced from `info.x-examplecorp`. +- **Mono-repo Location**: The designated filename-pattern convention (e.g. `*-openapi.yaml`, `*-asyncapi.yaml`) matched anywhere within the repo, used as the scope for discovery — not tied to a fixed folder. +- **`x-*` Metadata Mapping**: The documented mapping from the `info.x-examplecorp` object's fields to catalog entity fields/annotations, including default values applied when a field is absent, and the explicit list of fields sourced natively rather than from `x-*` (name, description, tags). + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: A learner can add a new API definition file to the mono-repo location and see it appear as a registered catalog entity within one discovery cycle, without writing any catalog metadata file by hand. +- **SC-002**: 100% of the owning-team, lifecycle, and visibility metadata visible on a registered API's catalog entity originates from the `x-*` object in its definition file, and 100% of the name/description/tag metadata originates from that same file's native `info.title`/`info.description`/`tags` fields — with zero fields duplicated between the two, and zero manual duplication of any of this metadata elsewhere. +- **SC-006**: A learner can set an API's visibility via its `x-*` field alone (no hand-authored catalog file) and observe the same access behavior Lab 2 documented for that visibility value (private vs. shared) when viewing the catalog as different users, confirming the auto-sourced value drives the existing permission policy rather than a separate mechanism. +- **SC-003**: A learner who introduces a malformed API definition file can identify, from a visible error surfaced in both the Backstage catalog processing errors UI and backend log output, exactly which file and problem caused registration to fail, without needing to inspect system internals. +- **SC-004**: A learner following the lab can successfully repoint discovery and metadata sourcing at a mono-repo layout that differs from the example (different folder structure or `x-*` field names) using only the lab documentation. +- **SC-005**: Removing an API definition file from the mono-repo results in its catalog entity no longer being presented as active, without any manual catalog cleanup step. + +## Assumptions + +- The "Git mono-repo" referenced in Lab 4 is simulated as a filename-pattern convention (not a fixed folder) within the lab's own repository (or a local sibling repository the learner creates), consistent with the zero-cost, no-external-network constraints established for all labs — it is not assumed to be a real remote Git hosting integration or webhook-driven trigger. +- "Automatic" registration is achieved via a periodic/on-demand discovery and refresh cycle consistent with Backstage's standard catalog processing model, not a real-time push notification from a Git host. +- The default `x-*` convention is a single vendor-namespaced object, `info.x-examplecorp`, containing `owner`, `lifecycle`, and `visibility` keys (`examplecorp` being this lab's fictional company namespace); tool-specific naming (e.g. `backstage`) is deliberately excluded from the field name since this metadata is not tool-specific, and fields with a natural home elsewhere in the spec (name, description, tags) are deliberately not duplicated into it. Learners are expected to rename the `x-examplecorp` namespace to their own organization's convention, per Constitution Principle VIII. +- The ownership model and set of valid owning teams established in Lab 2 (002-lab-2-users-roles) is the source of truth referenced when validating an `x-*` owner field — Lab 4 does not introduce a new/parallel ownership mechanism. +- The visibility model established in Lab 2/3 (the `example.com/visibility` annotation and its `private`/`shared` values, enforced by the existing permission policy) is the source of truth referenced when mapping the `x-*` visibility field — Lab 4 does not introduce a new/parallel visibility mechanism, only a new way of setting the same annotation. +- The default visibility applied when a definition omits the `x-*` visibility field is configured per mono-repo/source location, not globally fixed — different source repositories (e.g. a pre-production repo vs. the platform mono-repo) may reasonably want different defaults. A source that does not configure its own default falls back to `private`, matching the permission policy's existing behavior when no visibility annotation is present at all. +- When both a hand-authored `catalog-info.yaml` and sourced `x-*` metadata exist for the same API, the hand-authored `catalog-info.yaml` is assumed to take precedence, treating auto-sourced metadata as a default that manual configuration can override; this is documented explicitly for learners. +- Removal of a catalog entity when its source file disappears is treated as removal from active/discoverable presentation within Backstage's catalog processing, not necessarily an irreversible deletion of historical records. diff --git a/specs/004-lab-4-auto-registration/tasks.md b/specs/004-lab-4-auto-registration/tasks.md new file mode 100644 index 0000000..2ae6c85 --- /dev/null +++ b/specs/004-lab-4-auto-registration/tasks.md @@ -0,0 +1,226 @@ +--- + +description: "Task list for Lab 4 — Auto Registration" +--- + +# Tasks: Lab 4 — Auto Registration + +**Input**: Design documents from `/specs/004-lab-4-auto-registration/` + +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md (all present; no `contracts/` — this lab produces no external API contracts) + +**Tests**: Not requested. Per plan.md's Testing section, verification is manual (browser + backend logs) against quickstart.md's steps — consistent with Labs 1–3. No automated test tasks are generated; quickstart validation is folded into each story's checkpoint and the final Polish phase. + +**Organization**: Tasks are grouped by user story (spec.md priorities P1/P2/P3) to enable independent implementation and testing of each story. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (US1, US2, US3) + +## Path Conventions + +This is a tutorial-lab project (not a generic web/mobile app). Two path roots are used: + +- **Backstage instance** (modified in place, from Lab 1): `labs/lab-01-base-backstage/backstage/` +- **Lab 4 teaching content** (new, committed alongside README): `labs/lab-04-auto-registration/` +- **Speckit artifacts**: `specs/004-lab-4-auto-registration/` (this directory — no code changes here) + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Add dependencies and scaffold the directories every later task writes into. + +- [X] T001 Add `fast-glob@3.3.3`, `js-yaml@4.2.0`, and `chokidar` as direct dependencies of `labs/lab-01-base-backstage/backstage/packages/backend/package.json` (`yarn add` from that package directory; research.md R2/R6) +- [X] T002 [P] Create the lab content directories `labs/lab-04-auto-registration/apis/galaxy/` and `labs/lab-04-auto-registration/apis/precedence-demo/` +- [X] T003 [P] Create the backend module scaffold directory `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/` + +**Checkpoint**: Dependencies installed, directories exist — ready for foundational work. + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: Build the shared scanning/parsing/persistence/config plumbing that both US1 (discovery/CRUD) and US2 (metadata sourcing) sit on top of. No user story is independently testable until this phase is done — the EntityProvider skeleton it produces is what US1 wires up to catalog mutations and US2 extends with `x-*` extraction. + +**⚠️ CRITICAL**: No user story work can begin until this phase is complete. + +- [X] T004 Define the `autoApiRegistration` config schema (flat single-source shorthand + `sources[]` list form: `id`, `rootPath`, `patterns`, `ignore`, `mode`, `schedule`, `reconciliation`, `parseConcurrency`, `defaultOwner`, `defaultVisibility`, `xNamespace`) as TypeScript types in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts`, per data-model.md's Config section +- [X] T005 Implement config normalization (flat shorthand → single implicit `sources: [{ id: 'default', ... }]` entry; per-source `defaultVisibility` falling back to the global `'private'` "default default") in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R7, depends on T004) +- [X] T006 [P] Implement the scan-state cache DB migration (`source_id`, `file_path` composite key; `mtime_ms`, `content_hash`, `entity_name` indexed, `last_error` columns) via `coreServices.database` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistrationMigrations/001_scan_state_cache.ts` (data-model.md Scan-state cache section, research.md R6 Problem 2) +- [X] T007 [P] Implement the streamed, ignore-pattern-aware file discovery utility (`fastGlob.stream(...)` over `patterns`/`ignore`, per-source `rootPath`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T004) +- [X] T008 [P] Implement the YAML parse + shape-check utility (valid input = parses **and** has `openapi` or `asyncapi` **and** `info.title`; anything else is a log-only malformed-file error) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3, depends on T004) +- [X] T009 Implement the mtime/content-hash change-detection check against the scan-state cache (skip re-parse for unchanged files; sha1 hash only computed when `mtime_ms` differs) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R6 Problem 2, depends on T006, T007) +- [X] T010 Register the (not-yet-emitting) backend module via `createBackendModule` + `catalogProcessingExtensionPoint`, following the `permissionPolicy.ts` pattern, and wire it into `labs/lab-01-base-backstage/backstage/packages/backend/src/index.ts` with `backend.add(import('./extensions/autoApiRegistration'))` (research.md R1, depends on T004) +- [X] T011 Add the `autoApiRegistration` config block (single-source shorthand, matching the lab's default `rootPath`/`patterns`) to `labs/lab-01-base-backstage/backstage/app-config.yaml` (data-model.md Config section, depends on T004) + +**Checkpoint**: Config loads, files are discoverable and parseable, scan-state cache persists — foundation ready for US1. + +--- + +## Phase 3: User Story 1 - Automatic discovery and registration of API definitions (Priority: P1) 🎯 MVP + +**Goal**: A new OpenAPI/AsyncAPI file dropped anywhere under the mono-repo location appears in the Backstage catalog with no hand-authored `catalog-info.yaml`; edits update the entity in place; removing the file retracts it. + +**Independent Test**: Add a new, previously-unregistered OpenAPI spec file, trigger discovery, confirm a corresponding API entity appears — with default owner/lifecycle values (no `x-*` extraction required for this story) — with no manual `catalog-info.yaml` written for it; edit the file and confirm update-in-place; remove the file and confirm the entity is no longer active. + +### Implementation for User Story 1 + +- [X] T012 [US1] Implement candidate entity mapping using only natively-homed fields — `info.title` slugified → `metadata.name`, verbatim `info.title` → `metadata.title`, `info.description` → `metadata.description`, native `tags[].name` → `metadata.tags` (default `[]`), `openapi`/`asyncapi` presence → `spec.type`, `spec.definition.$text` → source file path/URL, `backstage.io/managed-by-location` annotation → source file path — with `spec.owner`/`spec.lifecycle` set to the per-source `defaultOwner` constant and the fixed `experimental` default (no `x-*` reading yet) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (data-model.md Catalog API Entity table, depends on T008) +- [X] T013 [US1] Implement the first-run full mutation: glob scan → parse + shape-check → map candidates (T012) → build scan-state cache rows → emit one `type: 'full'` `EntityProvider` mutation in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R1/R6, depends on T009, T012) +- [X] T014 [US1] Implement subsequent-cycle delta mutations: diff current scan against scan-state cache rows, emit `type: 'delta'` (`added`/`removed`) mutations for changed/new/removed files only, update cache rows in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-002/003/004, research.md R6 Problem 1, depends on T013) +- [X] T015 [US1] Wire the `coreServices.scheduler` poll loop (`schedule.frequencySeconds`, default 30s) to run the discovery cycle (T013 first run, then T014) per configured source in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R2, depends on T014) +- [X] T016 [US1] Vendor the Scalar Galaxy API to `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (trimmed copy of the MIT-licensed `https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml`, no `x-examplecorp` object yet, no pre-existing `catalog-info.yaml`) — used to manually verify create/update/remove for this story (research.md R5) +- [X] T017 [US1] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5–7): Galaxy API appears within one poll cycle with no hand-authored catalog file (SC-001); editing `info.description` updates the entity in place, not a duplicate (FR-003); renaming the file out of the discovery pattern retracts the entity as active (FR-004/SC-005) + +**Checkpoint**: User Story 1 fully functional and independently testable — discovery, create, update, and removal all work end-to-end with default metadata. + +--- + +## Phase 4: User Story 2 - Sourcing catalog metadata from spec `x-*` fields (Priority: P2) + +**Goal**: `spec.owner`, `spec.lifecycle`, and the `example.com/visibility` annotation are sourced from each file's `info.x-examplecorp` object (with documented defaults when absent), invalid values/references surface visible errors on both backend logs and the catalog processing-errors UI, name collisions and hand-authored `catalog-info.yaml` precedence are respected. + +**Independent Test**: Author an API definition with `x-examplecorp` owner/lifecycle/visibility fields, run discovery, confirm the resulting entity's `spec.owner`, `spec.lifecycle`, and `example.com/visibility` annotation reflect those values (not defaults); omit fields and confirm documented defaults apply without blocking registration; set an invalid owner/visibility and confirm a visible error on that entity without affecting other entities; confirm a hand-authored `catalog-info.yaml` wins over auto-sourced metadata for the same API. + +### Implementation for User Story 2 + +- [X] T018 [US2] Extend the candidate entity mapping (T012) to read `info.x-.owner` → `spec.owner` (default: per-source `defaultOwner`) and `info.x-.lifecycle` → `spec.lifecycle` (default: `experimental`) using each source's configured `xNamespace`, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-005/FR-006/FR-007, data-model.md Mapping Rule 1, depends on T012) +- [X] T019 [US2] Extend the candidate entity mapping to read `info.x-.visibility` → `metadata.annotations['example.com/visibility']` (default: per-source `defaultVisibility`, falling back to `'private'`) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-006a, data-model.md Mapping Rule 2, depends on T018) +- [X] T020 [US2] Implement owner validation against known `User`/`Group` catalog entities (fetched via `CatalogService`, defaulting kind to `group:default/` when unprefixed); unresolvable owners produce a marker/error entity instead of a normally-registered one, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4, depends on T019) +- [X] T021 [US2] Implement visibility value validation (static enum check against `{private, shared}`); an unrecognized present value produces a marker/error entity rather than a silent default, in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R3/R4, depends on T019) +- [X] T022 [US2] Implement cross-source name-collision detection using the scan-state cache's `entity_name` index (queried across all sources, not scoped to the current one; deterministic first-by-path winner, second emitted as a `-collision` marker entity) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (research.md R4/R7 Problem 3, depends on T020, T021) +- [X] T023 [US2] Implement the `catalog-info.yaml` precedence check (query the catalog for an existing non-auto-sourced entity of the same `kind: API`/`metadata.name`; skip the auto-sourced candidate if found) in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-011, research.md R4, depends on T022) +- [X] T024 [US2] Implement the `apiportal-lab.io/registration-error` marker annotation on error entities plus the companion `CatalogProcessor` (`preProcessEntity` hook throwing `InputError` when the annotation is present) and backend-logger error lines for every skipped/errored file, registered alongside the `EntityProvider` in `labs/lab-01-base-backstage/backstage/packages/backend/src/extensions/autoApiRegistration.ts` (FR-008, research.md R4, depends on T023) +- [X] T025 [US2] Add `info.x-examplecorp` (`owner`, `lifecycle`, `visibility: shared`) to the vendored `labs/lab-04-auto-registration/apis/galaxy/galaxy-openapi.yaml` (depends on T016) +- [X] T026 [P] [US2] Create the minimal precedence-demo spec `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-openapi.yaml` (~2 paths, distinct `info.title`) (research.md R5, depends on T002) +- [X] T027 [P] [US2] Create the hand-authored `labs/lab-04-auto-registration/apis/precedence-demo/precedence-demo-catalog-info.yaml` (owner a non-platform team, `example.com/visibility: private`) that must win over the auto-sourced candidate for the same file (FR-011, depends on T026) +- [X] T028 [US2] Manually verify against `labs/lab-01-base-backstage/backstage/` (quickstart.md steps 5, 8): Galaxy entity's `spec.owner`/`spec.lifecycle`/tags match `x-examplecorp`/native-`tags` values (SC-002); Galaxy (`shared`) vs. precedence-demo (`private`, hand-authored) show the documented visibility contrast when signed in as a non-owning, non-platform user (SC-006); precedence-demo entity reflects the hand-authored file, not auto-sourced values (FR-011); a broken-YAML file logs an identifying error; a nonexistent owner reference and an unrecognized visibility value each surface a catalog processing error on that entity (FR-008, SC-003) (depends on T024, T025, T027) + +**Checkpoint**: User Stories 1 AND 2 both work independently — metadata sourcing, defaults, validation errors, collisions, and precedence are all verified. + +--- + +## Phase 5: User Story 3 - Learner can extend the mono-repo convention to their own structure (Priority: P3) + +**Goal**: Lab documentation clearly separates adaptable conventions (file location pattern, `x-*` field names, multi-source config) from fixed discovery mechanics, so a learner can repoint the lab at their own layout. + +**Independent Test**: Follow the lab README's configuration section to identify which settings are adaptable vs. fixed, and confirm the documented `sources[]` multi-repo example is internally consistent with data-model.md's schema. + +### Implementation for User Story 3 + +- [X] T029 [US3] Write `labs/lab-04-auto-registration/README.md` Overview/Prerequisites/Steps sections (Constitution Principle II process-oriented documentation) covering: install deps, add the backend module, configure `autoApiRegistration`, add sample files, start Backstage, verify discovery/update/removal/errors — following quickstart.md steps 1–8 (depends on T015, T024) +- [X] T030 [US3] Add the README's "Adaptable conventions vs. fixed mechanics" section (FR-010) explicitly identifying `rootPath`, `patterns`, and `xNamespace` as learner-adaptable, and the `EntityProvider`/full-then-delta mutation mechanism as fixed, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T031 [P] [US3] Add the README's "Scaling to a real mono-repo" note (`mode: 'watch'`, `reconciliation.frequencySeconds`, mandatory `ignore` patterns, persisted scan-state cache vs. writing back into the mono-repo) per research.md R6, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T032 [P] [US3] Add the README's "Scaling to multiple source repositories" note, including the explicit `sources: [{ id: default, ... }, { id: team-checkout-api, ... }]` example from data-model.md, and why `defaultOwner`/`xNamespace` have no global fallback while `defaultVisibility` does, per research.md R7, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T033 [US3] Add the README's precedence-rule statement (FR-011: hand-authored `catalog-info.yaml` wins over auto-sourced metadata) and the Security Note required by Constitution Principle IX only if Lab 4 introduces new credential/auth configuration — confirm per plan.md's Constitution Check (none expected) and omit if not applicable, in `labs/lab-04-auto-registration/README.md` (depends on T029) +- [X] T034 [US3] Add the README's Verification and Troubleshooting sections (Lab Structure Standards) covering the manual checks already performed in T017/T028 plus common failure modes (stale scan-state cache after manual DB edits, glob pattern typos, wrong `xNamespace`), in `labs/lab-04-auto-registration/README.md` (depends on T030) + +**Checkpoint**: All three user stories independently functional; README fully documents adaptable vs. fixed conventions. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +**Purpose**: Final repo-wide consistency checks that span multiple stories. + +- [X] T035 [P] Confirm Labs 1–3's existing registered APIs (museum, streetlights, train-travel) remain untouched and still registered after the new backend module is added, per Constitution Principle VI +- [X] T036 Run the full quickstart.md validation sequence (steps 1–8) end-to-end against a clean `yarn start` of `labs/lab-01-base-backstage/backstage/` to confirm the lab works from a cold start, not just incrementally during development +- [X] T037 [P] Re-verify the Constitution Check table in plan.md against the final implementation (all 9 principles) and correct plan.md if any gate's justification no longer matches what was built + +--- + +## Phase 7: Bug Fixes (from checklists/issues.md — Run 1, 2026/07/03) + +**Purpose**: Fix the reported "No APIs were available in the catalog after starting backstage" issue. Root cause analysis of the pasted log: + +- The `group:default/platform-team` owner-resolution failure (T020's validator, `InputError` on `galaxy-openapi.yaml`) is a *symptom*: `teams.yaml` — which defines `platform-team` — never loads, because its catalog location URL is 404ing. +- That 404, and the four other "Unable to read url" warnings (`teams.yaml`, `users.yaml`, `museum-api.yaml`, `streetlights-api.yaml`, `train-travel-api.yaml`... actually `platform-user.yaml` too), all point at `raw.githubusercontent.com/DawMatt/backstage-apiportal-lab/003-api-quality/...` — a stale feature-branch name left over from when Lab 3's instructions were followed. That content has since been merged to `main` (verified: present at `main:labs/lab-02-users-roles/catalog/teams.yaml` and `main:labs/lab-03-api-quality/catalog/platform-user.yaml`), so the branch-pinned URLs should point at `main` instead. +- With `teams.yaml` unreachable, `platform-team` never registers as a `Group`, so the Lab 4 owner validator (correctly) rejects `galaxy-openapi.yaml`'s `owner: group:default/platform-team` and emits a marker/error entity instead of a real one — leaving the catalog with no visible APIs. +- The one remaining branch-pinned location (`precedence-demo-catalog-info.yaml`, pointing at `004-lab-4-auto-registration`) is *not* a bug — it is new-on-this-branch content that genuinely requires the branch to be pushed to GitHub first, exactly as documented in `labs/lab-04-auto-registration/README.md` (the "This only works once you've pushed your branch" note). No code or config change fixes that one; it's a sequencing step the learner has to perform. + +- [X] T038 Fix the five stale `003-api-quality`-branch catalog location URLs in `labs/lab-01-base-backstage/backstage/app-config.yaml` (lines ~130, 136, 142, 148, 154, 160, plus the two `spectralLinter` ruleset URLs at ~206–207) to point at `main` instead, since that content is already merged to `main` +- [X] T039 [P] Add a troubleshooting entry to `labs/lab-04-auto-registration/README.md`'s Troubleshooting section explaining the observed failure chain (stale/unpushed branch in an org-data or API catalog location → that entity never registers → any auto-sourced API whose `x-examplecorp.owner` references a group defined only in that unreachable location fails owner validation → looks like "no APIs in the catalog" even though the real cause is upstream of Lab 4's own code) so learners can self-diagnose instead of assuming the `EntityProvider` is broken +- [X] T040 Restart `labs/lab-01-base-backstage/backstage/` from a clean `yarn start` after T038 and confirm via backend logs and the catalog UI: `platform-team` resolves (no more `InputError` on `galaxy-openapi.yaml`), museum/streetlights/train-travel APIs and the `platform-team`/`platform-user` org entities all load without "Unable to read url" warnings, and the Galaxy API entity appears with `spec.owner: group:default/platform-team` (SC-001/SC-002); note that `precedence-demo-api` will still be absent unless the current branch has been pushed to GitHub, per T039's documented caveat +- [X] T041 Update `specs/004-lab-4-auto-registration/checklists/issues.md`: mark the Run 1 item resolved with a one-line pointer to T038–T040 once T040's re-verification passes + +**Checkpoint**: Catalog populates on a clean start; the only remaining "gap" (precedence-demo-api before the branch is pushed) is expected and documented, not a bug. + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies — start immediately +- **Foundational (Phase 2)**: Depends on Setup completion — BLOCKS all user stories +- **User Story 1 (Phase 3)**: Depends on Foundational completion — no dependency on US2/US3 +- **User Story 2 (Phase 4)**: Depends on Foundational completion; extends the same file US1 built (T012), so in practice follows US1 sequentially even though it introduces no new *architectural* dependency +- **User Story 3 (Phase 5)**: Depends on US1 (T015) and US2 (T024) being implemented, since the README documents both the discovery mechanism and the metadata-sourcing/error behavior +- **Polish (Phase 6)**: Depends on all user stories being complete +- **Bug Fixes (Phase 7)**: Independent of Phases 3–6's code (config/docs-only fix) but logically follows Polish since it was found during Polish-phase (T036) cold-start validation + +### User Story Dependencies + +- **User Story 1 (P1)**: Can start after Foundational — no dependency on other stories +- **User Story 2 (P2)**: Can start after Foundational, but shares `autoApiRegistration.ts` with US1 (same file, sequential edits) — implement after US1 for a working MVP checkpoint first +- **User Story 3 (P3)**: Documentation-only; depends on US1 and US2 being functionally complete so the README describes real, working behavior + +### Within Each User Story + +- Mapping/extraction logic before mutation-emission logic before scheduling +- Sample files before manual verification steps +- Verification task last in each phase (checkpoint gate) + +### Parallel Opportunities + +- T002, T003 (Setup) in parallel +- T006, T007, T008 (Foundational: cache migration, discovery utility, parse utility) in parallel — distinct concerns, though ultimately land in the same file, so treat [P] as "logically independent," and serialize the actual edits if working solo +- T026, T027 (US2 sample files) in parallel with each other +- T031, T032 (US3 README sections) in parallel with each other +- T035, T037 (Polish) in parallel + +--- + +## Parallel Example: Foundational Phase + +```bash +# Logically independent foundational concerns (same target file — serialize edits if solo): +Task: "Implement scan-state cache DB migration in autoApiRegistrationMigrations/001_scan_state_cache.ts" +Task: "Implement streamed file discovery utility in autoApiRegistration.ts" +Task: "Implement YAML parse + shape-check utility in autoApiRegistration.ts" +``` + +## Parallel Example: User Story 2 Sample Files + +```bash +Task: "Create precedence-demo-openapi.yaml" +Task: "Create precedence-demo-catalog-info.yaml" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup +2. Complete Phase 2: Foundational (CRITICAL — blocks all stories) +3. Complete Phase 3: User Story 1 +4. **STOP and VALIDATE**: T017 confirms discovery/create/update/remove work end-to-end with default metadata +5. Demo: a new spec file appears in the catalog with zero hand-authored `catalog-info.yaml` + +### Incremental Delivery + +1. Setup + Foundational → scanning/parsing/persistence plumbing ready +2. Add User Story 1 → validate independently (T017) → MVP demo +3. Add User Story 2 → validate independently (T028) → owner/lifecycle/visibility sourcing, error surfacing, precedence all work +4. Add User Story 3 → README documents adaptability → lab is learner-ready +5. Polish → cold-start validation, constitution re-check + +### Notes + +- US1 and US2 share one implementation file (`autoApiRegistration.ts`) because the architecture (research.md R1) is a single `EntityProvider` — this is a deliberate single-module design, not an accidental cross-story coupling. Each story still has an independently verifiable acceptance test (T017 vs. T028) even though the code lands in the same file. +- No test tasks are included (Tests: Not requested — see header). Verification is manual, per quickstart.md, folded into each story's final task. +- `watch` mode (research.md R6) and the multi-source sync-step extension point (research.md R7 Problem 4) are documented (T031/T032) but not built or exercised end-to-end — consistent with quickstart.md's explicit "Out of scope for this quickstart" section.