diff --git a/appinfo/info.xml b/appinfo/info.xml index 9ed3f9871..d24f35cc5 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -23,7 +23,7 @@ - 📋 Pas bedrijfsregels toe op endpoint-verkeer en houd een audit trail per object bij ]]> - 0.3.13-unstable.20260831053109 + 0.3.16-unstable.20260831214826 EUPL-1.2 Conduction Integriq @@ -229,6 +229,17 @@ Last in the block, and post-migration only: it removes rows and creates nothing, so no step depends on it, and a fresh install has nothing to remove. --> + + OCA\Integriq\Repair\MigrateFlowStepsToGraph OCA\Integriq\Repair\RemoveRetiredCronJobs OCA\Integriq\Command\RuleToFlow + + OCA\Integriq\Command\FlowStepsToGraph diff --git a/docs/features/README.md b/docs/features/README.md index 3cbd7cea2..70cb12cbf 100644 --- a/docs/features/README.md +++ b/docs/features/README.md @@ -13,6 +13,7 @@ Integriq is an API gateway and integration hub for Nextcloud. It brings enterpri | [Reliability](reliability.md) | Retry policy, per-Source circuit breaker, sync dead letters | Implemented | | [Rules](rules.md) | Authentication, file handling, locking, and audit trail rules | Implemented | | [Jobs](jobs.md) | Cron-based scheduled task execution | Implemented | +| [Flow nodes](flow-nodes.md) | Contributed step types for OpenRegister's flow engine | Implemented | | [Events & Webhooks](events.md) | CloudEvents emission, subscription, and consumer processing | Implemented | | [Logging & Monitoring](logging.md) | Call logs, sync logs, and Prometheus metrics | Implemented | | [Configuration Management](configuration-management.md) | Import/export, configuration groups, slug-based references | Implemented | diff --git a/docs/features/flow-nodes.md b/docs/features/flow-nodes.md new file mode 100644 index 000000000..573ddc446 --- /dev/null +++ b/docs/features/flow-nodes.md @@ -0,0 +1,99 @@ +# Flow nodes + +## Overview + +OpenRegister runs the fleet's one flow engine. Integriq does not run its own graphs. It contributes step types, so a flow can do what Integriq is good at: call an API, run a synchronization, apply a mapping, ask a person, emit an event. + +You build the flow in OpenRegister's flow editor. The Integriq steps appear in the palette when both apps are enabled. + +| Node | What the step does | +|------|--------------------| +| `openconnector.source-call` | Make one governed API call per item through a configured Source | +| `openconnector.synchronization-run` | Run a configured Synchronization and hand each synchronised object onward | +| `openconnector.source-paginate` | Fetch one page of objects from a Source | +| `openconnector.apply-mapping` | Apply a configured Mapping to every item | +| `openconnector.contract` / `contract-commit` / `contract-sweep` | The decomposed synchronization's contract steps | +| `openconnector.fetch-file` | Fetch a file referenced by an item | +| `openconnector.approval-request` | Pause the run until someone approves or rejects | +| `openconnector.event-emit` | Emit a CloudEvent for every item | + +## Call an API from a flow + +Add a `source-call` step. Pick a Source, give it a path and a method: + +```json +{ + "id": "step-apply-label", + "type": "openconnector.source-call", + "config": { + "source": "demo-forge-api", + "endpoint": "/issues/{{issue.number}}/labels", + "method": "POST", + "body": { "labels": ["{{triage.proposedLabel}}"] }, + "output": "labelResult" + } +} +``` + +The step runs once per item. `{{dotted.path}}` placeholders resolve from each item's record, and the response lands under the key you name in `output`. The call goes through `CallService`, so the Source's enablement, host guard, rate limits and call logging all apply unchanged. + +## Why there is no raw-URL node + +You cannot type a URL into a flow step. The step names a Source, and the endpoint is a path inside that Source's location. An absolute URL, a `//host` path or a `../` escape is rejected before any request goes out. + +This is the whole security model, not a missing convenience. A Source is where an administrator decides which hosts may be called, how often, and with which credential. A URL field in a flow document would hand that decision to every flow author and turn the editor into a request forger. If a host is worth calling, give it a Source first. + +Credentials follow the same line. A step has no token field. Authentication comes from the Source's `credentialRef`, resolved by the credential broker at call time. No secret ever sits in a flow document. + +## Why an unattributed run fails closed + +Every call runs as the flow run's owner, read from the run context. When no owner resolves, the step refuses and raises. There is no fallback to an admin, to the Source's creator, or to nobody. + +An anonymous authenticated outbound call is the failure we refuse to ship. A loud error names the gap; a silent fallback hides it behind someone else's identity. + +## Ask a person: the approval step + +`openconnector.approval-request` parks the run and creates a pending approval request. The approvers see it on the Pending approvals page and in their shared task list, like every other Integriq approval. + +```json +{ + "id": "approve-publish", + "type": "openconnector.approval-request", + "config": { + "question": "Publish this dataset?", + "approverGroup": "data-stewards", + "ttlSeconds": 86400 + } +} +``` + +- **Approved.** The run resumes. The decision, the approver and the comment land on every item under `approval`, so a later step can route on them. +- **Rejected.** By default the run continues and your reject edge reads `approval.decision`. Set `failOnReject: true` when a no should fail the run. +- **Expired.** The run fails. An approval nobody answered never counts as answered. + +An answer wakes the run immediately. If that wake-up is ever lost, the step re-checks the approval request itself on its next heartbeat, so a decision is never stranded. + +## Emit an event + +`openconnector.event-emit` sends one CloudEvent per item through the existing event pipeline. Name a `type` and a `source`, and subscriptions pick it up exactly as they would for any other Integriq event. + +## Migrate old step-list flows + +Flows built in Integriq's earlier step-list editor still exist as ordered `steps[]`. One command translates them onto the engine's graph shape: + +```bash +occ integriq:flow:steps-to-graph # dry run: reports what would happen +occ integriq:flow:steps-to-graph --apply # writes nodes/edges onto each flow +``` + +The migration is additive and repeatable. `steps` stays on the object, a flow that already carries `nodes` is skipped, and a flow the translator cannot express faithfully is refused with the reasons listed. The same pass also runs automatically on upgrade. + +Changed your mind? Roll it back: + +```bash +occ integriq:flow:steps-to-graph --rollback --apply +``` + +## Next steps + +Create a [Source](sources.md) for the API you want to call, then open OpenRegister's flow editor and add a `source-call` step against it. diff --git a/eslint.config.mjs b/eslint.config.mjs index a4bbb8c76..d6b672728 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -144,11 +144,44 @@ export default [ 'no-console': 'off', 'n/no-process-exit': 'off', 'n/hashbang': 'off', - // `_` / `__` as a deliberate throwaway binding — `catch (_)`, a - // discarded destructuring slot. Narrow on purpose: the pattern matches - // UNDERSCORES ONLY, so a real name that happens to start with `_` is - // still reported. v9 drives plain `.js` through the CORE rule (the - // `@typescript-eslint` swap is per-file-type), so it is set here. + // Tests import devDependencies by definition; this rule is about what + // ships in the published package, which tests/ never does. + 'n/no-unpublished-import': 'off', + // 🔴 COMMENTS ONLY, and the exception is load-bearing. Four + // api-direct specs document the `testIgnore` glob that excludes them + // from the gate-19 UI run, and a block comment cannot contain the + // literal `**` + `/` because that closes it at `*/`. The files carry a + // ZERO-WIDTH SPACE (U+200B) between the two to break the sequence. + // + // Deleting the character to satisfy the rule would terminate the + // comment early and break the file. In CODE an invisible character is + // a genuine hazard — a look-alike inside an identifier or a string — + // and the rule still catches that. In a comment it cannot change + // behaviour. + 'no-irregular-whitespace': ['error', { skipComments: true }], + }, + }, + + { + // `_` / `__` as a deliberate throwaway binding — `catch (_)`, a discarded + // destructuring slot. Narrow on purpose: the pattern matches UNDERSCORES + // ONLY, so a real name that happens to start with `_` is still reported. + // + // 🔴 `.js` / `.mjs` ONLY, NOT `.ts`. The CORE rule is not TypeScript-aware: + // applied to a `.ts` file it reads the parameter NAMES inside a function + // TYPE as bindings and reports them unused. Measured on humaniq — + // + // t?: (app: string, key: string) => string + // + // produced four `no-unused-vars` errors for `app` and `key`, which are + // documentation, not variables. The same mis-scoping made every unused + // `catch (e)` in a `.ts` spec report TWICE, once per rule. + // + // v9 already turns the core rule off for `.ts` and drives + // `@typescript-eslint/no-unused-vars` instead; naming `.ts` here switched + // it back on. TypeScript files are handled by the block below. + files: ['tests/**/*.js', 'tests/**/*.mjs'], + rules: { 'no-unused-vars': [ 'error', { @@ -165,12 +198,117 @@ export default [ ignoreRestSiblings: true, }, ], - // Tests import devDependencies by definition; this rule is about what - // ships in the published package, which tests/ never does. - 'n/no-unpublished-import': 'off', }, }, + { + // The TypeScript half of the block above. Same intent, same patterns, on + // the rule that actually understands the language: it knows a name inside + // a function type is not a binding, so type annotations stay quiet while a + // genuinely dead local is still reported. + files: ['tests/**/*.ts', 'tests/**/*.tsx'], + rules: { + '@typescript-eslint/no-unused-vars': [ + 'error', + { + varsIgnorePattern: '^_+$', + caughtErrors: 'all', + caughtErrorsIgnorePattern: '^_+$', + argsIgnorePattern: '^_', + ignoreRestSiblings: true, + }, + ], + }, + }, + + { + // 🔴 Node-side CLI tooling under `scripts/`, which is COMMONJS. Flat + // config defaults every `.js` to ESM with browser-ish globals, so without + // this block eslint reports the CommonJS wrapper itself as undefined + // identifiers. Measured on this app: 52 of the 233 errors under + // `tests/` + `scripts/` were `no-undef`, ALL of them in `scripts/`, and + // all five names were the environment rather than a typo — `process` 23, + // `require` 20, `__dirname` 6, `__filename` 2, `module` 1. + // + // This is describing the environment, not relaxing a rule, and it is the + // same argument the test-globals block below makes: declaring them keeps + // `no-undef` able to do its real job, which is catching a genuinely + // misspelled identifier. Suppressing the rule instead would bury that. + // + // `no-console` is off because printing its report is what a CLI checker + // is FOR. + // + // 🔴 NO `n/*` ENTRIES HERE, DELIBERATELY. `eslint-plugin-n` is NOT + // registered for these files under eslint 10 + @nextcloud/eslint-config + // 9, so `'n/no-process-exit': 'off'` would be dead config that reads as + // if it were doing something. Measured both ways on this app: 0 `n/` + // findings with the entries and 0 without. + // + // What DID report was the opposite — four `scripts/*.js` carried + // `/* eslint-disable n/no-process-exit */` and `/* eslint-disable + // n/shebang */` left over from the eslintrc era, and an inline disable + // naming an unregistered plugin is itself an error ("Definition for rule + // 'n/shebang' was not found"). Those 8 comments are removed; do not add + // `n/*` rules back to replace them. + // + // ⚠️ `.js` and `.cjs` ONLY. A `scripts/*.mjs` is genuinely ESM and must + // keep the default `sourceType`, or `import` stops parsing there. + files: ['scripts/**/*.js', 'scripts/**/*.cjs'], + languageOptions: { + sourceType: 'commonjs', + globals: { + require: 'readonly', + module: 'writable', + exports: 'writable', + process: 'readonly', + __dirname: 'readonly', + __filename: 'readonly', + console: 'readonly', + Buffer: 'readonly', + global: 'readonly', + URL: 'readonly', + TextEncoder: 'readonly', + TextDecoder: 'readonly', + }, + }, + rules: { + 'no-console': 'off', + }, + }, + + { + // The ESM half of the block above. A `scripts/*.mjs` is genuinely a module + // and must keep the default `sourceType`, so it gets Node's globals but + // none of the CommonJS wrapper. Measured: `process` reported undefined 2x + // in hermiq's generate-opengemeenten-icons.mjs and 4x in openregister's + // l10n/runtime-check.mjs, which the `.js`/`.cjs` block deliberately does + // not match. + files: ['scripts/**/*.mjs', 'tests/**/*.mjs'], + languageOptions: { + globals: { + process: 'readonly', + console: 'readonly', + Buffer: 'readonly', + global: 'readonly', + URL: 'readonly', + TextEncoder: 'readonly', + TextDecoder: 'readonly', + }, + }, + rules: { + 'no-console': 'off', + }, + }, + + { + // eslint must not try to PARSE a shell script. `tests/e2e/seed.test.sh` + // matches the `**/*.test.*` glob some presets use, and eslint then reads + // it as JavaScript and reports "Parsing error: Unexpected character" — + // a finding about a file it should never have opened. + ignores: ['**/*.sh', '**/*.bash'], + }, + + // eslint-config-prettier LAST OF THE PRESETS, and it has to be: it only turns // rules OFF, and what it turns off is everything prettier owns — including the // `@stylistic/*` family v9 introduces (`indent`, `quotes`, `semi`). diff --git a/l10n/nl.js b/l10n/nl.js index a457d105a..abf663c11 100644 --- a/l10n/nl.js +++ b/l10n/nl.js @@ -1516,7 +1516,38 @@ OC.L10N.register( "Open the documentation to keep going": "Open de documentatie om verder te gaan", "Where the automation lives": "Waar de automatisering zit", "Flows are what happens without anyone clicking: a synchronization that starts on a schedule, a webhook that fires when a record changes. This is where you read and edit them. Nothing to build now.": "Flows zijn wat er gebeurt zonder dat iemand klikt: een synchronisatie die op een schema start, een webhook die afgaat wanneer een record verandert. Hier leest en bewerkt u ze. U hoeft nu niets te bouwen.", - "Open Flows in the menu": "Open Flows in het menu" + "Open Flows in the menu": "Open Flows in het menu", + "Ask for approval": "Vraag om goedkeuring", + "Pause the flow until someone in the approver group approves or rejects. An expired request fails the run.": "Pauzeer de flow tot iemand uit de goedkeurdersgroep goedkeurt of afwijst. Een verlopen verzoek laat de run mislukken.", + "What is being asked": "Wat wordt er gevraagd", + "Shown to the approvers and written on the request, so a paused flow explains itself.": "Getoond aan de goedkeurders en vastgelegd op het verzoek, zodat een gepauzeerde flow zichzelf uitlegt.", + "Members of this group (and admins) may answer. Required: a request nobody owns is a request nobody answers.": "Leden van deze groep (en beheerders) mogen antwoorden. Verplicht: een verzoek zonder eigenaar wordt nooit beantwoord.", + "Expires after (seconds)": "Verloopt na (seconden)", + "An unanswered request expires and fails the run. Defaults to 24 hours.": "Een onbeantwoord verzoek verloopt en laat de run mislukken. Standaard 24 uur.", + "Treat a rejection as a failure": "Behandel een afwijzing als een fout", + "Off by default: a \"no\" continues the flow with the decision on the items, so a later step can route on it.": "Standaard uit: een \"nee\" laat de flow doorgaan met het besluit op de items, zodat een latere stap erop kan routeren.", + "Field to store the decision in": "Veld waarin het besluit wordt opgeslagen", + "The decision is written onto every item under this field. Defaults to \"approval\".": "Het besluit wordt onder dit veld op elk item geschreven. Standaard \"approval\".", + "Re-check every (minutes)": "Controleer opnieuw elke (minuten)", + "Safety net for a lost answer. Lower is not faster: a decision wakes the run immediately either way.": "Vangnet voor een verloren antwoord. Lager is niet sneller: een besluit wekt de run hoe dan ook direct.", + "Say what is being asked (\"question\"), or nobody can answer it.": "Zeg wat er wordt gevraagd (\"question\"), anders kan niemand antwoorden.", + "Name the approver group (\"approverGroup\"): an approval without an audience never resolves.": "Noem de goedkeurdersgroep (\"approverGroup\"): een goedkeuring zonder publiek wordt nooit afgerond.", + "The \"ttlSeconds\" field must be a positive number of seconds when set.": "Het veld \"ttlSeconds\" moet een positief aantal seconden zijn wanneer het is ingesteld.", + "The approval step has no resume slot; the engine did not dispatch it as a resumable node.": "De goedkeuringsstap heeft geen hervattingsslot; de engine heeft hem niet als hervatbare stap aangeroepen.", + "This run carries no uuid, so an approval could never answer it. The approval step is only usable in a persisted flow run.": "Deze run heeft geen uuid, dus een goedkeuring zou hem nooit kunnen beantwoorden. De goedkeuringsstap werkt alleen in een opgeslagen flowrun.", + "Emit an event": "Verstuur een gebeurtenis", + "Emit a CloudEvent for every item, delivered through the configured event subscriptions.": "Verstuur een CloudEvent voor elk item, afgeleverd via de geconfigureerde abonnementen.", + "The CloudEvent \"type\", e.g. \"nl.example.object.updated\". Subscriptions match on it.": "Het CloudEvent-\"type\", bijvoorbeeld \"nl.example.object.updated\". Abonnementen matchen erop.", + "Event source": "Gebeurtenisbron", + "The CloudEvent \"source\" URI identifying the emitter.": "De CloudEvent-\"source\"-URI die de verzender identificeert.", + "Subject": "Onderwerp", + "Optional CloudEvent \"subject\". Supports {{dotted.path}} placeholders resolved from each item.": "Optioneel CloudEvent-\"subject\". Ondersteunt {{dotted.path}}-plaatshouders, opgelost per item.", + "Output key": "Uitvoersleutel", + "Item key the emit summary is written under. Defaults to \"eventResult\".": "Itemsleutel waaronder de verstuursamenvatting wordt geschreven. Standaard \"eventResult\".", + "Name the event \"type\": an event without a type matches no subscription.": "Noem het gebeurtenis-\"type\": een gebeurtenis zonder type matcht geen enkel abonnement.", + "Name the event \"source\": a CloudEvent must say where it came from.": "Noem de gebeurtenis-\"source\": een CloudEvent moet zeggen waar hij vandaan komt.", + "Step \"%1$s\" failed to emit event \"%2$s\": %3$s": "Stap \"%1$s\" kon gebeurtenis \"%2$s\" niet versturen: %3$s", + "The flow \"%1$s\" cannot be migrated to a graph yet: %2$s unsupported feature(s).": "De flow \"%1$s\" kan nog niet naar een graaf worden gemigreerd: %2$s niet-ondersteunde functie(s)." }, "nplurals=2; plural=(n != 1);" ) diff --git a/l10n/nl.json b/l10n/nl.json index b7ef76044..439b5d0d5 100644 --- a/l10n/nl.json +++ b/l10n/nl.json @@ -1515,7 +1515,38 @@ "Open the documentation to keep going": "Open de documentatie om verder te gaan", "Where the automation lives": "Waar de automatisering zit", "Flows are what happens without anyone clicking: a synchronization that starts on a schedule, a webhook that fires when a record changes. This is where you read and edit them. Nothing to build now.": "Flows zijn wat er gebeurt zonder dat iemand klikt: een synchronisatie die op een schema start, een webhook die afgaat wanneer een record verandert. Hier leest en bewerkt u ze. U hoeft nu niets te bouwen.", - "Open Flows in the menu": "Open Flows in het menu" + "Open Flows in the menu": "Open Flows in het menu", + "Ask for approval": "Vraag om goedkeuring", + "Pause the flow until someone in the approver group approves or rejects. An expired request fails the run.": "Pauzeer de flow tot iemand uit de goedkeurdersgroep goedkeurt of afwijst. Een verlopen verzoek laat de run mislukken.", + "What is being asked": "Wat wordt er gevraagd", + "Shown to the approvers and written on the request, so a paused flow explains itself.": "Getoond aan de goedkeurders en vastgelegd op het verzoek, zodat een gepauzeerde flow zichzelf uitlegt.", + "Members of this group (and admins) may answer. Required: a request nobody owns is a request nobody answers.": "Leden van deze groep (en beheerders) mogen antwoorden. Verplicht: een verzoek zonder eigenaar wordt nooit beantwoord.", + "Expires after (seconds)": "Verloopt na (seconden)", + "An unanswered request expires and fails the run. Defaults to 24 hours.": "Een onbeantwoord verzoek verloopt en laat de run mislukken. Standaard 24 uur.", + "Treat a rejection as a failure": "Behandel een afwijzing als een fout", + "Off by default: a \"no\" continues the flow with the decision on the items, so a later step can route on it.": "Standaard uit: een \"nee\" laat de flow doorgaan met het besluit op de items, zodat een latere stap erop kan routeren.", + "Field to store the decision in": "Veld waarin het besluit wordt opgeslagen", + "The decision is written onto every item under this field. Defaults to \"approval\".": "Het besluit wordt onder dit veld op elk item geschreven. Standaard \"approval\".", + "Re-check every (minutes)": "Controleer opnieuw elke (minuten)", + "Safety net for a lost answer. Lower is not faster: a decision wakes the run immediately either way.": "Vangnet voor een verloren antwoord. Lager is niet sneller: een besluit wekt de run hoe dan ook direct.", + "Say what is being asked (\"question\"), or nobody can answer it.": "Zeg wat er wordt gevraagd (\"question\"), anders kan niemand antwoorden.", + "Name the approver group (\"approverGroup\"): an approval without an audience never resolves.": "Noem de goedkeurdersgroep (\"approverGroup\"): een goedkeuring zonder publiek wordt nooit afgerond.", + "The \"ttlSeconds\" field must be a positive number of seconds when set.": "Het veld \"ttlSeconds\" moet een positief aantal seconden zijn wanneer het is ingesteld.", + "The approval step has no resume slot; the engine did not dispatch it as a resumable node.": "De goedkeuringsstap heeft geen hervattingsslot; de engine heeft hem niet als hervatbare stap aangeroepen.", + "This run carries no uuid, so an approval could never answer it. The approval step is only usable in a persisted flow run.": "Deze run heeft geen uuid, dus een goedkeuring zou hem nooit kunnen beantwoorden. De goedkeuringsstap werkt alleen in een opgeslagen flowrun.", + "Emit an event": "Verstuur een gebeurtenis", + "Emit a CloudEvent for every item, delivered through the configured event subscriptions.": "Verstuur een CloudEvent voor elk item, afgeleverd via de geconfigureerde abonnementen.", + "The CloudEvent \"type\", e.g. \"nl.example.object.updated\". Subscriptions match on it.": "Het CloudEvent-\"type\", bijvoorbeeld \"nl.example.object.updated\". Abonnementen matchen erop.", + "Event source": "Gebeurtenisbron", + "The CloudEvent \"source\" URI identifying the emitter.": "De CloudEvent-\"source\"-URI die de verzender identificeert.", + "Subject": "Onderwerp", + "Optional CloudEvent \"subject\". Supports {{dotted.path}} placeholders resolved from each item.": "Optioneel CloudEvent-\"subject\". Ondersteunt {{dotted.path}}-plaatshouders, opgelost per item.", + "Output key": "Uitvoersleutel", + "Item key the emit summary is written under. Defaults to \"eventResult\".": "Itemsleutel waaronder de verstuursamenvatting wordt geschreven. Standaard \"eventResult\".", + "Name the event \"type\": an event without a type matches no subscription.": "Noem het gebeurtenis-\"type\": een gebeurtenis zonder type matcht geen enkel abonnement.", + "Name the event \"source\": a CloudEvent must say where it came from.": "Noem de gebeurtenis-\"source\": een CloudEvent moet zeggen waar hij vandaan komt.", + "Step \"%1$s\" failed to emit event \"%2$s\": %3$s": "Stap \"%1$s\" kon gebeurtenis \"%2$s\" niet versturen: %3$s", + "The flow \"%1$s\" cannot be migrated to a graph yet: %2$s unsupported feature(s).": "De flow \"%1$s\" kan nog niet naar een graaf worden gemigreerd: %2$s niet-ondersteunde functie(s)." }, "plurals": {} } diff --git a/lib/AppInfo/Application.php b/lib/AppInfo/Application.php index 1f8c71612..f5fcb1a28 100644 --- a/lib/AppInfo/Application.php +++ b/lib/AppInfo/Application.php @@ -42,7 +42,9 @@ use OCA\Integriq\Capabilities; use OCA\Integriq\Controller\HealthController; use OCA\Integriq\Controller\MetricsController; +use OCA\Integriq\Event\DeliveryRequestedEvent; use OCA\Integriq\EventListener\CloudEventListener; +use OCA\Integriq\EventListener\DeliveryRequestedListener; use OCA\Integriq\EventListener\EndpointCacheInvalidationListener; use OCA\Integriq\EventListener\NextcloudCalendarEventListener; use OCA\Integriq\EventListener\NextcloudFileEventListener; @@ -199,6 +201,11 @@ function ($c) { $dispatcher->addServiceListener(eventName: ObjectCreatedEvent::class, className: CloudEventListener::class); $dispatcher->addServiceListener(eventName: ObjectUpdatedEvent::class, className: CloudEventListener::class); $dispatcher->addServiceListener(eventName: ObjectDeletedEvent::class, className: CloudEventListener::class); + // ADR-041 cross-app delivery seam: a sibling app (dossiq, ...) raises + // a typed DeliveryRequestedEvent; this listener ingests it into the + // same CloudEvents pipeline (subscription routing, retry, dead-letter, + // replay) and writes the synchronous result slot back on the event. + $dispatcher->addServiceListener(eventName: DeliveryRequestedEvent::class, className: DeliveryRequestedListener::class); // Nextcloud-core-event triggers (nextcloud-event-hub). Each family // normalizes its NC event into the SAME `event` CloudEvents envelope // shape the OR-object pipeline above already uses, then hands off to diff --git a/lib/Command/FlowStepsToGraph.php b/lib/Command/FlowStepsToGraph.php new file mode 100644 index 000000000..6e8719d30 --- /dev/null +++ b/lib/Command/FlowStepsToGraph.php @@ -0,0 +1,154 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Command; + +use OCA\Integriq\Service\FlowGraphMigrationService; +use Symfony\Component\Console\Command\Command; +use Symfony\Component\Console\Input\InputInterface; +use Symfony\Component\Console\Input\InputOption; +use Symfony\Component\Console\Output\OutputInterface; +use Symfony\Component\Console\Style\SymfonyStyle; + +/** + * Migrates (or rolls back) live flow objects between steps and graph shape. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ +class FlowStepsToGraph extends Command { + + /** + * Constructor. + * + * @param FlowGraphMigrationService $migration The shared migration behaviour. + */ + public function __construct( + private readonly FlowGraphMigrationService $migration, + ) { + parent::__construct(); + + }//end __construct() + + /** + * Configure the command name, description and options. + * + * @return void + * + * @spec exclude Symfony console wiring — framework metadata, no domain behavior. + */ + protected function configure(): void { + $this->setName(name: 'integriq:flow:steps-to-graph') + ->setDescription( + 'Translate live flow objects from steps[] to the OpenRegister nodes/edges graph (dry run unless --apply)' + ) + ->addOption( + 'apply', + null, + InputOption::VALUE_NONE, + 'Write the changes; without it the command only reports what would happen' + ) + ->addOption( + 'rollback', + null, + InputOption::VALUE_NONE, + 'Remove the written nodes/edges again, leaving steps[] as the only shape' + ); + + }//end configure() + + /** + * Run the migration (or its rollback) and print one row per flow. + * + * @param InputInterface $input Console input. + * @param OutputInterface $output Console output. + * + * @return integer 0 when every flow migrated or was already done; 1 when any flow was refused. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + protected function execute(InputInterface $input, OutputInterface $output): int { + $io = new SymfonyStyle($input, $output); + $apply = ($input->getOption('apply') === true); + $report = $this->reportFor(input: $input, apply: $apply); + + if ($apply === false) { + $io->note('Dry run — nothing was written. Re-run with --apply to write.'); + } + + $refusals = 0; + foreach ($report as $row) { + $line = sprintf('[%s] %s (%s)', $row['action'], $row['name'], $row['id']); + $output->writeln($line); + + if ($row['action'] === FlowGraphMigrationService::REFUSED) { + $refusals++; + foreach ($row['reasons'] as $reason) { + $output->writeln(' - ' . $reason); + } + } + } + + $io->success(sprintf('%d flow(s) inspected, %d refused.', count($report), $refusals)); + + if ($refusals > 0) { + return Command::FAILURE; + } + + return Command::SUCCESS; + }//end execute() + + /** + * Run the direction the flags asked for. + * + * @param InputInterface $input Console input. + * @param bool $apply Whether to write. + * + * @return array}> One row per flow. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function reportFor(InputInterface $input, bool $apply): array { + if ($input->getOption('rollback') === true) { + return $this->migration->rollback(apply: $apply); + } + + return $this->migration->migrate(apply: $apply); + + }//end reportFor() +}//end class diff --git a/lib/Controller/ApprovalsController.php b/lib/Controller/ApprovalsController.php index 112466732..8588e6b4b 100644 --- a/lib/Controller/ApprovalsController.php +++ b/lib/Controller/ApprovalsController.php @@ -38,6 +38,7 @@ use OCA\Integriq\Service\ActionAuthService; use OCA\Integriq\Service\ApprovalService; use OCA\Integriq\Service\EndpointService; +use OCA\Integriq\Service\EngineSignalService; use OCA\Integriq\Service\FlowRunnerService; use OCA\Integriq\Service\SynchronizationService; use OCA\OpenRegister\Db\ObjectEntity; @@ -80,6 +81,10 @@ class ApprovalsController extends Controller { * @param IUserSession $userSession The user session. * @param IL10N $l The localization service. * @param LoggerInterface $logger Logger for non-fatal diagnostics. + * @param EngineSignalService|null $engineSignal Delivers approval decisions to suspended + * OpenRegister engine runs (retire-integriq-flow-schema + * Task 1). Nullable + defaulted so pre-existing + * positional test instantiations keep working. */ public function __construct( string $appName, @@ -93,6 +98,7 @@ public function __construct( private readonly IUserSession $userSession, private readonly IL10N $l, private readonly LoggerInterface $logger, + private readonly ?EngineSignalService $engineSignal = null, ) { parent::__construct(appName: $appName, request: $request); @@ -198,7 +204,26 @@ public function approve(string $id): JSONResponse { return new JSONResponse(['error' => $e->getMessage()], $e->getHttpStatus()); } - $comment = $this->request->getParam('comment'); + return $this->routeApproval( + approvalRequest: $approvalRequest, + user: $user, + comment: $this->request->getParam('comment') + ); + + }//end approve() + + /** + * Dispatch an authorized approve to the resume path its FK selects. + * + * @param ObjectEntity $approvalRequest The pending, authorized-to-act-on request. + * @param IUser $user The approving user. + * @param string|null $comment Optional approve comment. + * + * @return JSONResponse + * + * @spec openspec/specs/approval-workflow/spec.md + */ + private function routeApproval(ObjectEntity $approvalRequest, IUser $user, ?string $comment): JSONResponse { $data = $approvalRequest->getObject(); if (empty($data['endpointId']) === false) { @@ -213,9 +238,16 @@ public function approve(string $id): JSONResponse { return $this->approveFlowSuspension(approvalRequest: $approvalRequest, user: $user, comment: $comment); } - $this->logger->error('ApprovalsController: approval_request has neither endpointId, synchronizationId nor flowRunId', ['id' => $id]); + if (empty($data['engineRunUuid']) === false) { + return $this->approveEngineSuspension(approvalRequest: $approvalRequest, data: $data, user: $user, comment: $comment); + } + + $this->logger->error( + 'ApprovalsController: approval_request has neither endpointId, synchronizationId, flowRunId nor engineRunUuid', + ['id' => $approvalRequest->getUuid()] + ); return new JSONResponse(['error' => $this->l->t('Malformed approval request')], Http::STATUS_INTERNAL_SERVER_ERROR); - }//end approve() + }//end routeApproval() /** * Reject a `pending`, non-expired approval_request. Self-contained in @@ -267,13 +299,7 @@ public function reject(string $id): JSONResponse { $data = $approvalRequest->getObject(); - // Flow-sourced suspension (flowRunId set): stop the flow_run — no - // pipeline to re-invoke here (self-contained, per ApprovalService::reject()'s - // own docblock), but the flow_run's OWN status must still reflect the - // rejection (flow-orchestration REQ-005). - if (empty($data['flowRunId']) === false) { - $this->flowRunnerService->stopFromApprovalOutcome(approvalRequest: $approvalRequest); - } + $this->propagateRejection(approvalRequest: $approvalRequest, data: $data, user: $user, comment: $comment); return new JSONResponse( [ @@ -286,6 +312,40 @@ public function reject(string $id): JSONResponse { }//end reject() + /** + * Let the suspended run reflect a rejection, per its FK kind. + * + * Flow-sourced suspension (flowRunId): stop the app-local flow_run — no + * pipeline to re-invoke (self-contained, per `ApprovalService::reject()`'s + * own docblock), but the flow_run's OWN status must still reflect the + * rejection (flow-orchestration REQ-005). + * + * Engine-run suspension (engineRunUuid): wake the suspended OpenRegister + * run with the rejection so the approval node routes or fails it now. + * Best-effort by design — the record IS the decision, and the node's + * heartbeat re-reads it, so a lost signal costs one heartbeat rather + * than the flow. + * + * @param ObjectEntity $approvalRequest The just-rejected request. + * @param array $data The approval_request's object data. + * @param IUser $user The rejecting user. + * @param string $comment The rejection comment. + * + * @return void + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function propagateRejection(ObjectEntity $approvalRequest, array $data, IUser $user, string $comment): void { + if (empty($data['flowRunId']) === false) { + $this->flowRunnerService->stopFromApprovalOutcome(approvalRequest: $approvalRequest); + } + + if (empty($data['engineRunUuid']) === false) { + $this->signalEngineRun(data: $data, decision: 'rejected', user: $user, comment: $comment); + } + + }//end propagateRejection() + /** * Resume a suspended endpoint rule-pipeline run and finalize the * approval_request with the resumed chain's outcome. @@ -454,6 +514,87 @@ private function approveFlowSuspension(ObjectEntity $approvalRequest, IUser $use return new JSONResponse($flowRunData, $statusCode); }//end approveFlowSuspension() + /** + * Resolve an ENGINE-run approval: finalize the approval_request, then + * wake the suspended OpenRegister flow run with the decision. + * + * The order is deliberate. The record is resolved FIRST because it is + * the system of record — the approval node's heartbeat re-reads it, so + * a signal that fails to deliver (OpenRegister mid-upgrade, run already + * woken) only delays the resume by one heartbeat instead of losing the + * decision. `resumeResult` therefore reports the DELIVERY, not the run's + * eventual outcome, which the engine owns. + * + * @param ObjectEntity $approvalRequest The pending, authorized-to-act-on request. + * @param array $data The approval_request's object data. + * @param IUser $user The approving user. + * @param string|null $comment Optional approve comment. + * + * @return JSONResponse + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function approveEngineSuspension(ObjectEntity $approvalRequest, array $data, IUser $user, ?string $comment): JSONResponse { + $signalled = $this->signalEngineRun(data: $data, decision: 'approved', user: $user, comment: $comment); + + $resumeResult = 'error'; + if ($signalled === true) { + $resumeResult = 'success'; + } + + $approvalRequest = $this->approvalService->completeApproval( + approvalRequest: $approvalRequest, + approver: $user, + resumeResult: $resumeResult, + comment: $comment + ); + + $approvalRequestData = $approvalRequest->getObject(); + + return new JSONResponse( + [ + 'engineRunUuid' => (string)($data['engineRunUuid'] ?? ''), + 'signalled' => $signalled, + '_approval' => [ + 'id' => $approvalRequest->getUuid(), + 'status' => ($approvalRequestData['status'] ?? 'approved'), + 'resumedAt' => ($approvalRequestData['approvedAt'] ?? null), + ], + ] + ); + + }//end approveEngineSuspension() + + /** + * Deliver a decision to a suspended OpenRegister engine run, guarded. + * + * Delegates to {@see EngineSignalService::deliver()} so the approve and + * reject paths ship the identical signal. The service dependency is + * defaulted (nullable) so pre-existing positional test instantiations + * keep working; the container always injects it in production. + * + * @param array $data The approval_request's object data (`engineRunUuid`/`signalNodeId`). + * @param string $decision `approved` or `rejected`. + * @param IUser $user The deciding user. + * @param string|null $comment Optional decision comment. + * + * @return boolean True when the signal was delivered. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function signalEngineRun(array $data, string $decision, IUser $user, ?string $comment): bool { + if ($this->engineSignal === null) { + $this->logger->warning( + 'ApprovalsController: no EngineSignalService wired; the engine run resumes on its next heartbeat instead', + ['engineRunUuid' => ($data['engineRunUuid'] ?? '')] + ); + return false; + } + + return $this->engineSignal->deliver(data: $data, decision: $decision, user: $user, comment: $comment); + + }//end signalEngineRun() + /** * Build the `_approval`-enveloped response body for a resumed endpoint response. * diff --git a/lib/Controller/MappingsController.php b/lib/Controller/MappingsController.php index 187960771..231633960 100644 --- a/lib/Controller/MappingsController.php +++ b/lib/Controller/MappingsController.php @@ -295,11 +295,26 @@ public function saveObject(): ?JSONResponse { return new JSONResponse(['error' => $this->l->t('Missing required `object` field')], 400); } + // 🔴 THE MAPPING RESULT MAY NOT ADDRESS AN EXISTING OBJECT. + // + // This endpoint saves the OUTPUT OF A MAPPING TEST as a new object — + // the UI button is "save result as object". A mapping transforms source + // data, and source data very often carries an `id`. `saveObject()` + // resolves its target from the payload (`@self.id` first, then `id`) + // and the write is PUT-semantic, so a result carrying either would + // silently REPLACE whatever object shares that identifier, nulling + // every field the result omitted, and report success. + // + // Identity here belongs to the new object, not to the source record the + // mapping happened to read. + $object = (array)$data['object']; + unset($object['id'], $object['uuid'], $object['@self']); + // OR's ObjectService::saveObject signature is `(object, register?, // schema?)`. Prior code passed the register slug as the first arg // — a TypeError under the new signature, which surfaced as 500. $saved = $openRegisters->saveObject( - object: $data['object'], + object: $object, register: ($data['register'] ?? 'openconnector'), schema: ($data['schema'] ?? 'mapping') ); diff --git a/lib/Event/DeliveryConcludedEvent.php b/lib/Event/DeliveryConcludedEvent.php new file mode 100644 index 000000000..545ce7da1 --- /dev/null +++ b/lib/Event/DeliveryConcludedEvent.php @@ -0,0 +1,198 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-License-Identifier: EUPL-1.2 + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * + * @version GIT: + * + * @link https://conduction.nl + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Event; + +use OCP\EventDispatcher\Event; + +/** + * Terminal outcome of a cross-app delivery request. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + * + * @SuppressWarnings(PHPMD.ExcessiveParameterList) -- the ADR-041 event contract is a flat + * readonly provenance envelope (sourceApp, subject coordinates, kind/channel, correlation); + * folding fields into an array would untype the contract the consumer stubs must mirror + * verbatim. Mirrors the decidiq DecisionRequestedEvent precedent. + */ +class DeliveryConcludedEvent extends Event { + /** + * Terminal status: the delivery succeeded. + */ + public const STATUS_DELIVERED = 'delivered'; + + /** + * Terminal status: the retry budget is spent, no further attempts. + */ + public const STATUS_ABANDONED = 'abandoned'; + + /** + * Constructor. + * + * @param string $sourceApp The app that raised the original request. + * @param string $correlationId The caller's correlation id, echoed verbatim. + * @param string $subjectId The subject object id from the original request. + * @param string $channel The delivery channel from the original request. + * @param string $status Terminal status: {@see self::STATUS_DELIVERED} or {@see self::STATUS_ABANDONED}. + * @param string $eventId Uuid of the CloudEvent `event` object. + * @param string $messageId Uuid of the `event_message` delivery record. + * @param int $attempts How many delivery attempts were made. + * @param string|null $error The last delivery error, or null on success. + * @param string $concludedAt ISO 8601 timestamp of the terminal transition. + * + * @return void + */ + public function __construct( + private readonly string $sourceApp, + private readonly string $correlationId, + private readonly string $subjectId, + private readonly string $channel, + private readonly string $status, + private readonly string $eventId, + private readonly string $messageId, + private readonly int $attempts, + private readonly ?string $error, + private readonly string $concludedAt, + ) { + parent::__construct(); + }//end __construct() + + /** + * The app that raised the original request. + * + * @return string The source app id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSourceApp(): string { + return $this->sourceApp; + }//end getSourceApp() + + /** + * The caller's correlation id. + * + * @return string The correlation id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getCorrelationId(): string { + return $this->correlationId; + }//end getCorrelationId() + + /** + * The subject object id from the original request. + * + * @return string The subject id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSubjectId(): string { + return $this->subjectId; + }//end getSubjectId() + + /** + * The delivery channel from the original request. + * + * @return string The channel. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getChannel(): string { + return $this->channel; + }//end getChannel() + + /** + * Terminal status of the delivery. + * + * @return string One of the STATUS_* constants. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getStatus(): string { + return $this->status; + }//end getStatus() + + /** + * Uuid of the CloudEvent `event` object. + * + * @return string The event uuid. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getEventId(): string { + return $this->eventId; + }//end getEventId() + + /** + * Uuid of the `event_message` delivery record. + * + * @return string The message uuid. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getMessageId(): string { + return $this->messageId; + }//end getMessageId() + + /** + * How many delivery attempts were made. + * + * @return int The attempt count. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getAttempts(): int { + return $this->attempts; + }//end getAttempts() + + /** + * The last delivery error. + * + * @return string|null The error, or null on success. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getError(): ?string { + return $this->error; + }//end getError() + + /** + * When the delivery reached its terminal state. + * + * @return string ISO 8601 timestamp. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getConcludedAt(): string { + return $this->concludedAt; + }//end getConcludedAt() +}//end class diff --git a/lib/Event/DeliveryRequestedEvent.php b/lib/Event/DeliveryRequestedEvent.php new file mode 100644 index 000000000..be051530c --- /dev/null +++ b/lib/Event/DeliveryRequestedEvent.php @@ -0,0 +1,301 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-License-Identifier: EUPL-1.2 + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * + * @version GIT: + * + * @link https://conduction.nl + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Event; + +use OCP\EventDispatcher\Event; + +/** + * Typed cross-app command: "deliver this payload on my behalf". + * + * Carries provenance (which app, which subject object), a delivery payload + * reference, and a synchronous result slot the in-process listener writes: + * `isHandled()` + `getResultId()` (the persisted CloudEvent uuid) + + * `getMatchedSubscriptions()` (how many delivery routes picked it up — zero + * means the request was accepted but nothing is configured to deliver it, + * which a fail-closed consumer records as a refusal, not a success). + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + * + * @SuppressWarnings(PHPMD.ExcessiveParameterList) -- the ADR-041 event contract is a flat + * readonly provenance envelope (sourceApp, subject coordinates, kind/channel, correlation); + * folding fields into an array would untype the contract the consumer stubs must mirror + * verbatim. Mirrors the decidiq DecisionRequestedEvent precedent. + */ +class DeliveryRequestedEvent extends Event { + /** + * Whether an Integriq listener handled the request. + * + * @var bool + */ + private bool $handled = false; + + /** + * Uuid of the persisted CloudEvent `event` object, once handled. + * + * @var string|null + */ + private ?string $resultId = null; + + /** + * How many active event subscriptions matched the delivery request. + * + * @var int + */ + private int $matchedSubscriptions = 0; + + /** + * Constructor. + * + * @param string $sourceApp The requesting app id (e.g. `dossiq`). + * @param string $subjectRegister The OpenRegister register slug/id of the subject object. + * @param string $subjectSchema The schema slug/id of the subject object. + * @param string $subjectId The subject object id/uuid (e.g. the case id). + * @param string $subjectLabel Human-readable label for the subject. + * @param string $deliveryKind What is being delivered (e.g. `besluit-publication`). + * @param string $channel The requested delivery channel (e.g. `gemeenteblad`). + * @param array $payload The delivery payload reference (composed by the source app). + * @param string $correlationId Caller-generated id echoed on the concluded event. + * @param string|null $externalReference Optional external reference (e.g. besluit identificatie). + * @param string|null $userId The acting Nextcloud user, or null for system-produced requests. + * + * @return void + */ + public function __construct( + private readonly string $sourceApp, + private readonly string $subjectRegister, + private readonly string $subjectSchema, + private readonly string $subjectId, + private readonly string $subjectLabel, + private readonly string $deliveryKind, + private readonly string $channel, + private readonly array $payload, + private readonly string $correlationId, + private readonly ?string $externalReference = null, + private readonly ?string $userId = null, + ) { + parent::__construct(); + }//end __construct() + + /** + * The requesting app id. + * + * @return string The source app id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSourceApp(): string { + return $this->sourceApp; + }//end getSourceApp() + + /** + * The subject object's register. + * + * @return string The register slug/id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSubjectRegister(): string { + return $this->subjectRegister; + }//end getSubjectRegister() + + /** + * The subject object's schema. + * + * @return string The schema slug/id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSubjectSchema(): string { + return $this->subjectSchema; + }//end getSubjectSchema() + + /** + * The subject object id. + * + * @return string The object id/uuid. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSubjectId(): string { + return $this->subjectId; + }//end getSubjectId() + + /** + * Human-readable subject label. + * + * @return string The label. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getSubjectLabel(): string { + return $this->subjectLabel; + }//end getSubjectLabel() + + /** + * What is being delivered. + * + * @return string The delivery kind. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getDeliveryKind(): string { + return $this->deliveryKind; + }//end getDeliveryKind() + + /** + * The requested delivery channel. + * + * @return string The channel. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getChannel(): string { + return $this->channel; + }//end getChannel() + + /** + * The delivery payload reference. + * + * @return array The payload. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getPayload(): array { + return $this->payload; + }//end getPayload() + + /** + * The caller's correlation id. + * + * @return string The correlation id. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getCorrelationId(): string { + return $this->correlationId; + }//end getCorrelationId() + + /** + * Optional external reference. + * + * @return string|null The external reference. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getExternalReference(): ?string { + return $this->externalReference; + }//end getExternalReference() + + /** + * The acting Nextcloud user. + * + * @return string|null The user id, or null for system-produced requests. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getUserId(): ?string { + return $this->userId; + }//end getUserId() + + /** + * Mark the request as handled by an Integriq listener. + * + * @param bool $handled Whether the request was handled. + * + * @return void + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function setHandled(bool $handled): void { + $this->handled = $handled; + }//end setHandled() + + /** + * Whether an Integriq listener handled the request. + * + * @return bool True when handled. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function isHandled(): bool { + return $this->handled; + }//end isHandled() + + /** + * Record the persisted CloudEvent uuid. + * + * @param string $resultId The event object uuid. + * + * @return void + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function setResultId(string $resultId): void { + $this->resultId = $resultId; + }//end setResultId() + + /** + * The persisted CloudEvent uuid, once handled. + * + * @return string|null The event object uuid. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getResultId(): ?string { + return $this->resultId; + }//end getResultId() + + /** + * Record how many subscriptions matched. + * + * @param int $matchedSubscriptions The matched subscription count. + * + * @return void + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function setMatchedSubscriptions(int $matchedSubscriptions): void { + $this->matchedSubscriptions = $matchedSubscriptions; + }//end setMatchedSubscriptions() + + /** + * How many active subscriptions matched the delivery request. + * + * @return int The matched subscription count. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function getMatchedSubscriptions(): int { + return $this->matchedSubscriptions; + }//end getMatchedSubscriptions() +}//end class diff --git a/lib/EventListener/DeliveryRequestedListener.php b/lib/EventListener/DeliveryRequestedListener.php new file mode 100644 index 000000000..d9b05c26b --- /dev/null +++ b/lib/EventListener/DeliveryRequestedListener.php @@ -0,0 +1,100 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-License-Identifier: EUPL-1.2 + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * + * @version GIT: + * + * @link https://conduction.nl + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\EventListener; + +use OCA\Integriq\Event\DeliveryRequestedEvent; +use OCA\Integriq\Service\EventService; +use OCP\EventDispatcher\Event; +use OCP\EventDispatcher\IEventListener; +use Psr\Log\LoggerInterface; + +/** + * Listener that ingests cross-app delivery requests into the CloudEvents + * pipeline. + * + * On success it marks the event handled, records the persisted CloudEvent + * uuid as the result id, and reports the matched-subscription count so a + * fail-closed consumer can distinguish "accepted and routed" from "accepted + * but no delivery route is configured". On ingest failure the event stays + * unhandled — the consumer's fail-closed guard then records a refusal. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ +class DeliveryRequestedListener implements IEventListener { + /** + * Constructor. + * + * @param EventService $eventService The CloudEvents pipeline entry point. + * @param LoggerInterface $logger Logger for ingest failures. + * + * @return void + */ + public function __construct( + private readonly EventService $eventService, + private readonly LoggerInterface $logger, + ) { + }//end __construct() + + /** + * Handle a cross-app delivery request. + * + * @param Event $event The dispatched event. + * + * @return void + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function handle(Event $event): void { + if (($event instanceof DeliveryRequestedEvent) === false) { + return; + } + + try { + $result = $this->eventService->ingestDeliveryRequest(request: $event); + } catch (\Throwable $e) { + // Leave the event unhandled: the consumer's fail-closed guard + // records the refusal on its own domain record. + $this->logger->error( + 'Delivery request ingest failed: ' . $e->getMessage(), + [ + 'exception' => $e, + 'sourceApp' => $event->getSourceApp(), + 'correlationId' => $event->getCorrelationId(), + ] + ); + return; + }//end try + + $event->setResultId(resultId: (string)$result['event']->getUuid()); + $event->setMatchedSubscriptions(matchedSubscriptions: count($result['messages'])); + $event->setHandled(handled: true); + }//end handle() +}//end class diff --git a/lib/Flow/ApprovalRequestNode.php b/lib/Flow/ApprovalRequestNode.php new file mode 100644 index 000000000..000147d36 --- /dev/null +++ b/lib/Flow/ApprovalRequestNode.php @@ -0,0 +1,642 @@ +.decision` — being told "no" is the flow working. + * - **expired** — fails closed, always (`FlowStop`, error). An approval nobody + * answered must never quietly count as answered. + * + * WHY THE HEARTBEAT RE-READS THE RECORD + * ------------------------------------- + * A signal can be delivered while the run has not suspended yet, or its + * delivery can simply fail. `AwaitSignalNode`'s answer to that is a heartbeat + * that re-asks; this node has something better to re-ask than "did anything + * arrive?" — the approval_request row, which `ApprovalsController` resolves + * regardless of whether the signal made it. A lost signal therefore costs one + * heartbeat, never the flow. + * + * @category Flow + * @package OCA\Integriq\Flow + * + * @author Conduction Development Team + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Flow; + +use DateTime; +use OCA\Integriq\Exception\FlowNodeException; +use OCA\Integriq\Service\ApprovalService; +use OCA\OpenRegister\Service\Flow\FlowItems; +use OCA\OpenRegister\Service\Flow\FlowNodeResumeState; +use OCA\OpenRegister\Service\Flow\FlowStop; +use OCA\OpenRegister\Service\Flow\FlowSuspension; +use OCA\OpenRegister\Service\Flow\IFlowNode; +use OCA\OpenRegister\Service\Flow\IFlowNodeConfigForm; +use OCA\OpenRegister\Service\Flow\IFlowNodeConfigKeys; +use OCP\IL10N; +use OCP\IURLGenerator; +use OCP\WorkflowEngine\IManager; +use Psr\Log\LoggerInterface; +use Throwable; +use UnexpectedValueException; + +/** + * Asks a person, parks the run, and carries their answer onto the items. + * + * @SuppressWarnings(PHPMD.CouplingBetweenObjects) The count is the engine + * vocabulary itself — FlowSuspension/FlowStop/FlowItems/FlowNodeResumeState + * plus the three node interfaces — the same fan-in every sibling node + * carries (they sit in phpmd.baseline.xml for the identical reason). + * @SuppressWarnings(PHPMD.StaticAccess) FlowConfigGuard and FlowNodeSupport + * are the shared static config guards every Integriq node validates + * through; instantiating them would add state to say the same thing. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ +class ApprovalRequestNode implements IFlowNode, IFlowNodeConfigKeys, IFlowNodeConfigForm { + + /** + * The step type this node answers to. + * + * FROZEN on `openconnector.*` — the id is written into stored flow + * documents, so it survives the openconnector -> integriq app-id rename. + * + * @var string + */ + public const NODE_ID = 'openconnector.approval-request'; + + /** + * The context key the engine delivers a signal payload under. + * + * Mirrors `FlowRunService::SIGNAL_CONTEXT_KEY`. Declared locally (like + * `FlowNodeSupport::ON_ERROR_POLICIES`) so reading it never pulls the run + * service into scope on an instance without the flow engine. + * + * @var string + */ + private const SIGNAL_CONTEXT_KEY = 'signal'; + + /** + * The context key carrying the engine run's uuid. + * + * Mirrors `FlowRunContext::CONTEXT_RUN`. The uuid is what the approval + * resolution later addresses through `FlowRunSignalService::signalAs()`, + * so a run that cannot name itself cannot be approved and is refused. + * + * @var string + */ + private const RUN_CONTEXT_KEY = 'x-openregister-attribution-run'; + + /** + * Minutes between heartbeats when the step does not choose. + * + * Matches `AwaitSignalNode`'s default: short enough that a lost signal is + * an inconvenience, long enough that a fortnight-long approval stays + * cheap. + * + * @var int + */ + private const DEFAULT_HEARTBEAT_MINUTES = 15; + + /** + * The floor a configured heartbeat is clamped to. + * + * The stock system cron runs every five minutes; asking for less buys the + * same behaviour while looking like it bought more. + * + * @var int + */ + private const MIN_HEARTBEAT_MINUTES = 5; + + /** + * The item key the decision payload is written under by default. + * + * @var string + */ + private const DEFAULT_SIGNAL_KEY = 'approval'; + + /** + * Constructor. + * + * @param ApprovalService $approvalService The HITL state machine — persistence, mirror task, notifications. + * @param IL10N $l10n Translations. + * @param IURLGenerator $urlGenerator For the palette icon. + * @param LoggerInterface $logger Run diagnostics. + */ + public function __construct( + private readonly ApprovalService $approvalService, + private readonly IL10N $l10n, + private readonly IURLGenerator $urlGenerator, + private readonly LoggerInterface $logger, + ) { + + }//end __construct() + + /** + * The step type. + * + * @return string The type identifier. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getId(): string { + return self::NODE_ID; + }//end getId() + + /** + * Palette name. + * + * @return string The display name. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getDisplayName(): string { + return $this->l10n->t('Ask for approval'); + }//end getDisplayName() + + /** + * Palette description. + * + * @return string The description. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getDescription(): string { + return $this->l10n->t( + 'Pause the flow until someone in the approver group approves or rejects. An expired request fails the run.' + ); + + }//end getDescription() + + /** + * Palette icon. + * + * @return string The icon URL. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getIcon(): string { + return $this->urlGenerator->imagePath('core', 'actions/confirm.svg'); + }//end getIcon() + + /** + * Asking for an approval grants no privilege by itself. + * + * @param int $scope The scope constant. + * + * @return boolean Whether it is available. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function isAvailableForScope(int $scope): bool { + return in_array($scope, [IManager::SCOPE_ADMIN, IManager::SCOPE_USER], true); + }//end isAvailableForScope() + + /** + * The node's whole config vocabulary. + * + * @return array The accepted top-level config keys. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function configKeys(): array { + return [ + 'question', + 'approverGroup', + 'ttlSeconds', + 'failOnReject', + 'signalKey', + 'heartbeatMinutes', + 'onError', + ]; + }//end configKeys() + + /** + * The fields this node's configuration is edited through. + * + * @return array> The field descriptions. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function configForm(): array { + return [ + [ + 'key' => 'question', + 'label' => $this->l10n->t('What is being asked'), + 'type' => 'text', + 'help' => $this->l10n->t('Shown to the approvers and written on the request, so a paused flow explains itself.'), + 'required' => true, + ], + [ + 'key' => 'approverGroup', + 'label' => $this->l10n->t('Approver group'), + 'type' => 'text', + 'help' => $this->l10n->t('Members of this group (and admins) may answer. Required: a request nobody owns is a request nobody answers.'), + 'required' => true, + ], + [ + 'key' => 'ttlSeconds', + 'label' => $this->l10n->t('Expires after (seconds)'), + 'type' => 'number', + 'help' => $this->l10n->t('An unanswered request expires and fails the run. Defaults to 24 hours.'), + ], + [ + 'key' => 'failOnReject', + 'label' => $this->l10n->t('Treat a rejection as a failure'), + 'type' => 'boolean', + 'help' => $this->l10n->t('Off by default: a "no" continues the flow with the decision on the items, so a later step can route on it.'), + ], + [ + 'key' => 'signalKey', + 'label' => $this->l10n->t('Field to store the decision in'), + 'type' => 'text', + 'help' => $this->l10n->t('The decision is written onto every item under this field. Defaults to "approval".'), + ], + [ + 'key' => 'heartbeatMinutes', + 'label' => $this->l10n->t('Re-check every (minutes)'), + 'type' => 'number', + 'help' => $this->l10n->t('Safety net for a lost answer. Lower is not faster: a decision wakes the run immediately either way.'), + ], + ]; + }//end configForm() + + /** + * Reject a configuration the author cannot have meant, at flow-save time. + * + * @param array $config The step's authored configuration. + * + * @return void + * + * @throws UnexpectedValueException When the configuration is unusable. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function validateConfig(array $config): void { + FlowConfigGuard::assertNoForbiddenFields(config: $config, l10n: $this->l10n); + + if (trim((string)($config['question'] ?? '')) === '') { + throw new UnexpectedValueException( + $this->l10n->t('Say what is being asked ("question"), or nobody can answer it.') + ); + } + + if (trim((string)($config['approverGroup'] ?? '')) === '') { + throw new UnexpectedValueException( + $this->l10n->t('Name the approver group ("approverGroup"): an approval without an audience never resolves.') + ); + } + + if (array_key_exists('ttlSeconds', $config) === true + && (is_numeric($config['ttlSeconds']) === false || ((int)$config['ttlSeconds']) < 1) + ) { + throw new UnexpectedValueException( + $this->l10n->t('The "ttlSeconds" field must be a positive number of seconds when set.') + ); + } + + FlowNodeSupport::assertOnError(config: $config, l10n: $this->l10n); + + }//end validateConfig() + + /** + * Ask, suspend, and carry the decision onto the items when it arrives. + * + * @param array $items The input items. + * @param array $config The step's authored configuration. + * @param array $context Run-level metadata (signal payload, resume slot, run uuid). + * + * @return array The items, each carrying the decision under the signal key. + * + * @throws FlowSuspension While the request is pending. + * @throws FlowStop When the request was rejected under `failOnReject`, or expired (fail closed). + * @throws FlowNodeException When the run cannot be addressed for a later answer. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function execute(array $items, array $config, array $context): array { + $this->validateConfig(config: $config); + + $decision = $this->decisionFrom(context: $context); + if ($decision !== null) { + return $this->applyDecision(items: $items, config: $config, decision: $decision); + } + + $resume = ($context[FlowNodeResumeState::CONTEXT_KEY] ?? null); + if ($resume instanceof FlowNodeResumeState === false) { + // Without a resume slot every heartbeat would open a fresh + // request. That is a broken dispatch, not a pending approval. + throw new FlowNodeException( + message: $this->l10n->t('The approval step has no resume slot; the engine did not dispatch it as a resumable node.') + ); + } + + if ($resume->has('approvalRequestId') === true) { + return $this->answerFromRecord(items: $items, config: $config, resume: $resume); + } + + $this->openRequest(config: $config, context: $context, resume: $resume); + + throw new FlowSuspension( + resumeAt: $this->heartbeatAt(config: $config), + reason: sprintf( + 'waiting for approval: %s', + trim((string)$config['question']) + ) + ); + + }//end execute() + + /** + * Persist the pending approval_request and stamp the resume slot. + * + * The slot's `assignee` is the approver group, which is what + * OpenRegister's own signal guard (`FlowRunAssignee`) checks a signaller + * against — so the engine-side guard and Integriq's own approver-group + * authorization name the same audience. + * + * @param array $config The step's authored configuration. + * @param array $context Run-level metadata. + * @param FlowNodeResumeState $resume This node's resume slot. + * + * @return void + * + * @throws FlowNodeException When the run has no uuid to answer at. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function openRequest(array $config, array $context, FlowNodeResumeState $resume): void { + $runUuid = trim((string)($context[self::RUN_CONTEXT_KEY] ?? '')); + if ($runUuid === '') { + // A request created for an unaddressable run could be approved and + // still resume nothing. Refuse loudly instead. + throw new FlowNodeException( + message: $this->l10n->t( + 'This run carries no uuid, so an approval could never answer it. The approval step is only usable in a persisted flow run.' + ) + ); + } + + $record = $this->approvalService->suspendForEngineRun( + engineRunUuid: $runUuid, + signalNodeId: $resume->nodeId(), + config: $config, + requesterUid: trim((string)($context['triggeredBy'] ?? '')) + ); + + $data = $record->getObject(); + $resume->merge( + values: [ + 'approvalRequestId' => $record->getUuid(), + 'askedAt' => (new DateTime())->format('c'), + 'question' => trim((string)$config['question']), + 'assignee' => trim((string)$config['approverGroup']), + 'expiresAt' => (string)($data['expiresAt'] ?? ''), + ] + ); + + }//end openRequest() + + /** + * The heartbeat's answer when no signal made it: ask the record itself. + * + * The approval_request is the system of record and `ApprovalsController` + * resolves it whether or not the signal delivery succeeded, so a resolved + * record with no delivered signal means the answer exists and only the + * wake-up was lost. + * + * @param array $items The input items. + * @param array $config The step's authored configuration. + * @param FlowNodeResumeState $resume This node's resume slot. + * + * @return array The items carrying the decision, when the record resolved. + * + * @throws FlowSuspension While the record is still pending and unexpired. + * @throws FlowStop When the record expired, was dead-lettered, or was rejected under `failOnReject`. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function answerFromRecord(array $items, array $config, FlowNodeResumeState $resume): array { + $requestId = (string)$resume->get(key: 'approvalRequestId', default: ''); + + try { + $record = $this->approvalService->find(id: $requestId); + $data = $record->getObject(); + } catch (Throwable $e) { + // A vanished record can never resolve; waiting longer cannot fix it. + throw new FlowStop( + reason: sprintf('Approval request %s no longer exists; failing closed.', $requestId), + isError: true + ); + } + + $status = (string)($data['status'] ?? 'pending'); + + if ($status === 'approved') { + return $this->applyDecision( + items: $items, + config: $config, + decision: [ + 'decision' => 'approved', + 'decidedBy' => (string)($data['approverUserId'] ?? ''), + 'comment' => (string)($data['comment'] ?? ''), + 'approvalRequestId' => $requestId, + ] + ); + } + + if ($status === 'rejected') { + return $this->applyDecision( + items: $items, + config: $config, + decision: [ + 'decision' => 'rejected', + 'decidedBy' => (string)($data['approverUserId'] ?? ''), + 'comment' => (string)($data['comment'] ?? ''), + 'approvalRequestId' => $requestId, + ] + ); + } + + if ($status === 'dead_letter') { + throw new FlowStop( + reason: sprintf('Approval request %s was dead-lettered.', $requestId), + isError: true + ); + } + + if ($status !== 'pending' || $this->hasExpired(data: $data, resume: $resume) === true) { + // `expired` from the sweep, a past `expiresAt` the sweep has not + // reached yet, or any state this node does not know: fail closed. + throw new FlowStop( + reason: sprintf( + 'Approval request %s was not answered in time (status: %s); failing closed.', + $requestId, + $status + ), + isError: true + ); + } + + throw new FlowSuspension( + resumeAt: $this->heartbeatAt(config: $config), + reason: sprintf( + 'still waiting for approval: %s', + (string)$resume->get(key: 'question', default: 'approval') + ) + ); + + }//end answerFromRecord() + + /** + * Whether the pending record's deadline has passed. + * + * @param array $data The approval_request's object data. + * @param FlowNodeResumeState $resume This node's resume slot (fallback deadline). + * + * @return boolean True when expired. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function hasExpired(array $data, FlowNodeResumeState $resume): bool { + $expiresAt = trim((string)($data['expiresAt'] ?? $resume->get(key: 'expiresAt', default: ''))); + if ($expiresAt === '') { + return false; + } + + try { + return new DateTime($expiresAt) < new DateTime(); + } catch (Throwable $e) { + // An unreadable deadline must not read as "never expires". + return true; + } + + }//end hasExpired() + + /** + * Write the decision onto every item, honouring `failOnReject`. + * + * @param array $items The input items. + * @param array $config The step's authored configuration. + * @param array $decision The decision payload. + * + * @return array The items, each carrying the decision under the signal key. + * + * @throws FlowStop When rejected and the step asked to fail on rejection. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function applyDecision(array $items, array $config, array $decision): array { + $verdict = strtolower(trim((string)($decision['decision'] ?? ''))); + + if (($verdict === 'reject' || $verdict === 'rejected') && ($config['failOnReject'] ?? false) === true) { + throw new FlowStop( + reason: sprintf( + 'Rejected: %s', + trim((string)($decision['comment'] ?? $config['question'] ?? 'no reason given')) + ), + isError: true + ); + } + + $key = trim((string)($config['signalKey'] ?? '')); + if ($key === '') { + $key = self::DEFAULT_SIGNAL_KEY; + } + + // Into every item's record (`json`), like the engine's own + // await-signal node: the steps that follow route per item and read + // `json.`; an envelope-level key is invisible to a Switch. + foreach ($items as $index => $item) { + if (is_array($item) === false) { + continue; + } + + $json = (array)($item[FlowItems::JSON] ?? []); + $json[$key] = $decision; + $item[FlowItems::JSON] = $json; + $items[$index] = $item; + } + + return $items; + + }//end applyDecision() + + /** + * The decision this node is waiting for, if a signal delivered it. + * + * Null covers three cases that must all mean "keep waiting": no signal, a + * signal that is not a value bag, and a signal carrying no `decision` — + * the last so a stray empty resume cannot approve anything. + * + * @param array $context Run-level metadata. + * + * @return array|null The decision payload, or null while unanswered. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function decisionFrom(array $context): ?array { + $signal = ($context[self::SIGNAL_CONTEXT_KEY] ?? null); + if (is_array($signal) === false) { + return null; + } + + if (trim((string)($signal['decision'] ?? '')) === '') { + return null; + } + + return $signal; + + }//end decisionFrom() + + /** + * When the next heartbeat should wake the run. + * + * @param array $config The step's authored configuration. + * + * @return DateTime The wake-up time. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function heartbeatAt(array $config): DateTime { + $minutes = (int)($config['heartbeatMinutes'] ?? self::DEFAULT_HEARTBEAT_MINUTES); + if ($minutes < self::MIN_HEARTBEAT_MINUTES) { + $minutes = self::MIN_HEARTBEAT_MINUTES; + } + + return new DateTime(sprintf('+%d minutes', $minutes)); + + }//end heartbeatAt() +}//end class diff --git a/lib/Flow/EventEmitNode.php b/lib/Flow/EventEmitNode.php new file mode 100644 index 000000000..0307cbbd5 --- /dev/null +++ b/lib/Flow/EventEmitNode.php @@ -0,0 +1,379 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Flow; + +use OCA\Integriq\Exception\FlowNodeException; +use OCA\Integriq\Service\EventService; +use OCA\OpenRegister\Service\Flow\FlowItems; +use OCA\OpenRegister\Service\Flow\IFlowNode; +use OCA\OpenRegister\Service\Flow\IFlowNodeConfigForm; +use OCA\OpenRegister\Service\Flow\IFlowNodeConfigKeys; +use OCP\IL10N; +use OCP\IURLGenerator; +use OCP\WorkflowEngine\IManager; +use Psr\Container\ContainerInterface; +use Psr\Log\LoggerInterface; +use Throwable; +use UnexpectedValueException; + +/** + * Emits one CloudEvent per item through the existing event pipeline. + * + * @SuppressWarnings(PHPMD.CouplingBetweenObjects) The count is the engine + * vocabulary itself — the three node interfaces plus FlowItems and the + * shared guards — the same fan-in every sibling node carries (they sit in + * phpmd.baseline.xml for the identical reason). + * @SuppressWarnings(PHPMD.StaticAccess) FlowConfigGuard, FlowNodeSupport + * and FlowTemplate are the shared static helpers every Integriq node + * validates and templates through; instantiating them would add state to + * say the same thing. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ +class EventEmitNode implements IFlowNode, IFlowNodeConfigKeys, IFlowNodeConfigForm { + + /** + * The step type this node answers to. + * + * FROZEN on `openconnector.*` — the id is written into stored flow + * documents, so it survives the openconnector -> integriq app-id rename. + * + * @var string + */ + public const NODE_ID = 'openconnector.event-emit'; + + /** + * Constructor. + * + * `EventService` is resolved lazily through the container rather than + * constructor-injected — the same idiom `FlowRunnerService` uses — + * because `EventService`'s delivery path can dispatch flow work of its + * own, and an eager constructor edge from the flow palette into it drags + * the whole delivery graph into every palette build. + * + * @param ContainerInterface $container Lazily resolves EventService. + * @param IL10N $l10n Translations. + * @param IURLGenerator $urlGenerator For the palette icon. + * @param LoggerInterface $logger Run diagnostics. + */ + public function __construct( + private readonly ContainerInterface $container, + private readonly IL10N $l10n, + private readonly IURLGenerator $urlGenerator, + private readonly LoggerInterface $logger, + ) { + + }//end __construct() + + /** + * The step type. + * + * @return string The type identifier. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getId(): string { + return self::NODE_ID; + }//end getId() + + /** + * Palette name. + * + * @return string The display name. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getDisplayName(): string { + return $this->l10n->t('Emit an event'); + }//end getDisplayName() + + /** + * Palette description. + * + * @return string The description. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getDescription(): string { + return $this->l10n->t( + 'Emit a CloudEvent for every item, delivered through the configured event subscriptions.' + ); + + }//end getDescription() + + /** + * Palette icon. + * + * @return string The icon URL. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function getIcon(): string { + return $this->urlGenerator->imagePath('core', 'actions/share.svg'); + }//end getIcon() + + /** + * Whether the node is offered in the given scope. + * + * @param int $scope The scope constant. + * + * @return boolean Whether it is available. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function isAvailableForScope(int $scope): bool { + return in_array($scope, [IManager::SCOPE_ADMIN, IManager::SCOPE_USER], true); + }//end isAvailableForScope() + + /** + * The node's whole config vocabulary. + * + * @return array The accepted top-level config keys. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function configKeys(): array { + return ['type', 'source', 'subject', 'output', 'onError']; + }//end configKeys() + + /** + * The fields this node's configuration is edited through. + * + * @return array> The field descriptions. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function configForm(): array { + return [ + [ + 'key' => 'type', + 'label' => $this->l10n->t('Event type'), + 'type' => 'text', + 'help' => $this->l10n->t('The CloudEvent "type", e.g. "nl.example.object.updated". Subscriptions match on it.'), + 'required' => true, + ], + [ + 'key' => 'source', + 'label' => $this->l10n->t('Event source'), + 'type' => 'text', + 'help' => $this->l10n->t('The CloudEvent "source" URI identifying the emitter.'), + 'required' => true, + ], + [ + 'key' => 'subject', + 'label' => $this->l10n->t('Subject'), + 'type' => 'text', + 'help' => $this->l10n->t('Optional CloudEvent "subject". Supports {{dotted.path}} placeholders resolved from each item.'), + ], + [ + 'key' => 'output', + 'label' => $this->l10n->t('Output key'), + 'type' => 'text', + 'help' => $this->l10n->t('Item key the emit summary is written under. Defaults to "eventResult".'), + ], + ]; + }//end configForm() + + /** + * Reject a configuration the author cannot have meant, at flow-save time. + * + * @param array $config The step's authored configuration. + * + * @return void + * + * @throws UnexpectedValueException When the configuration is unusable. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function validateConfig(array $config): void { + FlowConfigGuard::assertNoForbiddenFields(config: $config, l10n: $this->l10n); + + if (trim((string)($config['type'] ?? '')) === '') { + throw new UnexpectedValueException( + $this->l10n->t('Name the event "type": an event without a type matches no subscription.') + ); + } + + if (trim((string)($config['source'] ?? '')) === '') { + throw new UnexpectedValueException( + $this->l10n->t('Name the event "source": a CloudEvent must say where it came from.') + ); + } + + if (array_key_exists('output', $config) === true) { + FlowConfigGuard::assertOutputKeyAllowed(outputKey: (string)$config['output'], l10n: $this->l10n); + } + + FlowNodeSupport::assertOnError(config: $config, l10n: $this->l10n); + + }//end validateConfig() + + /** + * Emit one CloudEvent per item through the existing pipeline. + * + * @param array $items The input items. + * @param array $config The step's authored configuration. + * @param array $context Run-level metadata. + * + * @return array The items, each carrying the emit summary under the output key. + * + * @throws FlowNodeException On a failure the `onError` policy does not absorb. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function execute(array $items, array $config, array $context): array { + // An empty page emits nothing and produces no items — the filter + // contract, not a failure. + if ($items === []) { + return []; + } + + $this->validateConfig(config: $config); + + $stepId = FlowNodeSupport::stepId(config: $config, context: $context, nodeId: self::NODE_ID); + $onError = FlowNodeSupport::onErrorPolicy(config: $config, context: $context); + $eventService = $this->container->get(EventService::class); + + $outputKey = trim((string)($config['output'] ?? '')); + if ($outputKey === '') { + $outputKey = 'eventResult'; + } + + $out = []; + foreach ($items as $index => $item) { + $json = []; + $rebuilt = []; + if (is_array($item) === true) { + $json = (array)($item[FlowItems::JSON] ?? []); + $rebuilt = $item; + } + + $rebuilt[FlowItems::JSON] = $this->emitForItem( + eventService: $eventService, + json: $json, + config: $config, + stepId: $stepId, + onError: $onError, + outputKey: $outputKey + ); + if (array_key_exists(FlowItems::PAIRED_ITEM, $rebuilt) === false) { + $rebuilt[FlowItems::PAIRED_ITEM] = ['item' => $index]; + } + + $out[] = $rebuilt; + }//end foreach + + return $out; + + }//end execute() + + /** + * Emit one item's event; success lands under the output key, failure is + * explicit — a raise, or `__error` state under `continue`. + * + * @param EventService $eventService The resolved event pipeline. + * @param array $json The item's record. + * @param array $config The step's authored configuration. + * @param string $stepId The step id, for error messages. + * @param string $onError The step's error policy. + * @param string $outputKey The key the emit summary lands under. + * + * @return array The item's record, carrying the summary or the error state. + * + * @throws FlowNodeException On a failure the `onError` policy does not absorb. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + private function emitForItem( + EventService $eventService, + array $json, + array $config, + string $stepId, + string $onError, + string $outputKey, + ): array { + try { + $subject = trim(FlowTemplate::renderString( + template: (string)($config['subject'] ?? ''), + json: $json + )); + if ($subject === '') { + $subject = null; + } + + $messages = $eventService->emitCloudEvent( + type: (string)$config['type'], + source: (string)$config['source'], + subject: $subject, + data: $json + ); + + $json[$outputKey] = [ + 'emitted' => true, + 'messageCount' => count($messages), + ]; + } catch (Throwable $e) { + if ($onError !== 'continue') { + throw new FlowNodeException( + message: $this->l10n->t( + 'Step "%1$s" failed to emit event "%2$s": %3$s', + [$stepId, (string)$config['type'], $e->getMessage()] + ), + details: ['stepId' => $stepId, 'type' => (string)$config['type']], + previous: $e + ); + } + + $this->logger->warning( + 'EventEmitNode: item failed to emit, carried as error state (onError: continue)', + ['stepId' => $stepId, 'exception' => $e] + ); + + // Explicit error state, never a success-shaped empty summary. + $json[FlowNodeSupport::ERROR_KEY] = [ + 'failed' => true, + 'stepId' => $stepId, + 'message' => $e->getMessage(), + 'type' => (string)$config['type'], + ]; + }//end try + + return $json; + + }//end emitForItem() +}//end class diff --git a/lib/Flow/FlowNodeListener.php b/lib/Flow/FlowNodeListener.php index a90146bab..6569c2b3a 100644 --- a/lib/Flow/FlowNodeListener.php +++ b/lib/Flow/FlowNodeListener.php @@ -50,6 +50,11 @@ * * @template-implements IEventListener * + * @SuppressWarnings(PHPMD.ExcessiveParameterList) One constructor parameter + * per contributed node is the whole job of this class: it is the single + * registration fan-in, and hiding the nodes behind a collection would trade + * a visible list for an invisible one. + * * @spec openspec/changes/integriq-flow-nodes/tasks.md#task-1-flow-node-scaffolding-guarded-registration-shared-helpers */ class FlowNodeListener implements IEventListener { @@ -64,6 +69,8 @@ class FlowNodeListener implements IEventListener { * @param ContractCommitNode $contractCommitNode The page-level contract-upsert node. * @param ContractSweepNode $contractSweepNode The guarded stale-object sweep node. * @param FetchFileNode $fetchFileNode The fetch-file rule node. + * @param ApprovalRequestNode $approvalRequestNode The HITL approval step (retire-integriq-flow-schema). + * @param EventEmitNode $eventEmitNode The CloudEvent emit step (retire-integriq-flow-schema). */ public function __construct( private readonly SourceCallNode $sourceCallNode, @@ -74,6 +81,8 @@ public function __construct( private readonly ContractCommitNode $contractCommitNode, private readonly ContractSweepNode $contractSweepNode, private readonly FetchFileNode $fetchFileNode, + private readonly ApprovalRequestNode $approvalRequestNode, + private readonly EventEmitNode $eventEmitNode, ) { }//end __construct() @@ -104,6 +113,8 @@ public function handle(Event $event): void { $event->registerNode(node: $this->contractCommitNode); $event->registerNode(node: $this->contractSweepNode); $event->registerNode(node: $this->fetchFileNode); + $event->registerNode(node: $this->approvalRequestNode); + $event->registerNode(node: $this->eventEmitNode); }//end handle() }//end class diff --git a/lib/Repair/MigrateFlowStepsToGraph.php b/lib/Repair/MigrateFlowStepsToGraph.php new file mode 100644 index 000000000..a718803e4 --- /dev/null +++ b/lib/Repair/MigrateFlowStepsToGraph.php @@ -0,0 +1,135 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Repair; + +use OCA\Integriq\Service\FlowGraphMigrationService; +use OCP\Migration\IOutput; +use OCP\Migration\IRepairStep; +use Psr\Container\ContainerInterface; +use Psr\Log\LoggerInterface; +use Throwable; + +/** + * Applies the steps-to-graph flow migration on install/upgrade. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ +class MigrateFlowStepsToGraph implements IRepairStep { + + /** + * Constructor. + * + * @param ContainerInterface $container Resolves the migration service lazily, + * so the OpenRegister class_exists guard + * can short-circuit before anything + * referencing OR types is constructed. + * @param LoggerInterface $logger For refusals and non-fatal failures. + */ + public function __construct( + private readonly ContainerInterface $container, + private readonly LoggerInterface $logger, + ) { + + }//end __construct() + + /** + * Human-readable name surfaced by `occ` during install / upgrade. + * + * @return string + * + * @spec exclude Repair-step display name for occ output — framework metadata, no domain behavior. + */ + public function getName(): string { + return 'Write the OpenRegister nodes/edges graph onto legacy Integriq flow objects (retire-integriq-flow-schema)'; + }//end getName() + + /** + * Migrate every live flow, additively and idempotently. + * + * @param IOutput $output Repair output channel. + * + * @return void + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + public function run(IOutput $output): void { + if (class_exists('OCA\\OpenRegister\\Service\\ObjectService') === false) { + // No OpenRegister, no register, no flows — nothing to migrate. + return; + } + + try { + $migration = $this->container->get(FlowGraphMigrationService::class); + $report = $migration->migrate(apply: true); + } catch (Throwable $e) { + // Non-fatal by contract: the legacy runner still executes steps[], + // and the occ command re-runs the migration on demand. + $this->logger->warning( + 'MigrateFlowStepsToGraph: migration pass failed, flows stay on steps[]: ' . $e->getMessage(), + ['exception' => $e] + ); + + return; + } + + $migrated = 0; + $refused = 0; + foreach ($report as $row) { + if ($row['action'] === FlowGraphMigrationService::MIGRATED) { + $migrated++; + } + + if ($row['action'] === FlowGraphMigrationService::REFUSED) { + $refused++; + $this->logger->warning( + 'MigrateFlowStepsToGraph: flow refused, staying on steps[]', + ['flowId' => $row['id'], 'name' => $row['name'], 'reasons' => $row['reasons']] + ); + } + } + + if (($migrated + $refused) > 0) { + $output->info(sprintf( + 'Flow steps-to-graph migration: %d migrated, %d refused (see the log), %d total.', + $migrated, + $refused, + count($report) + )); + } + + }//end run() +}//end class diff --git a/lib/Service/ApprovalService.php b/lib/Service/ApprovalService.php index ed24743fb..238658518 100644 --- a/lib/Service/ApprovalService.php +++ b/lib/Service/ApprovalService.php @@ -42,6 +42,7 @@ use OCA\Integriq\Service\Helper\FlowToken; use OCA\OpenRegister\Db\ObjectEntity; use OCA\OpenRegister\Service\ObjectService as ORObjectService; +use OCA\OpenRegister\Service\Task\TaskService as ORTaskService; use OCP\AppFramework\Db\DoesNotExistException; use OCP\IGroupManager; use OCP\IURLGenerator; @@ -106,6 +107,11 @@ class ApprovalService { * @param ExecutionTraceService|null $executionTraceService Persists the traced run's execution_trace at * suspension/resume (execution-trace REQ-004). Nullable + defaulted so * pre-existing positional test instantiations keep working unmodified. + * @param ORTaskService|null $taskService OpenRegister's shared task service: every suspension mirrors ONE + * shared task through it and every decision closes that mirror + * (hitl-on-shared-tasks D-1). Nullable + defaulted for the same + * positional-test reason; absent, no mirror exists and the approval + * flow is unchanged. */ public function __construct( private readonly ORObjectService $objectService, @@ -115,6 +121,7 @@ public function __construct( private readonly IURLGenerator $urlGenerator, private readonly LoggerInterface $logger, private readonly ?ExecutionTraceService $executionTraceService = null, + private readonly ?ORTaskService $taskService = null, ) { }//end __construct() @@ -195,6 +202,7 @@ public function suspend(ObjectEntity $endpoint, ObjectEntity $rule, FlowToken $f } } + $record = $this->mirrorIntoSharedTask(approvalRequest: $record); $this->notifyApprovers(approvalRequest: $record); return $record; @@ -243,6 +251,7 @@ public function suspendForSynchronization( schema: self::SCHEMA ); + $record = $this->mirrorIntoSharedTask(approvalRequest: $record); $this->notifyApprovers(approvalRequest: $record); return $record; @@ -296,11 +305,86 @@ public function suspendForFlow(ObjectEntity $flowRun, int $resumeStepOrder, arra schema: self::SCHEMA ); + $record = $this->mirrorIntoSharedTask(approvalRequest: $record); $this->notifyApprovers(approvalRequest: $record); return $record; }//end suspendForFlow() + /** + * Suspend an OpenRegister ENGINE flow run on an + * `openconnector.approval-request` step: persist a `pending` + * `approval_request` carrying `engineRunUuid`/`signalNodeId` instead of + * `flowRunId`/`resumeStepOrder`, and notify the configured approver + * group. The engine run resumes through + * `FlowRunSignalService::signalAs()` (delivered by + * `ApprovalsController`), never through a FlowToken rehydration — the + * engine holds the run's own state, so `snapshot` stays empty and this + * record remains purely the human-decision system of record + * (hitl-on-shared-tasks D-1 unchanged: the mirror task and + * notifications behave exactly as for every other suspension kind). + * + * @param string $engineRunUuid The suspended OpenRegister flow run's uuid. + * @param string $signalNodeId The graph node id awaiting the decision, so the + * signal addresses the right resume slot. + * @param array $config The approval step's config (`question`/`approverGroup`/`ttlSeconds`/`failOnReject`). + * @param string $requesterUid The run owner's uid (`context.triggeredBy`), or '' when unattributed. + * + * @return ObjectEntity The created, `pending` approval_request. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function suspendForEngineRun( + string $engineRunUuid, + string $signalNodeId, + array $config, + string $requesterUid = '', + ): ObjectEntity { + $ttlSeconds = (int)($config['ttlSeconds'] ?? self::DEFAULT_TTL_SECONDS); + + $now = new DateTime(); + $expiresAt = (clone $now)->add(new DateInterval('PT' . max($ttlSeconds, 1) . 'S')); + + $requesterUserId = $requesterUid; + if ($requesterUserId === '') { + $requesterUserId = ($this->userSession->getUser()?->getUID() ?? ''); + } + + // `onReject` mirrors the node's `failOnReject` into the legacy + // vocabulary the sweep and the UI already read: a step that fails on + // rejection is `error`, one that routes the rejection onward is + // `skip`. `onTimeout` is always `error` — the node fails closed on + // expiry by requirement, so the record must not promise otherwise. + $onReject = 'skip'; + if (($config['failOnReject'] ?? false) === true) { + $onReject = 'error'; + } + + $record = $this->objectService->saveObject( + object: [ + 'status' => 'pending', + 'engineRunUuid' => $engineRunUuid, + 'signalNodeId' => $signalNodeId, + 'question' => trim((string)($config['question'] ?? '')), + 'timing' => 'before', + 'snapshot' => [], + 'requesterUserId' => $requesterUserId, + 'approverGroup' => (string)($config['approverGroup'] ?? ''), + 'onReject' => $onReject, + 'onTimeout' => 'error', + 'createdAt' => $now->format('c'), + 'expiresAt' => $expiresAt->format('c'), + ], + register: self::REGISTER, + schema: self::SCHEMA + ); + + $record = $this->mirrorIntoSharedTask(approvalRequest: $record); + $this->notifyApprovers(approvalRequest: $record); + + return $record; + }//end suspendForEngineRun() + /** * Create the `approval_request` gating an `api_product_subscription` * whose chosen tier has `requiresApproval: true` (api-product-gateway @@ -349,6 +433,7 @@ public function suspendForSubscription( schema: self::SCHEMA ); + $record = $this->mirrorIntoSharedTask(approvalRequest: $record); $this->notifyApprovers(approvalRequest: $record); return $record; @@ -575,13 +660,16 @@ public function completeApproval( $data['comment'] = $comment; } - return $this->objectService->saveObject( + $saved = $this->objectService->saveObject( object: $data, register: self::REGISTER, schema: self::SCHEMA, uuid: $approvalRequest->getUuid() ); + $this->closeSharedTask(data: $data, outcome: 'transition:approved', actorUid: $approver->getUID()); + + return $saved; }//end completeApproval() /** @@ -618,13 +706,24 @@ public function reject(ObjectEntity $approvalRequest, IUser $approver, string $c $data['rejectedAt'] = (new DateTime())->format('c'); $data['comment'] = $comment; - return $this->objectService->saveObject( + $saved = $this->objectService->saveObject( object: $data, register: self::REGISTER, schema: self::SCHEMA, uuid: $approvalRequest->getUuid() ); + // The mirror ends the way the record did: dead-lettered when + // onReject routed the record there, plainly rejected otherwise + // (hitl-on-shared-tasks D-4). + $mirrorOutcome = 'transition:rejected'; + if ($data['status'] === 'dead_letter') { + $mirrorOutcome = 'dead_letter'; + } + + $this->closeSharedTask(data: $data, outcome: $mirrorOutcome, actorUid: $approver->getUID()); + + return $saved; }//end reject() /** @@ -791,6 +890,148 @@ public function notifyApprovers(ObjectEntity $approvalRequest): void { }//end notifyApprovers() + /** + * Mirror a just-created, pending approval_request into ONE shared + * OpenRegister task (hitl-on-shared-tasks D-1/D-2): approver group as + * candidate group, requester, expiry, and the record's + * onTimeout/onReject when they are in the shared vocabulary, so the + * shared timer sweep owns the mirror's expiry (D-3). The created task's + * uuid is written back onto the record as `taskUuid`. + * + * A failure here is logged and swallowed: the approval flow is the + * system of record and MUST NOT be gated by the mirror (D-5). + * + * @param ObjectEntity $approvalRequest The pending approval_request. + * + * @return ObjectEntity The record, carrying `taskUuid` when the mirror was created. + * + * @spec openspec/changes/hitl-on-shared-tasks/specs/hitl-on-shared-tasks/spec.md#requirement-every-suspension-mirrors-one-shared-task + */ + private function mirrorIntoSharedTask(ObjectEntity $approvalRequest): ObjectEntity { + if ($this->taskService === null) { + return $approvalRequest; + } + + $data = $approvalRequest->getObject(); + $actor = (string)($data['requesterUserId'] ?? ''); + if ($actor === '') { + $actor = 'integriq'; + } + + try { + $task = $this->taskService->import( + data: $this->sharedTaskData(data: $data, approvalRequestId: (string)$approvalRequest->getUuid()), + actor: $actor + ); + + $data['taskUuid'] = (string)$task->getUuid(); + + return $this->objectService->saveObject( + object: $data, + register: self::REGISTER, + schema: self::SCHEMA, + uuid: $approvalRequest->getUuid() + ); + } catch (Throwable $e) { + $this->logger->warning( + 'ApprovalService: could not mirror the approval into the shared task service: ' . $e->getMessage(), + ['approvalRequest' => $approvalRequest->getUuid()] + ); + + return $approvalRequest; + } + }//end mirrorIntoSharedTask() + + /** + * The shared-task payload a pending approval_request mirrors to. + * + * `onTimeout`/`onReject` travel only when they are in the shared + * vocabulary (`skip`|`error`|`dead_letter`); anything else stays an + * app-local behaviour and the mirror carries none. + * + * @param array $data The approval_request object data. + * @param string $approvalRequestId The record uuid the task links back to. + * + * @return array The task creation payload. + * + * @spec openspec/changes/hitl-on-shared-tasks/specs/hitl-on-shared-tasks/spec.md#requirement-every-suspension-mirrors-one-shared-task + */ + private function sharedTaskData(array $data, string $approvalRequestId): array { + $payload = [ + 'state' => 'enabled', + 'title' => 'Approval request', + 'description' => 'Approve or reject this request in Integriq. Your decision resumes the suspended run.', + 'performerType' => 'user', + 'appId' => 'integriq', + 'metadata' => [ + 'kind' => 'approval_request', + 'approvalRequestId' => $approvalRequestId, + ], + ]; + + if ((string)($data['approverGroup'] ?? '') !== '') { + $payload['candidateGroups'] = [(string)$data['approverGroup']]; + } + + if ((string)($data['requesterUserId'] ?? '') !== '') { + $payload['requester'] = (string)$data['requesterUserId']; + } + + if ((string)($data['expiresAt'] ?? '') !== '') { + $payload['expiresAt'] = (string)$data['expiresAt']; + $onTimeout = (string)($data['onTimeout'] ?? ''); + if (in_array($onTimeout, ['skip', 'error', 'dead_letter'], true) === true) { + $payload['onTimeout'] = $onTimeout; + } + } + + $onReject = (string)($data['onReject'] ?? ''); + if (in_array($onReject, ['skip', 'error', 'dead_letter'], true) === true) { + $payload['onReject'] = $onReject; + } + + return $payload; + }//end sharedTaskData() + + /** + * Close the mirrored shared task after a decision resolved the record + * (hitl-on-shared-tasks D-4), through the shared outcome path: the + * decision was already authorized by this service's own two-layer model, + * and the mirror has no assignee for a completion check to pass. + * + * A missing mirror (`taskUuid` absent: pre-seam rows, or a failed + * mirror) and a mirror already closed by the shared sweep are both + * fine; any failure is logged and swallowed (D-5). + * + * @param array $data The resolved approval_request object data. + * @param string $outcome The shared outcome (`transition:approved`, `transition:rejected` or `dead_letter`). + * @param string $actorUid The deciding user's uid, recorded as the source. + * + * @return void + * + * @spec openspec/changes/hitl-on-shared-tasks/specs/hitl-on-shared-tasks/spec.md#requirement-a-decision-closes-the-mirrored-task + */ + private function closeSharedTask(array $data, string $outcome, string $actorUid): void { + $taskUuid = (string)($data['taskUuid'] ?? ''); + if ($this->taskService === null || $taskUuid === '') { + return; + } + + try { + $this->taskService->applyTimerOutcome( + uuid: $taskUuid, + outcome: $outcome, + source: 'integriq:' . $actorUid, + reason: sprintf("Approval request resolved as '%s'.", (string)($data['status'] ?? '')) + ); + } catch (Throwable $e) { + $this->logger->warning( + 'ApprovalService: could not close the mirrored shared task: ' . $e->getMessage(), + ['taskUuid' => $taskUuid] + ); + } + }//end closeSharedTask() + /** * Strip sensitive headers (at minimum `Authorization`) from a FlowToken * snapshot's request slots before persisting it — security-hard diff --git a/lib/Service/EngineSignalService.php b/lib/Service/EngineSignalService.php new file mode 100644 index 000000000..228b36030 --- /dev/null +++ b/lib/Service/EngineSignalService.php @@ -0,0 +1,128 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Service; + +use OCP\IUser; +use Psr\Log\LoggerInterface; +use Throwable; + +/** + * Guarded `FlowRunSignalService::signalAs()` delivery for approval decisions. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ +class EngineSignalService { + + /** + * The engine's signal service, referenced by NAME only. + * + * The `BrokeredCallService::BROKER_CLASS` idiom: a string keeps the + * compile-time reference out of this app, so Integriq keeps working + * against an OpenRegister that predates the signal service. + * + * @var string + */ + private const SIGNAL_SERVICE_CLASS = 'OCA\\OpenRegister\\Service\\Flow\\FlowRunSignalService'; + + /** + * Constructor. + * + * @param LoggerInterface $logger Delivery diagnostics. + */ + public function __construct( + private readonly LoggerInterface $logger, + ) { + + }//end __construct() + + /** + * Deliver a decision to the suspended engine run an approval_request gates. + * + * Uses `FlowRunSignalService::signalAs()` so the engine's own assignee + * guard applies — the same audience check Integriq already made through + * `isAuthorizedApprover()`, enforced a second time by the engine against + * the resume slot's recorded approver group. + * + * @param array $data The approval_request's object data (`engineRunUuid`/`signalNodeId`). + * @param string $decision `approved` or `rejected`. + * @param IUser $user The deciding user. + * @param string|null $comment Optional decision comment. + * + * @return boolean True when the signal was delivered. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#1-the-missing-node + */ + public function deliver(array $data, string $decision, IUser $user, ?string $comment): bool { + if (class_exists(self::SIGNAL_SERVICE_CLASS) === false) { + $this->logger->warning( + 'EngineSignalService: OpenRegister has no FlowRunSignalService; the engine run resumes on its next heartbeat instead', + ['engineRunUuid' => ($data['engineRunUuid'] ?? '')] + ); + return false; + } + + $nodeId = trim((string)($data['signalNodeId'] ?? '')); + if ($nodeId === '') { + $nodeId = null; + } + + try { + \OCP\Server::get(self::SIGNAL_SERVICE_CLASS)->signalAs( + runUuid: (string)($data['engineRunUuid'] ?? ''), + payload: [ + 'decision' => $decision, + 'decidedBy' => $user->getUID(), + 'comment' => (string)($comment ?? ''), + ], + actorUid: $user->getUID(), + nodeId: $nodeId + ); + + return true; + } catch (Throwable $e) { + // NOT_SUSPENDED and RUN_NOT_FOUND included: the record already + // carries the decision, so the node's heartbeat picks it up. + $this->logger->warning( + 'EngineSignalService: could not signal engine run, it will resume on its heartbeat: ' . $e->getMessage(), + ['engineRunUuid' => ($data['engineRunUuid'] ?? ''), 'exception' => $e] + ); + + return false; + }//end try + + }//end deliver() +}//end class diff --git a/lib/Service/EventService.php b/lib/Service/EventService.php index 9985c7bcc..b8296833e 100644 --- a/lib/Service/EventService.php +++ b/lib/Service/EventService.php @@ -22,6 +22,8 @@ use DateTime; use Exception; use JWadhams\JsonLogic; +use OCA\Integriq\Event\DeliveryConcludedEvent; +use OCA\Integriq\Event\DeliveryRequestedEvent; use OCA\Integriq\Exception\FormsFeatureDisabledException; use OCA\Integriq\Exception\InvalidMessageStateException; use OCA\Integriq\Service\Forms\FormsAnswerResolver; @@ -30,6 +32,7 @@ use OCA\Integriq\Service\Security\SensitiveFieldRegistry; use OCA\OpenRegister\Db\ObjectEntity; use OCA\OpenRegister\Service\ObjectService as ORObjectService; +use OCP\EventDispatcher\IEventDispatcher; use OCP\Http\Client\IClientService; use Psr\Log\LoggerInterface; use Symfony\Component\ExpressionLanguage\ExpressionLanguage; @@ -116,6 +119,17 @@ class EventService { */ public const NEXTCLOUD_SOURCE_PREFIX = '/nextcloud/'; + /** + * CloudEvents `type` for cross-app delivery requests ingested through the + * ADR-041 typed-event seam ({@see DeliveryRequestedEvent}). Subscriptions + * route on this type plus `data.delivery.*` provenance filters. + * + * @var string + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public const DELIVERY_REQUESTED_TYPE = 'nl.conduction.delivery.requested'; + /** * Constructor. * @@ -143,6 +157,10 @@ class EventService { * so pre-existing positional test * instantiations keep working * unmodified. + * @param IEventDispatcher|null $eventDispatcher Dispatches {@see DeliveryConcludedEvent} when a + * provenance-carrying delivery reaches a terminal + * state (ADR-041 seam). Nullable + defaulted for + * the same test-compatibility reason as above. * * @spec openspec/specs/events-cloudevents/spec.md#requirement-a-subscription-s-action-dispatch-must-support-webhook-synchronization-or-job-kinds-req-008 * @spec openspec/specs/events-cloudevents/spec.md#requirement-a-subscription-s-action-dispatch-must-support-a-notificaties-kind-for-zgw-notificaties-api-publishing-req-010 @@ -162,6 +180,7 @@ public function __construct( private readonly ?FormsAnswerResolver $formsAnswerResolver = null, private readonly ?FormsSyncAdapter $formsSyncAdapter = null, private readonly ?ExecutionTraceService $executionTraceService = null, + private readonly ?IEventDispatcher $eventDispatcher = null, ) { }//end __construct() @@ -787,6 +806,15 @@ private function recordFailure( uuid: $message->getUuid() ); + if ($messageData['status'] === 'abandoned') { + $this->dispatchDeliveryConcluded( + message: $message, + messageData: $messageData, + status: DeliveryConcludedEvent::STATUS_ABANDONED, + concludedAt: $nowIso + ); + } + }//end recordFailure() /** @@ -1819,8 +1847,88 @@ private function recordDeliverySuccess(ObjectEntity $message): void { uuid: $message->getUuid() ); + $this->dispatchDeliveryConcluded( + message: $message, + messageData: $messageData, + status: DeliveryConcludedEvent::STATUS_DELIVERED, + concludedAt: $now + ); + }//end recordDeliverySuccess() + /** + * Dispatch the terminal {@see DeliveryConcludedEvent} for a + * provenance-carrying delivery message (ADR-041 seam). + * + * Gated to messages whose originating event was ingested through + * {@see ingestDeliveryRequest} — the `data.delivery.sourceApp` + + * `correlationId` provenance block is the gate, so ordinary CloudEvent + * traffic never produces a concluded event. Dispatch failures are logged + * and swallowed: the message's own status record is the source of truth + * and must not be rolled back by a consumer-side listener error. + * + * @param ObjectEntity $message The event_message row that reached a terminal state. + * @param array $messageData The message's persisted data (post-transition). + * @param string $status Terminal status: {@see DeliveryConcludedEvent::STATUS_DELIVERED} + * or {@see DeliveryConcludedEvent::STATUS_ABANDONED}. + * @param string $concludedAt ISO 8601 timestamp of the terminal transition. + * + * @return void + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + private function dispatchDeliveryConcluded( + ObjectEntity $message, + array $messageData, + string $status, + string $concludedAt, + ): void { + if ($this->eventDispatcher === null) { + return; + } + + $payload = (array)($messageData['payload'] ?? []); + $data = (array)($payload['data'] ?? []); + $delivery = (array)($data['delivery'] ?? []); + $sourceApp = (string)($delivery['sourceApp'] ?? ''); + $correlationId = (string)($delivery['correlationId'] ?? ''); + if ($sourceApp === '' || $correlationId === '') { + // Not an ADR-041 delivery request — nothing to conclude. + return; + } + + $error = null; + if (isset($messageData['error']) === true && (string)$messageData['error'] !== '') { + $error = (string)$messageData['error']; + } + + try { + $this->eventDispatcher->dispatchTyped( + new DeliveryConcludedEvent( + sourceApp: $sourceApp, + correlationId: $correlationId, + subjectId: (string)($delivery['subjectId'] ?? ''), + channel: (string)($delivery['channel'] ?? ''), + status: $status, + eventId: (string)($messageData['event'] ?? ''), + messageId: (string)$message->getUuid(), + attempts: count((array)($messageData['attempts'] ?? [])), + error: $error, + concludedAt: $concludedAt, + ) + ); + } catch (\Throwable $e) { + $this->logger->error( + 'DeliveryConcludedEvent dispatch failed: ' . $e->getMessage(), + [ + 'exception' => $e, + 'messageId' => $message->getUuid(), + 'sourceApp' => $sourceApp, + ] + ); + }//end try + }//end dispatchDeliveryConcluded() + /** * Persist a configuration-error failure (e.g. an unrecognised * `action.kind`): `status='failed'` with a descriptive error, WITHOUT @@ -2289,6 +2397,67 @@ public function emitCloudEvent(string $type, string $source, ?string $subject, a return $this->processEvent(event: $event); }//end emitCloudEvent() + /** + * Ingest an ADR-041 cross-app delivery request into the CloudEvents + * pipeline. + * + * Persists a {@see self::DELIVERY_REQUESTED_TYPE} `event` OR object whose + * `data.delivery` block carries the request's provenance (sourceApp, + * subject, channel, correlationId) and whose `data.payload` carries the + * caller-composed delivery payload, then fans it out via + * {@see processEvent} so admin-configured `event_subscription`s route it + * to a webhook / flow / synchronization / notificaties action with the + * pipeline's retry, dead-letter and replay semantics. + * + * The provenance block is what later gates the terminal + * {@see DeliveryConcludedEvent} dispatch in + * {@see dispatchDeliveryConcluded} — ordinary CloudEvent traffic carries + * no `data.delivery` and never produces one. + * + * @param DeliveryRequestedEvent $request The typed cross-app delivery request. + * + * @return array{event: ObjectEntity, messages: ObjectEntity[]} The persisted event and its created delivery messages. + * + * @throws Exception On event processing failure. + * @throws \OCP\DB\Exception On persistence failure. + * + * @spec openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md + */ + public function ingestDeliveryRequest(DeliveryRequestedEvent $request): array { + $event = $this->objectService->saveObject( + object: [ + 'source' => ('/apps/' . $request->getSourceApp() . '/delivery'), + 'type' => self::DELIVERY_REQUESTED_TYPE, + 'time' => (new DateTime())->format('c'), + 'subject' => $request->getSubjectId(), + 'data' => [ + 'delivery' => [ + 'sourceApp' => $request->getSourceApp(), + 'subjectRegister' => $request->getSubjectRegister(), + 'subjectSchema' => $request->getSubjectSchema(), + 'subjectId' => $request->getSubjectId(), + 'subjectLabel' => $request->getSubjectLabel(), + 'deliveryKind' => $request->getDeliveryKind(), + 'channel' => $request->getChannel(), + 'correlationId' => $request->getCorrelationId(), + 'externalReference' => $request->getExternalReference(), + ], + 'payload' => $request->getPayload(), + ], + 'userId' => $request->getUserId(), + ], + register: 'integriq', + schema: 'event' + ); + + $messages = $this->processEvent(event: $event); + + return [ + 'event' => $event, + 'messages' => $messages, + ]; + }//end ingestDeliveryRequest() + /** * Normalize a Nextcloud-native core event (files/calendar/Tables/Forms) * into the same CloudEvents `event` OR-object shape diff --git a/lib/Service/FlowGraphMigrationService.php b/lib/Service/FlowGraphMigrationService.php new file mode 100644 index 000000000..f492464d0 --- /dev/null +++ b/lib/Service/FlowGraphMigrationService.php @@ -0,0 +1,267 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Service; + +use OCA\Integriq\Exception\EntityNotMigratableException; +use OCA\OpenRegister\Db\ObjectEntity; +use OCA\OpenRegister\Service\ObjectService as OrObjectService; +use Psr\Log\LoggerInterface; +use Throwable; + +/** + * Reads, translates and rewrites live `flow` objects, both directions. + * + * @SuppressWarnings(PHPMD.BooleanArgumentFlag) `$apply` is the dry-run/write + * switch the occ command exposes as its own flag — the same + * dry-run-by-default contract MigrateInlineSecrets and DedupeContracts + * already carry (both suppress this rule for the same reason). + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ +class FlowGraphMigrationService { + + /** + * Result marker: the flow's graph was written (or would be, on a dry run). + * + * @var string + */ + public const MIGRATED = 'migrated'; + + /** + * Result marker: the flow already carries `nodes` and was left alone. + * + * @var string + */ + public const SKIPPED = 'skipped'; + + /** + * Result marker: the translator refused the flow; reasons attached. + * + * @var string + */ + public const REFUSED = 'refused'; + + /** + * Result marker: the graph was removed (or would be, on a dry run). + * + * @var string + */ + public const ROLLED_BACK = 'rolled_back'; + + /** + * Constructor. + * + * @param FlowStepsToGraphTranslator $translator The pure steps-to-graph translation. + * @param OrObjectService $orObjectService OpenRegister object persistence. + * @param LoggerInterface $logger Migration diagnostics. + */ + public function __construct( + private readonly FlowStepsToGraphTranslator $translator, + private readonly OrObjectService $orObjectService, + private readonly LoggerInterface $logger, + ) { + + }//end __construct() + + /** + * Translate every live `flow` object's steps into a graph, in place. + * + * @param bool $apply False (the default posture for the occ command) reports + * what WOULD happen without writing anything. + * + * @return array}> One row per flow. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + public function migrate(bool $apply = false): array { + $report = []; + + foreach ($this->liveFlows() as $flow) { + $data = $flow->getObject(); + $row = [ + 'id' => (string)$flow->getUuid(), + 'name' => (string)($data['name'] ?? ''), + 'action' => self::MIGRATED, + 'reasons' => [], + ]; + + if (empty($data['nodes']) === false) { + // Idempotence: a graph already written (by this migration or + // by hand) is never overwritten — refusing is the rule the + // tasks file states verbatim. + $row['action'] = self::SKIPPED; + $report[] = $row; + continue; + } + + try { + $graph = $this->translator->translate(flow: $data); + } catch (EntityNotMigratableException $e) { + $row['action'] = self::REFUSED; + $row['reasons'] = $e->getReasons(); + $this->logger->warning( + 'FlowGraphMigrationService: flow refused by the translator: ' . $e->getMessage(), + ['flowId' => $row['id'], 'reasons' => $e->getReasons()] + ); + $report[] = $row; + continue; + } + + if ($apply === true) { + $data['nodes'] = $graph['nodes']; + $data['edges'] = $graph['edges']; + $this->orObjectService->saveObject( + object: $data, + register: FlowRunnerService::REGISTER, + schema: FlowRunnerService::SCHEMA_FLOW, + uuid: (string)$flow->getUuid() + ); + } + + $report[] = $row; + }//end foreach + + return $report; + + }//end migrate() + + /** + * Remove the written graph from every live `flow` object, in place. + * + * The rollback of a migration whose forward direction is additive: it + * deletes `nodes`/`edges` and leaves `steps` as the only shape again. A + * flow whose `steps` are gone is REFUSED — deleting its graph would + * leave nothing executable at all, and how its steps vanished is a + * question for a person, not a rollback. + * + * @param bool $apply False reports what WOULD happen without writing. + * + * @return array}> One row per flow. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + public function rollback(bool $apply = false): array { + $report = []; + + foreach ($this->liveFlows() as $flow) { + $data = $flow->getObject(); + $row = [ + 'id' => (string)$flow->getUuid(), + 'name' => (string)($data['name'] ?? ''), + 'action' => self::ROLLED_BACK, + 'reasons' => [], + ]; + + if (empty($data['nodes']) === true && empty($data['edges']) === true) { + $row['action'] = self::SKIPPED; + $report[] = $row; + continue; + } + + if (empty($data['steps']) === true) { + $row['action'] = self::REFUSED; + $row['reasons'] = ['The flow has a graph but no steps; removing the graph would leave it with no executable shape.']; + $report[] = $row; + continue; + } + + if ($apply === true) { + unset($data['nodes'], $data['edges']); + $this->orObjectService->saveObject( + object: $data, + register: FlowRunnerService::REGISTER, + schema: FlowRunnerService::SCHEMA_FLOW, + uuid: (string)$flow->getUuid() + ); + } + + $report[] = $row; + }//end foreach + + return $report; + + }//end rollback() + + /** + * Every live `flow` object, unfiltered by tenancy — a migration walks + * the whole table or it is not a migration. + * + * @return array The flow objects. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function liveFlows(): array { + try { + $matches = $this->orObjectService->findAll( + config: [ + 'filters' => [ + 'register' => FlowRunnerService::REGISTER, + 'schema' => FlowRunnerService::SCHEMA_FLOW, + ], + 'limit' => 1000, + ], + _rbac: false, + _multitenancy: false + ); + } catch (Throwable $e) { + // A register that does not exist yet (fresh install ordering) has + // no flows to migrate; that is a no-op, not a failure. + $this->logger->info( + 'FlowGraphMigrationService: could not list flows (register not initialised yet?): ' . $e->getMessage() + ); + + return []; + } + + $results = ($matches['results'] ?? $matches); + + return array_values( + array_filter( + (array)$results, + static fn ($row): bool => $row instanceof ObjectEntity + ) + ); + + }//end liveFlows() +}//end class diff --git a/lib/Service/FlowStepsToGraphTranslator.php b/lib/Service/FlowStepsToGraphTranslator.php new file mode 100644 index 000000000..eed239f36 --- /dev/null +++ b/lib/Service/FlowStepsToGraphTranslator.php @@ -0,0 +1,645 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + * + * @version GIT: + * + * @link https://www.Integriq.nl + * + * @spec openspec/changes/retire-integriq-flow-schema/specs/flow-orchestration/spec.md + */ + +declare(strict_types=1); + +namespace OCA\Integriq\Service; + +use OCA\Integriq\Exception\EntityNotMigratableException; +use OCP\IL10N; + +/** + * Pure `steps[]` -> `nodes`/`edges` translation for the flow migration. + * + * @SuppressWarnings(PHPMD.ExcessiveClassComplexity) The class is a total + * function over the closed six-type step vocabulary: every type contributes + * its own mapping arm and its own refusal checks, and that enumeration IS + * the migration. Splitting it across classes would scatter one closed + * decision table without removing a single decision. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ +class FlowStepsToGraphTranslator { + + /** + * Legacy step type -> contributed/built-in node type. + * + * Every entry is a node that exists: the two `integriq-flow-nodes` nodes, + * the two nodes `retire-integriq-flow-schema` contributes, and the + * engine's own switch anchor for `branch`. + * + * @var array + */ + private const TYPE_MAP = [ + 'call' => 'openconnector.source-call', + 'mapping' => 'openconnector.apply-mapping', + 'synchronization' => 'openconnector.synchronization-run', + 'event' => 'openconnector.event-emit', + 'approval' => 'openconnector.approval-request', + 'branch' => 'openregister.switch', + ]; + + /** + * The graph's entry node id. + * + * @var string + */ + private const TRIGGER_ID = 'trigger'; + + /** + * The graph's terminal node id. + * + * @var string + */ + private const END_ID = 'end'; + + /** + * Constructor. + * + * @param IL10N $l10n Translations for the refusal summary. + */ + public function __construct( + private readonly IL10N $l10n, + ) { + + }//end __construct() + + /** + * Translate one flow document's `steps[]` into `nodes` and `edges`. + * + * @param array $flow The flow's serialised record (needs `steps`, uses `name` in messages). + * + * @return array{nodes: array>, edges: array>} The graph. + * + * @throws EntityNotMigratableException When the flow uses a feature the graph cannot express. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + public function translate(array $flow): array { + $steps = $this->sortedSteps(steps: (array)($flow['steps'] ?? [])); + + $reasons = $this->refusalsFor(steps: $steps); + if ($reasons !== []) { + throw new EntityNotMigratableException( + subject: 'flow', + message: $this->l10n->t( + 'The flow "%1$s" cannot be migrated to a graph yet: %2$s unsupported feature(s).', + [(string)($flow['name'] ?? ($flow['uuid'] ?? 'unnamed')), (string)count($reasons)] + ), + reasons: $reasons + ); + } + + $nodes = [ + ['id' => self::TRIGGER_ID, 'type' => 'openregister.trigger-manual', 'config' => []], + ]; + $edges = []; + + $orders = array_map(static fn (array $step): int => (int)$step['order'], $steps); + $previousId = self::TRIGGER_ID; + + foreach ($steps as $index => $step) { + $id = (string)((int)$step['order']); + $nodes[] = $this->nodeFor(step: $step, id: $id); + + // The edge INTO this step from the sequential predecessor. A + // branch's own outgoing edges are conditioned below; every other + // step chains to the next order, which is exactly the runner's + // sequential walk. + if ($previousId !== null) { + $edges[] = ['id' => $previousId . '-' . $id, 'from' => $previousId, 'to' => $id]; + } + + $nextId = self::END_ID; + if (isset($orders[($index + 1)]) === true) { + $nextId = (string)$orders[($index + 1)]; + } + + if ((string)($step['type'] ?? '') === 'branch') { + foreach ($this->branchEdges(step: $step, id: $id, nextId: $nextId) as $edge) { + $edges[] = $edge; + } + + // The branch's outgoing edges are complete; nothing chains + // sequentially out of it. + $previousId = null; + continue; + } + + $previousId = $id; + }//end foreach + + $nodes[] = ['id' => self::END_ID, 'type' => 'openregister.end', 'config' => []]; + + if ($previousId !== null) { + $edges[] = ['id' => $previousId . '-' . self::END_ID, 'from' => $previousId, 'to' => self::END_ID]; + } + + return [ + 'nodes' => $nodes, + 'edges' => $edges, + ]; + + }//end translate() + + /** + * Every feature of this flow the graph translation cannot express. + * + * An empty list means the flow is migratable. A non-empty one is the + * refusal — migrating anyway would swap declared behaviour for silence + * or approximation. + * + * @param array $steps The steps, sorted by `order`. + * + * @return array One sentence per unsupported feature. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + public function refusalsFor(array $steps): array { + $reasons = []; + + if ($steps === []) { + $reasons[] = 'The flow has no steps; there is nothing to migrate.'; + } + + $seen = []; + $orders = []; + foreach ($steps as $step) { + $order = (int)($step['order'] ?? 0); + if (isset($seen[$order]) === true) { + $reasons[] = sprintf( + 'Duplicate step order %d — FlowRunnerService::run() rejects this flow today, and a graph would silently lose one of the two nodes.', + $order + ); + } + + $seen[$order] = true; + $orders[] = $order; + } + + foreach ($steps as $step) { + $order = (int)($step['order'] ?? 0); + $type = (string)($step['type'] ?? ''); + + if (isset(self::TYPE_MAP[$type]) === false) { + $reasons[] = sprintf('Step %d has unsupported type "%s".', $order, $type); + continue; + } + + if (empty($step['condition']) === false) { + $reasons[] = sprintf( + 'Step %d carries a run-if `condition`; the graph expresses conditions on branch edges, not as step skips, so this flow needs a manual re-model.', + $order + ); + } + + $reasons = array_merge($reasons, $this->stepRefusals(step: $step, order: $order, type: $type, orders: $orders)); + }//end foreach + + return $reasons; + + }//end refusalsFor() + + /** + * Per-type refusals for one step. + * + * @param array $step The step definition. + * @param int $order The step's order. + * @param string $type The step's (supported) type. + * @param array $orders Every declared order, for branch-target checks. + * + * @return array One sentence per unsupported feature. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function stepRefusals(array $step, int $order, string $type, array $orders): array { + $config = (array)($step['config'] ?? []); + + return array_merge( + $this->referenceRefusals(step: $step, order: $order, type: $type), + $this->configRefusals(config: $config, order: $order, type: $type), + $this->branchRefusals(step: $step, order: $order, type: $type, orders: $orders) + ); + + }//end stepRefusals() + + /** + * The refusal for a step whose node requires a `configRef` it lacks. + * + * @param array $step The step definition. + * @param int $order The step's order. + * @param string $type The step's (supported) type. + * + * @return array Zero or one sentence. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function referenceRefusals(array $step, int $order, string $type): array { + if (in_array($type, ['call', 'mapping', 'synchronization'], true) === false) { + return []; + } + + if (trim((string)($step['configRef'] ?? '')) !== '') { + return []; + } + + return [sprintf('Step %d (%s) names no `configRef`; the node it maps to requires the referenced entity.', $order, $type)]; + + }//end referenceRefusals() + + /** + * The per-type refusals living in a step's `config` block. + * + * @param array $config The step's config block. + * @param int $order The step's order. + * @param string $type The step's (supported) type. + * + * @return array One sentence per unsupported feature. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function configRefusals(array $config, int $order, string $type): array { + return match ($type) { + 'call' => $this->callRefusals(config: $config, order: $order), + 'synchronization' => $this->synchronizationRefusals(config: $config, order: $order), + 'event' => $this->eventRefusals(config: $config, order: $order), + 'approval' => $this->approvalRefusals(config: $config, order: $order), + default => [], + }; + + }//end configRefusals() + + /** + * The `call` step features the source-call node cannot express. + * + * @param array $config The step's config block. + * @param int $order The step's order. + * + * @return array Zero or one sentence. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function callRefusals(array $config, int $order): array { + if (empty($config['requestConfig']) === true) { + return []; + } + + return [ + sprintf( + 'Step %d (call) carries a raw `requestConfig`; the source-call node expresses requests as ' + . 'endpoint/method/query/body/headers, so this step needs a manual re-model.', + $order + ), + ]; + + }//end callRefusals() + + /** + * The `synchronization` step features the synchronization-run node lacks. + * + * @param array $config The step's config block. + * @param int $order The step's order. + * + * @return array One sentence per unsupported feature. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function synchronizationRefusals(array $config, int $order): array { + $reasons = []; + + if (($config['isTest'] ?? false) === true) { + $reasons[] = sprintf('Step %d (synchronization) runs in `isTest` mode, which the synchronization-run node does not offer.', $order); + } + + if (trim((string)($config['mutationType'] ?? '')) !== '') { + $reasons[] = sprintf('Step %d (synchronization) sets a `mutationType`, which the synchronization-run node does not offer.', $order); + } + + return $reasons; + + }//end synchronizationRefusals() + + /** + * The `event` step configuration the event-emit node would reject. + * + * @param array $config The step's config block. + * @param int $order The step's order. + * + * @return array Zero or one sentence. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function eventRefusals(array $config, int $order): array { + if (trim((string)($config['type'] ?? '')) !== '') { + return []; + } + + return [sprintf('Step %d (event) names no event `type`.', $order)]; + + }//end eventRefusals() + + /** + * The `approval` step configuration the approval-request node would reject. + * + * @param array $config The step's config block. + * @param int $order The step's order. + * + * @return array Zero or one sentence. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function approvalRefusals(array $config, int $order): array { + if (trim((string)($config['approverGroup'] ?? '')) !== '') { + return []; + } + + return [ + sprintf( + 'Step %d (approval) names no `approverGroup`; the approval-request node requires an audience, ' + . 'because a request nobody owns never resolves.', + $order + ), + ]; + + }//end approvalRefusals() + + /** + * The refusals for branch targets that resolve to no step. + * + * @param array $step The step definition. + * @param int $order The step's order. + * @param string $type The step's (supported) type. + * @param array $orders Every declared order. + * + * @return array One sentence per dangling target. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function branchRefusals(array $step, int $order, string $type, array $orders): array { + if ($type !== 'branch') { + return []; + } + + $reasons = []; + foreach ($this->branchTargets(step: $step) as $target) { + if (in_array($target, $orders, true) === false) { + $reasons[] = sprintf('Step %d (branch) targets step order %d, which does not exist.', $order, $target); + } + } + + return $reasons; + + }//end branchRefusals() + + /** + * The graph node standing in for one step. + * + * @param array $step The step definition. + * @param string $id The node id (the step's order, verbatim). + * + * @return array The node. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function nodeFor(array $step, string $id): array { + $type = (string)$step['type']; + $node = [ + 'id' => $id, + 'type' => self::TYPE_MAP[$type], + 'config' => $this->configFor(step: $step, type: $type), + ]; + + // The engine reads the policy from the STEP definition + // (`$step['onError']`), same key, same vocabulary — carry it over + // verbatim so stop/continue/dead_letter behave as authored. + if (trim((string)($step['onError'] ?? '')) !== '') { + $node['onError'] = (string)$step['onError']; + } + + return $node; + + }//end nodeFor() + + /** + * The node config standing in for one step's `configRef`/`config`. + * + * @param array $step The step definition. + * @param string $type The step's type. + * + * @return array The node config. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function configFor(array $step, string $type): array { + $config = (array)($step['config'] ?? []); + $configRef = trim((string)($step['configRef'] ?? '')); + + switch ($type) { + case 'call': + return [ + 'source' => $configRef, + 'endpoint' => (string)($config['endpoint'] ?? ''), + 'method' => (string)($config['method'] ?? 'GET'), + 'output' => 'response', + ]; + case 'mapping': + // No `input`/`output`: the node then maps the whole item and + // replaces it with the result, which is exactly the runner's + // output-becomes-next-input threading. + return ['mapping' => $configRef]; + case 'synchronization': + $node = [ + 'synchronization' => $configRef, + 'output' => 'syncResult', + ]; + if (array_key_exists('force', $config) === true) { + $node['force'] = (bool)$config['force']; + } + + return $node; + case 'event': + $node = [ + 'type' => (string)($config['type'] ?? ''), + 'source' => (string)($config['source'] ?? ''), + ]; + if (trim((string)($config['subject'] ?? '')) !== '') { + $node['subject'] = (string)$config['subject']; + } + + return $node; + case 'approval': + return [ + 'question' => $this->approvalQuestion(step: $step, config: $config), + 'approverGroup' => (string)($config['approverGroup'] ?? ''), + 'ttlSeconds' => (int)($config['ttlSeconds'] ?? ApprovalService::DEFAULT_TTL_SECONDS), + // The runner's `onReject` vocabulary: anything but `skip` + // (error, dead_letter) ended the run, so it maps to the + // node failing on rejection. + 'failOnReject' => ((string)($config['onReject'] ?? 'error')) !== 'skip', + ]; + default: + // The branch case: the switch anchor carries no config — its + // routing lives on the conditioned edges. + return []; + }//end switch + + }//end configFor() + + /** + * The question an approval step asks, synthesised when the step has none. + * + * @param array $step The step definition. + * @param array $config The step's config block. + * + * @return string The question. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function approvalQuestion(array $step, array $config): string { + $question = trim((string)($config['question'] ?? '')); + if ($question !== '') { + return $question; + } + + return sprintf('Approve step %d of this flow.', (int)($step['order'] ?? 0)); + + }//end approvalQuestion() + + /** + * The conditioned edges leaving a branch step. + * + * Each `branches[]` entry becomes an edge carrying its JsonLogic + * condition verbatim — the engine evaluates edge conditions with the + * same JsonLogic the runner used. The default edge (no condition) goes + * to `defaultNextStepOrder` when declared and otherwise to the next + * sequential step, which is the runner's own fallthrough. + * + * @param array $step The branch step definition. + * @param string $id The branch node's id. + * @param string $nextId The sequential successor's node id (or the end node). + * + * @return array> The edges. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function branchEdges(array $step, string $id, string $nextId): array { + $edges = []; + + foreach ((array)($step['branches'] ?? []) as $index => $branch) { + if (is_array($branch) === false || empty($branch['condition']) === true || isset($branch['nextStepOrder']) === false) { + continue; + } + + $target = (string)((int)$branch['nextStepOrder']); + $edges[] = [ + 'id' => sprintf('%s-%s-%d', $id, $target, (int)$index), + 'from' => $id, + 'to' => $target, + 'condition' => $branch['condition'], + ]; + } + + $defaultTarget = $nextId; + if (isset($step['defaultNextStepOrder']) === true) { + $defaultTarget = (string)((int)$step['defaultNextStepOrder']); + } + + $edges[] = [ + 'id' => sprintf('%s-%s-default', $id, $defaultTarget), + 'from' => $id, + 'to' => $defaultTarget, + ]; + + return $edges; + + }//end branchEdges() + + /** + * The branch targets a branch step declares, for existence checks. + * + * @param array $step The branch step definition. + * + * @return array Every referenced step order. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function branchTargets(array $step): array { + $targets = []; + + foreach ((array)($step['branches'] ?? []) as $branch) { + if (is_array($branch) === true && isset($branch['nextStepOrder']) === true) { + $targets[] = (int)$branch['nextStepOrder']; + } + } + + if (isset($step['defaultNextStepOrder']) === true) { + $targets[] = (int)$step['defaultNextStepOrder']; + } + + return $targets; + + }//end branchTargets() + + /** + * Sort steps by `order` ascending — the runner's own execution sequence. + * + * @param array $steps Raw `steps[]` from the flow record. + * + * @return array Steps sorted ascending by `order`. + * + * @spec openspec/changes/retire-integriq-flow-schema/tasks.md#2-the-step-to-graph-migration + */ + private function sortedSteps(array $steps): array { + $steps = array_values(array_filter($steps, static fn ($step): bool => is_array($step))); + usort( + $steps, + static fn (array $a, array $b): int => (((int)($a['order'] ?? 0)) <=> ((int)($b['order'] ?? 0))) + ); + + return $steps; + + }//end sortedSteps() +}//end class diff --git a/lib/Settings/integriq_mock_register.json b/lib/Settings/integriq_mock_register.json index 03c485d42..129d4186f 100644 --- a/lib/Settings/integriq_mock_register.json +++ b/lib/Settings/integriq_mock_register.json @@ -2,7 +2,7 @@ "openapi": "3.0.0", "info": { "title": "integriq demo data", - "version": "1.0.0", + "version": "1.1.0", "description": "Demo data covering every schema this app supplies, offered as the first step of the app's setup walkthrough. Generated from the schemas themselves, so every object satisfies the schema that will validate it." }, "x-openregister": { @@ -3004,141 +3004,6 @@ "appendOnly": false, "immutable": false }, - "flow": { - "slug": "flow", - "title": "Flow", - "icon": "Sitemap", - "version": "1.0.0", - "summary": "A declarative, ordered multi-step pipeline referencing existing Source/Mapping/Synchronization/Endpoint/Approval entities by id", - "description": "An ordered list of steps (call/mapping/synchronization/event/approval/branch), each a thin reference to an existing entity. Executed by FlowRunnerService, which calls the referenced entity's own existing service method — no step type reimplements that logic. See openspec/specs/flow-orchestration/spec.md REQ-001.", - "required": [ - "name", - "steps" - ], - "type": "object", - "properties": { - "uuid": { - "type": "string", - "description": "Canonical UUID assigned by OpenRegister", - "title": "UUID" - }, - "name": { - "type": "string", - "description": "Human-readable flow name", - "title": "Name" - }, - "description": { - "type": "string", - "description": "What this flow does", - "title": "Description" - }, - "isEnabled": { - "type": "boolean", - "default": true, - "description": "When false, cron/endpoint/event triggers skip this flow; a manual Run still executes it (matches job/synchronization isEnabled precedent)", - "title": "Enabled" - }, - "steps": { - "type": "array", - "title": "Steps", - "description": "Ordered list of flow steps. Execution order is each item's own `order` field (a stable identifier), not array position, so `branch` targets (nextStepOrder/defaultNextStepOrder) stay valid across insert/delete/reorder edits in the step-list editor. `order` values MUST be unique within a flow — FlowRunnerService::run() rejects a flow with duplicate step `order` values as a fatal configuration error before executing any step, and the step-list editor validates the same rule client-side before save.", - "items": { - "type": "object", - "required": [ - "order", - "type", - "onError" - ], - "properties": { - "order": { - "type": "integer", - "description": "Stable step identifier and default execution sequence (ascending). branch steps' nextStepOrder/defaultNextStepOrder reference this value, not array position. MUST be unique within the flow.", - "title": "Order" - }, - "type": { - "type": "string", - "enum": [ - "call", - "mapping", - "synchronization", - "event", - "approval", - "branch" - ], - "description": "Which existing service this step dispatches to: call -> CallService::call(), mapping -> MappingService::executeMapping(), synchronization -> SynchronizationService::synchronize(), event -> EventService::emitCloudEvent(), approval -> suspend/resume via ApprovalService, branch -> JsonLogic-selected next step (no service call).", - "title": "Type" - }, - "configRef": { - "type": "string", - "format": "uuid", - "description": "Id of the existing Source (call) / Mapping (mapping) / Synchronization (synchronization) entity this step invokes. Not applicable to event/approval (config carries their parameters) or branch steps.", - "title": "Config Reference" - }, - "condition": { - "type": "object", - "description": "Optional JsonLogic run-if rule, evaluated via JWadhams\\JsonLogic::apply() against the current step context. Step runs only when this evaluates loosely true. Absent/empty = always run.", - "title": "Condition" - }, - "onError": { - "type": "string", - "enum": [ - "stop", - "continue", - "dead_letter" - ], - "default": "stop", - "description": "stop: flow_run.status becomes stopped, no later step runs. continue: the run proceeds to the next step. dead_letter: flow_run.status becomes dead_letter (distinct from stopped), no later step runs.", - "title": "On Error" - }, - "config": { - "type": "object", - "description": "Type-specific parameters this step's dispatch needs beyond configRef: call={endpoint?, method?, requestConfig?}; event={source, subject?, type} (EventService::emitCloudEvent() args, design.md Decision 5); approval={approverGroup, onReject?, onTimeout?, ttlSeconds?} (mirrors the approval rule action's own config shape); synchronization={isTest?, force?, mutationType?} (optional overrides, default off so sync-safety guards are never bypassed by a flow step).", - "title": "Step Configuration" - }, - "branches": { - "type": "array", - "description": "branch steps only. Evaluated in array order via JsonLogic::apply(); the first matching entry's nextStepOrder is selected.", - "title": "Branches", - "items": { - "type": "object", - "properties": { - "condition": { - "type": "object", - "description": "JsonLogic rule for this branch", - "title": "Branch condition" - }, - "nextStepOrder": { - "type": "integer", - "description": "The order of the step to jump to when this branch's condition matches", - "title": "Next step order" - } - } - } - }, - "defaultNextStepOrder": { - "type": "integer", - "description": "branch steps only. Used when no branches[].condition matches. When absent, execution continues to the next step in order sequence.", - "title": "Default Next Step Order" - } - } - } - }, - "created": { - "type": "string", - "format": "date-time", - "description": "OR-managed creation timestamp", - "title": "Created" - }, - "updated": { - "type": "string", - "format": "date-time", - "description": "OR-managed update timestamp", - "title": "Updated" - } - }, - "appendOnly": false, - "immutable": false - }, "flow_run": { "slug": "flow_run", "title": "Flow Run", @@ -8975,87 +8840,6 @@ "updated": "2026-03-03T09:00:00+00:00", "expires": "2026-03-03T09:00:00+00:00" }, - { - "@self": { - "register": "integriq", - "schema": "flow", - "slug": "flow-voorbeeld-name-1-1" - }, - "name": "Voorbeeld Name 1", - "steps": [ - { - "order": 1, - "type": "call", - "onError": "stop", - "configRef": "00000000-0000-4000-8000-000000000000", - "condition": {}, - "config": {}, - "branches": [ - {} - ], - "defaultNextStepOrder": 1 - } - ], - "uuid": "Voorbeeld Uuid 1", - "description": "Voorbeeld Description 1", - "isEnabled": true, - "created": "2026-03-01T09:00:00+00:00", - "updated": "2026-03-01T09:00:00+00:00" - }, - { - "@self": { - "register": "integriq", - "schema": "flow", - "slug": "flow-voorbeeld-name-2-2" - }, - "name": "Voorbeeld Name 2", - "steps": [ - { - "order": 2, - "type": "mapping", - "onError": "continue", - "configRef": "00000000-0000-4000-8000-000000000001", - "condition": {}, - "config": {}, - "branches": [ - {} - ], - "defaultNextStepOrder": 2 - } - ], - "uuid": "Voorbeeld Uuid 2", - "description": "Voorbeeld Description 2", - "isEnabled": true, - "created": "2026-03-02T09:00:00+00:00", - "updated": "2026-03-02T09:00:00+00:00" - }, - { - "@self": { - "register": "integriq", - "schema": "flow", - "slug": "flow-voorbeeld-name-3-3" - }, - "name": "Voorbeeld Name 3", - "steps": [ - { - "order": 3, - "type": "synchronization", - "onError": "dead_letter", - "configRef": "00000000-0000-4000-8000-000000000002", - "condition": {}, - "config": {}, - "branches": [ - {} - ], - "defaultNextStepOrder": 3 - } - ], - "uuid": "Voorbeeld Uuid 3", - "description": "Voorbeeld Description 3", - "isEnabled": true, - "created": "2026-03-03T09:00:00+00:00", - "updated": "2026-03-03T09:00:00+00:00" - }, { "@self": { "register": "integriq", @@ -11209,6 +10993,183 @@ "translatedAt": "2026-03-03T09:00:00+00:00", "created": "2026-03-03T09:00:00+00:00", "updated": "2026-03-03T09:00:00+00:00" + }, + { + "@self": { + "register": "integriq", + "schema": "flow", + "slug": "flow-demo-1" + }, + "name": "Demo: enrich and store", + "description": "Calls a source, maps the answer and stores it. Demo data, safe to delete.", + "isEnabled": false, + "steps": [ + { + "order": 10, + "type": "call", + "configRef": "00000000-0000-4000-8000-000000000000", + "onError": "stop", + "config": { + "endpoint": "/organisations/123", + "method": "GET" + } + }, + { + "order": 20, + "type": "mapping", + "configRef": "00000000-0000-4000-8000-000000000000", + "onError": "stop" + }, + { + "order": 30, + "type": "synchronization", + "configRef": "00000000-0000-4000-8000-000000000000", + "onError": "dead_letter" + } + ] + }, + { + "@self": { + "register": "integriq", + "schema": "flow", + "slug": "flow-demo-2" + }, + "name": "Demo: gated publish", + "description": "Asks the approver group before emitting the publish event. Demo data, safe to delete.", + "isEnabled": false, + "steps": [ + { + "order": 10, + "type": "approval", + "onError": "stop", + "config": { + "approverGroup": "demo-approvers", + "onReject": "error", + "ttlSeconds": 86400 + } + }, + { + "order": 20, + "type": "event", + "onError": "continue", + "config": { + "type": "nl.example.dataset.published", + "source": "https://example.org/integriq" + } + } + ] + }, + { + "@self": { + "register": "integriq", + "schema": "flow", + "slug": "flow-demo-3" + }, + "name": "Demo: branched routing (migrated shape)", + "description": "A branch step choosing between two mappings, carrying the migrated nodes/edges graph beside its steps. Demo data, safe to delete.", + "isEnabled": false, + "steps": [ + { + "order": 10, + "type": "branch", + "onError": "stop", + "branches": [ + { + "condition": { + "==": [ + { + "var": "syncInputAmended.kind" + }, + "a" + ] + }, + "nextStepOrder": 40 + } + ], + "defaultNextStepOrder": 50 + }, + { + "order": 40, + "type": "mapping", + "configRef": "00000000-0000-4000-8000-000000000000", + "onError": "stop" + }, + { + "order": 50, + "type": "mapping", + "configRef": "00000000-0000-4000-8000-000000000000", + "onError": "stop" + } + ], + "nodes": [ + { + "id": "trigger", + "type": "openregister.trigger-manual", + "config": {} + }, + { + "id": "10", + "type": "openregister.switch", + "config": {}, + "onError": "stop" + }, + { + "id": "40", + "type": "openconnector.apply-mapping", + "config": { + "mapping": "00000000-0000-4000-8000-000000000000" + }, + "onError": "stop" + }, + { + "id": "50", + "type": "openconnector.apply-mapping", + "config": { + "mapping": "00000000-0000-4000-8000-000000000000" + }, + "onError": "stop" + }, + { + "id": "end", + "type": "openregister.end", + "config": {} + } + ], + "edges": [ + { + "id": "trigger-10", + "from": "trigger", + "to": "10" + }, + { + "id": "10-40-0", + "from": "10", + "to": "40", + "condition": { + "==": [ + { + "var": "syncInputAmended.kind" + }, + "a" + ] + } + }, + { + "id": "10-50-default", + "from": "10", + "to": "50" + }, + { + "id": "40-end", + "from": "40", + "to": "end" + }, + { + "id": "50-end", + "from": "50", + "to": "end" + } + ] } ] } diff --git a/lib/Settings/register.d/visual-flow-orchestration.json b/lib/Settings/register.d/visual-flow-orchestration.json index df1943d57..65184c021 100644 --- a/lib/Settings/register.d/visual-flow-orchestration.json +++ b/lib/Settings/register.d/visual-flow-orchestration.json @@ -1,5 +1,5 @@ { - "$comment": "ADR-037 register fragment (visual-flow-orchestration). Declares the `flow` schema (an ordered, declarative multi-step pipeline of existing Source/Mapping/Synchronization/Endpoint/Approval references) plus its `flow_run`/`flow_run_log` execution-trace schemas, and additively extends `approval_request` (declared by hitl-approval-rule-action.json) with `flowRunId`/`resumeStepOrder` so an `approval` flow step can suspend/resume through the same approval_request state machine. See openspec/changes/archive/2026-07-15-visual-flow-orchestration/design.md Decisions 1/4/6. Deep-merge note: this fragment only adds the two new `approval_request` properties (InitializeRegister::deepMergeConfig unions `components.schemas.approval_request.properties` by key with hitl-approval-rule-action.json's — glob sort places this file after the `h...` fragment alphabetically, so the union always sees the base schema first); it does not repeat approval_request's existing properties/required/seed data.", + "$comment": "ADR-037 register fragment (visual-flow-orchestration). Declares the `flow` schema (an ordered, declarative multi-step pipeline of existing Source/Mapping/Synchronization/Endpoint/Approval references) plus its `flow_run`/`flow_run_log` execution-trace schemas, and additively extends `approval_request` (declared by hitl-approval-rule-action.json) with `flowRunId`/`resumeStepOrder` so an `approval` flow step can suspend/resume through the same approval_request state machine. See openspec/changes/archive/2026-07-15-visual-flow-orchestration/design.md Decisions 1/4/6. Deep-merge note: this fragment only adds the two new `approval_request` properties (InitializeRegister::deepMergeConfig unions `components.schemas.approval_request.properties` by key with hitl-approval-rule-action.json's — glob sort places this file after the `h...` fragment alphabetically, so the union always sees the base schema first); it does not repeat approval_request's existing properties/required/seed data. retire-integriq-flow-schema additions: approval_request additionally gains engineRunUuid/signalNodeId/question (engine-run approvals via the openconnector.approval-request node), and flow gains nodes/edges (the steps-to-graph migration writes the graph IN PLACE next to steps, so the two engines dual-run until FlowRunnerService is retired; steps is kept as the rollback shape).", "components": { "registers": { "integriq": { @@ -119,6 +119,18 @@ "format": "date-time", "description": "OR-managed update timestamp", "title": "Updated" + }, + "nodes": { + "type": "array", + "title": "Nodes", + "description": "OpenRegister flow-engine graph nodes ({id, type, config, onError}), written IN PLACE by the steps-to-graph migration (retire-integriq-flow-schema Task 2). While both shapes are present, `steps` remains what FlowRunnerService executes and `nodes`/`edges` is what OpenRegister's engine executes; the migration refuses to overwrite an object that already carries nodes, and `steps` is kept as the rollback shape.", + "items": {"type": "object"} + }, + "edges": { + "type": "array", + "title": "Edges", + "description": "OpenRegister flow-engine graph edges ({id, from, to, condition?}), written by the steps-to-graph migration alongside `nodes`. Branch steps' JsonLogic conditions ride on the edges, matching the engine's edge-condition routing.", + "items": {"type": "object"} } }, "appendOnly": false, @@ -269,6 +281,21 @@ "type": "integer", "description": "The `order` of the flow step to resume at once this request is approved — the step immediately after the suspending `approval` step (flow-orchestration REQ-005). Not applicable to the endpoint-rule or Synchronization batch-gate cases.", "title": "Resume Step Order" + }, + "engineRunUuid": { + "type": "string", + "description": "Opaque handle naming the suspended OpenRegister ENGINE flow run this request gates (openconnector.approval-request node case); set instead of endpointId/ruleId/synchronizationId/flowRunId. Deliberately NOT a $ref: engine runs live in OpenRegister's own oc_openregister_flow_runs table, not as objects of any schema this register could reference, so there is nothing to relate to (ADR-062 rule 7 does not apply). The decision is delivered through FlowRunSignalService::signalAs(), and the node's heartbeat re-reads this record when that delivery is lost (retire-integriq-flow-schema Task 1).", + "title": "Engine Run Uuid" + }, + "signalNodeId": { + "type": "string", + "description": "Graph node id of the openconnector.approval-request step awaiting the decision, so the signal addresses that node's resume slot (and the engine's assignee guard checks ITS recorded approver group).", + "title": "Signal Node Id" + }, + "question": { + "type": "string", + "description": "What is being asked, verbatim from the approval step's config. Shown to approvers; makes a pending request explain itself.", + "title": "Question" } } } diff --git a/openapi.json b/openapi.json index 6de7d4b06..812593d62 100644 --- a/openapi.json +++ b/openapi.json @@ -2,7 +2,7 @@ "openapi": "3.0.3", "info": { "title": "integriq", - "version": "0.3.13-unstable.20260831053109", + "version": "0.3.16-unstable.20260831214826", "description": "open connector", "license": { "name": "EUPL-1.2" diff --git a/openspec/changes/absorb-dossiq-deliveries/design.md b/openspec/changes/absorb-dossiq-deliveries/design.md new file mode 100644 index 000000000..80172c2a0 --- /dev/null +++ b/openspec/changes/absorb-dossiq-deliveries/design.md @@ -0,0 +1,47 @@ +# Design — absorb-dossiq-deliveries + +## Landing zone: the CloudEvents pipeline, not a new engine + +The integriq audit ranked four landing zones for a sibling app's delivery: a Flow node, an +`event_subscription` action, a bespoke provider quintet, and raw `CallService`. The seam lands on +the **event pipeline** because the requesting context is a backend transition handler (no user +session for the sibling-push controllers, no admin-authored flow at the dispatch site), and because +the pipeline already owns exactly the semantics a delivery needs: per-subscription retry policy, +exponential backoff, dead-letter + replay UI, HMAC signing, and status bookkeeping on +`event_message`. A flow can still do the actual transport — `action.kind = 'flow'` on the matching +subscription — so the seam composes with the wave-3 direction instead of competing with it. + +The legacy runners (`SynchronizationService`, `RuleService`, `JobService`, `FlowRunnerService`) are +never called directly by the seam; they are reachable only as subscription actions that already +existed. + +## The provenance gate + +`ingestDeliveryRequest()` writes the request's provenance into the event's `data.delivery` block. +`createEventMessage()` embeds the event's serialization in `event_message.payload`, so the terminal +hooks (`recordDeliverySuccess`, terminal `recordFailure`) can read +`payload.data.delivery.{sourceApp,correlationId}` without a second lookup. That block is the gate: +present → dispatch `DeliveryConcludedEvent`; absent → ordinary CloudEvent traffic, no conclusion. +`recordConfigurationError` does not conclude — a config error is operator-fixable and replayable, +not terminal. + +## Result-slot honesty + +`setMatchedSubscriptions()` exists so "accepted" and "will actually travel" are distinguishable. +Zero matches means the instance has no route for this delivery — the consumer records `unrouted` +and an operator configures a subscription; nothing pretends to deliver. This is the +fail-closed-refusal shape the fleet ruling requires. + +## Constructor compatibility + +`IEventDispatcher` joins the constructor as a nullable, defaulted final parameter — the same +pattern `ExecutionTraceService` used — so every pre-existing positional test instantiation keeps +working and DI supplies the real dispatcher in production. A null dispatcher simply skips +conclusions (unit-test contexts); the listener half is unaffected. + +## Replay semantics + +`replayMessage()` can revive an abandoned message. If the replay succeeds, a second conclusion +(`delivered`) is dispatched and supersedes the earlier `abandoned` at the consumer — consumers +MUST project last-terminal-state-wins (dossiq's listener does). This is deliberate: the message +record and the consumer's projection converge without a tombstone protocol. diff --git a/openspec/changes/absorb-dossiq-deliveries/proposal.md b/openspec/changes/absorb-dossiq-deliveries/proposal.md new file mode 100644 index 000000000..6e3f36b8d --- /dev/null +++ b/openspec/changes/absorb-dossiq-deliveries/proposal.md @@ -0,0 +1,74 @@ +# Proposal: absorb-dossiq-deliveries + +kind: capability — cites **ADR-041** (hydra org-wide: cross-app commands via typed events), +**ADR-013** (event-bus model) and the `events-cloudevents` spec. Coupled to the dossiq change +`dossiq-delivers-nothing`, which ships the requesting half. Train order: this PR merges first — it +defines the event contract dossiq's `class_exists()`-guarded dispatch resolves; dossiq's half fails +closed until then, so no ordering breakage either way. + +## Summary + +Fleet ruling: **case apps keep no delivery code — integrations belong to integriq.** This change +gives integriq the receiving half of the ADR-041 delivery seam so a sibling app (dossiq first) can +hand over an outbound delivery and get an honest, terminal answer back: + +1. **`OCA\Integriq\Event\DeliveryRequestedEvent`** — the typed cross-app command ("deliver this + payload on my behalf"), carrying provenance (`sourceApp`, subject register/schema/id/label), + `deliveryKind`, `channel`, a caller-composed payload, a `correlationId`, and a synchronous + result slot (`isHandled` / `getResultId` / `getMatchedSubscriptions`). +2. **`DeliveryRequestedListener`** — ingests the request into the existing CloudEvents pipeline as + a `nl.conduction.delivery.requested` event whose `data.delivery` block carries the provenance; + admin-configured `event_subscription`s route it to a webhook / flow / synchronization / + notificaties action and inherit retry, backoff, dead-letter, replay and HMAC signing unchanged. + Zero matched subscriptions is reported honestly so the consumer fail-closes as "unrouted". +3. **`OCA\Integriq\Event\DeliveryConcludedEvent`** — dispatched from the `event_message` state + machine when a provenance-carrying delivery reaches a terminal state: `delivered` on success, + `abandoned` when the retry budget is spent. Ordinary CloudEvent traffic (no provenance block) + never produces one. The consumer projects the outcome onto its own domain record (dossiq: the + case's publication entry). + +No new transport, no new engine: the seam is a thin typed-event skin over `EventService`, and it +deliberately does NOT touch the wave-3 retirement targets (`SynchronizationService`, `RuleService`, +`JobService`, `FlowRunnerService` are not called directly — a flow can still be the *subscription's +action*). + +## Why + +ADR-041 requires cross-app commands to travel as typed events defined by the target app; integriq +had no such contract (no ADR-041 recipe existed in this repo before this change). Meanwhile every +delivery-shaped surface dossiq carries is either unreachable, mocked, or retry-less, and integriq +already operates the machinery all of them need. The seam lets sibling apps shed transport without +integriq growing bespoke per-app code: one contract, provenance-routed subscriptions. + +The sibling-push controllers (`stufZkn#outbound`, `iwmoIjw#createMessage`, ...) stay: they serve +session-carrying frontend calls. The event seam serves backend/flow contexts where a server-side +HTTP call would 401 (the exact phantom ADR-041 documents). + +## What + +1. `lib/Event/DeliveryRequestedEvent.php` + `lib/Event/DeliveryConcludedEvent.php` (new). +2. `lib/EventListener/DeliveryRequestedListener.php` (new), registered in `Application::boot()`. +3. `EventService::ingestDeliveryRequest()` (new public method): persists the provenance-carrying + `event` object, fans out via `processEvent()`, returns event + created messages. +4. `EventService` terminal-state hooks: `recordDeliverySuccess()` and the terminal branch of + `recordFailure()` dispatch `DeliveryConcludedEvent` via a new nullable `IEventDispatcher` + constructor dependency (nullable + defaulted, same test-compatibility pattern as + `ExecutionTraceService`). Dispatch failures are logged and swallowed — the message record stays + the source of truth. +5. Unit tests: `EventServiceDeliverySeamTest` (ingest shape, delivered/abandoned dispatch, no + dispatch without provenance or on non-terminal failure), `DeliveryRequestedListenerTest` + (result-slot write-back, unhandled-on-ingest-failure, foreign-event ignore). + +## Follow-ups staged in tasks.md + +Phase 2 tracks the integriq-side halves of dossiq's staged extractions: StUF endpoint/credential +migration intake, a per-callback notificaties routing decision, and (on commission) real +Berichtenbox / DROP-LVBB transports as provider quintets. Each carries its blocker honestly. + +## Non-goals + +- A delivery-specific message schema: `event_message` + the CloudEvent `data.delivery` block + already carry everything the seam needs (the `*_message` quintet pattern stays reserved for + bespoke wire protocols with their own inbound leg). +- Replay semantics changes: a replayed abandoned message that later succeeds simply dispatches a + second, superseding `delivered` conclusion — consumers project last-terminal-state-wins. diff --git a/openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md b/openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md new file mode 100644 index 000000000..e3ed41897 --- /dev/null +++ b/openspec/changes/absorb-dossiq-deliveries/specs/delivery-intake/spec.md @@ -0,0 +1,86 @@ +# delivery-intake Specification + +**Status:** proposed +**Scope:** integriq +**Tier:** V1 +**Depends on:** `events-cloudevents` spec (the `event`/`event_subscription`/`event_message` +pipeline this seam rides), Nextcloud `OCP\EventDispatcher\IEventDispatcher`. + +## Purpose + +The ADR-041 cross-app delivery seam: a sibling Conduction app composes WHAT must be delivered and +raises a typed event; integriq owns HOW it travels by fanning the request out through its +CloudEvents pipeline, and answers with a terminal conclusion the consumer projects onto its own +domain record. + +@e2e exclude The seam is a backend-only in-process typed-event exchange with no integriq browser +surface of its own: requests and conclusions surface in the existing Events / DeadLetters pages, +which have their own coverage. The seam behaviours are proven by the PHPUnit suites +(EventServiceDeliverySeamTest, DeliveryRequestedListenerTest) on this side and dossiq's +PublicationServiceTest / DeliveryConcludedListenerTest on the consumer side. + +## ADDED Requirements + +### Requirement: A delivery request is a typed event with a synchronous result slot + +Integriq SHALL expose `OCA\Integriq\Event\DeliveryRequestedEvent` carrying provenance +(`sourceApp`, `subjectRegister`, `subjectSchema`, `subjectId`, `subjectLabel`), a `deliveryKind`, +a `channel`, a caller-composed `payload`, a `correlationId`, and optional `externalReference` / +`userId`. The in-process listener SHALL write the result slot: `setHandled(true)`, the persisted +CloudEvent uuid via `setResultId()`, and the matched-subscription count via +`setMatchedSubscriptions()`. On ingest failure the event SHALL stay unhandled so the consumer +fail-closes. + +#### Scenario: A handled request carries the result slot + +- **GIVEN** the CloudEvents pipeline persists the request and one subscription matches +- **WHEN** the listener handles a `DeliveryRequestedEvent` +- **THEN** `isHandled()` MUST be true, `getResultId()` MUST be the event uuid, and + `getMatchedSubscriptions()` MUST be 1 + +#### Scenario: An ingest failure leaves the request unhandled + +- **WHEN** persisting or fanning out the request throws +- **THEN** the event MUST stay unhandled and MUST carry no result id + +### Requirement: Delivery requests ride the CloudEvents pipeline unchanged + +The listener SHALL persist the request as an `event` object of type +`nl.conduction.delivery.requested` with source `/apps//delivery`, the subject id as the +CloudEvents subject, and a `data.delivery` block carrying the full provenance, then fan it out via +`processEvent()`. Routing, retry, backoff, dead-letter, replay and HMAC signing SHALL be the +existing `event_subscription` / `event_message` machinery — no delivery-specific engine, and no +direct call into the legacy synchronization/rule/job runners. + +#### Scenario: The persisted event carries provenance + +- **WHEN** a request from `dossiq` for channel `gemeenteblad` is ingested +- **THEN** the persisted event MUST have type `nl.conduction.delivery.requested`, source + `/apps/dossiq/delivery`, and `data.delivery.sourceApp = 'dossiq'` with the correlation id + +### Requirement: A provenance-carrying delivery concludes with a typed terminal event + +When an `event_message` whose originating event carries a `data.delivery` provenance block reaches +a terminal state, integriq SHALL dispatch `OCA\Integriq\Event\DeliveryConcludedEvent` — +`delivered` from the success path, `abandoned` when the retry budget is spent — echoing +`sourceApp`, `correlationId`, `subjectId` and `channel`, with the attempt count, the last error (or +null) and the terminal timestamp. Ordinary CloudEvent traffic without the provenance block SHALL +never produce a conclusion, a non-terminal failure SHALL not conclude, and a conclusion-dispatch +failure SHALL be logged and swallowed — the message record stays the source of truth. + +#### Scenario: Success concludes delivered + +- **GIVEN** a pending message whose event data carries `delivery.sourceApp` and `correlationId` +- **WHEN** delivery succeeds +- **THEN** a `DeliveryConcludedEvent` with status `delivered` and the echoed correlation id MUST be + dispatched + +#### Scenario: A spent retry budget concludes abandoned + +- **WHEN** a provenance-carrying message fails with no retries remaining +- **THEN** a `DeliveryConcludedEvent` with status `abandoned` and the last error MUST be dispatched + +#### Scenario: Ordinary traffic never concludes + +- **WHEN** a message without a `data.delivery` provenance block reaches any terminal state +- **THEN** no `DeliveryConcludedEvent` is dispatched diff --git a/openspec/changes/absorb-dossiq-deliveries/tasks.md b/openspec/changes/absorb-dossiq-deliveries/tasks.md new file mode 100644 index 000000000..58a6fcf46 --- /dev/null +++ b/openspec/changes/absorb-dossiq-deliveries/tasks.md @@ -0,0 +1,42 @@ +# Tasks — absorb dossiq deliveries: the ADR-041 delivery seam + +## Phase 1: The delivery seam (this PR) + +- [x] `lib/Event/DeliveryRequestedEvent.php` — provenance + payload + synchronous result slot + (`setHandled`/`isHandled`, `setResultId`/`getResultId`, `setMatchedSubscriptions`). +- [x] `lib/Event/DeliveryConcludedEvent.php` — terminal outcome envelope (`delivered` / + `abandoned`, attempts, error, concludedAt), echoing sourceApp + correlationId + subject. +- [x] `lib/EventListener/DeliveryRequestedListener.php` — ingest via + `EventService::ingestDeliveryRequest()`, write the result slot; leave unhandled on ingest + failure so the consumer fail-closes. +- [x] `EventService::ingestDeliveryRequest()` — persist the `nl.conduction.delivery.requested` + CloudEvent with the `data.delivery` provenance block, fan out via `processEvent()`. +- [x] `EventService::dispatchDeliveryConcluded()` — dispatched from `recordDeliverySuccess()` and + the terminal (`abandoned`) branch of `recordFailure()`, gated to provenance-carrying + messages; new nullable `IEventDispatcher` constructor dependency. +- [x] Register the listener in `Application::boot()`. +- [x] Unit tests: ingest event shape, delivered dispatch, abandoned dispatch with error, + no dispatch on non-terminal failure, no dispatch without provenance, listener result-slot + write-back, listener unhandled-on-failure, foreign-event ignore. + +## Phase 2: Intake halves of dossiq's staged extractions — staged + +- [ ] **StUF endpoint/credential migration intake.** Blocked on: migration design — dossiq's + `stufEndpoint` objects hold `vault://` refs resolved via dossiq `IAppConfig`; integriq + sources resolve through the OpenRegister credential broker. Needs a documented mapping + (dossiq repair step writes `source` objects `type=stuf-zkn` with broker refs; secrets are + re-entered or brokered, never copied blind). Tracked jointly with dossiq + `dossiq-delivers-nothing` phase 2. +- [ ] **Per-callback ZGW notificaties routing.** Blocked on: a design decision — dossiq's + notificaties fan-out is per-abonnement callback URLs; the seam carries one delivery request, + while subscriptions are admin-configured. Either the `notificaties` action kind gains + callback-from-payload support, or dossiq raises one request per callback. Decide before + dossiq phase 3 lands. +- [ ] **Berichtenbox (MijnOverheid) transport.** Blocked on: commissioning — no production + transport exists anywhere in the fleet (dossiq ships only a MockAdapter). When built, it is + an integriq provider quintet (controller + provider seam + sync service + `*_message` schema + + retry job, the StufZkn/IwmoIjw pattern) addressed by `deliveryKind: 'berichtenbox'`. +- [ ] **DROP/LVBB publication transport.** Blocked on: commissioning — no DROP/LVBB transport + exists in dossiq to move (its PublicationService was record-only); a real + bekendmaking-via-DROP delivery is new integriq work, addressed by the existing + `deliveryKind: 'besluit-publication'` routing on channel `gemeenteblad`. diff --git a/openspec/changes/api-product-gateway/design.md b/openspec/changes/api-product-gateway/design.md deleted file mode 100644 index 822de9e17..000000000 --- a/openspec/changes/api-product-gateway/design.md +++ /dev/null @@ -1,336 +0,0 @@ -# Design: api-product-gateway - -## Architecture Overview - -``` - ┌────────────────────┐ - Consumer subscribes │ api_product │ groups N Endpoints (uuid[]) - ───────────────────► │ - tiers{name:policy}│ version + status + sunsetDate - │ - defaultTier │ visibility - └─────────┬───────────┘ - │ 1:N - ┌─────────▼───────────┐ requiresApproval? ──► ApprovalService - │ api_product_ │ (approval_request, generic) - │ subscription │ - │ - consumer, tier │ - │ - status │ - └─────────┬───────────┘ - │ resolved at request time - inbound request ──► EndpointService::doHandleRequest() - │ - ├─ resolveProductTierPolicy() ──► InboundRateLimitService::enforce() - │ (NEW — reads api_product+subscription; falls back to (UNCHANGED) - │ Consumer.rateLimit/quota when no product/tier applies) - │ - ├─ recordInboundCallLog() ──► call_log (product, endpoint, responseTime) - │ (NEW — only for product-attached endpoints) - │ - └─ handleRequest() header-merge loop - ├─ RateLimit-* / Retry-After (UNCHANGED, REQ-CON-RL-003) - └─ Sunset / Deprecation (NEW, RFC 8594) - - GET /api/metrics ──► MetricsController - ├─ declarative groupBy(call_log, product) → request/error counts - └─ IntegriqMetricsProvider (escape hatch) → p50/p95/p99 gauges -``` - -## API Design - -`api_product` and `api_product_subscription` are plain OpenRegister-backed -schemas; CRUD goes through OpenRegister's generic object API -(`/api/objects/integriq/api_product`, `/api/objects/integriq/ -api_product_subscription`) exactly like `endpoint` and `consumer` today — no -bespoke controller (`openconnector-direct-or-usage` / redundant-controller -avoidance). Two small custom endpoints are needed for the approval-gated -subscribe flow, mirroring `ApprovalsController`'s shape: - -### `POST /api/products/{productId}/subscriptions` - -**Request:** -```json -{ "consumerId": "", "tier": "gold" } -``` -**Response (201, auto-approved tier):** -```json -{ "uuid": "", "status": "active", "tier": "gold", "product": "", "consumer": "" } -``` -**Response (202, approval-required tier):** -```json -{ "uuid": "", "status": "pending_approval", "approvalRequestId": "" } -``` - -### `GET /api/products/{productId}/analytics` - -**Request:** (query params `window` seconds, default 3600) - -**Response (200):** -```json -{ - "requestCount": 4213, - "errorRate": 0.012, - "latency": { "p50": 42, "p95": 180, "p99": 410 } -} -``` - -## Database Changes - -Shipped as one `register.d` fragment (ADR-037): -`lib/Settings/register.d/api-product-gateway.json`. - -- **New schema `api_product`** — `uuid`, `name` (required), `description`, - `productSlug` (required — groups version-rows of the same logical - product), `version` (semver, default `1.0.0`), `visibility` - (`public`|`private`, default `public`), `status` - (`active`|`deprecated`, default `active`), `sunsetDate` (date-time, - required when `status: deprecated`), `endpoints` (array of Endpoint uuid - strings — same array-of-string-ref pattern as `endpoint.rules`), `tiers` - (object map `tierName -> {rateLimit, quota, requiresApproval}`, same - `rateLimit`/`quota` shape as the `consumer` schema), `defaultTier` - (string), `created`/`updated`. -- **New schema `api_product_subscription`** — `uuid`, `product` (uuid FK → - `api_product`, onDelete CASCADE), `consumer` (uuid FK → `consumer`, - onDelete CASCADE), `tier` (string), `status` - (`pending_approval`|`active`|`rejected`|`revoked`, default - `pending_approval`), `approvalRequestId` (uuid FK → `approval_request`, - onDelete SET_NULL, nullable), `requesterUserId`, `createdAt`, - `activatedAt`, `revokedAt`. -- **Deep-merge onto existing `call_log`** (per `99-source-secrets-writeonly.json` - precedent — a fragment can add properties to a pre-existing schema without - touching the monolith): `product` (uuid FK → `api_product`, onDelete - SET_NULL), `endpoint` (uuid FK → `endpoint`, onDelete SET_NULL), - `responseTime` (integer, milliseconds — top-level so it is directly - aggregatable for percentile queries, unlike the outbound path's nested - `response.responseTime`). - -Full migration plan: see `migration.md`. - -## Decisions - -### Decision 1: `api_product` rows are per-(product, version), not nested version arrays - -**Choice:** One `api_product` OR object per product **version**, grouped by -a shared `productSlug`. Deprecating "v1" means setting `status: deprecated` -+ `sunsetDate` on the v1 row; v2 is untouched. - -**Alternative considered:** A single `api_product` row per product name with -a nested `versions[]` array (each carrying its own `endpoints`/`status`/ -`sunsetDate`). Rejected: every existing versioned entity in this schema -(`endpoint.version`, `source` has no version) uses a flat row + slug/ -reference-grouping pattern, not nested version arrays; nesting would be the -first of its kind in this register and complicates the tier/rate-limit -resolution query (`resolveProductTierPolicy()` would need to reach *into* a -JSON array instead of a direct object lookup by uuid). - -### Decision 2: A subscription in `pending_approval` blocks access (403), it does not fall back to `defaultTier` - -**Choice:** A request from a Consumer whose only subscription to the -product is `pending_approval` (or has none at all) is rejected with 403 — -"subscribe" is opt-in access, and falling back to `defaultTier` would let an -unapproved consumer bypass the approval gate the operator explicitly -configured for that tier. - -**Alternative considered:** Silently applying `defaultTier`'s policy while -approval is pending, so the consumer isn't blocked. Rejected: this defeats -the entire purpose of `requiresApproval` — an operator who gates a tier -behind approval expects zero access until approved, not degraded-but-open -access. - -### Decision 3: Percentiles are computed from a bounded per-product `call_log` window at scrape time, not pre-aggregated storage - -**Choice:** `IntegriqMetricsProvider::metrics()` gains one new sample -producer that, per `api_product`, fetches the most recent N (bound: 1000, -matching the existing `REQ-PROM-007` top-100-cardinality-class precedent -scaled for a per-row not per-series cap) inbound `call_log` rows with that -`product` uuid within the query window, sorts their `responseTime` values, -and computes p50/p95/p99 by index. No new aggregate table, no background -job — consistent with every other `REQ-PROM-*` metric ("computed at query -time from existing tables"). - -**Alternative considered:** A dedicated `api_product_latency_bucket` -pre-aggregation table updated on every request (true HDR-histogram style). -Rejected: violates the "no new data model entities" precedent every other -metric in this app follows, and adds a write on the hot request path for a -metric that only needs to be right at scrape granularity (15s per -`REQ-PROM-001`), not per-request. - -### Decision 4: Subscription approval reuses `ApprovalService`'s generic state machine via one new creation method, not `suspend()` - -**Choice:** Add `ApprovalService::suspendForSubscription(string -$subscriptionId, string $approverGroup, string $onReject, int $ttlSeconds): -ObjectEntity`, structurally identical to the existing -`suspendForSynchronization()` (empty `snapshot`, no FlowToken, persists -`pending` `approval_request`, calls the existing `notifyApprovers()`). -`SubscriptionsController::approve()`/`reject()` call the existing, already -subject-agnostic `ApprovalService::completeApproval()` / -`ApprovalService::reject()` directly — no new approve/reject logic. On -`completeApproval()`, the controller (not `ApprovalService`, keeping the -orchestration split `ApprovalService`'s own docblock already documents) -flips the `api_product_subscription.status` to `active` and stamps -`activatedAt`. - -**Alternative considered:** Generalizing `ApprovalService::suspend()` itself -to accept an arbitrary "subject" instead of `(endpoint, rule, flowToken)`. -Rejected: `suspend()`'s snapshot-stripping and `resumeOrder` fields are -meaningless for a subscription and would become dead parameters on this call -path — `suspendForSynchronization()` already proved the "small dedicated -creation method, shared everything else" pattern is the lower-risk fork -point. - -### Decision 5: Per-tier rate-limit resolution is a new step ahead of `enforceInboundRateLimit()`, `InboundRateLimitService::enforce()` is untouched - -**Choice:** `EndpointService` gains a private -`resolveProductTierPolicy(ObjectEntity $endpoint, ObjectEntity $consumer): -?array` returning `['key' => string, 'rateLimit' => ?array, 'quota' => -?array]` or `null`. When non-null, `enforceInboundRateLimit()` uses its -`key`/`rateLimit`/`quota` instead of deriving them from the Consumer -directly; when `null` (endpoint isn't in any `api_product`, or the consumer -has no `active` subscription to that product), behaviour is byte-for-byte -today's `REQ-CON-RL-002` path. The resolved `key` is namespaced -`product:{productUuid}:consumer:{consumerKey}` so it never collides with a -plain `consumer:{key}`/`ip:{addr}` counter in the same distributed cache. - -**Alternative considered:** A new `ProductRateLimitService` wrapping -`InboundRateLimitService`. Rejected: `enforce()`'s contract (`consumerKey`, -`rateLimit`, `quota` → `RateLimitDecision`) is already exactly what's -needed; a wrapper service would just forward three resolved values to the -same method, adding a layer with no behaviour of its own — the resolution -logic belongs next to where the Consumer's own `rateLimit`/`quota` are -already read (`enforceInboundRateLimit()`), not behind a new service -boundary. - -### Decision 6: Deprecation headers reuse the existing `handleRequest()` header-merge choke point - -**Choice:** `$this->deprecationHeaders` (new instance array, same lifecycle -as the existing `$this->rateLimitHeaders`) is populated when the matched -endpoint belongs to an `api_product` with `status: deprecated`: -`Deprecation: true` and `Sunset: ` (RFC 8594). -`handleRequest()`'s existing header-merge `foreach` loop (today only over -`rateLimitHeaders`) iterates both bags. - -**Alternative considered:** A new `after`-timing rule type -(`deprecation_headers`), dispatched from `processRules()` like -`selfurl_hal` (`REQ-EP-006`). Rejected: `selfurl_hal` needs to be opt-in -per-Endpoint because it's a general-purpose output helper; deprecation -headers are not opt-in — they are a direct, non-optional consequence of the -product's own `status` field, so gating them behind a Rule an operator must -remember to attach on every endpoint is the wrong default and an easy way to -silently under-deliver RFC 8594 compliance. - -## Risks / Trade-offs - -- [Risk] Product-scoped inbound `call_log` writes add one extra OR - `saveObject()` call per request on product-attached endpoints → [Mitigation] - scoped only to product-attached endpoints (see proposal.md Risk 1); the - write is best-effort (same try/catch-and-log pattern as - `recordInboundThrottle()` — a logging failure never blocks the response). -- [Risk] A consumer with an `active` subscription whose product is later - deleted leaves `api_product_subscription.product` null (`SET_NULL`) → - [Mitigation] `resolveProductTierPolicy()` treats a subscription with a - null `product` as "no policy" (falls back to Consumer-level), so the - subscription row becomes an inert audit record rather than a crash. -- [Risk] Percentile computation reads up to 1000 rows per product per scrape - → [Mitigation] see Decision 3; a query failure falls back to a zero-value - sample with a warning logged (matches `REQ-PROM-011`'s existing degraded - pattern), never a 500. - -## Migration Plan - -See `migration.md`. - -## Nextcloud Integration - -- Controllers: `lib/Controller/ProductSubscriptionsController.php` (new, - thin — subscribe/approve/reject/analytics; everything else is generic OR - object CRUD). -- Services: `lib/Service/EndpointService.php` (extended), - `lib/Service/ApprovalService.php` (extended, one new method). -- Mappers/Entities: none new — everything is an OpenRegister object, no - app-local `Db\` entity/mapper per `openconnector-direct-or-usage`. -- Observability: `lib/Observability/IntegriqMetricsProvider.php` - (extended), `src/manifest.json` `observability.metrics[]` (extended - `calls_total` groupBy + one new declarative descriptor). - -## Security Considerations - -- Subscription creation/approve/reject follow the same two-layer - authorization `ApprovalService` already enforces - (`isAuthorizedApprover()` — NC admin or `approverGroup` member); no new - authorization primitive is introduced. -- The `POST /api/products/{id}/subscriptions` endpoint requires an - authenticated NC admin session (creating subscriptions on behalf of a - Consumer is an administrative action, same posture as Consumer - create/edit today) — `#[NoAdminRequired]` is deliberately NOT used here, - matching `ConsumersController`'s existing posture. -- Per-tier rate-limit counters use the same hashed, TTL-bound distributed - cache keys as the existing consumer-level limiter — no new attack surface - (no new secret, no new plaintext credential field). -- Sunset/Deprecation headers disclose only information the operator already - configured (a product's own deprecation status) — no information - disclosure risk. - -## File Structure - -``` -lib/ - Controller/ - ProductSubscriptionsController.php (new) - Service/ - EndpointService.php (modified — tier resolution, deprecation headers, inbound logging) - ApprovalService.php (modified — suspendForSubscription()) - Observability/ - IntegriqMetricsProvider.php (modified — percentile gauges) - Settings/ - register.d/ - api-product-gateway.json (new) -src/ - manifest.json (modified — pages, menu, observability.metrics) - views/ - ApiProducts/ - ApiProductsIndex.vue (new, if custom list chrome is needed beyond generic index) - ApiProductDetail.vue (new — endpoint picker, tier editor, analytics panel, subscriptions) -tests/ - Unit/Service/EndpointServiceTierPolicyTest.php (new) - Unit/Observability/IntegriqMetricsProviderTest.php (extended) - postman/ (Newman collection additions — over-tier 429, deprecated headers) -``` - -## Seed Data - -### Schema: `api_product` - -| Field | Object 1 | Object 2 | Object 3 | -|-------|----------|----------|----------| -| slug | `api-product-woo-publications-v1` | `api-product-woo-publications-v2` | `api-product-kvk-lookup-v1` | -| name | WOO Publications API | WOO Publications API | KVK Lookup API | -| productSlug | `woo-publications` | `woo-publications` | `kvk-lookup` | -| version | 1.0.0 | 2.0.0 | 1.0.0 | -| status | deprecated | active | active | -| sunsetDate | 2026-10-01T00:00:00+00:00 | — | — | -| visibility | public | public | private | -| defaultTier | free | free | gold | -| tiers | `{free:{rateLimit:{requestsPerWindow:60,windowSeconds:60}},gold:{rateLimit:{requestsPerWindow:600,windowSeconds:60},requiresApproval:true}}` | same shape | `{gold:{rateLimit:{requestsPerWindow:1000,windowSeconds:60},requiresApproval:true}}` | - -### Schema: `api_product_subscription` - -| Field | Object 1 | Object 2 | -|-------|----------|----------| -| slug | `sub-acme-woo-v2-free` | `sub-acme-kvk-gold-pending` | -| product | (woo-publications v2 uuid) | (kvk-lookup v1 uuid) | -| consumer | (existing seeded Consumer uuid) | (existing seeded Consumer uuid) | -| tier | free | gold | -| status | active | pending_approval | - -**Related items per object:** none (Files/Notes/Tasks/Contacts not -applicable to this domain). - -## Trade-offs - -- Chose flat versioned rows over nested version arrays (Decision 1) — - trades a small amount of query-time joining (resolve "the deprecated - version of product X" by `productSlug` + `status`) for consistency with - every other entity in this register and a much simpler tier-resolution - lookup. -- Chose scoped (product-attached-only) inbound logging over universal - inbound logging — trades "analytics only exist for product-fronted - endpoints" for avoiding an unbounded volume/retention change to every - endpoint in the app (see discovery.md Risk Uncovered). diff --git a/openspec/changes/api-product-gateway/discovery.md b/openspec/changes/api-product-gateway/discovery.md deleted file mode 100644 index a0dbecee9..000000000 --- a/openspec/changes/api-product-gateway/discovery.md +++ /dev/null @@ -1,143 +0,0 @@ -# Discovery: api-product-gateway - -## Question - -The context brief assumes gateway latency percentiles can be "computed from -existing CallLog inbound entries" and that per-tier rate-limit policy should -"extend the existing `InboundRateLimitService`". Both are underspecified at -HEAD: does `call_log` already carry enough inbound rows/fields for -percentiles, and what's the correct extension seam on -`InboundRateLimitService` that doesn't fork it? - -## Approach Taken - -- Read `lib/Service/RateLimit/InboundRateLimitService.php` in full. -- Read the `consumer-management`, `endpoint-runtime`, and `prometheus-metrics` - specs in full, plus ADR-003 (CallLog is the primary observability surface). -- Grepped every `'direction' =>` write site in `lib/` to find every place - that produces an inbound `call_log` row. -- Read `lib/Service/EndpointService.php`'s `enforceInboundRateLimit()` / - `recordInboundThrottle()` / `handleRequest()` (the RateLimit-header - choke point). -- Read the `call_log` schema block in `lib/Settings/integriq_register.json` - and the outbound `responseTime` write site in `CallService::buildResponseData()`. -- Read `openspec/specs/openconnector-storage-migration/spec.md` to confirm - `call_log` is now an OpenRegister object (not the legacy `lib/Db/CallLog.php` - entity, which no longer exists on disk). -- Read the archived `2026-07-14-consumer-apikey-enforcement` proposal and - `lib/Service/AuthorizationService::getResolvedConsumer()` to confirm how a - request's Consumer is resolved today. -- Read `lib/Service/ApprovalService.php` and the archived - `2026-07-15-hitl-approval-rule-action` `approval-workflow` spec in full to - find a subject-agnostic (non-FlowToken) approval seam. -- Read `lib/Observability/IntegriqMetricsProvider.php` and - `src/manifest.json`'s `observability.metrics` block to find the declarative - vs. escape-hatch split for Prometheus gauges. -- Read `lib/Settings/register.d/hitl-approval-rule-action.json` and - `register.d/99-source-secrets-writeonly.json` to confirm the register - fragment mechanism can both add new schemas and deep-merge new fields onto - an existing schema. -- Read `src/manifest.json`'s `pages` array to confirm the SPA is - manifest-driven (index/detail/custom page types), not hand-routed Vue views. - -## Findings - -1. **`call_log` inbound rows are NOT a general request log today — brief's - assumption is wrong.** The only code path that writes a `direction: - inbound` `call_log` row is `EndpointService::recordInboundThrottle()`, - called exclusively from `enforceInboundRateLimit()` on the 429 branch - (`REQ-CON-RL-004`). Every *successful* inbound endpoint request writes - nothing to `call_log`. There is no `responseTime`, `endpoint`, or - `product` field on the schema at all — only `statusCode`, `statusMessage`, - `direction`, `created`. Computing latency percentiles "from existing - CallLog inbound entries" is therefore not possible without first adding - general-purpose inbound logging with a duration field. **Deviation from - brief, followed the code**: this change adds inbound `call_log` writes - (with a new `responseTime` field) for every request dispatched through an - `api_product`-scoped endpoint — not every endpoint, to bound volume - growth (see design.md Decision 3 and proposal.md Risk 1). - -2. **`InboundRateLimitService::enforce()` is already policy-agnostic** — it - takes a `consumerKey` string and plain `rateLimit`/`quota` arrays, with no - knowledge of Consumer, Endpoint, or any product concept. The correct - "extend, don't fork" seam is one level up, in - `EndpointService::enforceInboundRateLimit()`, which today derives its - `$key`/`$rateLimit`/`$quota` from the resolved Consumer. Adding a - tier-resolution step ahead of that derivation (falling back to the - existing Consumer-level values when no product/tier applies) requires - zero changes to `InboundRateLimitService` itself. - -3. **`ApprovalService::suspend()` cannot be reused as-is for subscription - approval.** It's tightly coupled to a `FlowToken` snapshot and an - in-flight endpoint rule-pipeline suspension (`EndpointService:: - doHandleRequest()`'s `JSONResponse` short-circuit). But - `ApprovalService::suspendForSynchronization()` already establishes the - pattern this change needs: create a `pending` `approval_request` with an - empty `snapshot`, no FlowToken, notify the `approverGroup`, and let a - *different* subject (a Synchronization batch gate there; an - `api_product_subscription` here) resolve on `approve()`/`reject()`. - `completeApproval()`, `reject()`, `isAuthorizedApprover()`, - `assertActionable()`, `sweepExpired()`, and `notifyApprovers()` are - already fully generic (no FlowToken coupling). Only the *creation* - method needs a subscription-specific sibling of - `suspendForSynchronization()`. - -4. **Prometheus gauges split cleanly into two existing mechanisms.** - `src/manifest.json`'s declarative `observability.metrics[].source.kind: - "tableCount"` (with `groupBy`) already produces `calls_total{status, - direction}` from `call_log` — extending `groupBy` to include a `product` - label is a manifest-only change once `call_log.product` exists. But a - *percentile* is not a row count — it requires sorting/indexing values - within a group — so it cannot be expressed by the declarative - `tableCount`/`objectCount`/`orAvailable` `source.kind` vocabulary. This is - exactly the situation `circuit_breaker_state` already solved: it uses - `source.kind: "provider"`, resolved to `IntegriqMetricsProvider` - (`OCA\OpenRegister\AppHost\IMetricsProvider::integriq` container - alias). Latency percentiles will use the same escape hatch. - -5. **The register fragment mechanism (ADR-037, - `lib/Settings/register.d/*.json`) supports both new-schema declaration - and deep-merge onto an existing schema's `properties`** — - `99-source-secrets-writeonly.json` deep-merges `writeOnly: true` onto five - existing `source` properties without touching the monolith. This is the - documented reason to avoid editing `integriq_register.json` directly - (a `SchemaMapper` `$ref`-resolution bug on re-parse). The new - `api_product`/`api_product_subscription` schemas and the three new - `call_log` fields (`product`, `endpoint`, `responseTime`) will all ship in - one fragment for this change, following the `hitl-approval-rule-action` - precedent of one fragment per change. - -6. **The SPA is manifest-driven**, not hand-routed Vue views — `src/ - manifest.json`'s `pages` array declares `index` (generic list), - `detail` (generic detail), and `custom` (bespoke component) page types. - `Consumers` is a plain `index` page over the `consumer` schema; `Approvals` - is `custom` because it needs non-CRUD actions (approve/reject) and a - scoped API. `API Products` needs the same: a plain index for browsing, but - the detail view needs endpoint-picker + tier editor + analytics panel + - subscription-approval actions that a generic CRUD detail can't express — - it will be `custom`, mirroring `ApprovalDetail`. - -## Recommendation - -Proceed with the approach in proposal.md/design.md: extend -`EndpointService`'s inbound-rate-limit call site (not the service), extend -`ApprovalService` with one new creation method (not the FlowToken-coupled -`suspend()`), add general-but-scoped inbound `call_log` logging restricted to -product-attached endpoints, and split analytics across the declarative -`groupBy` mechanism (counts) and the `IMetricsProvider` escape hatch -(percentiles). All four extension seams already exist in the codebase for -an analogous purpose — none require inventing a new pattern. - -## Risks Uncovered - -- Scoping inbound logging to product-attached endpoints only (rather than - every endpoint) means a consumer's plain (non-product) endpoint calls stay - invisible to analytics — acceptable per proposal.md's in-scope framing - ("API Products GROUP them"; analytics is a product-level, not - endpoint-level, capability), but worth flagging: a future "make analytics - available to every endpoint" change would need to revisit the volume - trade-off in Risk 1. - -## Next Steps - -Proceed to design.md and the spec deltas. diff --git a/openspec/changes/api-product-gateway/migration.md b/openspec/changes/api-product-gateway/migration.md deleted file mode 100644 index 103b634bc..000000000 --- a/openspec/changes/api-product-gateway/migration.md +++ /dev/null @@ -1,116 +0,0 @@ -# Migration: api-product-gateway - -## Current State - -`integriq_register.json` (loaded via `ConfigurationService::importFromApp`, -`openconnector-storage-migration`) declares 15 core schemas, including -`call_log` with `uuid`, `statusCode`, `statusMessage`, `direction` -(`inbound`|`outbound`), `request`, `response`, `sourceId`/`source`, -`actionId`, `synchronizationId`/`synchronization`, `userId`, `sessionId`, -`expires`, `created`, `size`. There is no `api_product` or -`api_product_subscription` schema. `call_log` carries no `product`, -`endpoint`, or top-level `responseTime` field. `lib/Settings/register.d/` -holds one fragment per prior change (`hitl-approval-rule-action.json`, -`99-source-secrets-writeonly.json`, etc.), merged at load per ADR-037. - -## Target State - -- New register.d fragment `lib/Settings/register.d/api-product-gateway.json` - declaring: - - `api_product` schema (new). - - `api_product_subscription` schema (new). - - A deep-merge onto the existing `call_log` schema's `properties` adding - `product` (uuid FK → `api_product`, `onDelete: SET_NULL`), `endpoint` - (uuid FK → `endpoint`, `onDelete: SET_NULL`), and `responseTime` - (integer, milliseconds). -- No existing `call_log` row is touched — the three new fields are simply - absent (`null`/undefined) on every row written before this change deploys; - `endpoint-runtime` `REQ-EP-009`'s new inbound logging only affects rows - written *after* deploy, for endpoints attached to an `api_product` (which - cannot exist before this change either, since the schema is new). - -## Migration Class - -``` -Version: N/A — no PHP Nextcloud migration class needed. -File: none. -Key operations: -- Register fragments are merged into the in-memory register definition at - application boot / `ConfigurationService::importFromApp()` time (see - openconnector-storage-migration#REQ — "Migration class MUST provision the - register via importFromApp"), which already re-runs on every app upgrade - and is idempotent. Adding a new register.d/*.json file requires no new - Version*.php migration class — the existing migration class that calls - importFromApp picks it up automatically on next `occ upgrade` / - `occ app:update integriq`. -``` - -No `Version*.php` class is added by this change. The existing storage -migration's `postSchemaChange` hook already re-imports the full merged -register (base + all `register.d/*.json` fragments) on every upgrade, -per `openconnector-storage-migration` "Migration class MUST provision the -register via importFromApp" — idempotent by design (re-running is a no-op -for schemas/fields that already exist, additive for new ones). - -## Migration Steps - -1. Add `lib/Settings/register.d/api-product-gateway.json` with the two new - schema declarations and the `call_log` deep-merge block (verifiable: - `git diff` shows only new-file addition + no edits to - `integriq_register.json`). -2. Deploy the app version carrying the fragment; on `occ app:update - integriq` (or fresh `occ app:enable`), the existing migration class - re-runs `ConfigurationService::importFromApp()`, which merges the - fragment into the live OpenRegister register/schema tables (verifiable: - `oc_openregister_schemas` gains 2 rows for `api_product` and - `api_product_subscription`; the existing `call_log` schema row's - `properties` JSON gains the 3 new keys, in place — no new row). -3. No data backfill is needed or attempted — existing `call_log` rows simply - have the 3 new fields absent; nothing reads them as required (all three - are optional/nullable and every read site added by this change - null-coalesces). - -## Data Impact - -- **Records affected:** 0 existing rows are modified. The 2 new schemas - start empty. The `call_log` schema **definition** row gains 3 optional - properties; existing `call_log` **object** rows are untouched (their - `object` JSON blobs are not rewritten — OpenRegister schemas are - additive-by-default for optional properties, no `NOT NULL` backfill - required). -- **Data loss:** none. -- **Live-data safe:** yes — purely additive schema changes; no column type - changes, no destructive DDL. Import runs within the existing idempotent - `importFromApp()` call already exercised on every upgrade. - -## Rollback Procedure - -1. Remove `lib/Settings/register.d/api-product-gateway.json` and redeploy - the prior app version. -2. On the next `importFromApp()` run, the 2 new schemas remain registered - (OpenRegister does not auto-drop schemas absent from a re-import by - default) but become unreferenced/orphaned — acceptable for a rollback, - since no other code path depends on their absence. If a clean rollback of - the schema rows themselves is required, an operator runs OpenRegister's - existing schema-deletion tooling manually against `api_product` and - `api_product_subscription` (out of band — this change does not ship an - automated schema-deletion step, consistent with every other register.d - fragment in this app, none of which ship a reverse migration either). -3. The `call_log` deep-merged properties (`product`, `endpoint`, - `responseTime`) similarly remain on the schema definition but stop being - written to once the code that populates them (`EndpointService`'s new - logging) is reverted — no functional impact, since they were - optional/nullable throughout. - -## Validation - -- `SELECT COUNT(*) FROM oc_openregister_schemas WHERE title IN ('Api - Product', 'Api Product Subscription');` → expect `2` after deploy. -- `SELECT properties::jsonb ? 'responseTime' FROM oc_openregister_schemas - WHERE slug = 'call_log';` (Postgres) → expect `true` after deploy. -- Re-run `occ app:update integriq` a second time → expect no error, no - duplicate schema rows (idempotency, per - openconnector-storage-migration's "Idempotent re-run" scenario). -- Create one `api_product` and one `api_product_subscription` via the OR - generic object API → expect both persist and are retrievable, confirming - the fragment merged correctly. diff --git a/openspec/changes/api-product-gateway/proposal.md b/openspec/changes/api-product-gateway/proposal.md index ade624c28..2b22d3d81 100644 --- a/openspec/changes/api-product-gateway/proposal.md +++ b/openspec/changes/api-product-gateway/proposal.md @@ -1,165 +1,44 @@ -# Proposal: api-product-gateway - -## Summary - -Integriq already lets an administrator define individual inbound -`Endpoint`s and gate them behind a `Consumer`'s authentication, per-consumer -rate limit, and quota (`consumer-management`, `endpoint-runtime`). It has no -concept of an **API Product** — a named, versioned bundle of endpoints that a -Consumer can discover and subscribe to at a rate-limit tier, with an optional -approval gate. This change adds API Products, tiered subscriptions, per-tier -rate-limit enforcement, RFC 8594 deprecation headers on sunset product -versions, and gateway analytics (request count, error rate, and p50/p95/p99 -latency per product) — the multi-tenancy and analytics table stakes every -competing API gateway ships. - -## Motivation - -Specter user-story synthesis for the API-product cluster surfaces five -recurring asks: "create an API product definition in the gateway", "rate -limit per consumer tier", "view current rate-limit usage per consumer", -"gateway latency percentiles per API product", and "deprecate an old API -version". `consumer-rate-limiting` (archived) already delivered per-consumer -limits; `consumer-apikey-enforcement` (archived 2026-07-14) just made -Consumer-backed apiKey auth real. What is still missing is the *product* -layer on top: grouping endpoints into a sellable/discoverable unit, giving -each subscribing consumer a named tier instead of one flat per-consumer -limit, and surfacing usage/latency so an operator can actually run the -gateway as a product. - -## Affected Projects - -- [x] Project: `integriq` — new `api_product` and - `api_product_subscription` OR schemas, extended `call_log` schema, extended - `InboundRateLimitService` call site (not the service itself), - `EndpointService` dispatch (deprecation headers + product-scoped inbound - logging), `ApprovalService` (one new subscription-approval creation - method), `IntegriqMetricsProvider` (latency percentile gauges), SPA - manifest (`API Products` page). - -## Scope - -### In Scope - -1. `api_product` OR schema: a named, versioned bundle of `Endpoint`s (by - uuid) with a `visibility` (public/private), a set of named `tiers` (each - carrying its own `rateLimit`/`quota`, mirroring the existing Consumer - shape, plus a `requiresApproval` flag), a `defaultTier`, and a - `status`/`sunsetDate` pair for version deprecation. -2. `api_product_subscription`: a Consumer's subscription to a Product at a - named tier, gated by an approval workflow when the tier requires it - (reusing `hitl-approval-rule-action`'s `ApprovalService` state machine — - see design.md Decision 4), auto-activated otherwise. -3. Per-tier rate-limit/quota enforcement at the endpoint runtime: extends - `InboundRateLimitService::enforce()`'s call site in `EndpointService` - (the service itself is unchanged) to resolve a subscription's tier policy - ahead of the existing consumer-level policy, keyed on - `(consumer, product, tier)` so product-tier counters are independent of a - consumer's plain per-endpoint counters. A request past its tier's - `rateLimit`/`quota` receives HTTP 429 exactly like today's consumer-level - 429 (`REQ-CON-RL-002`/`003`). -4. Gateway analytics: per-product request count and error rate (declarative - Prometheus gauge, extending the existing `calls_total` groupBy) and - p50/p95/p99 latency (AppHost provider escape hatch, computed from - inbound `call_log.responseTime` — see design.md Decision 3), surfaced on - a new **API Products** SPA page and via `/api/metrics`. -5. API version deprecation: marking an `api_product` `status: deprecated` - with a `sunsetDate` makes every response served through that product's - endpoints carry `Sunset` (RFC 8594) and `Deprecation` headers. -6. Tests: PHPUnit for tier-policy resolution, percentile calculation, and - Sunset/Deprecation header emission; Newman for consumer-over-tier → 429 - and deprecated-product → header scenarios. - -### Out of Scope - -- A full self-service developer portal or API-key issuance UI for - prospective consumers — follow-up, filed as an issue at apply time. -- Monetization/billing on top of tiers. -- Multi-version endpoint routing (an endpoint moving between product - versions) — a product version references the endpoints that exist today; - endpoint versioning itself is unchanged. - -## Approach - -New OR schemas (`api_product`, `api_product_subscription`) shipped as a -per-change `register.d` fragment (ADR-037), deep-merging two new fields -(`product`, `endpoint`) plus a `responseTime` field onto the existing -`call_log` schema. Tier-policy resolution is a new private method in -`EndpointService` that runs *before* today's `enforceInboundRateLimit()` and -substitutes its `rateLimit`/`quota` inputs and cache key — `enforce()` on -`InboundRateLimitService` is not touched. Subscription approval reuses -`ApprovalService`'s generic `approval_request` state machine via one new -creation method mirroring the existing `suspendForSynchronization()` -(no FlowToken, no rule-pipeline coupling). Deprecation headers reuse the -existing `handleRequest()` header-merge choke point that already attaches -`RateLimit-*` headers. Analytics split across the two existing observability -mechanisms: declarative `groupBy` for counts, the `IMetricsProvider` -escape hatch for percentiles (the same split the codebase already uses for -`calls_total` vs `circuit_breaker_state`). - -## New Dependencies - -None. No new packages, libraries, or external services. - -## Impact - -- **Schema**: `integriq_register.json` gains `product`/`endpoint`/ - `responseTime` on `call_log` (register.d fragment). New `api_product`, - `api_product_subscription` schemas. -- **Backend**: `lib/Service/EndpointService.php` (tier-policy resolution, - deprecation headers, product-scoped inbound logging), - `lib/Service/ApprovalService.php` (one new method), - `lib/Observability/IntegriqMetricsProvider.php` (percentile gauges), - `src/manifest.json` (declarative `calls_total` groupBy extension + new - metric descriptor). -- **Frontend**: new `ApiProducts` (index) and `ApiProductDetail` (custom) - manifest pages, new `ConnectionsGroup` menu entry. -- **No changes** to `InboundRateLimitService`, the `consumer` schema, or any - existing endpoint's dispatch behaviour when it is not part of a product. - -## Cross-Project Dependencies - -None. This is entirely within Integriq; the API Products surface is -consumed by external API clients, not by other apps-extra projects. - -## Risks - -### Risk 1: Inbound call_log volume growth from product-scoped logging - -**Severity:** Medium — **Mitigation:** Logging is scoped to endpoints that -belong to an `api_product` only (today's non-product endpoints keep their -current behaviour — only 429s are logged, per `REQ-CON-RL-004`). Retention -follows the existing `expires` convention (ADR-004); no new retention floor -is introduced. - -### Risk 2: Percentile computation cost at scrape time - -**Severity:** Medium — **Mitigation:** Bounded per-product row window (last -N inbound rows, consistent with `REQ-PROM-007`'s top-100 cardinality cap and -`REQ-PROM-001`'s 500ms scrape budget); a query failure falls back to a -zero-value sample, matching every other `REQ-PROM-*` degraded-not-broken -pattern. - -### Risk 3: Two overlapping rate-limit keys per consumer - -**Severity:** Low — **Mitigation:** Product-tier keys -(`product:{uuid}:consumer:{key}`) are namespaced separately from plain -consumer keys (`consumer:{key}` / `ip:{addr}`) in the same distributed -cache, so they cannot collide or double-count against each other. - -## Rollback Strategy - -Revert the `register.d` fragment (drops the two new schemas and the three -new `call_log` fields — additive fields, no data loss for existing rows), -revert the `EndpointService`/`ApprovalService`/`IntegriqMetricsProvider` -changes, and remove the manifest page/menu entries. No endpoint that is not -attached to an `api_product` observes any behaviour change, so rollback is -safe on a live instance with active traffic. - -## Open Questions - -- Should a `api_product_subscription` in `pending_approval` block the - consumer from calling the product's endpoints entirely, or fall back to - the product's `defaultTier`? Resolved in design.md Decision 2 — it - blocks (403), consistent with "subscribe" implying opt-in access, not - ambient access. +--- +kind: spec-only +depends_on: [] +--- + +# Proposal: api-product-gateway (superseded — retired 2026-09-02) + +This directory double-counted a change that had already shipped. The API +product gateway was implemented and archived on 2026-07-15 +(`archive/2026-07-15-api-product-gateway`, 21/33 tasks checked with +per-task evidence), yet this live copy was resurrected at 0/33: the +openconnector→integriq rename applied to the prose, the evidence notes +stripped, every box reset. The machinery exists at HEAD: +`lib/Controller/ProductSubscriptionsController.php`, the api_product +routes in `appinfo/routes.php`, the schema fragment in +`lib/Settings/register.d/api-product-gateway.json`, gateway enforcement in +`lib/Service/EndpointService.php`, product metrics in +`lib/Observability/IntegriqMetricsProvider.php` (the fleet rename moved it +from `OpenConnectorMetricsProvider.php` — a move, not a gap), and the UI +(manifest pages `ApiProducts` / `ApiProductDetail`, +`src/views/ApiProducts/ApiProductDetail.vue`). + +No live `@spec` tags point into this directory (`appinfo/routes.php` +mentions it in a prose comment only, which this retirement keeps valid by +leaving the directory in place). + +## Disposition of the original scope + +| Original scope | Where it went | +| --- | --- | +| `api_product` schema + tiers, subscription lifecycle (subscribe/approve/reject), over-tier enforcement + deprecation headers in the endpoint runtime, analytics, product metrics, API Products pages + Consumer subscription widget, seed data | **Already shipped and archived**: `archive/2026-07-15-api-product-gateway` (21/33 boxes checked), code at HEAD | +| Residual verification: live schema-import and seed run, Playwright for the API Products pages and Consumer widget, Newman for subscribe/approve/reject/analytics + 429 + deprecation headers, feature docs, screenshot, `nl_NL` catalog entries | Open, and honestly unticked in the archived twin (no live instance in that session; each open box carries its reason). Same shape as `approvals-verification-pack`; pick up in a verification pass, not by resurrecting this change | + +## Sequencing + +Nothing remains to implement from this change directly. The residual +live-instance verification, docs and l10n belong to a +verification-pack-style follow-up. + +## Archival + +This directory is retired in place (not moved or renamed) to keep the diff +reviewable and the prose pointer in `appinfo/routes.php` valid; archive it +via the normal flow at the next sweep. diff --git a/openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md b/openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md deleted file mode 100644 index 3ef8e8268..000000000 --- a/openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md +++ /dev/null @@ -1,250 +0,0 @@ -# api-product-gateway Specification - -**Status**: in-progress -**Scope**: integriq -**OpenSpec changes**: -- [api-product-gateway](../../changes/api-product-gateway/) - -## Purpose - -Integriq exposes individual `Endpoint`s gated by `Consumer` auth and -per-consumer rate limits (`consumer-management`, `endpoint-runtime`), but has -no concept of a **API Product** — a named, versioned bundle of endpoints a -consumer can subscribe to at a rate-limit tier, with gateway analytics and -version-deprecation signalling. This capability adds that product layer on -top of the existing endpoint/consumer primitives, per ADR-008 (polymorphic -target dispatch is unchanged — products group existing endpoints, they do -not introduce a new target kind). - -## ADDED Requirements - -### Requirement: API Product groups Endpoints into a named, versioned bundle (REQ-APG-001) - -The system MUST provide an `api_product` OpenRegister schema representing a -named bundle of existing `Endpoint`s at a specific `version`, with a -`productSlug` grouping multiple version-rows of the same logical product, a -`visibility` (`public`|`private`), a `status` (`active`|`deprecated`), and a -`tiers` map of named rate-limit/quota policies with a `defaultTier`. An -`api_product`'s `endpoints` array MUST reference existing `Endpoint` uuids; -creating or updating an `api_product` MUST NOT create, modify, or delete the -`Endpoint`s it references. - -@e2e exclude backend schema definition — covered by PHPUnit, no browser UI - -#### Scenario: an API Product groups multiple endpoints - -- GIVEN three existing `Endpoint`s serving `/publications`, `/publications/{id}`, and `/publications/{id}/attachments` -- WHEN an administrator creates an `api_product` with `productSlug: "woo-publications"`, `version: "2.0.0"`, and those three endpoint uuids in `endpoints` -- THEN the `api_product` is persisted with all three endpoint uuids AND none of the three `Endpoint` objects are modified - -#### Scenario: a product version is independent of other versions of the same product - -- GIVEN two `api_product` rows sharing `productSlug: "woo-publications"` — one `version: "1.0.0"`, one `version: "2.0.0"` -- WHEN the `1.0.0` row's `endpoints` array is edited -- THEN the `2.0.0` row's `endpoints` array is unaffected - -### Requirement: API Products management UI (REQ-APG-002) - -Integriq MUST provide an **API Products** section in its SPA where an -administrator can browse, create, edit, and delete API Products, pick which -existing Endpoints belong to a product, and define/edit its named tiers. - -#### Scenario: API Products list page mounts and shows content - -- GIVEN an authenticated admin visits the integriq app -- WHEN they navigate to the API Products section via the sidebar nav or direct URL `/apps/integriq/products` -- THEN the API Products index page renders inside the main content area with content visible - -#### Scenario: product detail page exposes an endpoint picker and tier editor - -- GIVEN at least one `api_product` and at least one `Endpoint` exist -- WHEN the administrator opens the product's detail page -- THEN they can add/remove Endpoints from the product's `endpoints` array and add/edit named tiers with a `rateLimit`/`quota`/`requiresApproval` configuration - -### Requirement: Consumer subscribes to an API Product at a tier (REQ-APG-003) - -The system MUST let a Consumer be subscribed to an `api_product` at one of -its named `tiers` via `POST /api/products/{productId}/subscriptions`, -creating an `api_product_subscription` referencing the product, the -consumer, and the chosen tier. The chosen tier MUST exist in the product's -`tiers` map; an unknown tier MUST be rejected with HTTP 400. - -@e2e exclude backend subscription creation — covered by Newman, not browser UI - -#### Scenario: subscribing to a tier that requires no approval activates immediately - -- GIVEN an `api_product` with a `free` tier where `requiresApproval` is absent (falsy) -- WHEN a Consumer subscribes at the `free` tier -- THEN an `api_product_subscription` is created with `status: active` and HTTP 201 is returned - -#### Scenario: subscribing to an unknown tier is rejected - -- GIVEN an `api_product` whose `tiers` map contains only `free` and `gold` -- WHEN a subscription request names tier `platinum` -- THEN the response is HTTP 400 and no `api_product_subscription` is created - -### Requirement: Subscription approval gate reuses the HITL ApprovalService (REQ-APG-004) - -When the chosen tier's `requiresApproval` is `true`, subscribing MUST create -the `api_product_subscription` with `status: pending_approval`, create a -`pending` `approval_request` via `ApprovalService::suspendForSubscription()` -(no `FlowToken` snapshot — see design.md Decision 4), notify the configured -`approverGroup`, and return HTTP 202 with the subscription id and the -approval_request id. Approving the request MUST flip the subscription's -`status` to `active` and stamp `activatedAt`; rejecting it MUST flip it to -`rejected`. A subscription that is not `active` MUST NOT receive its tier's -rate-limit/quota policy (`REQ-APG-005`); requests from a consumer with no -`active` subscription to a product's endpoint MUST receive HTTP 403. - -@e2e exclude backend approval-gated subscription flow — covered by PHPUnit/Newman, not browser UI - -#### Scenario: a gold tier requiring approval creates a pending subscription - -- GIVEN an `api_product` whose `gold` tier has `requiresApproval: true` -- WHEN a Consumer subscribes at the `gold` tier -- THEN an `api_product_subscription` is created with `status: pending_approval`, a `pending` `approval_request` is created, the configured `approverGroup` is notified, and HTTP 202 is returned - -#### Scenario: approving the request activates the subscription - -- GIVEN a `pending_approval` subscription with its linked `pending` `approval_request` -- WHEN an authorized approver approves the request -- THEN the subscription's `status` becomes `active` with `activatedAt` set - -#### Scenario: a pending subscription grants no access - -- GIVEN a Consumer with only a `pending_approval` subscription to a product -- WHEN that consumer calls one of the product's endpoints -- THEN the response is HTTP 403 and no rate-limit policy from that product is applied - -### Requirement: Per-tier rate-limit enforcement extends the inbound rate limiter (REQ-APG-005) - -The system MUST, for a request to an `Endpoint` that belongs to an -`api_product`, resolve the caller's `active` subscription to that product -and enforce the subscription's tier `rateLimit`/`quota` via the existing -`InboundRateLimitService::enforce()` (unmodified — see design.md Decision 5), -keyed on `(consumer, product)` so product-tier counters never share a bucket -with the consumer's plain per-endpoint counters. A request exceeding the -tier's `rateLimit.requestsPerWindow` or `quota.limit` MUST receive HTTP 429 -with the same `RateLimit-*`/`Retry-After` header contract as -`consumer-management` `REQ-CON-RL-003`. When the endpoint is not part of any -`api_product`, or the consumer has no `active` subscription to that product, -today's Consumer-level `rateLimit`/`quota` (`REQ-CON-RL-002`) applies -unchanged. - -@e2e exclude backend enforcement — covered by PHPUnit/Newman, not browser UI - -#### Scenario: over-tier request returns 429 - -- GIVEN an `active` subscription at the `free` tier (`rateLimit {requestsPerWindow: 2, windowSeconds: 60}`) -- WHEN the subscribed consumer makes 3 requests to the product's endpoint within the same window -- THEN the first 2 succeed and the 3rd receives HTTP 429 with `Retry-After` - -#### Scenario: product-tier counters are independent of the consumer's own rateLimit - -- GIVEN a Consumer with its own `rateLimit {requestsPerWindow: 100, windowSeconds: 60}` AND an `active` subscription to a product's `free` tier `{requestsPerWindow: 2, windowSeconds: 60}` -- WHEN the consumer calls the product's endpoint 3 times in the window -- THEN the 3rd request receives HTTP 429 from the tier limit even though the consumer's own 100-request budget is far from exhausted - -#### Scenario: a non-product endpoint is unaffected - -- GIVEN an `Endpoint` that belongs to no `api_product` -- WHEN its consumer calls it repeatedly within its own `rateLimit` -- THEN enforcement follows `consumer-management` `REQ-CON-RL-002` exactly as before this change - -### Requirement: Deprecated product version carries Sunset and Deprecation headers (REQ-APG-006) - -The system MUST, when an `api_product`'s `status` is `deprecated`, ensure -every response served through any of that product's `endpoints` carries a -`Deprecation: true` header and a `Sunset` header (RFC 8594, HTTP-date -format) reflecting the product's `sunsetDate`. An `api_product` with -`status: active` MUST NOT add either header. - -@e2e exclude backend response headers — covered by Newman, not browser UI - -#### Scenario: a deprecated product version's endpoint responses carry Sunset and Deprecation - -- GIVEN an `api_product` with `status: deprecated` and `sunsetDate: "2026-10-01T00:00:00+00:00"`, grouping an endpoint `/publications` -- WHEN a request is served through `/publications` -- THEN the response carries `Deprecation: true` and `Sunset: Thu, 01 Oct 2026 00:00:00 GMT` - -#### Scenario: an active product version's endpoint responses carry neither header - -- GIVEN an `api_product` with `status: active` grouping an endpoint `/publications` -- WHEN a request is served through `/publications` -- THEN the response carries neither `Deprecation` nor `Sunset` - -#### Scenario: an endpoint shared by an active and a deprecated version reflects only the version it was dispatched through - -- GIVEN `productSlug: "woo-publications"` has a `deprecated` `1.0.0` row and an `active` `2.0.0` row, each grouping its own endpoint set -- WHEN a request is served through the `1.0.0` row's endpoint -- THEN Deprecation/Sunset headers are present, regardless of `2.0.0`'s status - -### Requirement: Gateway analytics per API Product (REQ-APG-007) - -The system MUST compute, per `api_product`, a request count, an error rate -(share of requests with `statusCode >= 400`), and p50/p95/p99 response-time -latency percentiles from inbound `call_log` rows carrying that product's -uuid (see `endpoint-runtime` `REQ-EP-009` for how those rows are produced), -and surface them both on the API Products detail page -(`GET /api/products/{productId}/analytics`) and as Prometheus gauges (see -`prometheus-metrics` `REQ-PROM-012`/`REQ-PROM-013`). - -@e2e exclude backend analytics computation — covered by PHPUnit, no browser UI - -#### Scenario: analytics reflect recent traffic - -- GIVEN a product with 100 recorded inbound `call_log` rows in the last hour, 5 with `statusCode >= 400` -- WHEN `GET /api/products/{productId}/analytics` is called -- THEN `requestCount` is 100 and `errorRate` is 0.05 - -#### Scenario: latency percentiles are computed from responseTime - -- GIVEN a product's recent inbound `call_log` rows with `responseTime` values ranging 10ms-500ms -- WHEN the analytics endpoint (or the Prometheus scrape) computes percentiles -- THEN `p50`/`p95`/`p99` reflect the 50th/95th/99th percentile of the recorded `responseTime` values - -#### Scenario: a product with no recorded traffic reports zero, not an error - -- GIVEN a newly created product with no inbound `call_log` rows yet -- WHEN analytics are requested -- THEN `requestCount` is 0, `errorRate` is 0, and latency percentiles are 0 — no error is raised - -## Non-Functional Requirements - -- **Performance:** analytics computation (both the REST endpoint and the - Prometheus provider) is bounded to the most recent 1000 inbound rows per - product and completes within the existing `REQ-PROM-001` 500ms scrape - budget. -- **Accessibility:** the API Products SPA pages meet WCAG 2.2 AA — endpoint - picker and tier editor controls carry accessible labels (`NcSelect` - `inputLabel`, per the established `ncvue` gotcha); analytics charts carry - a text-equivalent summary (request count / error rate / percentiles as - plain text alongside any chart). -- **Internationalization:** Dutch and English MUST be supported (hydra - ADR-007) for all new SPA strings (product/tier labels, subscription - status, deprecation notices). - -## Acceptance Criteria - -- [ ] An administrator can create an `api_product`, pick its endpoints, and - define tiers with independent rate-limit/quota policies. -- [ ] A consumer can subscribe to a product at a tier; approval-gated tiers - block access until approved. -- [ ] An over-tier request receives HTTP 429 without affecting the - consumer's own plain rate limit. -- [ ] A deprecated product version's endpoints carry Sunset/Deprecation - headers; an active version's do not. -- [ ] Per-product request count, error rate, and p50/p95/p99 latency are - visible on the product detail page and via `/api/metrics`. - -## Notes - -- Self-service API key issuance / a public developer portal is explicitly - out of scope (proposal.md) — filed as a follow-up issue at apply time. -- See `discovery.md` for the verified-against-HEAD finding that inbound - `call_log` did not previously carry general request data — this - capability's analytics requirement (`REQ-APG-007`) depends on the new - product-scoped inbound logging added by `endpoint-runtime` `REQ-EP-009`. -- Related ADRs: ADR-003 (CallLog is the primary observability surface), - ADR-008 (polymorphic endpoint target dispatch, unchanged), ADR-037 - (register fragments). diff --git a/openspec/changes/api-product-gateway/specs/consumer-management/spec.md b/openspec/changes/api-product-gateway/specs/consumer-management/spec.md deleted file mode 100644 index b16304c57..000000000 --- a/openspec/changes/api-product-gateway/specs/consumer-management/spec.md +++ /dev/null @@ -1,52 +0,0 @@ -# consumer-management Specification (Delta) - -## ADDED Requirements - -### Requirement: Consumer detail surfaces its API Product subscriptions (REQ-CON-SUB-001) - -The Consumer detail view in the *Consumers* section MUST list the -consumer's `api_product_subscription` rows (product name, tier, status), -read-only, alongside the authentication and rate-limit/quota configuration -it already renders (`REQ-CON-RL-005`). This requirement adds visibility -only; subscription creation/approval happens on the API Products pages -(`api-product-gateway` `REQ-APG-003`/`REQ-APG-004`), not here. - -@e2e exclude consumer detail subscription list — Playwright regression added in the implementation phase alongside the existing Consumer detail journey - -#### Scenario: an operator sees a consumer's active and pending subscriptions - -- GIVEN a Consumer with one `active` subscription to "WOO Publications API" at tier `free` and one `pending_approval` subscription to "KVK Lookup API" at tier `gold` -- WHEN the operator opens that Consumer's detail view -- THEN both subscriptions are listed with their product name, tier, and status - -#### Scenario: a consumer with no subscriptions shows an empty state - -- GIVEN a Consumer with no `api_product_subscription` rows -- WHEN the operator opens that Consumer's detail view -- THEN the subscriptions section renders an empty state, not an error - -### Requirement: Per-product-tier policy takes precedence over the consumer-level rate limit (REQ-CON-SUB-002) - -The inbound rate-limit/quota policy applied MUST be the subscription's tier -policy, not the Consumer's own `rateLimit`/`quota` (`REQ-CON-RL-001`), when -a request targets an `Endpoint` that belongs to an `api_product` and the -resolved consumer has an `active` `api_product_subscription` to that -product. The Consumer's own `rateLimit`/`quota` remains the -policy for every other endpoint the same consumer calls that is not part of -that product. This requirement states the precedence rule from the -Consumer's perspective; the resolution mechanism lives in `endpoint-runtime` -and is specified by `api-product-gateway` `REQ-APG-005`. - -@e2e exclude backend precedence rule — covered by PHPUnit, no browser UI - -#### Scenario: tier policy overrides the consumer's own rate limit on a product endpoint - -- GIVEN a Consumer with `rateLimit {requestsPerWindow: 1000, windowSeconds: 60}` AND an `active` subscription to a product's `free` tier `{requestsPerWindow: 2, windowSeconds: 60}` -- WHEN that consumer calls the product's endpoint -- THEN the `free` tier's 2-requests-per-window limit is enforced, not the consumer's 1000-requests-per-window limit - -#### Scenario: the consumer's own rate limit still governs non-product endpoints - -- GIVEN the same Consumer as above, also calling an unrelated `Endpoint` that belongs to no `api_product` -- WHEN that consumer calls the unrelated endpoint -- THEN the consumer's own `rateLimit {requestsPerWindow: 1000, windowSeconds: 60}` is enforced unchanged diff --git a/openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md b/openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md deleted file mode 100644 index 60333375a..000000000 --- a/openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md +++ /dev/null @@ -1,78 +0,0 @@ -# endpoint-runtime Specification (Delta) - -## ADDED Requirements - -### Requirement: Deprecated-product-version dispatch attaches Sunset/Deprecation headers (REQ-EP-008) - -The system MUST, via `EndpointService::handleRequest()`'s existing -header-merge choke point (the same one that attaches the -`RateLimit-*`/`Retry-After` headers per `consumer-management` -`REQ-CON-RL-003`), when the dispatched endpoint belongs to an `api_product` -whose `status` is `deprecated`, merge a `Deprecation: true` header and a -`Sunset` header (RFC 8594, HTTP-date format derived from the product's -`sunsetDate`) into the response, for every method and every dispatch path -(simple fast-path `REQ-EP-002` and full pipeline `REQ-EP-003` alike). This -is the runtime mechanism backing `api-product-gateway` `REQ-APG-006`; this -requirement documents where in the dispatch pipeline it is wired, not the -product-level contract itself. - -@e2e exclude backend header attachment — covered by Newman, no browser UI - -#### Scenario: the fast path also carries deprecation headers - -- GIVEN a "simple" endpoint (`REQ-EP-002`) that belongs to a `deprecated` `api_product` -- WHEN a GET request is served via the fast path -- THEN the response carries `Deprecation: true` and `Sunset` despite bypassing the full rule pipeline - -#### Scenario: the full pipeline path also carries deprecation headers - -- GIVEN a non-simple endpoint (`REQ-EP-003`) that belongs to a `deprecated` `api_product` -- WHEN a request runs the full before/dispatch/after pipeline -- THEN the response carries `Deprecation: true` and `Sunset` alongside any rule-produced headers - -#### Scenario: an endpoint in no product carries neither header - -- GIVEN an endpoint that belongs to no `api_product` -- WHEN a request is dispatched -- THEN the response carries neither `Deprecation` nor `Sunset` — no change from pre-change behaviour - -### Requirement: Inbound observability logging for API-product-scoped endpoints (REQ-EP-009) - -The system MUST, for a request dispatched through an `Endpoint` that -belongs to at least one `api_product`, persist a `direction: inbound` -`call_log` row on completion (success or error) carrying the resolved -`product` uuid, -the dispatched `endpoint` uuid, the final `statusCode`, and a `responseTime` -in milliseconds measured from dispatch start to response ready — extending -the existing 429-only inbound logging (`consumer-management` -`REQ-CON-RL-004`) to every outcome, but scoped to product-attached endpoints -only (an endpoint in no `api_product` continues to log inbound rows only on -429, exactly as today). The write MUST be best-effort: a logging failure -MUST NOT block or alter the response (same pattern as -`recordInboundThrottle()`). - -@e2e exclude backend inbound logging — covered by PHPUnit, no browser UI - -#### Scenario: a successful product-scoped request is logged with duration - -- GIVEN an endpoint that belongs to an `api_product` -- WHEN a request to it completes with HTTP 200 in 42ms -- THEN a `call_log` row is persisted with `direction: inbound`, `statusCode: 200`, `product: `, `endpoint: `, and `responseTime: 42` - -#### Scenario: an errored product-scoped request is logged too - -- GIVEN an endpoint that belongs to an `api_product` -- WHEN a request to it fails with HTTP 500 -- THEN a `call_log` row is persisted with `direction: inbound`, `statusCode: 500`, and the product/endpoint linkage, so it counts toward that product's error rate - -#### Scenario: a non-product endpoint's successful requests are still not logged - -- GIVEN an endpoint that belongs to no `api_product` -- WHEN a request to it completes with HTTP 200 -- THEN no `call_log` row is persisted for it (unchanged from pre-change behaviour; only its 429s would be, per `REQ-CON-RL-004`) - -#### Scenario: a logging failure never blocks the response - -- GIVEN the `call_log` write raises an exception (e.g. OpenRegister temporarily unavailable) -- WHEN a product-scoped request otherwise succeeds -- THEN the response is still returned successfully and the logging failure is recorded only in the application log diff --git a/openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md b/openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md deleted file mode 100644 index 7b0be26b3..000000000 --- a/openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md +++ /dev/null @@ -1,70 +0,0 @@ -# prometheus-metrics Specification (Delta) - -## ADDED Requirements - -### Requirement: Per-API-Product request and error gauges (REQ-PROM-012) - -The app MUST expose `integriq_api_product_requests_total` as a gauge -with labels `product` (the `api_product`'s `productSlug`) and `status` -(HTTP status code), and `integriq_api_product_errors_total` as a gauge -with label `product`, both computed declaratively from inbound `call_log` -rows carrying a `product` uuid (`endpoint-runtime` `REQ-EP-009`), the same -`source.kind: "tableCount"` + `groupBy` mechanism that already produces -`calls_total{status,direction}` (`REQ-PROM-005`) — extended with a `product` -label, resolved from the `call_log.product` uuid to its `productSlug` via -the existing label-resolution join pattern. - -#### Scenario: request counts exposed per product and status - -- GIVEN 40 inbound `call_log` rows for product `woo-publications` with status 200 and 3 with status 429 -- WHEN the metrics endpoint is called -- THEN the output includes `integriq_api_product_requests_total{product="woo-publications",status="200"} 40` and `...{product="woo-publications",status="429"} 3` - -#### Scenario: error count reflects statusCode >= 400 rows - -- GIVEN a product with 100 inbound rows, 5 with `statusCode >= 400` -- WHEN the metrics endpoint is called -- THEN `integriq_api_product_errors_total{product=""} 5` - -#### Scenario: a product with no inbound traffic emits a zero placeholder - -- GIVEN an `api_product` with no inbound `call_log` rows yet -- WHEN the metrics endpoint is called -- THEN `integriq_api_product_requests_total{product="",status="200"} 0` is emitted, consistent with every other `REQ-PROM-*` zero-placeholder scenario - -### Requirement: Per-API-Product latency percentile gauges (REQ-PROM-013) - -The app MUST expose `integriq_api_product_latency_seconds` as a gauge -with labels `product` and `quantile` (`0.5`|`0.95`|`0.99`), produced by the -`IntegriqMetricsProvider` `IMetricsProvider` escape hatch (the same -mechanism `circuit_breaker_state` uses, `REQ-PROM-011`) — a percentile -cannot be expressed by the declarative `tableCount`/`objectCount` `groupBy` -vocabulary used by `REQ-PROM-012`, since it requires sorting values within a -group rather than counting rows. Per product, the provider MUST read at -most the most recent 1000 inbound `call_log` rows carrying that product's -uuid and compute p50/p95/p99 from their `responseTime` values (milliseconds, -converted to seconds for the gauge per Prometheus convention). - -#### Scenario: latency gauge exposes p50/p95/p99 per product - -- GIVEN a product's 1000 most recent inbound `call_log` rows with `responseTime` ranging 10-500ms -- WHEN the metrics endpoint is called -- THEN the output includes `integriq_api_product_latency_seconds{product="",quantile="0.5"}`, `...quantile="0.95"`, and `...quantile="0.99"` reflecting those percentiles in seconds - -#### Scenario: a product with no traffic reports zero latency, not a missing series - -- GIVEN a product with zero inbound `call_log` rows -- WHEN the metrics endpoint is called -- THEN all three quantile samples for that product are emitted as `0`, not omitted - -#### Scenario: provider query failure falls back to zero, degraded not broken - -- GIVEN the `call_log` query for a product's percentile computation raises an exception -- WHEN the metrics endpoint collects this metric -- THEN a zero-value fallback is emitted with a warning logged, and the overall endpoint still returns HTTP 200 (same degraded-but-not-broken contract as `REQ-PROM-001`'s partial-failure scenario and `REQ-PROM-011`'s query-failure scenario) - -#### Scenario: percentile computation stays within the scrape performance budget - -- GIVEN 50 active `api_product` rows each with up to 1000 inbound rows -- WHEN the metrics endpoint is called -- THEN percentile computation for all products completes within the existing `REQ-PROM-001` 500ms budget (bounded row count per product, in-memory sort, no additional joins) diff --git a/openspec/changes/api-product-gateway/tasks.md b/openspec/changes/api-product-gateway/tasks.md index 4d8c0cc4f..409b83071 100644 --- a/openspec/changes/api-product-gateway/tasks.md +++ b/openspec/changes/api-product-gateway/tasks.md @@ -1,127 +1,9 @@ -# Tasks: api-product-gateway - -## Implementation Tasks - -### Task 1: Add the api-product-gateway register fragment (schema/migration) -- **spec_ref**: `openspec/changes/api-product-gateway/migration.md`, `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-product-groups-endpoints-into-a-named-versioned-bundle-req-apg-001` -- **files**: `lib/Settings/register.d/api-product-gateway.json` -- **acceptance_criteria**: - - GIVEN the app is upgraded WHEN `occ app:update integriq` runs THEN `api_product` and `api_product_subscription` schemas are registered AND `call_log` gains `product`/`endpoint`/`responseTime` properties without any existing `call_log` row being modified - - GIVEN the migration re-runs a second time WHEN `occ app:update integriq` runs again THEN no error occurs and no duplicate schema rows are created -- [ ] Implement -- [ ] Test - -### Task 2: Resolve per-tier rate-limit policy ahead of the inbound limiter -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-per-tier-rate-limit-enforcement-extends-the-inbound-rate-limiter-req-apg-005`, `openspec/changes/api-product-gateway/specs/consumer-management/spec.md#requirement-per-product-tier-policy-takes-precedence-over-the-consumer-level-rate-limit-req-con-sub-002` -- **files**: `lib/Service/EndpointService.php` -- **acceptance_criteria**: - - GIVEN a consumer with an active subscription to a product's `free` tier `{requestsPerWindow:2,windowSeconds:60}` WHEN it makes 3 requests to the product's endpoint in one window THEN the 3rd receives HTTP 429 with `Retry-After` - - GIVEN the same consumer also calling an endpoint outside any product WHEN it calls that endpoint THEN its own Consumer-level `rateLimit` applies unchanged - - GIVEN a request to an endpoint in no `api_product` WHEN it is dispatched THEN behaviour is byte-for-byte identical to before this change (no regression on `REQ-CON-RL-002`) -- [ ] Implement -- [ ] Test - -### Task 3: Add Sunset/Deprecation headers for deprecated product versions -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-deprecated-product-version-carries-sunset-and-deprecation-headers-req-apg-006`, `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-deprecated-product-version-dispatch-attaches-sunset-deprecation-headers-req-ep-008` -- **files**: `lib/Service/EndpointService.php` -- **acceptance_criteria**: - - GIVEN an endpoint that belongs to an `api_product` with `status: deprecated` and a `sunsetDate` WHEN a request is served (fast path or full pipeline) THEN the response carries `Deprecation: true` and `Sunset: ` - - GIVEN an endpoint that belongs to an active or no product WHEN a request is served THEN neither header is present -- [ ] Implement -- [ ] Test - -### Task 4: Log inbound requests for product-scoped endpoints -- **spec_ref**: `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-inbound-observability-logging-for-api-product-scoped-endpoints-req-ep-009` -- **files**: `lib/Service/EndpointService.php` -- **acceptance_criteria**: - - GIVEN an endpoint that belongs to an `api_product` WHEN a request completes (2xx or error) THEN a `call_log` row is persisted with `direction: inbound`, the product/endpoint uuids, `statusCode`, and `responseTime` - - GIVEN an endpoint in no `api_product` WHEN a successful request completes THEN no `call_log` row is written for it (unchanged from today — only its 429s are logged) - - GIVEN the `call_log` write throws WHEN a product-scoped request otherwise succeeds THEN the response is still returned and the failure is only logged -- [ ] Implement -- [ ] Test - -### Task 5: Add subscription-approval creation to ApprovalService -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-subscription-approval-gate-reuses-the-hitl-approvalservice-req-apg-004`, `openspec/changes/api-product-gateway/design.md#decision-4-subscription-approval-reuses-approvalservices-generic-state-machine-via-one-new-creation-method-not-suspend` -- **files**: `lib/Service/ApprovalService.php` -- **acceptance_criteria**: - - GIVEN a tier with `requiresApproval: true` WHEN `suspendForSubscription()` is called THEN a `pending` `approval_request` is created with no FlowToken snapshot and the configured `approverGroup` is notified -- [ ] Implement -- [ ] Test - -### Task 6: Add ProductSubscriptionsController (subscribe / approve / reject / analytics) -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-consumer-subscribes-to-an-api-product-at-a-tier-req-apg-003`, `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-subscription-approval-gate-reuses-the-hitl-approvalservice-req-apg-004`, `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-gateway-analytics-per-api-product-req-apg-007` -- **files**: `lib/Controller/ProductSubscriptionsController.php`, `appinfo/routes.php` -- **acceptance_criteria**: - - GIVEN `POST /api/products/{id}/subscriptions` with a tier that requires no approval WHEN called THEN HTTP 201 and `status: active` are returned - - GIVEN the same call with a tier requiring approval WHEN called THEN HTTP 202, `status: pending_approval`, and an `approvalRequestId` are returned - - GIVEN an unknown tier name WHEN subscribing THEN HTTP 400 is returned and no subscription is created - - GIVEN `GET /api/products/{id}/analytics` on a product with no traffic WHEN called THEN `requestCount: 0`, `errorRate: 0`, all percentiles `0` — no error -- [ ] Implement -- [ ] Test - -### Task 7: Add per-product latency percentile gauges (AppHost provider escape hatch) -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-latency-percentile-gauges-req-prom-013` -- **files**: `lib/Observability/IntegriqMetricsProvider.php` -- **acceptance_criteria**: - - GIVEN a product's inbound `call_log` rows with varying `responseTime` WHEN `/api/metrics` is scraped THEN `integriq_api_product_latency_seconds{product,quantile}` reflects p50/p95/p99 in seconds - - GIVEN a product with zero traffic WHEN scraped THEN all three quantile samples are `0`, not omitted - - GIVEN the underlying query throws WHEN scraped THEN a zero-value fallback is emitted with a warning logged and the endpoint still returns HTTP 200 -- [ ] Implement -- [ ] Test - -### Task 8: Extend declarative request/error gauges with a product label -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-request-and-error-gauges-req-prom-012` -- **files**: `src/manifest.json` -- **acceptance_criteria**: - - GIVEN inbound `call_log` rows carrying a `product` uuid WHEN `/api/metrics` is scraped THEN `integriq_api_product_requests_total{product,status}` and `integriq_api_product_errors_total{product}` are exposed - - GIVEN a product with no traffic WHEN scraped THEN a zero-value placeholder is emitted -- [ ] Implement -- [ ] Test - -### Task 9: Add the API Products SPA pages (index + detail) -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-products-management-ui-req-apg-002` -- **files**: `src/manifest.json`, `src/views/ApiProducts/ApiProductDetail.vue` -- **acceptance_criteria**: - - GIVEN an authenticated admin WHEN they navigate to `/apps/integriq/products` THEN the API Products index page renders with content visible - - GIVEN a product's detail page WHEN opened THEN the admin can add/remove endpoints and add/edit named tiers, and see the analytics panel (request count, error rate, p50/p95/p99) and pending subscriptions with approve/reject actions -- [ ] Implement -- [ ] Test - -### Task 10: Surface subscriptions on the Consumer detail view -- **spec_ref**: `openspec/changes/api-product-gateway/specs/consumer-management/spec.md#requirement-consumer-detail-surfaces-its-api-product-subscriptions-req-con-sub-001` -- **files**: `src/manifest.json` (Consumer detail config), or the Consumer detail component it references -- **acceptance_criteria**: - - GIVEN a consumer with active and pending subscriptions WHEN its detail view is opened THEN both are listed with product name, tier, and status - - GIVEN a consumer with no subscriptions WHEN its detail view is opened THEN an empty state renders, not an error -- [ ] Implement -- [ ] Test - -### Task 11: Seed data for api_product and api_product_subscription -- **spec_ref**: `openspec/changes/api-product-gateway/design.md#seed-data` -- **files**: `lib/Settings/register.d/api-product-gateway.json` (`x-openregister-seed`) -- **acceptance_criteria**: - - GIVEN a fresh install WHEN the app is enabled THEN 3 `api_product` rows (one deprecated) and 2 `api_product_subscription` rows (one active, one pending_approval) exist, using the general organization/publications domain data already established by this app's other seeds -- [ ] Implement -- [ ] Test - -## Verification -- [ ] All tasks checked off -- [ ] `openspec validate` passes -- [ ] Manual testing against acceptance criteria -- [ ] Code review against spec requirements - -## Tests (company-wide ADR-009) - -- [ ] PHPUnit unit tests for new/changed business logic (`tests/Unit/Service/EndpointServiceTierPolicyTest.php`, `tests/Unit/Service/ApprovalServiceSubscriptionTest.php`, `tests/Unit/Observability/IntegriqMetricsProviderTest.php`, `tests/Unit/Controller/ProductSubscriptionsControllerTest.php`) -- [ ] Newman/Postman tests for the new API endpoints (subscribe/approve/reject/analytics; over-tier 429; deprecated-product headers) -- [ ] Browser tests (Playwright MCP) for the API Products index/detail pages and the Consumer detail subscription list -- [ ] All tests pass (`composer test`, `newman run`) - -## Documentation (company-wide ADR-010) - -- [ ] Feature documentation updated in `docs/` (API Products concept, tier configuration, deprecation headers) -- [ ] Screenshot captured and committed to `docs/images/` - -## i18n (company-wide hydra ADR-007) - -- [ ] Dutch (`nl_NL`) and English (`en_US`) translation strings added for all new SPA strings (product/tier labels, subscription status, deprecation notices, analytics panel) +# Tasks: api-product-gateway (superseded) + +The original 11-task / 33-checkbox list was removed with the 2026-09-02 +retirement (see proposal.md for the disposition; the list survives in +`archive/2026-07-15-api-product-gateway/tasks.md`, where 21/33 boxes are +checked with per-task evidence, and in git history). The residual +live-instance verification, docs and `nl_NL` l10n work is listed there with +per-box reasons and belongs to a verification-pack-style follow-up. There +is nothing to implement from this change directly. diff --git a/openspec/changes/api-product-gateway/test-plan.md b/openspec/changes/api-product-gateway/test-plan.md deleted file mode 100644 index ed8977d9e..000000000 --- a/openspec/changes/api-product-gateway/test-plan.md +++ /dev/null @@ -1,237 +0,0 @@ -# Test Plan: api-product-gateway - -## Test Cases - -### TC-1: API Product groups endpoints without mutating them -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-product-groups-endpoints-into-a-named-versioned-bundle-req-apg-001` -- **type**: api -- **preconditions**: three existing `Endpoint`s -- **steps**: create an `api_product` referencing all three endpoint uuids -- **expected result**: the product persists with all three uuids; none of the three `Endpoint` objects change -- **test command**: /test-api - -### TC-2: Product versions are independent -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-product-groups-endpoints-into-a-named-versioned-bundle-req-apg-001` -- **type**: api -- **preconditions**: two `api_product` rows sharing a `productSlug`, different `version` -- **steps**: edit one version's `endpoints` array -- **expected result**: the other version's `endpoints` array is unaffected -- **test command**: /test-api - -### TC-3: API Products index page renders -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-products-management-ui-req-apg-002` -- **type**: functional -- **preconditions**: authenticated admin session -- **steps**: navigate to `/apps/integriq/products` via sidebar nav -- **expected result**: index page renders inside main content area with content visible -- **test command**: /test-functional - -### TC-4: Product detail exposes endpoint picker and tier editor -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-api-products-management-ui-req-apg-002` -- **type**: functional -- **preconditions**: one `api_product` and one `Endpoint` exist -- **steps**: open the product's detail page; add an endpoint; add a tier with a rateLimit -- **expected result**: the endpoint appears in the product's `endpoints`; the tier persists with its policy -- **test command**: /test-functional - -### TC-5: Endpoint picker and tier editor are accessible -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#non-functional-requirements` -- **type**: accessibility -- **preconditions**: product detail page loaded -- **steps**: run WCAG 2.2 AA audit against the endpoint picker (`NcSelect`) and tier editor controls -- **expected result**: no missing accessible-name violations; `inputLabel`/`ariaLabelCombobox` present on every `NcSelect` -- **test command**: /test-accessibility - -### TC-6: Subscribing to a no-approval tier activates immediately -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-consumer-subscribes-to-an-api-product-at-a-tier-req-apg-003` -- **type**: api -- **preconditions**: product with a `free` tier, `requiresApproval` absent -- **steps**: `POST /api/products/{id}/subscriptions` with `{consumerId, tier:"free"}` -- **expected result**: HTTP 201, `status: active` -- **test command**: /test-api - -### TC-7: Subscribing to an unknown tier is rejected -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-consumer-subscribes-to-an-api-product-at-a-tier-req-apg-003` -- **type**: api -- **preconditions**: product with tiers `free`/`gold` only -- **steps**: `POST /api/products/{id}/subscriptions` with `tier:"platinum"` -- **expected result**: HTTP 400, no subscription created -- **test command**: /test-api - -### TC-8: Approval-gated tier creates a pending subscription and notifies approvers -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-subscription-approval-gate-reuses-the-hitl-approvalservice-req-apg-004` -- **type**: api -- **preconditions**: product with a `gold` tier, `requiresApproval: true`, `approverGroup: "gateway-approvers"` -- **steps**: `POST /api/products/{id}/subscriptions` with `tier:"gold"` -- **expected result**: HTTP 202, `status: pending_approval`, `approvalRequestId` present; every member of `gateway-approvers` receives an NC notification -- **test command**: /test-api - -### TC-9: Approving a subscription activates it -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-subscription-approval-gate-reuses-the-hitl-approvalservice-req-apg-004` -- **type**: api -- **preconditions**: a `pending_approval` subscription with its linked `pending` approval_request -- **steps**: an authorized approver calls the approve action -- **expected result**: subscription `status` becomes `active`, `activatedAt` set -- **test command**: /test-api - -### TC-10: A pending subscription grants no access -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-subscription-approval-gate-reuses-the-hitl-approvalservice-req-apg-004` -- **type**: api -- **preconditions**: consumer with only a `pending_approval` subscription to a product -- **steps**: call the product's endpoint -- **expected result**: HTTP 403 -- **test command**: /test-api - -### TC-11: Over-tier request returns 429 without exhausting the consumer's own limit -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-per-tier-rate-limit-enforcement-extends-the-inbound-rate-limiter-req-apg-005`, `openspec/changes/api-product-gateway/specs/consumer-management/spec.md#requirement-per-product-tier-policy-takes-precedence-over-the-consumer-level-rate-limit-req-con-sub-002` -- **type**: api -- **preconditions**: consumer with `rateLimit{1000,60}` AND an active subscription to a product's `free` tier `{2,60}` -- **steps**: call the product's endpoint 3 times within one window -- **expected result**: requests 1-2 succeed, request 3 returns HTTP 429 with `Retry-After`; the consumer's own 1000-budget is unaffected -- **test command**: /test-api - -### TC-12: Non-product endpoints are unaffected by tier enforcement -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-per-tier-rate-limit-enforcement-extends-the-inbound-rate-limiter-req-apg-005` -- **type**: regression -- **preconditions**: an endpoint that belongs to no `api_product` -- **steps**: call it repeatedly within the consumer's own `rateLimit` -- **expected result**: behaviour identical to pre-change `REQ-CON-RL-002` -- **test command**: /test-regression - -### TC-13: Deprecated product version emits Sunset/Deprecation headers -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-deprecated-product-version-carries-sunset-and-deprecation-headers-req-apg-006`, `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-deprecated-product-version-dispatch-attaches-sunset-deprecation-headers-req-ep-008` -- **type**: api -- **preconditions**: `api_product` with `status: deprecated`, `sunsetDate` set, grouping an endpoint -- **steps**: call the endpoint -- **expected result**: response carries `Deprecation: true` and `Sunset: ` -- **test command**: /test-api - -### TC-14: Active product version emits neither header (both dispatch paths) -- **spec_ref**: `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-deprecated-product-version-dispatch-attaches-sunset-deprecation-headers-req-ep-008` -- **type**: api -- **preconditions**: one simple (fast-path) and one full-pipeline endpoint, both in an `active` product -- **steps**: call each -- **expected result**: neither response carries `Deprecation` or `Sunset` -- **test command**: /test-api - -### TC-15: Product-scoped requests are logged with duration (success and error) -- **spec_ref**: `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-inbound-observability-logging-for-api-product-scoped-endpoints-req-ep-009` -- **type**: api -- **preconditions**: an endpoint in an `api_product` -- **steps**: call it once successfully, once forcing a 500 -- **expected result**: two `call_log` rows persisted with `direction:inbound`, product/endpoint uuids, correct `statusCode`, and a positive `responseTime` -- **test command**: /test-api - -### TC-16: Non-product endpoint successful requests remain unlogged -- **spec_ref**: `openspec/changes/api-product-gateway/specs/endpoint-runtime/spec.md#requirement-inbound-observability-logging-for-api-product-scoped-endpoints-req-ep-009` -- **type**: regression -- **preconditions**: an endpoint in no `api_product` -- **steps**: call it successfully -- **expected result**: no new `call_log` row for that call -- **test command**: /test-regression - -### TC-17: Gateway analytics reflect recent traffic -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-gateway-analytics-per-api-product-req-apg-007` -- **type**: api -- **preconditions**: a product with a mix of recorded inbound rows (some errors) -- **steps**: `GET /api/products/{id}/analytics` -- **expected result**: `requestCount`, `errorRate`, and `latency.p50/p95/p99` reflect the recorded rows -- **test command**: /test-api - -### TC-18: Analytics for a traffic-free product report zero, not an error -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-gateway-analytics-per-api-product-req-apg-007` -- **type**: api -- **preconditions**: newly created product, no `call_log` rows -- **steps**: `GET /api/products/{id}/analytics` -- **expected result**: `requestCount:0`, `errorRate:0`, all percentiles `0`, HTTP 200 -- **test command**: /test-api - -### TC-19: Per-product Prometheus request/error gauges -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-request-and-error-gauges-req-prom-012` -- **type**: api -- **preconditions**: inbound rows across two products, mixed status codes -- **steps**: `GET /api/metrics` -- **expected result**: `integriq_api_product_requests_total{product,status}` and `integriq_api_product_errors_total{product}` present with correct counts; zero-value placeholder for a traffic-free product -- **test command**: /test-api - -### TC-20: Per-product Prometheus latency percentile gauges -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-latency-percentile-gauges-req-prom-013` -- **type**: api -- **preconditions**: a product with recorded `responseTime` values -- **steps**: `GET /api/metrics` -- **expected result**: `integriq_api_product_latency_seconds{product,quantile}` present for `0.5`/`0.95`/`0.99`, in seconds -- **test command**: /test-api - -### TC-21: Percentile gauge degrades gracefully on query failure -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-latency-percentile-gauges-req-prom-013` -- **type**: api -- **preconditions**: simulate the underlying `call_log` query throwing -- **steps**: `GET /api/metrics` -- **expected result**: zero-value fallback emitted, warning logged, endpoint still returns HTTP 200 -- **test command**: /test-api - -### TC-22: Metrics scrape stays within budget at moderate scale -- **spec_ref**: `openspec/changes/api-product-gateway/specs/prometheus-metrics/spec.md#requirement-per-api-product-latency-percentile-gauges-req-prom-013` -- **type**: performance -- **preconditions**: 50 active products, up to 1000 rows each -- **steps**: `GET /api/metrics`, measure wall time -- **expected result**: completes within the existing 500ms `REQ-PROM-001` scrape budget -- **test command**: /test-performance - -### TC-23: Consumer detail lists active and pending subscriptions -- **spec_ref**: `openspec/changes/api-product-gateway/specs/consumer-management/spec.md#requirement-consumer-detail-surfaces-its-api-product-subscriptions-req-con-sub-001` -- **type**: functional -- **preconditions**: a Consumer with one active and one pending subscription -- **steps**: open that Consumer's detail view -- **expected result**: both subscriptions listed with product name, tier, status -- **test command**: /test-functional - -### TC-24: Consumer detail shows an empty state with no subscriptions -- **spec_ref**: `openspec/changes/api-product-gateway/specs/consumer-management/spec.md#requirement-consumer-detail-surfaces-its-api-product-subscriptions-req-con-sub-001` -- **type**: functional -- **preconditions**: a Consumer with zero subscriptions -- **steps**: open that Consumer's detail view -- **expected result**: empty state renders, no error thrown -- **test command**: /test-functional - -### TC-25: Gateway operator persona — subscribe, over-limit, deprecate end to end -- **spec_ref**: `openspec/changes/api-product-gateway/specs/api-product-gateway/spec.md#requirement-consumer-subscribes-to-an-api-product-at-a-tier-req-apg-003` -- **type**: persona -- **persona**: Mark (MKB Software Vendor integrating against the gateway) -- **preconditions**: a published API Product with tiers -- **steps**: subscribe a consumer at a low tier, exceed its limit, observe the 429 and Retry-After, then have an admin deprecate the product version and re-call the endpoint -- **expected result**: the full lifecycle behaves per REQ-APG-003/005/006 from an integrator's point of view — clear 429 with retry guidance, clear deprecation signal -- **test command**: /test-persona-mark - -## Coverage Summary - -| Requirement | Covered by | -|---|---| -| api-product-gateway REQ-APG-001 | TC-1, TC-2 | -| api-product-gateway REQ-APG-002 | TC-3, TC-4, TC-5 | -| api-product-gateway REQ-APG-003 | TC-6, TC-7, TC-25 | -| api-product-gateway REQ-APG-004 | TC-8, TC-9, TC-10 | -| api-product-gateway REQ-APG-005 | TC-11, TC-12, TC-25 | -| api-product-gateway REQ-APG-006 | TC-13, TC-14, TC-25 | -| api-product-gateway REQ-APG-007 | TC-17, TC-18 | -| consumer-management REQ-CON-SUB-001 | TC-23, TC-24 | -| consumer-management REQ-CON-SUB-002 | TC-11 | -| endpoint-runtime REQ-EP-008 | TC-13, TC-14 | -| endpoint-runtime REQ-EP-009 | TC-15, TC-16 | -| prometheus-metrics REQ-PROM-012 | TC-19 | -| prometheus-metrics REQ-PROM-013 | TC-20, TC-21, TC-22 | - -All 13 ADDED requirements across the 4 spec deltas have at least one covering -test case; every requirement with an error/degraded-path scenario has a -dedicated negative test case (TC-7, TC-10, TC-12, TC-16, TC-18, TC-21). - -## Out of Scope - -- Self-service developer portal / API key issuance UI — out of scope for - this change (proposal.md), no test cases written. -- Monetization/billing on tiers — out of scope, no test cases written. -- Load/soak testing of the distributed rate-limit cache under concurrency - beyond what `consumer-management`'s existing "counters are correct under - concurrency" test already covers for the underlying - `InboundRateLimitService` — this change does not modify that service, so - its concurrency guarantee is inherited, not re-verified here. diff --git a/openspec/changes/approvals-verification-pack/.openspec.yaml b/openspec/changes/approvals-verification-pack/.openspec.yaml new file mode 100644 index 000000000..7b2c0fa7b --- /dev/null +++ b/openspec/changes/approvals-verification-pack/.openspec.yaml @@ -0,0 +1,2 @@ +schema: conduction +created: 2026-09-02 diff --git a/openspec/changes/approvals-verification-pack/proposal.md b/openspec/changes/approvals-verification-pack/proposal.md new file mode 100644 index 000000000..29bfd0a29 --- /dev/null +++ b/openspec/changes/approvals-verification-pack/proposal.md @@ -0,0 +1,79 @@ +--- +kind: code +depends_on: [] +--- + +# Proposal: approvals-verification-pack + +## Summary + +Close the verification debt the archived `hitl-approval-rule-action` change +left open with honest unticked boxes: the shipped approval surface +(`ApprovalService`, `ApprovalsController`, the Pending Approvals pages, the +rule editor's `approval` form) has unit coverage but no end-to-end proof, no +Newman coverage, no feature documentation and no compiled l10n catalog +entries. This change adds exactly that verification and nothing else: no +approval behavior changes here. + +## Motivation + +The archived twin (`archive/2026-07-15-hitl-approval-rule-action`) checked +31/42 boxes and left the rest open with reasons ("no live instance in this +environment", "not run in this finalization pass"). That debt has sat +invisible inside a superseded 0/42 umbrella ever since. It matters now +because `hitl-on-shared-tasks` is rewiring approval expiry and outcomes onto +OpenRegister's task service: a regression net around the existing +suspend-approve-resume behavior is the difference between that cutover being +verifiable and being hopeful. + +## Affected Projects + +- [x] Project: `integriq` — tests (PHPUnit integration, Newman, Playwright), + docs, l10n. No `lib/` behavior changes. + +## Scope + +### In Scope + +1. PHPUnit integration test: suspend → approve → resume through a real + endpoint rule chain (real `EndpointService` + `ApprovalService` wiring, + faked HTTP/OR edges only), per the archived change's open ADR-009 box. +2. Newman coverage for `/api/approvals*`: list, detail, approve, reject, and + the 403/404/409 error paths, added to the existing Postman collection. +3. Playwright specs for the Pending Approvals list + detail pages (approve + with comment, reject with comment) and the rule editor's `approval` action + form, traced to the scenarios in + `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` + per `hydra-gate-e2e-coverage`. +4. Feature documentation in `docs/`: the `approval` rule action type, the + Synchronization `requiresApproval` gate, the Pending Approvals UI; one + screenshot in `docs/images/`. +5. l10n: `en_US` source strings verified extractable and `nl_NL` catalog + entries for the Approvals UI and the `ApprovalForm.vue` fields. + +### Out of Scope + +- Any change to approval behavior, schemas or routes. +- Expiry/onTimeout/onReject semantics: `hitl-on-shared-tasks` owns their + move onto OR's task service. If that change alters resume mechanics before + this one runs, the tests here assert the behavior at HEAD when written. +- New approval features (delegation, escalation): OR `TaskService` / + `TaskSequenceService` territory, per the superseded umbrella's disposition. + +## Approach + +Test-only PR(s). The integration test lives in `tests/Integration/`, the +Newman additions in `tests/postman/`, the Playwright specs in +`tests/e2e/spec-coverage/`. Docs follow the existing `docs/` layout. + +## Impact + +- `tests/Integration/ApprovalRuleChainTest.php` — new. +- `tests/postman/*.postman_collection.json` — approval scenarios added. +- `tests/e2e/spec-coverage/approval-workflow.spec.ts` — new. +- `docs/` + `docs/images/` — new page + screenshot. +- `l10n/` — catalog entries. + +## Rollback Strategy + +Tests and docs only; revert the PR. No schema, route or service is touched. diff --git a/openspec/changes/approvals-verification-pack/tasks.md b/openspec/changes/approvals-verification-pack/tasks.md new file mode 100644 index 000000000..9f3525825 --- /dev/null +++ b/openspec/changes/approvals-verification-pack/tasks.md @@ -0,0 +1,69 @@ +# Tasks: approvals-verification-pack + +## 1. Integration test + +### Task 1: Suspend → approve → resume through a real rule chain +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `tests/Integration/ApprovalRuleChainTest.php` +- **acceptance_criteria**: + - GIVEN an endpoint with a `before`-phase `approval` rule followed by a later rule WHEN the endpoint is called THEN the response is `202` with a polling URL and an `approval_request` persists the FlowToken snapshot + - GIVEN that request is approved WHEN `ApprovalsController::approve()` runs THEN `processRules()` resumes at the rule after the approval rule and the chain completes + - GIVEN that request is rejected THEN the rule's configured `onReject` outcome is applied +- [ ] Implement +- [ ] Test + +## 2. Newman + +### Task 2: `/api/approvals*` scenarios in the Postman collection +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `tests/postman/` (existing collection) +- **acceptance_criteria**: + - GIVEN the collection runs against a live instance THEN list, detail, approve and reject succeed for an approver-group member + - GIVEN a non-member calls approve THEN `403`; GIVEN a missing id THEN `404`; GIVEN a double approve THEN `409` +- [ ] Implement +- [ ] Test + +## 3. Playwright + +### Task 3: Pending Approvals list + detail +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `tests/e2e/spec-coverage/approval-workflow.spec.ts` +- **acceptance_criteria**: + - GIVEN a pending request WHEN an approver opens the Approvals pages THEN they can approve with a comment and the row leaves the pending list + - GIVEN a pending request THEN reject with a comment records the rejection +- [ ] Implement +- [ ] Test + +### Task 4: Rule editor `approval` action form +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `tests/e2e/spec-coverage/approval-workflow.spec.ts` +- **acceptance_criteria**: + - GIVEN the rule editor WHEN `approval` is chosen as the action type THEN the approver-group, expiry and onReject/onTimeout fields render and persist on save +- [ ] Implement +- [ ] Test + +## 4. Docs & l10n + +### Task 5: Feature documentation + screenshot +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `docs/`, `docs/images/` +- **acceptance_criteria**: + - GIVEN `docs/` THEN a page describes the `approval` rule action, the `requiresApproval` sync gate and the Pending Approvals UI, with one committed screenshot +- [ ] Implement +- [ ] Test + +### Task 6: l10n catalog entries +- **spec_ref**: `openspec/changes/hitl-approval-rule-action/specs/approval-workflow/spec.md` +- **files**: `l10n/` +- **acceptance_criteria**: + - GIVEN the Approvals UI strings THEN `nl_NL` catalog entries exist (or the external localization pipeline demonstrably carries them; record which) +- [ ] Implement +- [ ] Test + +## Verification +- [ ] All tasks checked off +- [ ] Manual testing against acceptance criteria +- [ ] Code review against spec requirements + +## Tests (company-wide ADR-009) +- [ ] All tests pass (`composer test`, `newman run`, Playwright suite) diff --git a/openspec/changes/cdc-incremental-sync/design.md b/openspec/changes/cdc-incremental-sync/design.md deleted file mode 100644 index 7f5f0b8db..000000000 --- a/openspec/changes/cdc-incremental-sync/design.md +++ /dev/null @@ -1,276 +0,0 @@ -# Design: cdc-incremental-sync - -## Architecture Overview -Incremental sync is a mode flag on the existing Source → Synchronization → -SynchronizationContract triad (ADR-005) — it does not introduce a new -entity or a parallel sync path. It changes three things inside the existing -extern→intern flow in `SynchronizationService`: - -``` -synchronize() - └─ synchronizeExternToIntern() - ├─ Stage 2: getAllObjectsFromSource() → getAllObjectsFromApi() - │ [NEW] when syncMode=incremental: inject stored cursorWatermark - │ into the Twig context already used for {{ data.* }} - │ endpoint templating, extended to sourceConfig.query too - ├─ Stage 4: per-object processSynchronizationObject() loop (unchanged) - ├─ Stage 5: deleteInvalidObjects() gate - │ [NEW] syncMode=incremental short-circuits this call entirely, - │ at the same site fetchComplete (REQ-009/REQ-010) already - │ gates it - └─ end-of-run persistSynchronization() - [NEW] when syncMode=incremental AND fetchComplete: compute and - persist the new cursorWatermark from the fetched records -``` - -This mirrors exactly how `currentPage` (pagination-in-progress) and -`targetLastSynced` (last successful pass) already round-trip through -`$synchronization` as plain array/OR-object fields — `cursorWatermark` is a -third field in that same family, not a new subsystem. - -## Goals / Non-Goals - -**Goals:** -- Let large/high-volume `api` sources skip already-synced records on every - run after the first, via a stored high-watermark cursor. -- Guarantee the watermark can never advance past data the engine has not - durably processed (composes with REQ-009 fetch-completeness). -- Guarantee `deleteInvalidObjects()` never runs against a partial view of - the source (incremental mode is *always* a partial view, by definition). -- Give operators an explicit, auditable way back to a full baseline - (reset-cursor). - -**Non-Goals:** -- Log-based/binlog CDC (no DB-source adapter exists to attach to — see - proposal.md Out of Scope). -- Automatic cursor-field inference. -- Sub-run (per-page) watermark checkpointing — REQ-009's fetch-completeness - signal is already whole-fetch, all-or-nothing; incremental sync composes - with it as-is rather than adding a second, finer-grained completeness - concept. -- Deletion detection for incremental mode by any means (e.g. a "soft" - ratio-based partial guard) — REQ-010's existing ratio guard is explicitly - a *bulk full-fetch* diff mechanism; giving it a partial-fetch input would - silently reintroduce exactly the false-positive deletions the guardrails - change was written to prevent. Incremental mode's deletion answer is "not - supported, use `full` mode periodically or the reset-cursor action." - -## Decisions - -### Decision 1: Cursor watermark storage — Synchronization object field, not a `sync_cursor` object -**Choice:** Add `cursorWatermark` (string) and `syncMode` (string enum -`full`|`incremental`, default `full`) as top-level properties on the -existing `synchronization` schema in -`lib/Settings/integriq_register.json`, alongside the pre-existing -`currentPage` (pagination-in-progress cursor) and `targetLastSynced` fields. - -**Why:** The codebase already has an established, working convention for -exactly this kind of "small piece of per-pass state that belongs to one -Synchronization" data: `currentPage` is read/reset directly on -`$synchronization` inside `getAllObjectsFromApi()` -(`SynchronizationService.php` ~L3941-3980), and `targetLastSynced` is -written directly onto `$synchronization` at the end of -`synchronizeExternToIntern()` (~L1861-1864) via -`persistSynchronization()`. A cursor watermark is the same shape of fact — -one value, one owner, updated at the same point in the same method. Reusing -the field-on-Synchronization pattern means: -- No new OR schema, no new register entry, no new REST surface to fetch/ - list watermarks. -- No new join/lookup on every fetch — the watermark is already in memory - wherever `$synchronization` is (it is loaded once per run via - `toSynchronization()`). -- Rollback is free (see proposal.md Rollback Strategy) — an unset field is - just `null`/absent, exactly like every pre-existing Synchronization today - has no opinion on `currentPage` beyond its default. - -**Alternative considered — separate `sync_cursor` OR object (1 per -Synchronization, or 1 per Synchronization × cursor-field):** Rejected. -A separate object would need its own schema, its own find-or-create -resolution on every run (an extra OR round-trip per sync pass, on the hot -path), and — worse — a second place where "did this run's watermark update -actually commit" could diverge from whether the run itself committed, -reintroducing a variant of the exact split-state problem -sync-safety-guardrails REQ-011 (test runs make no writes) was written to -close for contracts/targets. A dedicated object would earn its keep if -watermarks needed independent versioning/history (e.g. audit trail of every -watermark value ever set) — not needed here; `targetLastSynced` doesn't get -that either, and this field is symmetric with it. - -### Decision 2: Cursor filter injection — extend the existing Twig endpoint-templating context, not a second templating mechanism -**Choice:** `getAllObjectsFromApi()` already Twig-renders -`sourceConfig.endpoint` when it contains `{{`/`}}`, via -`MappingService::renderTemplateString(template: $endpoint, context: ['data' -=> $contextData])` (SynchronizationService.php ~L3888-3904). This change -adds a `cursor` key to that same context — -`context: ['data' => $contextData, 'cursor' => $cursorWatermark]` — so an -admin can write `sourceConfig.endpoint: ".../items?updatedAfter={{ cursor -}}"`. It also extends the identical `{{`/`}}`-detection-then- -`renderTemplateString()` treatment to each scalar value in -`sourceConfig.query` (currently passed through to `$config['query']` -verbatim, untemplated), so an admin can instead write -`sourceConfig.query.updatedAfter: "{{ cursor }}"` when the source takes the -cursor as a query parameter rather than a path/endpoint segment. `cursor` -resolves to an empty string on a synchronization's first-ever incremental -run (no prior watermark) — sources whose API treats an absent/empty cursor -parameter as "give me everything" get a correct first full-ish incremental -baseline for free; sources that require a non-empty value document that in -their `sourceConfig.query` default (e.g. -`sourceConfig.query.updatedAfter: "{{ cursor|default('1970-01-01') }}"`, -which Twig's `default` filter already supports with no engine change). - -**Why:** "Reuse the existing request-config templating" is an explicit -proposal constraint. Endpoint templating is the only templating already -wired into the fetch path; the minimal, lowest-risk change is widening its -context by one key and widening its application from one field (endpoint) -to one more (query values) using the exact same -detect-`{{`-then-`renderTemplateString()` idiom already proven at -L3889-3904 — not introducing a second engine, a second context-building -function, or a bespoke cursor-substitution mini-language. - -**Alternative considered — a dedicated `{{cursor}}` placeholder syntax -resolved by string-replace, bypassing Twig:** Rejected. Twig is already a -hard dependency of this file (`use Twig\Error\LoaderError;` etc. at the top -of `SynchronizationService.php`) and `renderTemplateString()` already -supports filters/defaults for free (as shown above); a bespoke -string-replace would be strictly less capable while adding a second -code path to maintain and explain. - -**Alternative considered — a dedicated `cursorQueryParam` config key that -the engine sets directly into `$config['query']`, no templating:** -Rejected as the sole mechanism (though effectively a special case of the -templating approach still applies) because it cannot express -endpoint-path-segment cursors (e.g. `/items/since/{{ cursor }}`) or -composite values (e.g. a cursor embedded in a JSON request body via -`useDataAsRequestBody`), while the templating approach handles all three -injection points (endpoint, query, and — already possible today with zero -further change, since `$config['body']` is built from `$data` which callers -control — body) through one mechanism. - -### Decision 3: Composition point with the sync-safety guard — the existing `$fetchComplete` local in `synchronizeExternToIntern()` Stage 5 -**Choice:** Both new behaviors attach to the exact code that already exists -at `synchronizeExternToIntern()` Stage 5 (SynchronizationService.php -~L1766-1802): - -```php -$fetchComplete = ($rateLimitException === null && ($fetchInfo['complete'] ?? true)); - -$deletedCount = 0; -$guardInfo = null; -if ($isTest === false) { - // [NEW] incremental mode never runs deletion — checked BEFORE the - // existing fetchComplete-gated call, so it short-circuits deletion - // for its own explicit reason ('incremental_mode') rather than - // reusing fetchComplete's 'fetch_incomplete' reason, which would be - // misleading (the fetch can be perfectly complete for what it asked - // for — it just didn't ask for everything). - $syncMode = (string) ($synchronization['syncMode'] ?? 'full'); - if ($syncMode !== 'incremental') { - $deletedCount = $this->deleteInvalidObjects( - synchronization: $synchronization, - synchronizedTargetIds: $synchronizedTargetIds, - deleteRestriction: $deleteRestriction, - data: $deleteData, - fetchComplete: $fetchComplete, - forceDeletion: ($forceDeletion ?? false), - guardInfo: $guardInfo - ); - } else { - $guardInfo = ['guarded' => true, 'reason' => 'incremental_mode', 'ratio' => null, 'threshold' => null]; - } - - // [NEW] watermark advance — same $fetchComplete boolean REQ-010 - // already computed above; a rate-limited or otherwise incomplete - // fetch (REQ-009) blocks the watermark exactly as it blocks deletion. - if ($syncMode === 'incremental' && $fetchComplete === true) { - $newWatermark = $this->computeCursorWatermark(synchronization: $synchronization, objectList: $objectList); - if ($newWatermark !== null) { - $synchronization['cursorWatermark'] = $newWatermark; - } - } -} -``` - -**Why:** REQ-009 (fetch-completeness tracking) and REQ-010 (deletion -gating) already compute and thread a single `$fetchComplete` boolean to -exactly this point — it is the one place in the method that knows, with -certainty, "did this run's fetch see everything it was supposed to." Both -new invariants (never advance the watermark on an incomplete fetch; never -delete in incremental mode) are correctness rules *about that same fact*, -so attaching them here means: -- There is no way for a future change to update `$fetchComplete`'s - computation (e.g. adding a new failure mode) without both the deletion - guard and the watermark guard picking it up automatically — they read - the same variable. -- The incremental-mode deletion block is unconditional (checked first, not - folded into `$fetchComplete`) so it cannot be defeated by - `forceDeletion: true` the way the ratio guard can — deleting in - incremental mode is not a "the operator explicitly overrode a soft - guard" situation, it is "the data needed to make this decision correctly - was never fetched," which no override can fix (proposal.md Risk 3). -- `deleteInvalidObjects()` itself also gets a defense-in-depth check - (`$synchronization['syncMode'] === 'incremental'` → return 0 immediately, - mirroring its existing `$fetchComplete === false` early-return at - L2533-2551) so a future caller that reaches it directly (bypassing - `synchronizeExternToIntern()`) cannot accidentally delete against a - partial incremental fetch either. - -**Alternative considered — a separate `syncMode`-only guard clause -independent of `$fetchComplete`, placed earlier in the method (e.g. right -after Stage 2 fetch):** Rejected for the watermark half — advancing the -watermark logically depends on the fetch being *complete*, not merely on -mode, so it must read `$fetchComplete` regardless of where it's placed; -placing it right next to the deletion gate (which already needs the same -variable) avoids computing or threading `$fetchComplete` to two different -locations in the method. - -## Risks / Trade-offs -- [Risk] An incremental synchronization whose source has no reliable - monotonic field (flaky clocks, non-monotonic ids) silently misses - records → [Mitigation] Documented in `sourceConfig.cursorField`'s schema - description as an admin responsibility, same as `idPosition` today; - Risk 1 in proposal.md covers the missing-field case specifically (throws - rather than silently skipping). -- [Risk] Extending Twig templating to `sourceConfig.query` values is a - small surface-area increase (any query value containing `{{`/`}}` is now - template-evaluated, not passed through literally) → [Mitigation] Uses the - exact same evaluation function and trust boundary as the pre-existing - endpoint templating (both operate on admin-authored `sourceConfig`, never - on source-returned data), so this does not cross a new trust boundary — - it is the same boundary, one more field. -- [Risk] Operators may expect `reset-cursor` to also retroactively delete - now-possibly-stale target objects, or to restore deletion-based garbage - collection once the next fetch happens to cover the whole source → - [Mitigation] `reset-cursor` only clears `cursorWatermark`; it deliberately - does **not** change `syncMode`. Per Decision 3, `deleteInvalidObjects()` - is skipped unconditionally whenever `syncMode === 'incremental'`, with no - exception for "this particular fetch happened to be full" — the engine - has no reliable way to verify that an admin-templated `{{ cursor }}` - placeholder resolving to an empty string actually caused the source to - return its complete set (that is a semantic guarantee about the source's - API, not something the engine can structurally confirm). An operator who - wants deletion detection back MUST explicitly switch the Synchronization's - `syncMode` to `full` — a separate, deliberate action, not a side effect of - `reset-cursor`. Document both of these (what `reset-cursor` does and does - not do) explicitly in the SPA tooltip/help text (tasks.md). - -## Migration Plan -No Nextcloud database migration — see `migration.md` (skipped, with -rationale) and Decision 1: both new fields are additive, optional JSON -schema properties on an OpenRegister-persisted object, not columns on an -NC-managed table. Deploy is: ship the schema change + code together; -existing Synchronizations are unaffected (`syncMode` absent ⇒ treated as -`full`, byte-identical to current behavior — no code path changes for any -Synchronization that does not explicitly opt into `incremental`). - -## Open Questions -- Should the SPA surface the current `cursorWatermark` value read-only (for - operator visibility/debugging) in addition to the reset action? Deferred - to tasks.md as a small, low-risk addition — not a design decision, since - it changes no backend behavior. -- Should `deleteInvalidObjects()`'s defense-in-depth `syncMode` check log a - warning (mirroring the `fetchComplete === false` branch's warning + - `SynchronizationDeletionGuardedEvent` dispatch) if it is ever actually - reached via a direct caller, or silently return 0? Recommendation: mirror - the existing pattern exactly (warning + event, `reason: - 'incremental_mode'`) for observability parity — captured as a task - acceptance criterion rather than left open at implementation time. diff --git a/openspec/changes/cdc-incremental-sync/proposal.md b/openspec/changes/cdc-incremental-sync/proposal.md index a46328f32..3294be398 100644 --- a/openspec/changes/cdc-incremental-sync/proposal.md +++ b/openspec/changes/cdc-incremental-sync/proposal.md @@ -1,178 +1,38 @@ -# Proposal: cdc-incremental-sync +--- +kind: spec-only +depends_on: [] +--- -## Summary -Add a cursor-based `incremental` sync mode to Integriq's Synchronization -engine, alongside the existing (default, unchanged) `full` hash-diff mode. -When `syncMode: incremental`, an extern→intern run requests only source -records changed since a stored high-watermark cursor (via the engine's -existing Twig request-config templating), advances that watermark only after -a complete, successful fetch, and never runs the source-diff garbage -collection pass (`deleteInvalidObjects()`) — an incremental fetch never sees -the full source set, so absence from one page is not evidence of deletion. -An explicit reset-cursor action clears the watermark and forces the next run -back to a full sync. This closes a competitive gap against Airbyte-style -incremental/CDC sync for large, high-volume sources where full-scan-per-run -is prohibitively expensive. +# Proposal: cdc-incremental-sync (superseded — retired 2026-09-02) -## Motivation -Integriq's current sync model (`SynchronizationService:: -synchronizeExternToIntern()`) always fetches the entire source result set on -every run, computes an order-independent hash per object, and diffs against -stored `SynchronizationContract` hashes to detect changes — a correct but -O(source size) approach on every pass. For large or frequently-polled -sources (e.g. a `nextcloud-table`, TED, or registry-mirror source with tens -of thousands of records) this means every scheduled run re-fetches and -re-hashes records that have not changed since the last run, which is both -slow and — for rate-limited API sources — wasteful of a scarce quota -(`checkRateLimit()`/`rateLimitRemaining` in REQ-002). Airbyte and comparable -integration platforms offer cursor-based incremental sync as a first-class -mode specifically to avoid this. Now is the right time because the -sync-safety-guardrails change (archived 2026-07-14) already added the two -correctness primitives incremental sync must compose with without -regressing: fetch-completeness tracking (REQ-009) and deletion gating -(REQ-010) — this change reuses both rather than inventing parallel ones. +This directory double-counted a change that had already shipped. CDC-style +incremental synchronization was implemented and archived on 2026-07-15 +(`archive/2026-07-15-cdc-incremental-sync`, 16/24 tasks checked with +per-task evidence), yet this live copy was resurrected at 0/24: the +openconnector→integriq rename applied to the prose, the evidence notes +stripped, every box reset. The machinery exists at HEAD: the incremental +sync mode, cursor tracking and full-resync fallback in +`lib/Service/SynchronizationService.php` (23 incremental/CDC references), +the `reset-cursor` handling in +`lib/Controller/SynchronizationsController.php`, and the `syncMode` +configuration keys documented in the register descriptor. -## Affected Projects -- [ ] Project: `integriq` — new `syncMode` field + cursor watermark - field on the Synchronization schema, cursor-filtered fetch path in - `SynchronizationService`, incremental-aware deletion gating, and a - reset-cursor REST action + SPA control. +No live `@spec` tags point into this directory. -## Scope +## Disposition of the original scope -### In Scope -1. `syncMode` on a Synchronization: `full` (current default, unchanged - behavior) | `incremental`. -2. Cursor field configuration: which source field is the cursor (e.g. - `updatedAt`, an id, a page token), a comparator, and the stored - high-watermark value itself, persisted on the Synchronization OR object - (see design.md Decision 1). -3. On an incremental run: inject the stored watermark into the outbound - source request via the engine's existing Twig endpoint-templating - mechanism (extended to `sourceConfig.query` values — design.md Decision - 2), so the source is asked for only records newer than the watermark; - process the returned (delta-only) records through the existing - mapping/write pipeline unchanged; advance the watermark **only** after a - complete, successful fetch (composing with REQ-009's `fetchInfo` — - design.md Decision 3); and **never** invoke `deleteInvalidObjects()` for - an incremental run, regardless of fetch-completeness or deletion ratio — - an incremental fetch is a strict subset of the source, so non-appearance - is not deletion evidence. -4. A reset-cursor action (`POST /api/synchronizations/{id}/reset-cursor`) - that clears the stored watermark so the synchronization's next run - requests an unfiltered (empty-cursor) fetch — full-equivalent for a - source whose templated request treats an absent cursor as "no filter." - This action clears the watermark only; it does **not** change - `syncMode`, and therefore does **not** re-enable `deleteInvalidObjects()` - — that stays hard-disabled for as long as `syncMode` is `incremental` - (see item 3 and design.md Decision 3). Restoring deletion detection - requires explicitly switching `syncMode` back to `full`. -5. Tests: unit coverage for watermark-advances-only-on-complete-fetch, - watermark-does-not-advance-on-incomplete/failed fetch, and - no-deletion-in-incremental-mode; integration coverage for two successive - incremental runs fetching/writing only the delta between them. +| Original scope | Where it went | +| --- | --- | +| Incremental sync mode with change cursor, cursor persistence and reset endpoint, full-resync fallback, deletion-detection interplay with the guardrails, SPA sync-mode fields | **Already shipped and archived**: `archive/2026-07-15-cdc-incremental-sync` (16/24 boxes checked), code at HEAD | +| Residual verification: browser test for the sync-mode fields + reset-cursor action, Newman for `reset-cursor`, feature docs, screenshot | Open, and honestly unticked in the archived twin (no live instance in that pass). Same shape as `approvals-verification-pack`; pick up in a verification pass, not by resurrecting this change | -### Out of Scope -- Log-based CDC (database binlog / WAL tailing) — Integriq has no - DB-source adapter today (`getAllObjectsFromSource()`'s `database` branch - is a documented no-op per the base synchronization-engine spec), so - binlog-based CDC has no source to attach to. Filed as a follow-up once a - DB-source adapter exists. -- Automatic cursor-field discovery/inference from a source's schema — - `cursorField` is admin-configured, same convention as `idPosition` - (REQ-003's `getOriginId()`). -- Per-page/partial watermark checkpointing within a single in-progress run — - the watermark advances once, after the whole run's fetch completes (or not - at all); this is a deliberate consequence of composing with REQ-009's - all-or-nothing fetch-completeness signal, not an oversight. +## Sequencing -## Approach -Add two new fields to the Synchronization OR schema (`syncMode`, -`cursorWatermark`) following the existing convention already used for -`currentPage`/`targetLastSynced` (transient per-pass state stored directly -on the Synchronization object, no separate entity). Branch -`synchronizeExternToIntern()`'s fetch stage on `syncMode`: when -`incremental`, resolve the stored watermark and thread it into the same Twig -context (`{{ cursor }}`) `getAllObjectsFromApi()` already uses for -`{{ data.* }}` endpoint templating, extending that templating to -`sourceConfig.query` values as well (currently endpoint-only). After a -successful, complete fetch, compute the new high-watermark from the fetched -records' configured `cursorField` (a dotted-path extraction mirroring -`getOriginId()`) and persist it onto the Synchronization alongside -`targetLastSynced`. Gate `deleteInvalidObjects()` on `syncMode !== 'incremental'` -at the exact call site that already gates it on `fetchComplete` (REQ-010), -plus a defense-in-depth check inside `deleteInvalidObjects()` itself. Add a -`resetCursor()` controller action mirroring the existing `activate`/ -`deactivate` action pattern on `SynchronizationsController`. +Nothing remains to implement from this change directly. The residual +live-instance verification and docs belong to a verification-pack-style +follow-up. -## New Dependencies -None — reuses the existing Twig (`MappingService::renderTemplateString()`) -templating engine already wired for endpoint substitution; no new package. +## Archival -## Impact -- `lib/Service/SynchronizationService.php`: `synchronizeExternToIntern()`, - `getAllObjectsFromApi()`, `deleteInvalidObjects()`, plus new private - helpers for cursor extraction/persistence. -- `lib/Settings/integriq_register.json`: `synchronization` schema gains - `syncMode` and `cursorWatermark` properties; `sourceConfig`'s free-text - description gains the new recognised keys (`cursorField`, - `cursorComparator`). -- `lib/Controller/SynchronizationsController.php` + `appinfo/routes.php`: - new `resetCursor()` action / route. -- SPA: a "Sync mode" field and "Reset cursor" action on the Synchronization - edit form (src/modals or equivalent — implementation detail for tasks.md). -- `openspec/specs/synchronization-engine/spec.md`: new requirements above - REQ-015 (the current highest numbered requirement in this spec). - -## Cross-Project Dependencies -None — this is entirely internal to Integriq's own sync engine and REST -surface; no other apps-extra project consumes a new API from this change. - -## Risks - -### Risk 1: A misconfigured `cursorField` silently produces a monotonically-wrong watermark -**Severity:** High — **Mitigation:** `cursorField` extraction reuses -`getOriginId()`'s existing dotted-path-lookup-with-throw pattern (REQ-003): -a record missing the configured cursor field throws rather than silently -treating it as the lowest possible cursor value, which would otherwise -cause that record's siblings to be permanently skipped on every subsequent -run. Also require `cursorField` to be an ISO-8601 timestamp or a -lexicographically/numerically comparable value; document the constraint -in the schema description like `idPosition` already is. - -### Risk 2: Deleted-then-recreated source records are invisible to incremental sync -**Severity:** Medium — **Mitigation:** This is an inherent limitation of -cursor-based incremental sync (also true of Airbyte), not something this -change can special-case — document it explicitly in the schema description -and the spec's Notes so admins choose `full` mode for sources where -deletion detection matters, and use the reset-cursor action to periodically -force a full reconciliation pass. - -### Risk 3: Watermark advance and deletion-skip must never regress the sync-safety-guardrails invariants -**Severity:** Medium — **Mitigation:** Both new behaviors are implemented -at the exact call site that already threads REQ-009's `$fetchComplete` -through to REQ-010's deletion gate (`synchronizeExternToIntern()` Stage 5, -`lib/Service/SynchronizationService.php` ~line 1781) rather than as a -parallel code path, so the two concerns cannot drift apart. Unit tests -assert both the incomplete-fetch-blocks-watermark-advance case and the -incremental-mode-blocks-deletion case independently. - -## Rollback Strategy -`syncMode` defaults to `full` on the schema, and every existing -Synchronization object predates this field, so an unset `syncMode` is -treated as `full` — a no-op rollback requires no data migration; reverting -the code change alone restores prior behavior exactly, since no existing -Synchronization can already be in `incremental` mode. If an operator has -already opted synchronizations into `incremental` mode, running the -reset-cursor action (or manually clearing `syncMode`) before rollback avoids -any confusion from a since-orphaned `cursorWatermark` value being read by -older code (which will simply ignore it, as it is an unrecognised field). - -## Open Questions -- Should `cursorComparator` support anything beyond `gt`/`gte` (e.g. a - source-specific opaque page-token comparator that isn't numerically or - lexicographically ordered)? Deferred to design.md Decision 2 — `gt`/`gte` - covers the `updatedAt`-timestamp and monotonic-id cases in scope; a - token-cursor source can still work by treating the token as an opaque - string substituted via `{{ cursor }}` without the engine interpreting its - ordering at all. +This directory is retired in place (not moved or renamed) to keep the diff +reviewable; archive it via the normal flow at the next sweep. diff --git a/openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md b/openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md deleted file mode 100644 index 5426cdca4..000000000 --- a/openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md +++ /dev/null @@ -1,391 +0,0 @@ -# synchronization-engine Specification Delta — cdc-incremental-sync - -This delta adds a cursor-based `incremental` sync mode alongside the -existing (default, unchanged) `full` hash-diff mode. REQ numbers continue -from the highest currently claimed by any capability in this spec — -`hitl-approval-rule-action` claims REQ-015, the highest in use on `main` at -the time this change was authored — so this change takes REQ-016..REQ-019. - -## ADDED Requirements - -### Requirement: Incremental sync mode selects a cursor-filtered fetch request (REQ-016) - -`SynchronizationService` SHALL support `syncMode: incremental` on a -Synchronization, in addition to the existing (default, unchanged) `full` -mode. When `syncMode` is `incremental` and `sourceType` is `api`, -`getAllObjectsFromApi()` MUST make the Synchronization's stored -`cursorWatermark` — or an empty string when no watermark has been set yet — -available as a `cursor` key in the Twig context already passed to -`MappingService::renderTemplateString()` when rendering `sourceConfig.endpoint` -(alongside the existing `data` key). `getAllObjectsFromApi()` MUST also -apply the identical `{{`/`}}`-presence-detection-then-`renderTemplateString()` -treatment already used for `sourceConfig.endpoint` to each scalar value in -`sourceConfig.query`, using the same `['data' => ..., 'cursor' => ...]` -context, so a source that takes its cursor as a query parameter rather than -an endpoint path segment can also reference `{{ cursor }}`. A Synchronization -with `syncMode` absent or `full` MUST take the exact pre-existing code path — -no `cursor` context key is added and `sourceConfig.query` values are passed -through unrendered, unchanged from current behavior. - -#### Scenario: an incremental run injects the stored watermark into a templated endpoint - -- GIVEN a Synchronization with `syncMode: incremental`, `sourceConfig.endpoint: - ".../items?updatedAfter={{ cursor }}"`, and a stored `cursorWatermark` of - `"2026-07-01T00:00:00Z"` -- WHEN `getAllObjectsFromApi()` runs -- THEN the rendered request endpoint is - `.../items?updatedAfter=2026-07-01T00:00:00Z` - -#### Scenario: an incremental run injects the stored watermark into a templated query parameter - -- GIVEN a Synchronization with `syncMode: incremental`, - `sourceConfig.query.updatedAfter: "{{ cursor }}"`, and a stored - `cursorWatermark` of `"42"` -- WHEN `getAllObjectsFromApi()` runs -- THEN the outbound request's `updatedAfter` query parameter is rendered to - `"42"` before the call is made - -#### Scenario: an incremental run with no prior watermark passes an empty cursor - -- GIVEN a Synchronization with `syncMode: incremental` and no - `cursorWatermark` set (its first-ever incremental run) -- WHEN `getAllObjectsFromApi()` runs -- THEN `{{ cursor }}` renders to an empty string -- AND a source whose default/fallback (e.g. `{{ cursor|default('1970-01-01') - }}`, a plain Twig filter requiring no engine change) treats an empty - cursor as "everything" receives an effectively full fetch on this first - incremental run - -#### Scenario: a full-mode run is unaffected - -- GIVEN a Synchronization with `syncMode` absent or `full` -- WHEN `getAllObjectsFromApi()` runs -- THEN the Twig context passed to `sourceConfig.endpoint` templating - contains only `data` (no `cursor` key), and `sourceConfig.query` values - are used exactly as configured, byte-identical to pre-existing behavior - -**Notes:** - -- This requirement extends the fetch-request-shaping mechanics of REQ-002 - (source object fetching and pagination) for the `api` branch only; it does - not change REQ-002's pagination, rate-limiting, or next-page resolution - behavior. -- `sourceConfig.cursorField` (a dotted-path lookup mirroring REQ-003's - `idPosition`/`getOriginId()` convention) identifies which field of a - fetched record is the comparable cursor value; it is read by REQ-017's - watermark computation, not by this requirement. -- Methods: `getAllObjectsFromApi()` (extended), `MappingService:: - renderTemplateString()` (reused, unchanged). - -### Requirement: Cursor watermark advances only after a complete, successful fetch (REQ-017) - -`synchronizeExternToIntern()` MUST, for a Synchronization with `syncMode: -incremental`, compute a new high-watermark value from the fetched records' -configured `sourceConfig.cursorField` and persist it as the Synchronization's -`cursorWatermark` **only when** that run's fetch was marked complete per -REQ-009 (`fetchInfo.complete === true`) **and** no `TooManyRequestsHttpException` -was thrown during the fetch — the same `$fetchComplete` computation REQ-010 -already performs at the same point in the method. When the fetch was -incomplete for any reason (partial pagination, a failed page, a rate-limit -response, or the pagination safety cap), the system MUST NOT persist any -change to `cursorWatermark`, so the next run retries from the same -watermark rather than silently skipping the unfetched remainder. A run -invoked with `isTest: true` MUST NOT persist a watermark change regardless -of fetch completeness, consistent with REQ-011 (test runs make no writes). -A record whose configured `cursorField` resolves to `null` MUST cause the -run to throw, mirroring REQ-003's `getOriginId()` behavior for a missing -`idPosition` — silently computing a watermark from an incomplete field -would risk producing an incorrect (too-low) high-watermark that -permanently skips sibling records on every subsequent run. - -#### Scenario: watermark advances after a complete fetch - -- GIVEN a Synchronization with `syncMode: incremental`, - `sourceConfig.cursorField: "updatedAt"`, and a fetch that completes - successfully, returning records with `updatedAt` values up to - `"2026-07-15T09:00:00Z"` -- WHEN `synchronizeExternToIntern()` finishes the run -- THEN the Synchronization's `cursorWatermark` is persisted as - `"2026-07-15T09:00:00Z"` - -#### Scenario: watermark does not advance after a page failure mid-fetch - -- GIVEN a Synchronization with `syncMode: incremental` and an existing - `cursorWatermark` of `"2026-07-01T00:00:00Z"` -- WHEN a run's fetch is marked incomplete (REQ-009) because page 2 of 3 - returned HTTP 500 -- THEN the Synchronization's `cursorWatermark` remains - `"2026-07-01T00:00:00Z"` after the run, unchanged -- AND the next run requests records with `cursor` still resolving to - `"2026-07-01T00:00:00Z"` - -#### Scenario: watermark does not advance after a 429 rate-limit - -- GIVEN a Synchronization with `syncMode: incremental` -- WHEN the source returns HTTP 429 on the first page of a run -- THEN the run's fetch is treated as incomplete (REQ-009) -- AND the Synchronization's `cursorWatermark` is not modified -- AND the caller still receives the `TooManyRequestsHttpException` as before - (REQ-010's existing behavior for the deletion side is unchanged; this - requirement adds the equivalent guarantee for the watermark side) - -#### Scenario: watermark does not advance for a test run even when the fetch is complete - -- GIVEN a Synchronization with `syncMode: incremental` -- WHEN `POST .../synchronizations/{id}/test` runs and its fetch completes - successfully -- THEN the Synchronization's persisted `cursorWatermark` is unchanged - (REQ-011: test runs persist no Synchronization state) - -#### Scenario: a record missing the configured cursorField throws rather than silently computing a wrong watermark - -- GIVEN a Synchronization with `syncMode: incremental` and - `sourceConfig.cursorField: "updatedAt"` -- WHEN a fetched record has no value at the `updatedAt` path -- THEN the run throws an `Exception` naming the missing cursor field -- AND no `cursorWatermark` change is persisted for that run - -**Notes:** - -- This requirement composes directly with REQ-009/REQ-010's existing - `$fetchComplete` computation in `synchronizeExternToIntern()` — it does - not introduce a second completeness signal. -- Watermark computation takes the maximum `cursorField` value across all - fetched records in the run (not the last record processed), so - out-of-order pagination or concurrent per-page fetching (REQ-002's - optimized parallel mode) cannot regress the watermark. -- Methods added: `computeCursorWatermark()` (private, alongside the - existing `getOriginId()`/`hashObject()` identity helpers). - -### Requirement: Deletion garbage-collection never runs for an incremental sync (REQ-018) - -`synchronizeExternToIntern()` MUST NOT invoke `deleteInvalidObjects()` for -any run whose Synchronization has `syncMode: incremental` — unconditionally, -regardless of that run's fetch-completeness (REQ-009), the computed -deletion ratio (REQ-010), or an explicit `forceDeletion` override. An -incremental fetch is, by construction, a filtered subset of the source; the -absence of a target id from `$synchronizedTargetIds` on an incremental run -is not evidence that the corresponding source record was deleted — it may -simply be outside the cursor filter. `deleteInvalidObjects()` MUST also -independently refuse to run when passed a Synchronization whose `syncMode` -is `incremental`, regardless of caller, so a future caller that invokes it -directly (bypassing `synchronizeExternToIntern()`'s gate) cannot -accidentally delete against a partial incremental fetch. On this refusal, -`deleteInvalidObjects()` MUST log a warning-level message and dispatch a -`SynchronizationDeletionGuardedEvent` with `reason: incremental_mode`, -mirroring its existing `fetch_incomplete`-reason guard (REQ-010), and MUST -return `0`. - -#### Scenario: incremental mode blocks deletion even on a complete fetch - -- GIVEN a Synchronization with `syncMode: incremental`, 100 existing - contracts, and a run whose fetch completes successfully but — because it - is cursor-filtered — returns only 5 changed records -- WHEN `synchronizeExternToIntern()` reaches its cleanup stage -- THEN `deleteInvalidObjects()` is not invoked -- AND 0 objects are deleted -- AND the run's `result.objects.deletionGuard` records - `reason: incremental_mode` - -#### Scenario: forceDeletion cannot override the incremental-mode block - -- GIVEN the same Synchronization as above -- WHEN the run is invoked with `forceDeletion: true` -- THEN deletion is still not invoked — `forceDeletion` only overrides - REQ-010's ratio guard on a `full`-mode run and has no effect on this - unconditional incremental-mode block - -#### Scenario: deleteInvalidObjects() called directly against an incremental Synchronization still refuses - -- GIVEN a Synchronization with `syncMode: incremental` -- WHEN `deleteInvalidObjects()` is invoked directly (not via - `synchronizeExternToIntern()`) with `fetchComplete: true` and - `forceDeletion: true` -- THEN it still returns `0` and deletes nothing -- AND a warning is logged and a `SynchronizationDeletionGuardedEvent` with - `reason: incremental_mode` is dispatched - -#### Scenario: the deleteRestriction single-object delete path is unaffected - -- GIVEN an OpenRegister `ObjectDeletedEvent` triggers a synchronization run - with `mutationType: delete` and `sourceConfig.restrictDeletion: true` - against a Synchronization with `syncMode: incremental` -- WHEN `synchronizeExternToIntern()` runs -- THEN the single-object delete path (`$data !== null && $mutationType === - 'delete'`) is taken — this path never calls `deleteInvalidObjects()`'s - bulk source-diff branch regardless of `syncMode`, so this requirement - introduces no new behavior here; it is called out only to confirm the - event-driven single-object delete is not accidentally caught by this - guard - -**Notes:** - -- This requirement composes with, and takes priority over, REQ-010's - ratio/`forceDeletion` guard: the `syncMode` check happens first and - unconditionally, before REQ-010's `fetchComplete`/ratio logic is ever - reached, for an incremental Synchronization. -- Restoring deletion-based garbage collection for a Synchronization - currently in `incremental` mode requires explicitly switching its - `syncMode` back to `full` — REQ-019's reset-cursor action does not do - this (see REQ-019). -- Methods: `deleteInvalidObjects()` (extended with the new guard clause, - ahead of its existing `fetchComplete === false` early return), - `synchronizeExternToIntern()` (extended call-site check). - -### Requirement: Reset-cursor action clears the stored watermark (REQ-019) - -`SynchronizationsController` MUST expose `POST -/api/synchronizations/{id}/reset-cursor`, which clears the target -Synchronization's `cursorWatermark` (to `null`/absent) and persists that -change, without altering `syncMode` or any other Synchronization field. -This action MUST NOT itself delete, create, or update any target object or -`SynchronizationContract` — it only clears stored cursor state. Following a -reset, the Synchronization's next run resolves `{{ cursor }}` to an empty -string (REQ-016's "no prior watermark" case), which — for a source whose -templated request treats an absent cursor as unfiltered — yields a -full-equivalent fetch that re-evaluates every currently-reachable source -record for create/update via the existing hash-diff contract mechanism -(REQ-003). This action MUST NOT re-enable `deleteInvalidObjects()` for that -Synchronization: REQ-018's guard is keyed on `syncMode`, not on cursor -state, and a reset-cursor call does not change `syncMode`. - -#### Scenario: reset-cursor clears the watermark - -- GIVEN a Synchronization with `syncMode: incremental` and - `cursorWatermark: "2026-07-10T00:00:00Z"` -- WHEN `POST /api/synchronizations/{id}/reset-cursor` is called -- THEN the Synchronization's `cursorWatermark` is persisted as - `null`/absent -- AND `syncMode` remains `incremental`, unchanged - -#### Scenario: the next run after a reset requests an unfiltered fetch - -- GIVEN a Synchronization whose `cursorWatermark` was just cleared via - reset-cursor, with `sourceConfig.endpoint: ".../items?updatedAfter={{ - cursor }}"` -- WHEN the next `synchronize()` run's `getAllObjectsFromApi()` executes -- THEN the rendered endpoint is `.../items?updatedAfter=` (empty cursor - value) - -#### Scenario: reset-cursor does not perform or re-enable deletion - -- GIVEN a Synchronization with `syncMode: incremental` and 100 existing - contracts -- WHEN `POST /api/synchronizations/{id}/reset-cursor` is called, and then - the Synchronization's next run executes and — because the source - honored the empty cursor — refetches all currently-existing source - records -- THEN `reset-cursor` itself deletes nothing -- AND the subsequent run also does not invoke `deleteInvalidObjects()` - (REQ-018 still applies — `syncMode` is still `incremental`) -- AND restoring deletion detection requires a separate, explicit change of - `syncMode` to `full` - -#### Scenario: a missing synchronization id returns 404 - -- GIVEN no Synchronization exists with the given `id` -- WHEN `POST /api/synchronizations/{id}/reset-cursor` is called -- THEN the response is `404`, mirroring the existing `run()`/`test()` - action's not-found handling - -**Notes:** - -- This action follows the existing `activate`/`deactivate`/`run`/`test` - action-route convention on `SynchronizationsController` - (`/api/synchronizations/{id}/`, `POST`). -- **SECURITY:** per REQ-005's existing, pre-existing IDOR note on this - controller, `reset-cursor` inherits the same `@NoAdminRequired` + - `@NoCSRFRequired` + no-per-object-ownership-guard posture as every other - action on `SynchronizationsController` today. This is observed, - pre-existing behavior this change does not alter or worsen (clearing a - watermark is a low-severity action relative to `run`/`test`/`execute` - already available on the same unguarded surface) — flagged for the same - future authorization follow-up already noted under REQ-005, not - addressed here. -- Methods added: `SynchronizationsController::resetCursor()`. - -## MODIFIED Requirements - -### Requirement: Target write, deduplication and file handling (REQ-004) - -The system SHALL write each transformed object to its target, branching to an -OpenRegister-specific write when the target is an OR register/schema, and SHALL -maintain one `SynchronizationContract` per object carrying origin/target ids and -hashes for incremental change detection. The system SHALL cascade contract -creation and id rewrites to sub-objects. It SHALL garbage-collect target objects -no longer present in the source (`deleteInvalidObjects()`) unless `force` opts -out, **and unless the run's fetch was incomplete, the run is a test -(`isTest: true`), the computed deletion ratio exceeds the configured -guard threshold without an explicit `forceDeletion` override (REQ-009, -REQ-010, REQ-011), or the Synchronization's `syncMode` is `incremental` -(REQ-018 — this last guard is unconditional and is never bypassed by -`forceDeletion`)**. The system SHALL fetch, persist, and clean up files -referenced by sync objects: download a file via `CallService`, validate the -target object id is a UUID, persist to storage, optionally run async batch -fetching (ReactPHP), and remove orphaned files/attachments no longer -referenced after a sync. - - - -@e2e exclude backend target-write internals — covered by PHPUnit/Newman, not browser UI - -#### Scenario: OR target write records a contract - -- **GIVEN** a transformed object whose target is an OR register/schema -- **WHEN** `updateTarget()` runs -- **THEN** it delegates to `updateTargetOpenRegister()` and a `SynchronizationContract` records the resulting origin/target ids and hashes. - -#### Scenario: absent source objects are garbage-collected when the fetch was complete, within the deletion-ratio guard, and syncMode is full - -- **GIVEN** a source no longer returns objects that previously had contracts, a complete fetch (REQ-009), a non-test run, a deletion ratio within the configured threshold (REQ-010), and `syncMode` absent or `full` (REQ-018) -- **WHEN** `deleteInvalidObjects()` runs -- **THEN** the now-absent target objects are deleted (garbage-collected). - -#### Scenario: referenced file is fetched and persisted - -- **GIVEN** a sync object referencing a file URL -- **WHEN** `fetchFile()` runs -- **THEN** the file is downloaded via `CallService`, the object id is validated as a UUID before write, and the file is persisted to storage; a null response throws an `Exception`. - -#### Scenario: batch file fetch with cleanup - -- **GIVEN** a batch of file references -- **WHEN** `startAsyncFileFetching()` / `executeAsyncFileFetching()` / `processMultipleFilesWithCleanup()` run -- **THEN** files are fetched concurrently and orphaned files are cleaned up afterward via `cleanupOrphanedFiles()`. - -#### Scenario: unreferenced attachments are removed - -- **GIVEN** a previously-synced object whose attachments are no longer referenced -- **WHEN** `cleanupFilesFromAttachments()` runs -- **THEN** the stale attachments are removed from the object. - -**Notes:** - -- `fetchFile()` builds the request endpoint from source-supplied - `location`/`sourceConfiguration` and substitutes `{{ originId }}` into a - JSON-encoded config. The endpoint is attacker-influenceable via source - configuration; combined with `base64_decode` of the response body this is a - surface worth a dedicated SSRF/content-handling review (flagged, not changed). -- `fetchFileSafely()` wraps `fetchFile()` and swallows exceptions so an async - batch continues past individual file failures — a silent-fail path; failed - fetches are not surfaced to the caller as a structured error. -- `updateTargetOpenRegister()` is the only fully-wired target-write branch; - non-OR targets are handled generically by `writeObjectToTarget()`. -- **See REQ-009/REQ-010/REQ-011/REQ-012/REQ-013 (sync-safety-guardrails) for - the deletion-gating, test-run no-write, ad-hoc Source, and duplicate-contract - detection behaviour layered onto this requirement, and REQ-016/REQ-017/ - REQ-018/REQ-019 (cdc-incremental-sync) for the incremental-mode fetch - filtering, watermark, and unconditional deletion-block layered on top of - those.** -- Methods: `updateTarget()`, `updateTargetOpenRegister()`, - `writeObjectToTarget()`, `deleteInvalidObjects()`, `processSyncContract()`, - `updateContractsForSubObjects()`, `processSynchronizationObject()`, - `writeFile()`, `fetchFile()`, `fetchFileSafely()`, `startAsyncFileFetching()`, - `executeAsyncFileFetching()`, `processMultipleFilesWithCleanup()`, - `cleanupOrphanedFiles()`, `cleanupFilesFromAttachments()`, - `shouldPublishFile()`, `getFileContext()`, `getFilenameFromHeaders()`, - `synchronizeToTarget()`, `detectDuplicateContracts()`. diff --git a/openspec/changes/cdc-incremental-sync/tasks.md b/openspec/changes/cdc-incremental-sync/tasks.md index 0d8da13c1..daf320f97 100644 --- a/openspec/changes/cdc-incremental-sync/tasks.md +++ b/openspec/changes/cdc-incremental-sync/tasks.md @@ -1,151 +1,9 @@ -# Tasks: cdc-incremental-sync - -## Implementation Tasks - -### Task 1: Add `syncMode` and `cursorWatermark` fields to the Synchronization schema -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **files**: `lib/Settings/integriq_register.json` -- **acceptance_criteria**: - - GIVEN the `synchronization` schema WHEN it is inspected THEN it has a - `syncMode` string property (documented values `full`|`incremental`, - default `full`) and a `cursorWatermark` string property, following the - existing `currentPage`/`targetLastSynced` documentation style - - GIVEN the `sourceConfig` property's description THEN it documents the - new recognised keys `cursorField` and `cursorComparator`, alongside the - existing `deletionRatioThreshold`/`resultsPosition`/etc. documentation - - GIVEN an existing Synchronization object with no `syncMode` set WHEN it - is read THEN the application treats it as `full` (no migration/backfill - needed) -- [ ] Implement -- [ ] Test - -### Task 2: Extend Twig request-config templating with a `cursor` context key (REQ-016) -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **files**: `lib/Service/SynchronizationService.php` (`getAllObjectsFromApi()`) -- **acceptance_criteria**: - - GIVEN a Synchronization with `syncMode: incremental` and a templated - `sourceConfig.endpoint` referencing `{{ cursor }}` WHEN - `getAllObjectsFromApi()` runs THEN the rendered endpoint contains the - stored `cursorWatermark` value (or an empty string when unset) - - GIVEN the same Synchronization with a templated `sourceConfig.query` - value referencing `{{ cursor }}` WHEN the fetch runs THEN that query - value is rendered the same way endpoint values already are - - GIVEN a Synchronization with `syncMode` absent or `full` WHEN the fetch - runs THEN the Twig context has no `cursor` key and `sourceConfig.query` - values are passed through unrendered — byte-identical to current - behavior (regression check) -- [ ] Implement -- [ ] Test - -### Task 3: Compute and persist the cursor watermark, gated on fetch-completeness (REQ-017) -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **files**: `lib/Service/SynchronizationService.php` (new private - `computeCursorWatermark()`; `synchronizeExternToIntern()` Stage 5/end-of-run) -- **acceptance_criteria**: - - GIVEN an incremental run whose fetch completes (REQ-009 - `fetchInfo.complete === true`) WHEN the run finishes THEN - `cursorWatermark` is persisted as the maximum `sourceConfig.cursorField` - value seen across the fetched records - - GIVEN an incremental run whose fetch is marked incomplete (page - failure, rate-limit, or safety-cap per REQ-009) WHEN the run finishes - THEN `cursorWatermark` is left unchanged - - GIVEN an incremental `isTest: true` run whose fetch completes WHEN the - run finishes THEN `cursorWatermark` is left unchanged (REQ-011 parity) - - GIVEN a fetched record whose configured `cursorField` resolves to - `null` WHEN the run processes it THEN an `Exception` is thrown naming - the missing field, and no partial/incorrect watermark is persisted -- [ ] Implement -- [ ] Test - -### Task 4: Hard-block `deleteInvalidObjects()` for incremental synchronizations (REQ-018) -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-deletion-garbage-collection-never-runs-for-an-incremental-sync-req-018` -- **files**: `lib/Service/SynchronizationService.php` - (`synchronizeExternToIntern()` Stage 5 call site; `deleteInvalidObjects()`) -- **acceptance_criteria**: - - GIVEN a Synchronization with `syncMode: incremental` WHEN - `synchronizeExternToIntern()` reaches its cleanup stage THEN - `deleteInvalidObjects()` is never invoked, and `result.objects. - deletionGuard.reason` is `incremental_mode` - - GIVEN the same Synchronization and `forceDeletion: true` WHEN the run - executes THEN deletion is still blocked (unconditional — `forceDeletion` - has no effect on this guard) - - GIVEN `deleteInvalidObjects()` is invoked directly (bypassing - `synchronizeExternToIntern()`) against a Synchronization with `syncMode: - incremental` WHEN it runs THEN it returns `0`, logs a warning, and - dispatches `SynchronizationDeletionGuardedEvent` with `reason: - incremental_mode` - - GIVEN the event-driven single-object `deleteRestriction` path (REQ-010) - on an incremental Synchronization WHEN an `ObjectDeletedEvent` fires - THEN the single-object delete still runs unaffected (regression check — - this path never calls the bulk-diff branch this task guards) -- [ ] Implement -- [ ] Test - -### Task 5: Reset-cursor controller action and route (REQ-019) -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **files**: `lib/Controller/SynchronizationsController.php`, - `appinfo/routes.php` -- **acceptance_criteria**: - - GIVEN a Synchronization with a stored `cursorWatermark` WHEN `POST - /api/synchronizations/{id}/reset-cursor` is called THEN the watermark - is persisted as cleared and `syncMode` is unchanged - - GIVEN no Synchronization exists with the given id WHEN the action is - called THEN it responds `404`, matching `run()`/`test()`'s existing - not-found handling - - GIVEN a successful reset WHEN the response is inspected THEN it - reflects the cleared watermark (for SPA confirmation feedback) -- [ ] Implement -- [ ] Test - -### Task 6: Synchronization SPA — sync mode field + reset-cursor action -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **files**: Synchronization edit form component (e.g. - `src/modals/Synchronization/EditSynchronization.vue` or equivalent — match - existing modal location), Synchronization detail/actions view -- **acceptance_criteria**: - - GIVEN the Synchronization edit form WHEN an admin opens it THEN a "Sync - mode" field (full / incremental) and, when incremental, a "Cursor - field" + "Cursor comparator" configuration are shown - - GIVEN an incremental Synchronization's detail/actions view WHEN an - admin opens it THEN a "Reset cursor" action button is available, - labelled/tooltipped to make clear it clears the watermark only and does - **not** delete data or restore deletion detection (design.md Decision - 3 / Risks) - - GIVEN the "Reset cursor" action WHEN clicked THEN it calls `POST - .../reset-cursor` and shows a confirmation -- [ ] Implement -- [ ] Test - -## Verification -- [ ] All tasks checked off -- [ ] `openspec validate` passes -- [ ] Manual testing against acceptance criteria -- [ ] Code review against spec requirements - -## Tests (company-wide ADR-009) - -- [ ] PHPUnit unit tests for new/changed business logic (`tests/Unit/`) — - watermark advance/no-advance (Task 3), incremental deletion block - (Task 4), cursor templating (Task 2) -- [ ] Newman/Postman tests for new/changed API endpoints — `reset-cursor` - (Task 5) -- [ ] Browser tests (Playwright MCP) for UI changes — sync mode field + - reset-cursor action (Task 6) -- [ ] Integration test: two successive incremental runs against a - synthetic paginated source fetch/write only the delta between them - (proposal.md Scope item 5) -- [ ] All tests pass (`composer test`, `newman run`) - -## Documentation (company-wide ADR-010) - -- [ ] Feature documentation updated in `docs/` — incremental sync mode, - cursor field configuration, reset-cursor action, and the explicit - "incremental mode never deletes" caveat -- [ ] Screenshot captured and committed to `docs/images/` — Synchronization - edit form's new Sync mode field - -## i18n (company-wide hydra ADR-007) - -- [ ] Dutch (`nl_NL`) and English (`en_US`) translation strings added for: - "Sync mode", "Cursor field", "Cursor comparator", "Reset cursor" action - label + confirmation + tooltip text +# Tasks: cdc-incremental-sync (superseded) + +The original 24-checkbox list was removed with the 2026-09-02 retirement +(see proposal.md for the disposition; the list survives in +`archive/2026-07-15-cdc-incremental-sync/tasks.md`, where 16/24 boxes are +checked with per-task evidence, and in git history). The residual +live-instance verification and docs work is listed there with per-box +reasons and belongs to a verification-pack-style follow-up. There is +nothing to implement from this change directly. diff --git a/openspec/changes/cdc-incremental-sync/test-plan.md b/openspec/changes/cdc-incremental-sync/test-plan.md deleted file mode 100644 index a5fdc4941..000000000 --- a/openspec/changes/cdc-incremental-sync/test-plan.md +++ /dev/null @@ -1,224 +0,0 @@ -# Test Plan: cdc-incremental-sync - -## Test Cases - -### TC-1: incremental run injects stored watermark into a templated endpoint -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **type**: api -- **persona**: N/A (backend engine behavior) -- **preconditions**: Synchronization with `syncMode: incremental`, - `sourceConfig.endpoint: ".../items?updatedAfter={{ cursor }}"`, - `cursorWatermark: "2026-07-01T00:00:00Z"` -- **steps**: trigger a run (`POST /api/synchronizations/{id}/run`) against - a mocked/stub source that echoes the requested URL -- **expected result**: the source receives a request to - `.../items?updatedAfter=2026-07-01T00:00:00Z` -- **test command**: `/test-api` (PHPUnit unit test on - `getAllObjectsFromApi()` is the primary coverage; Newman covers the - outer `run` endpoint contract) - -### TC-2: incremental run injects stored watermark into a templated query parameter -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **type**: api -- **preconditions**: Synchronization with `syncMode: incremental`, - `sourceConfig.query.updatedAfter: "{{ cursor }}"`, `cursorWatermark: "42"` -- **steps**: trigger a run against a mocked source capturing outbound query - parameters -- **expected result**: outbound `updatedAfter` query parameter equals `"42"` -- **test command**: `/test-api` - -### TC-3: full-mode run is unaffected by the cursor templating extension -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **type**: regression -- **preconditions**: Synchronization with `syncMode` absent (pre-existing - fixture, unmodified) -- **steps**: trigger a run -- **expected result**: request endpoint/query are byte-identical to - pre-change behavior; no `cursor` context key present -- **test command**: `/test-regression` - -### TC-4: watermark advances after a complete fetch -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental`, - `sourceConfig.cursorField: "updatedAt"`, source returns records with - `updatedAt` up to `2026-07-15T09:00:00Z`, fetch completes normally -- **steps**: trigger a run; inspect persisted Synchronization afterward -- **expected result**: `cursorWatermark === "2026-07-15T09:00:00Z"` -- **test command**: `/test-functional` - -### TC-5: watermark does not advance after a page failure mid-fetch -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental` and - existing `cursorWatermark: "2026-07-01T00:00:00Z"`; mocked source returns - HTTP 500 on page 2 of 3 -- **steps**: trigger a run -- **expected result**: fetch marked incomplete (REQ-009); `cursorWatermark` - unchanged at `"2026-07-01T00:00:00Z"` after the run -- **test command**: `/test-functional` - -### TC-6: watermark does not advance after a 429 rate-limit -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental`; source - returns HTTP 429 on first page -- **steps**: trigger a run -- **expected result**: `TooManyRequestsHttpException` (429) thrown to - caller; `cursorWatermark` unchanged -- **test command**: `/test-functional` - -### TC-7: watermark does not advance for a test run -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental` -- **steps**: `POST .../synchronizations/{id}/test`, fetch completes - successfully -- **expected result**: `cursorWatermark` unchanged (REQ-011 parity) -- **test command**: `/test-functional` - -### TC-8: missing cursorField throws rather than computing a wrong watermark -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-cursor-watermark-advances-only-after-a-complete-successful-fetch-req-017` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental`, - `sourceConfig.cursorField: "updatedAt"`; one fetched record has no - `updatedAt` value -- **steps**: trigger a run -- **expected result**: `Exception` thrown naming the missing field; no - `cursorWatermark` change persisted -- **test command**: `/test-functional` - -### TC-9: incremental mode blocks deletion even on a complete fetch -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-deletion-garbage-collection-never-runs-for-an-incremental-sync-req-018` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental`, 100 - existing contracts; a complete incremental fetch returns 5 changed - records (cursor-filtered, so 95 are absent from this run by design) -- **steps**: trigger a run -- **expected result**: `deleteInvalidObjects()` not invoked; 0 objects - deleted; `result.objects.deletionGuard.reason === "incremental_mode"` -- **test command**: `/test-functional` - -### TC-10: forceDeletion cannot override the incremental-mode block -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-deletion-garbage-collection-never-runs-for-an-incremental-sync-req-018` -- **type**: functional -- **preconditions**: same as TC-9 -- **steps**: trigger a run with `forceDeletion: true` -- **expected result**: deletion still blocked; 0 objects deleted -- **test command**: `/test-functional` - -### TC-11: deleteInvalidObjects() called directly still refuses on incremental -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-deletion-garbage-collection-never-runs-for-an-incremental-sync-req-018` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental` -- **steps**: call `deleteInvalidObjects()` directly with - `fetchComplete: true, forceDeletion: true` -- **expected result**: returns `0`; warning logged; - `SynchronizationDeletionGuardedEvent` dispatched with - `reason: incremental_mode` -- **test command**: `/test-functional` (PHPUnit-level; exercised via a - direct service-layer test, not browser) - -### TC-12: event-driven single-object delete path unaffected on incremental -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-deletion-garbage-collection-never-runs-for-an-incremental-sync-req-018` -- **type**: regression -- **preconditions**: Synchronization with `syncMode: incremental`, - `sourceConfig.restrictDeletion: true` -- **steps**: fire an OpenRegister `ObjectDeletedEvent` for a synced object -- **expected result**: the single matching target object is deleted, - unaffected by REQ-018 (this path never reaches the bulk-diff branch) -- **test command**: `/test-regression` - -### TC-13: reset-cursor clears the watermark without touching syncMode -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **type**: api -- **preconditions**: Synchronization with `syncMode: incremental`, - `cursorWatermark: "2026-07-10T00:00:00Z"` -- **steps**: `POST /api/synchronizations/{id}/reset-cursor` -- **expected result**: `200`; persisted `cursorWatermark` is null/absent; - `syncMode` still `incremental` -- **test command**: `/test-api` - -### TC-14: next run after reset requests an unfiltered fetch -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **type**: functional -- **preconditions**: watermark just cleared via TC-13, - `sourceConfig.endpoint: ".../items?updatedAfter={{ cursor }}"` -- **steps**: trigger the next run -- **expected result**: rendered endpoint is `.../items?updatedAfter=` - (empty cursor) -- **test command**: `/test-functional` - -### TC-15: reset-cursor does not perform or re-enable deletion -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental`, 100 - existing contracts -- **steps**: `POST .../reset-cursor`, then trigger the next run (source - honors empty cursor and returns its full set) -- **expected result**: reset-cursor itself deletes nothing; the subsequent - run also does not invoke `deleteInvalidObjects()` (REQ-018 still applies) -- **test command**: `/test-functional` - -### TC-16: reset-cursor against a missing synchronization returns 404 -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **type**: api -- **preconditions**: no Synchronization with the given id -- **steps**: `POST /api/synchronizations/{bogus-id}/reset-cursor` -- **expected result**: `404` -- **test command**: `/test-api` - -### TC-17: two successive incremental runs fetch/write only the delta (integration) -- **spec_ref**: proposal.md Scope item 5 / `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **type**: functional -- **preconditions**: Synchronization with `syncMode: incremental` against a - synthetic paginated source with a mutable dataset -- **steps**: run 1 fetches/writes the full initial dataset and advances the - watermark; mutate the source (add N new/changed records with newer - `updatedAt`); run 2 executes -- **expected result**: run 2's fetch request is cursor-filtered to the new - watermark; only the N new/changed records are fetched and written; - contracts for the unrelated, unchanged records are untouched -- **test command**: `/test-functional` - -### TC-18: sync mode field and reset-cursor action render in the SPA -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-incremental-sync-mode-selects-a-cursor-filtered-fetch-request-req-016` -- **type**: functional -- **preconditions**: authenticated admin on the Synchronizations page -- **steps**: open the Synchronization edit form; select `incremental`; - configure a cursor field; save; open the detail/actions view -- **expected result**: the "Sync mode"/"Cursor field" fields persist - correctly; a "Reset cursor" action is visible with clarifying help text - for an incremental Synchronization -- **test command**: `/test-functional` - -### TC-19: reset-cursor action is reachable and labeled clearly for a non-technical operator -- **spec_ref**: `openspec/changes/cdc-incremental-sync/specs/synchronization-engine/spec.md#requirement-reset-cursor-action-clears-the-stored-watermark-req-019` -- **type**: persona -- **persona**: Noor Yilmaz (Municipal CISO / Functional Admin) — needs to - understand that reset-cursor does not delete data and does not restore - deletion detection, per design.md's explicit caveat -- **preconditions**: incremental Synchronization configured -- **steps**: locate and read the reset-cursor action's tooltip/help text -- **expected result**: the text makes clear (a) only the cursor is cleared, - (b) no data is deleted by this action, and (c) deletion detection stays - off until `syncMode` is explicitly switched to `full` -- **test command**: `/test-persona-noor` - -## Coverage Summary - -| Requirement | Covered by | -|---|---| -| REQ-016 (cursor-filtered fetch request) | TC-1, TC-2, TC-3, TC-18 | -| REQ-017 (watermark advance gating) | TC-4, TC-5, TC-6, TC-7, TC-8 | -| REQ-018 (deletion hard-blocked in incremental mode) | TC-9, TC-10, TC-11, TC-12 | -| REQ-019 (reset-cursor action) | TC-13, TC-14, TC-15, TC-16, TC-19 | -| REQ-004 (MODIFIED — deletion gate composition) | TC-9, TC-10, TC-12, TC-15 | -| Integration (two-run delta-only behavior) | TC-17 | - -## Out of Scope -- Log-based CDC / binlog tailing — no DB-source adapter exists to test - against (proposal.md Out of Scope); no test cases written. -- Automatic cursor-field inference — not implemented, nothing to test. -- Sub-run (per-page) watermark checkpointing — deliberately not - implemented (design.md Non-Goals); no test cases written. diff --git a/openspec/changes/connector-catalog-ui/context-brief.md b/openspec/changes/connector-catalog-ui/context-brief.md deleted file mode 100644 index c37bdc202..000000000 --- a/openspec/changes/connector-catalog-ui/context-brief.md +++ /dev/null @@ -1,24 +0,0 @@ -# Context Brief: connector-catalog-ui -Source: Specter deep-research 2026-07-14 (insights #1256, #1265). VERIFY every code claim against HEAD before writing artifacts. - -## Problem -Discovery and day-2 ops are API-only. Seeded connectors (PDOK etc.) sit dormant behind feature flags with no browsable surface; configuration export/import (OpenAPI JSON, slug translation, credential redaction via ConfigurationHandlers) exists with NO UI. Every competitor leads with a catalog/template gallery (n8n 600+ templates; Workato tens of thousands of recipes) — the #1 onboarding device. - -## Current state (verify at HEAD) -- Configuration groups bundling sources/endpoints/mappings/rules/jobs/syncs; export/import API endpoints (find exact routes in appinfo/routes.php + ConfigurationController). -- Seeds: PDOK sources behind pdok.feature_flag; in-flight seed changes (BRP, KVK, xWiki, messaging) will add more. -- UI: manifest v2, 26 pages; src/manifest.json; FeaturesRoadmap page exists (look at its pattern for a catalog-like page). -- Source types enum: json/xml/soap/ftp/sftp (+rest/wms/wfs seeded). - -## In scope -1. Catalog page (new manifest page "Catalog"): browsable cards of (a) connector types/adapters available (from a registry of adapter metadata: name, category, standards, status incl. feature-flagged/dormant), (b) seeded source templates, (c) importable configuration templates. Search + category filter. Detail modal with description + "Enable"/"Instantiate" action (creates the Source/Configuration from seed, respecting feature flags + action matrix authorization). -2. Configuration import/export UI: export a configuration group to file (redacted) from the UI; import with preview (what will be created/updated, slug collisions) + confirmation; surface redacted-credential placeholders needing re-entry after import. -3. Adapter metadata registry: PHP-side registry (attribute or service-based) describing each built-in adapter/connector for the catalog — single source, no hardcoded frontend list. -4. Tests: PHP unit for registry + import preview; vitest for catalog store; Playwright e2e for catalog browse + import flow (e2e-coverage gate). -## Out of scope -- Full environments/promotion with credential re-binding (deferred until source-broker-credentials lands). -- Community template marketplace (remote fetch) — local/seeded only. - -## Constraints -- Use nc-vue Cn* primitives (CnIndexPage/CnDataTable/cards) — NO nc-vue library changes; follow manifest-v2 typed pages where possible (#814 wants LESS custom pages, so prefer typed primitives; custom page only if unavoidable — mind hydra custom-widget-ratchet gate). -- Specs: new capability spec connector-catalog; delta to configuration-export-import (UI scenarios). diff --git a/openspec/changes/connector-catalog-ui/discovery.md b/openspec/changes/connector-catalog-ui/discovery.md deleted file mode 100644 index 880e7a0c1..000000000 --- a/openspec/changes/connector-catalog-ui/discovery.md +++ /dev/null @@ -1,39 +0,0 @@ -# Discovery: connector-catalog-ui - -## Question - -Three open feasibility questions from the context brief needed resolving before specs/design could be written with confidence: -1. Can the Catalog page be built from an existing manifest-v2 typed primitive (no `nextcloud-vue` change), or is a bespoke `type: "custom"` page unavoidable? -2. Does an adapter/connector metadata registry already exist in Integriq or OpenRegister that this change should extend rather than duplicate? -3. Is configuration export/import really "API-only" today (implying a route to wrap a UI around), or something else? - -## Approach Taken - -- Read `src/manifest.json` in full (26 pages) and traced `FeaturesRoadmap`'s `type: "roadmap"` through `src/main.js` / `src/registry.js` to confirm it is a library-supplied typed primitive, not a bespoke component — precedent that typed primitives beyond plain CRUD exist. -- Searched sibling `apps-extra` repos' `src/manifest.json` for `"cards"`/`"gallery"`/`"filters"` usage and found two live precedents: `openbuild`'s `VirtualApps` page (`type: "index"`, `viewMode: "cards"`, `cardComponent: "ApplicationCard"`) and `softwarecatalog`'s `Organisaties` page (same pattern, `cardComponent: "OrganisatieCard"`). Also found `openbuild`'s `Templates` page, which uses a genuinely bespoke `type: "custom"` component (`TemplateGallery`) — but only because it integrates a *remote* template-store search, a capability explicitly out of scope here. -- Read `@conduction/nextcloud-vue`'s `CnIndexPage.vue` source directly (checked out at `/home/rubenlinde/nextcloud-docker-dev/workspace/server/apps-extra/nextcloud-vue`) and confirmed `viewMode`/`viewModes`, `cardComponent`, `filters` (facet chips), and search are all config-driven props on the existing component — no library change needed to get a searchable, filterable card grid. -- Grepped `lib/AppInfo/Application.php` for `IntegrationRegistry`/`addProvider` and read `lib/Service/Adapter/AbstractCategoryAdapterProvider.php` to establish what registry machinery already exists. -- Grepped `appinfo/routes.php` for `configuration`/`Configuration` and read the "Import & Export" comment block plus `lib/Service/ConfigurationService.php` and its callers (via `tests/Unit/Service/ConfigurationServiceTest.php`, the only caller found) to establish the real current reachability of export/import. - -## Findings - -1. **Page type**: `type: "index"` + `config.viewMode: "cards"` + `config.cardComponent` + `config.filters: [...]` is an established, twice-shipped, config-only pattern for a browsable, filterable card catalog backed by an OpenRegister register/schema. It satisfies the "prefer typed primitives over custom pages" constraint directly — no `nextcloud-vue` change, no custom-widget-ratchet gate exposure. -2. **Adapter registry**: A registry already exists but is narrower than the brief assumed — `OCA\OpenRegister\Service\Integration\IntegrationRegistry` (OR-side) plus Integriq's `AbstractCategoryAdapterProvider` covers exactly 4 adapters (Azure Virtual Desktop, SharePoint Online, Microsoft 365, S3), registered by hand in `Application.php::registerIntegrationProviders()`. PDOK, Digikoppeling, Berichtenbox, DSO, and the `register.d`-seeded sources (BRP/KVK/xWiki/messaging/OpenCorporates) are **not** in this registry. Two seeding mechanisms exist and are not interchangeable: container-level `*.feature_flag` app-config (PDOK, Berichtenbox) vs. per-object `configuration.mock`/`isEnabled` on seeded Source objects (everything else). -3. **Configuration export/import**: There is no `ConfigurationController` and no route — `ConfigurationService::exportConfiguration()`/`importConfiguration()` are called only from PHPUnit tests today. The brief's framing ("API-only") is inaccurate; the correct framing is "fully implemented, fully tested, entirely unrouted." OpenRegister's generic `/api/registers/{id}/export` / `/api/configurations/{id}/import` endpoints (mentioned in the routes.php dead-code comment) operate at register granularity, not at Integriq's configuration-group granularity (`configurations[]` membership spanning 6 entity types) — they are not a drop-in substitute. -4. Source "type" has no enforced enum (contra the brief); the live vocabulary is `lib/Settings/integriq_register.json`'s free-form `type` field with recognised values `api, database, file, soap, dso, peppol, psd2, sms, payment`. -5. **Action-level authorization (ADR-023) already fully implemented in Integriq** (correction of an earlier draft that assumed it was absent): `lib/Service/ActionAuthService.php` (`requireAction()`/`can()` over an `IAppConfig` matrix, admin break-glass pass), `lib/Controller/ActionMatrixController.php` (admin matrix editor), `lib/Repair/InitializeActions.php`, and `lib/actions.seed.json` (38 actions, `.` convention — `source.test`, `job.run`, `pdok.suggest`) — already consumed by SourcesController, MappingsController, EventsController, JobsController and others. This change only appends three action keys to the existing seed; no new auth machinery. - -## Recommendation - -- **Catalog page**: build with `type: "index"` + `viewMode: "cards"`, backed by a new `catalog_item` register/schema. Do not request a new `nextcloud-vue` typed primitive and do not write a `type: "custom"` page — the cards pattern is proven and sufficient. -- **Adapter registry**: do not extend `IntegrationRegistry` in this change. Its `IntegrationProvider` interface is shaped for the 4 category adapters (auth requirements, storage strategy, health) and promoting PDOK/Digikoppeling/Berichtenbox/DSO into it is a larger, separate refactor with its own risk surface. Instead, build a lightweight Integriq-local `CatalogRegistryService` that (a) reads the 4 already-registered providers from `IntegrationRegistry` for their metadata, (b) hand-describes PDOK/Digikoppeling/Berichtenbox/DSO in a small static descriptor list colocated with each adapter's namespace, and (c) reads the `register.d/*-source.json` seed fragments for seeded-source templates. This is additive and reuses rather than duplicates; promoting (b) into full `IntegrationProvider`s is recorded as a follow-up, not done here. -- **Configuration import/export UI**: resurrect a thin `ConfigurationController` in Integriq wrapping the existing `ConfigurationService` unchanged, rather than building against OR's generic endpoints. This preserves Integriq's configuration-group semantics and reuses fully-tested logic; it does not reopen review of the underlying redaction/slug-translation behaviour (documented as retrofit-accurate in `configuration-export-import/spec.md` REQ-001–REQ-005), only adds a route + UI layer on top. - -## Risks Uncovered - -- The `ConfigurationController` import endpoint becomes a new privileged write surface (creates/updates Source/Endpoint/Mapping/Rule/Job/Synchronization objects from an uploaded, largely unvalidated OAS document — REQ-003 Notes: "Import performs no schema validation of the per-entity payload beyond the top-level `components` check"). Gated via the existing `ActionAuthService` with a `configuration.import` action seeded `["admin"]` (finding 5), matching the existing `99-source-lockdown.json` admin-only lock on the `source` schema. -- `catalog_item` materialisation (repair step) must not race with `register.d` fragment application at boot, since it reads seed fragment files as one of its inputs. - -## Next Steps - -Proceed to specs and design with the three decisions above locked in. diff --git a/openspec/changes/connector-catalog-ui/migration.md b/openspec/changes/connector-catalog-ui/migration.md deleted file mode 100644 index ca6041377..000000000 --- a/openspec/changes/connector-catalog-ui/migration.md +++ /dev/null @@ -1,51 +0,0 @@ -# Migration: connector-catalog-ui - -## Current State - -No `catalog_item` schema exists. `lib/Settings/integriq_register.json` + `lib/Settings/register.d/*.json` fragments define the `openconnector` register's other 15 schemas (source, endpoint, mapping, rule, job, synchronization, consumer, event, event_subscription, event_message, call_log, job_log, synchronization_log, synchronization_contract — see `openconnector-storage-migration` spec). Fragments are merged and imported via `OCA\OpenRegister\Service\ConfigurationService::importFromApp()`, invoked both from `lib/Repair\InitializeRegister` (repeatable repair step, runs on every `occ upgrade` and app enable) and, historically, one-shot from `lib/Migration\Version2Date20260520000001` for the chain-B storage cutover. There is no `catalog_item` register/schema, and no repair step materialises catalog data from the registries described in design.md. - -## Target State - -- A new schema fragment `lib/Settings/register.d/catalog-item-schema.json` defines `catalog_item` (fields: `name`, `description`, `category`, `kind`, `mechanism`, `flagKey`, `sourceTemplateSlug`, `standards[]`, `icon`) alongside the existing 15 schemas, merged by the existing `InitializeRegister` fragment-merge mechanism — **no new migration class is needed for the schema itself**, since `register.d/*.json` fragments are picked up automatically by the existing repair step on every run (same mechanism as the `99-source-lockdown.json` and `brp-haalcentraal-source.json` fragments already in the repo). -- A new repair step `lib/Repair/MaterializeCatalogItems.php` (implementing `\OCP\Migration\IRepairStep`, registered in `lib/AppInfo/Application.php` alongside `InitializeRegister`) runs `CatalogRegistryService::materialize()` on every `occ upgrade` / app enable, upserting one `catalog_item` OpenRegister object per real adapter/seed-source entry (see design.md Decisions), keyed by a stable `kind:slug` identifier so re-runs update in place. -- Three new action keys — `catalog.instantiate`, `configuration.export`, `configuration.import` — are appended to the **existing** `lib/actions.seed.json` (ADR-023 matrix seed, verified present at HEAD with 38 actions in `.` style, e.g. `source.test`, `job.run`, `pdok.suggest`), defaulting to `["admin"]`, applied by the existing `lib/Repair/InitializeActions.php` repair step. No new auth service, controller, or repair step is needed for authorization. - -## Migration Class - -No native-table `lib/Migration/VersionXXXXXXXXXX.php` schema migration is required — `catalog_item` is an OpenRegister-managed schema (JSON fragment + repeatable repair step), not a native Doctrine/QBMapper table, matching the pattern already used for every other Integriq entity (`openconnector-direct-or-usage`). If a one-shot trigger is later found necessary (e.g. to force an immediate materialisation on upgrade rather than waiting for the next repair-step pass), it would follow the `Version2Date20260520000001` pattern exactly: `preSchemaChange()` no-op, `changeSchema()` returns `null`, `postSchemaChange()` resolves `CatalogRegistryService` from the container and calls `materialize()` idempotently, guarded the same way (`class_exists` check for OpenRegister availability, try/catch around service resolution). This is deferred to the apply step's judgment — the repair step alone is expected to be sufficient since it already runs on every upgrade. - -``` -Version: (none required — see above) -File: lib/Repair/MaterializeCatalogItems.php (repair step, not a versioned migration) -Key operations: -- Read IntegrationRegistry-registered providers (4 category adapters) -- Read static descriptor list (PDOK, Digikoppeling, Berichtenbox, DSO) -- Read register.d/*-source.json seed fragments (BRP, KVK, xWiki, messaging, OpenCorporates, PDOK) -- Upsert one catalog_item object per entry, keyed by kind:slug -``` - -## Migration Steps - -1. Ship `lib/Settings/register.d/catalog-item-schema.json` — picked up automatically by the existing `InitializeRegister` repair step's fragment merge on the next `occ upgrade` or app enable. Verifiable: `catalog_item` appears as a schema under the `openconnector` register. -2. Ship `lib/Repair/MaterializeCatalogItems.php`, registered as an `IRepairStep` in `Application.php`. Verifiable: repair step name appears in `occ upgrade` output. -3. First repair-step run materialises `catalog_item` objects for every real adapter/seed source found (see design.md Seed Data — these are not fictional, they mirror already-shipped code). Verifiable: `GET /apps/openregister/api/objects/integriq/catalog_item` returns one object per entry. -4. Append `catalog.instantiate`, `configuration.export`, `configuration.import` (each `["admin"]`) to the existing `lib/actions.seed.json`; the existing `InitializeActions` repair step applies them on the next run. Verifiable: the existing admin action-matrix settings panel (`ActionMatrixController`) lists the three new actions. -5. Re-running steps 1–4 (idempotency check) produces no duplicate `catalog_item` objects and no duplicate action-matrix entries. - -## Data Impact - -- Additive only: creates new `catalog_item` objects (expected count: ~4 category adapters + 4 hand-described adapters [PDOK, Digikoppeling, Berichtenbox, DSO] + ~6 seeded source templates [PDOK, BRP, KVK, xWiki, messaging (grouped or per-channel), OpenCorporates] ≈ 12-16 objects at initial rollout). No existing Source/Endpoint/Mapping/Rule/Job/Synchronization/Consumer object is read, modified, or deleted by this migration. -- Safe on live data: the repair step only writes to the new `catalog_item` schema; it performs read-only queries against `IntegrationRegistry` and the existing seed fragments. -- No downtime: repair steps run as part of the normal `occ upgrade` flow, same as `InitializeRegister` today. - -## Rollback Procedure - -Remove `lib/Settings/register.d/catalog-item-schema.json`, `lib/Repair/MaterializeCatalogItems.php`, and its registration in `Application.php`. The `catalog_item` schema and its objects become orphaned (no longer written to) but are not automatically deleted — an operator MAY run `occ openregister:schema:delete openconnector catalog_item` (existing OpenRegister command) to remove them if a clean rollback is required. No other schema or object is touched, so rollback carries zero risk to existing Source/Endpoint/Configuration data (matches proposal.md Rollback Strategy). - -## Validation - -- `occ upgrade` completes without error; log output shows the `MaterializeCatalogItems` repair step ran. -- `GET /apps/openregister/api/objects/integriq/catalog_item` returns the expected object count (~12-16) with no duplicates. -- Re-running `occ upgrade` a second time produces the same object count (idempotency). -- The Catalog page (`/catalog`) renders all materialised items as cards without error. -- The admin action-matrix settings panel shows `catalog.instantiate`, `configuration.export`, `configuration.import` all defaulted to `["admin"]`. diff --git a/openspec/changes/connector-catalog-ui/proposal.md b/openspec/changes/connector-catalog-ui/proposal.md index cbeba1f7c..374cbce95 100644 --- a/openspec/changes/connector-catalog-ui/proposal.md +++ b/openspec/changes/connector-catalog-ui/proposal.md @@ -1,71 +1,46 @@ -# Proposal: connector-catalog-ui - -## Summary - -Integriq today has real integration capability — seeded PDOK/BRP/KVK/xWiki/messaging sources, a working (but unrouted) configuration export/import service with slug-translation and credential redaction, and four registered category adapters (Azure Virtual Desktop, SharePoint Online, Microsoft 365, S3) — but none of it is discoverable from the UI. Onboarding is 100% tribal-knowledge/API-only. This change adds a browsable **Catalog** page (connector adapters, seeded source templates, importable configuration templates, with search + category filter and an Enable/Instantiate action) and a **Configuration import/export UI** (export-to-file with redaction, import-with-preview and confirmation, redacted-credential re-entry flagging), built entirely from existing manifest-v2 typed primitives (`CnIndexPage` with `viewMode: "cards"`, matching the precedent already shipped in `openbuild`'s `VirtualApps` page and `softwarecatalog`'s `Organisaties` page) — no `nextcloud-vue` library changes. - -## Motivation - -Every competitor in this space leads with a template/catalog gallery as the #1 onboarding device (n8n: 600+ templates; Workato: tens of thousands of recipes). Integriq's seeded sources (PDOK behind `pdok.feature_flag`; BRP/KVK/xWiki/messaging/OpenCorporates seeded via `lib/Settings/register.d/*.json` in mock mode) sit dormant with no surface for an operator to find, understand, or enable them. Separately, `ConfigurationService` (`lib/Service/ConfigurationService.php`) already implements a complete, tested export/import/redaction/slug-translation pipeline (`openspec/specs/configuration-export-import/spec.md`, retrofit, status `done`) but has **no controller, no route, and no UI** — it is reachable only from PHPUnit tests. This is the highest-leverage, lowest-net-new-code opportunity in the app: surface what already exists rather than build new integration logic. - -## Affected Projects - -- [x] Project: `integriq` — new Catalog page (manifest + PHP catalog API + adapter metadata registry), new Configuration import/export UI (manifest + a thin `ConfigurationController` wrapping the existing `ConfigurationService`), new `catalog_item` register schema seeded by a repair step. - -## Scope - -### In Scope - -1. **Catalog page** (`src/manifest.json` page `id: "Catalog"`, `type: "index"`, `viewMode: "cards"`) browsing three kinds of catalog items — built-in connector adapters (PDOK, Berichtenbox, Digikoppeling, the four `IntegrationProvider` category adapters, DSO), seeded source templates (BRP/KVK/xWiki/messaging/OpenCorporates/PDOK), and importable configuration templates — each with category, status (`available` / `dormant` behind a feature flag), and a detail modal offering an "Enable" (flip `*.feature_flag` app-config) or "Instantiate" (create a Source/Configuration from a seed) action, gated by the existing ADR-023 authorization matrix and respecting feature-flag state. -2. **Adapter metadata registry** (PHP-side, single source of truth) — a new `CatalogRegistryService` in Integriq that assembles catalog entries from (a) the existing OR-side `IntegrationRegistry` for the 4 registered category adapters, (b) a small hand-written descriptor list for PDOK/Digikoppeling/Berichtenbox/DSO (not currently in any registry), and (c) the `register.d/*.json` seeded-source fragments, materialised into `catalog_item` OpenRegister objects by a repair step so the Catalog page can be a standard register/schema-backed `index` page. -3. **Configuration import/export UI** — a thin `ConfigurationController` (new route group) wrapping the existing, already-tested `ConfigurationService::exportConfiguration()` / `importConfiguration()`; export produces a redacted download; import shows a preview (creates vs. updates vs. slug collisions) and requires explicit confirmation before writing; imported Sources with redacted credential placeholders are flagged for operator re-entry. -4. **Tests**: PHPUnit for the catalog registry and import-preview diff logic; vitest for the catalog Pinia store; Playwright e2e for catalog-browse and the import flow (satisfies the `e2e-coverage` hydra gate). -5. **Specs**: new capability spec `connector-catalog`; delta to `configuration-export-import` adding the UI-facing scenarios (export-from-UI, import-preview, confirmation, redacted-credential flagging); delta to `openconnector-app-manifest` adding the `Catalog` page and menu entry (scoped to this addition — the base manifest spec is already stale against `src/manifest.json` at HEAD on unrelated axes; this change does not attempt a full resync). - -### Out of Scope - -- Full environments/promotion with credential re-binding across environments — deferred until `source-broker-credentials` lands (per context brief). -- A community/remote template marketplace (openbuild's `TemplateGallery` pattern of a remote registry search) — catalog entries are local/seeded only. -- Retrofitting the existing drift in `openconnector-app-manifest` spec (stale page count, phantom `Import` page, flat-vs-grouped menu) beyond the one addition this change makes. -- Registering PDOK/Digikoppeling/Berichtenbox/DSO into the OR `IntegrationRegistry` as first-class `IntegrationProvider`s — they are catalogued via a lighter descriptor list in this change; promoting them to full `IntegrationProvider`s is a separate follow-up (see design.md). - -## Approach - -Reuse the manifest-v2 `index` page type in `viewMode: "cards"` (the same pattern already shipped by `openbuild`'s `VirtualApps` page and `softwarecatalog`'s `Organisaties` page — a `cardComponent` override, `filters: [...]` for category/status facets, all config-only, zero `nextcloud-vue` changes) backed by a new `catalog_item` register/schema, itself populated at boot/repair time from a small new `CatalogRegistryService`. The Configuration import/export UI resurrects a thin, spec-referenced `ConfigurationController` over the pre-existing `ConfigurationService` rather than duplicating its logic or routing through OpenRegister's generic (register-scoped, not configuration-group-scoped) export/import endpoints, which do not model Integriq's own configuration-group semantics (sources+endpoints+mappings+rules+jobs+syncs bundled by `configurations[]` membership). Full technical detail in `design.md`. - -## New Dependencies - -None. No new npm/composer packages; reuses existing `@conduction/nextcloud-vue` primitives, existing `ConfigurationService`, existing OR `IntegrationRegistry`. - -## Impact - -- New: `lib/Controller/ConfigurationController.php`, `lib/Service/CatalogRegistryService.php`, `lib/Settings/register.d/catalog-item-schema.json` (or equivalent schema fragment), a repair/migration step to materialise `catalog_item` objects, `appinfo/routes.php` entries, `src/manifest.json` `Catalog` page + menu entry, a `CatalogItemCard.vue` card component, an import-preview modal, a catalog Pinia store. -- Touched: `lib/actions.seed.json` — three new ADR-023 action keys appended (`catalog.instantiate`, `configuration.export`, `configuration.import`, each `["admin"]`) to the existing 38-action matrix seed; enforced by the existing `lib/Service/ActionAuthService.php` and applied by the existing `lib/Repair/InitializeActions.php`, both reused unchanged. None of the existing Source/Configuration CRUD paths change behaviour — this is additive (a new read/browse surface + a new write path for import that reuses the existing `ConfigurationService` write logic unchanged). -- No database schema changes to existing Integriq tables; `catalog_item` is a new OpenRegister schema, not a native table. - -## Cross-Project Dependencies - -Depends on OpenRegister's `IntegrationRegistry` (`OCA\OpenRegister\Service\Integration\IntegrationRegistry`) for the 4 already-registered category adapters — read-only consumption, no changes requested to OpenRegister. No other apps are affected; nothing in `apps-extra` currently consumes Integriq's configuration export/import surface. - -## Risks - -### Risk 1: "Enable"/"Instantiate" actions bypass or duplicate authorization already enforced elsewhere -**Severity:** High — **Mitigation:** the Catalog page's actions call into the *existing* Source/Configuration create/update code paths (already governed by ADR-023's action matrix and the `99-source-lockdown.json` admin-only CRUD lock on the `source` schema) rather than introducing a new write path; the catalog action handler is a thin dispatcher, not a new authorization surface. Verified in design.md against `99-source-lockdown.json`. - -### Risk 2: Two incompatible "dormant" mechanisms (container `*.feature_flag` app-config vs. per-object `configuration.mock`/`isEnabled`) collapse into one UI affordance incorrectly -**Severity:** Medium — **Mitigation:** the catalog registry's status field distinguishes the mechanism explicitly (`flag-gated` vs. `mock-seeded`) and the detail-modal action dispatches to the correct handler per mechanism; specs enumerate both paths as separate scenarios. - -### Risk 3: Resurrecting a `ConfigurationController` reopens an unrouted, security-relevant surface (substring-based, not allowlist, credential redaction per REQ-005 Notes) -**Severity:** Medium — **Mitigation:** the controller is additive over already-audited logic; the import endpoint is admin-only (mirrors the `source` schema's admin-only lockdown), and the redaction gap is pre-existing and documented (not introduced by this change) — noted explicitly in the spec delta rather than silently relied upon. - -### Risk 4: `catalog_item` materialisation drifts from the live registry state (stale cards) -**Severity:** Low — **Mitigation:** materialisation runs on every repair-step pass (same cadence as existing `register.d` fragment application), and feature-flag/mock status is read live at request time by the catalog API endpoint, not baked into the stored object, so status badges cannot go stale even if the object list itself is momentarily behind. - -## Rollback Strategy - -Entirely additive: remove the `Catalog` page + menu entry from `src/manifest.json`, remove the `ConfigurationController` route registrations, and drop the `catalog_item` schema fragment (its repair step is idempotent and re-runnable). No existing Source/Endpoint/Configuration data or behaviour is touched, so rollback carries no data-migration risk. - -## Open Questions - -- Should PDOK/Digikoppeling/Berichtenbox/DSO be promoted to full OR `IntegrationRegistry` `IntegrationProvider`s in a follow-up so the catalog has one registry instead of two sources (registry + descriptor list)? Deferred — see design.md decision and DEFERRED_QUESTIONS. -- Should the import-preview UI surface the REQ-004 "unresolvable slug left verbatim" dangling-reference risk as a blocking warning, or an informational note? Proposed: blocking warning requiring explicit acknowledgement, since it is a silent-failure mode today (see configuration-export-import delta). +--- +kind: spec-only +depends_on: [] +--- + +# Proposal: connector-catalog-ui (superseded — retired 2026-09-02) + +This directory double-counted a change that had already shipped. The +connector catalog was implemented and archived on 2026-07-14 +(`archive/2026-07-14-connector-catalog-ui`, 32/41 tasks checked with +per-task evidence), yet this live copy was resurrected at 0/41: the +openconnector→integriq rename applied to the prose, the evidence notes +stripped, every box reset. The machinery exists at HEAD: +`lib/Controller/CatalogController.php`, `lib/Service/CatalogRegistryService.php`, +`lib/Repair/MaterializeCatalogItems.php`, the `catalog_item` schema in +`lib/Settings/register.d/catalog-item-schema.json`, the catalog routes in +`appinfo/routes.php`, the Catalog UI (`src/components/CatalogItemCard.vue`, +`src/dialogs/CatalogItemDetailDialog.vue`, import/export dialogs), and the +authored e2e specs (`tests/e2e/spec-coverage/connector-catalog.spec.ts`, +`configuration-import-export-ui.spec.ts`). + +`appinfo/routes.php` and `lib/Settings/register.d/catalog-item-schema.json` +reference this directory's `contract.md` and `design.md`, so those two files +stay exactly where they are as reference targets. The other artifacts +(context brief, discovery, migration, test plan, spec deltas) are removed; +they survive verbatim in the archived twin and in git history. + +## Disposition of the original scope + +| Original scope | Where it went | +| --- | --- | +| `catalog_item` schema, materialisation repair step, registry service, controller + routes, Catalog page, item detail dialog, import preview / export dialogs, ADR-023 `catalog.instantiate` entry, unit + vitest coverage, authored e2e specs | **Already shipped and archived**: `archive/2026-07-14-connector-catalog-ui` (32/41 boxes checked), code at HEAD | +| Residual verification: executing the two authored Playwright specs against a live instance, Newman coverage for the catalog endpoints, a Catalog-page screenshot in `docs/images/` | Open, and honestly unticked in the archived twin (each open box carries its reason: no live instance in that build environment). Same shape as `approvals-verification-pack`; pick up in a verification pass, not by resurrecting this change | + +## Sequencing + +Nothing remains to implement from this change directly. The residual +live-instance verification belongs to a verification-pack-style follow-up. + +## Archival + +This directory is retired in place (not moved or renamed): `contract.md` and +`design.md` are referenced from `appinfo/routes.php` and the register +fragment, and a rename would break those pointers and detonate every +diff-scoped gate. Archive it via the normal flow only after those comments +are repointed. diff --git a/openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md b/openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md deleted file mode 100644 index 383e92160..000000000 --- a/openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md +++ /dev/null @@ -1,62 +0,0 @@ -# configuration-export-import Specification (delta: connector-catalog-ui) - -## ADDED Requirements - -### Requirement: REQ-006 — Export a configuration from the UI - -The system SHALL expose the existing `ConfigurationService::exportConfiguration()` (REQ-001–REQ-005, unchanged) through a routed `POST /api/configurations/{id}/export` endpoint and a Configuration UI page action, so an operator can download a redacted configuration document without using the API directly. The endpoint SHALL be gated by Integriq's existing `ActionAuthService::requireAction()` (ADR-023) with a new action key `configuration.export`, seeded `["admin"]` in the existing `lib/actions.seed.json`. - -Notes: This requirement adds reachability only; it does not change REQ-001–REQ-005's export, slug-translation, or redaction behaviour, including the documented substring-match redaction gap (REQ-005 Notes) and the O(all entities) cost note (REQ-001 Notes). - -#### Scenario: Exporting a configuration from the UI produces a redacted downloadable file -- GIVEN a configuration group containing a Source with `apikey = "live_xyz"` -- WHEN an operator with the `configuration.export` action permission clicks "Export" on that configuration in the UI -- THEN the browser downloads a JSON file -- AND the file does not contain `apikey`, `secret`, or any other REQ-005 redacted field - -#### Scenario: A user without the configuration.export action permission cannot export -- GIVEN a non-admin user whose groups are not mapped to `configuration.export` in the admin-configured action matrix (admins always pass `requireAction()` — documented break-glass behaviour) -- WHEN that user calls the export endpoint -- THEN the request is rejected with `OCSForbiddenException` and no file is produced - -### Requirement: REQ-007 — Preview an import before writing anything - -The system SHALL expose a non-mutating `POST /api/configurations/import/preview` endpoint that, given an OAS document, computes and returns the same creates/updates/collisions classification that `importConfiguration()` (REQ-003) would perform, plus the set of unresolved slug references (REQ-004's "left verbatim" case) that would result, WITHOUT calling `saveObject()` on any entity. The preview SHALL reuse the existing handlers' slug-resolution logic (`resetMappings()`, per-type `import()` dry-run mode) rather than reimplementing it. - -#### Scenario: Preview classifies creates, updates and collisions -- GIVEN an OAS document containing one Source whose slug exists in the target environment and one Source whose slug does not -- WHEN `POST /api/configurations/import/preview` is called with that document -- THEN the response lists the existing-slug Source under `updates` and the new-slug Source under `creates` -- AND no Source object is created or modified by the preview call - -#### Scenario: Preview surfaces an unresolvable slug reference as a blocking warning -- GIVEN an OAS document containing a Rule whose nested configuration references a Source slug that does not exist in the target environment (the REQ-004 "unresolvable slug is left verbatim" case) -- WHEN the import is previewed -- THEN the response's `unresolvedReferences` array contains that Rule's slug and the unresolved field -- AND the import UI marks this as a blocking warning requiring explicit operator acknowledgement before the import can be confirmed - -### Requirement: REQ-008 — Import requires explicit confirmation after preview - -The system SHALL require a `confirmed: true` flag on `POST /api/configurations/import` and SHALL reject the request with HTTP 400 if it is absent, so that no import write occurs without the caller having first retrieved and (per the UI) displayed a preview. Both the import and preview endpoints SHALL be gated by the existing `ActionAuthService::requireAction()` (ADR-023) with a new action key `configuration.import` seeded `["admin"]` in the existing `lib/actions.seed.json`, and the underlying entity writes SHALL continue to pass through each entity type's existing OpenRegister data-layer authorization unchanged (e.g. Source writes remain admin-only per the `source` schema lock). - -#### Scenario: Import without confirmation is rejected -- GIVEN a valid OAS document -- WHEN `POST /api/configurations/import` is called with `confirmed` omitted or `false` -- THEN the response is HTTP 400 -- AND no entity is created or updated - -#### Scenario: Confirmed import proceeds and reuses the existing import pipeline unchanged -- GIVEN a valid OAS document and `confirmed: true` -- WHEN `POST /api/configurations/import` is called -- THEN the system delegates to the existing `ConfigurationService::importConfiguration()` (REQ-003) unmodified -- AND the response reflects what was actually created and updated - -### Requirement: REQ-009 — Imported Sources with redacted credentials are flagged for re-entry - -The system SHALL, in both the preview and post-import response, list every imported Source object whose credential fields were stripped by REQ-005's redaction (i.e. every Source in the import document, since export always redacts) under `credentialsNeedingReentry`, naming the fields that require operator re-entry, so the UI can direct the operator to the Source's edit form after import completes. - -#### Scenario: A newly created Source from import is flagged for credential re-entry -- GIVEN an OAS document containing a Source with no `apikey`/`secret`/`username`/`password` fields (because REQ-005 stripped them on export) -- WHEN the import is confirmed and the Source is created -- THEN the response's `credentialsNeedingReentry` array contains that Source's slug and the list of credential field names it is missing -- AND the created Source object itself contains no credential values, matching the existing REQ-005 "imported source has no credentials and needs re-entry" scenario diff --git a/openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md b/openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md deleted file mode 100644 index d11352c9f..000000000 --- a/openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -status: planned ---- - -# connector-catalog Specification - -**Status**: planned -**Scope**: integriq -**OpenSpec changes**: -- connector-catalog-ui - -## Purpose - -Integriq ships real integration capability — seeded PDOK/BRP/KVK/xWiki/messaging/OpenCorporates sources and four registered category adapters (Azure Virtual Desktop, SharePoint Online, Microsoft 365, S3) — but none of it is browsable. This capability defines a Catalog: a single, register/schema-backed, card-grid page that lists every built-in connector adapter, seeded source template, and importable configuration template, with search, category filtering, live status badges (`available` vs `dormant`), and an authorized Enable/Instantiate action. It is populated by a PHP-side `CatalogRegistryService` that reads the existing OpenRegister `IntegrationRegistry`, a small static descriptor list for adapters not yet in that registry, and the `register.d/*-source.json` seed fragments — no catalog entry is invented; every entry corresponds to real, already-shipped code. See ADR-023 (action-level authorization) for the Enable/Instantiate authorization model. - -## ADDED Requirements - -### Requirement: Catalog lists adapters, seeded source templates and configuration templates with category filter and status badges (REQ-001) - -The system MUST provide a Catalog page listing every registered `catalog_item` object, grouped by `kind` (`adapter`, `source-template`, `configuration-template`), each rendered as a card showing name, category, standards, and a live status badge (`available` or `dormant`). The page MUST support free-text search and a category facet filter, and MUST NOT require a bespoke `type: "custom"` manifest page to do so — the manifest-v2 `type: "index"` page with `config.viewMode: "cards"` MUST be used (see `openconnector-app-manifest` delta). - -#### Scenario: Catalog lists built-in adapters and seeded source templates by category -- GIVEN the `catalog_item` register contains entries for the PDOK WMS adapter (category "Geo / Maps"), the BRP HaalCentraal seeded source (category "Government registers"), and the S3 data-infra adapter (category "Data infrastructure") -- WHEN an operator opens the Catalog page -- THEN all three items are rendered as cards -- AND selecting the "Government registers" category filter narrows the grid to only the BRP HaalCentraal card - -#### Scenario: Status badge reflects a flag-gated dormant item -- GIVEN the PDOK WMS catalog item has `mechanism: "flag-gated"` and the `pdok.feature_flag` app-config value is unset (default off) -- WHEN the Catalog page renders the PDOK WMS card -- THEN its status badge reads "dormant" - -#### Scenario: Status badge reflects a mock-seeded available item -- GIVEN the BRP HaalCentraal catalog item has `mechanism: "mock-seeded"` and its underlying Source object has `isEnabled: true` and `configuration.mock: true` -- WHEN the Catalog page renders the BRP HaalCentraal card -- THEN its status badge reads "available" (mock mode is not treated as dormant — the source is reachable, just returning canned data) - -#### Scenario: Search narrows the catalog grid -- GIVEN the Catalog page is open with no filters applied -- WHEN an operator types "brp" into the search field -- THEN only catalog items whose name or description matches "brp" remain visible - -### Requirement: Catalog detail modal offers an authorized Enable or Instantiate action (REQ-002) - -The system MUST provide a detail modal for each catalog item, opened from its card, showing the item's full description and standards, plus a primary action: "Enable" for a `flag-gated` item, or "Instantiate" for a `mock-seeded` or `always-available` item. The action MUST be gated at the action layer by Integriq's existing ADR-023 implementation — `ActionAuthService::requireAction()` (`lib/Service/ActionAuthService.php`) against a new `catalog.instantiate` action key seeded `["admin"]` in the existing `lib/actions.seed.json` (following its established `.` naming, e.g. `source.test`, `job.run`) — and MUST still pass through the underlying OpenRegister data-layer authorization for the object being created or updated (e.g. the `source` schema's admin-only lock). The catalog action MUST NOT introduce a new authorization service or a bypass of existing data-layer authorization. - -#### Scenario: Enable action flips a feature flag for a flag-gated item -- GIVEN an operator with the `catalog.instantiate` action permission opens the PDOK WMS detail modal while it is dormant -- WHEN the operator clicks "Enable" -- THEN the system sets the `pdok.feature_flag` app-config value to enabled -- AND the catalog item's status badge updates to "available" on next status check - -#### Scenario: Instantiate action creates a Source from a seeded template -- GIVEN an operator with the `catalog.instantiate` action permission opens a seeded source-template catalog item that has not yet been instantiated as a live Source -- WHEN the operator clicks "Instantiate" -- THEN a new Source object is created in the `openconnector` register from the template -- AND the response indicates the created Source's id - -#### Scenario: A user without the catalog.instantiate action permission cannot enable or instantiate -- GIVEN a non-admin user whose groups are not mapped to the `catalog.instantiate` action in the admin-configured matrix (admins always pass `ActionAuthService::requireAction()` — documented break-glass behaviour) -- WHEN that user calls the instantiate endpoint for any catalog item -- THEN the request is rejected with `OCSForbiddenException` before any Source or app-config write occurs - -#### Scenario: Instantiate action still respects the Source schema's data-layer admin-only lock -- GIVEN an operator's groups ARE mapped to `catalog.instantiate` in the action matrix, but that operator is not a Nextcloud admin -- WHEN the operator calls the instantiate endpoint for a source-template catalog item -- THEN the underlying Source create call is rejected by OpenRegister's admin-only authorization on the `source` schema, independent of the action-matrix result - -### Requirement: A single PHP-side adapter metadata registry is the source of truth for catalog entries (REQ-003) - -The system MUST assemble catalog entries from exactly one service, `CatalogRegistryService`, which MUST source its data from (a) OpenRegister's existing `IntegrationRegistry` for adapters already registered there, (b) a static descriptor list for built-in adapters not registered there, and (c) the `register.d/*-source.json` seed fragments for seeded source templates. The frontend MUST NOT hardcode any catalog entry — every card rendered on the Catalog page MUST originate from a `catalog_item` OpenRegister object materialized by this service. - -#### Scenario: A newly registered IntegrationRegistry provider appears in the catalog without a frontend change -- GIVEN a fifth `IntegrationProvider` is registered into OpenRegister's `IntegrationRegistry` by Integriq -- WHEN the next `CatalogRegistryService` materialization repair-step run occurs -- THEN a corresponding `catalog_item` object is created or updated -- AND it appears on the Catalog page without any change to `CatalogItemCard.vue` or the manifest - -#### Scenario: Materialization is idempotent -- GIVEN a `catalog_item` object already exists for the PDOK WMS adapter with a given slug -- WHEN the materialization repair step runs again with no underlying change -- THEN the existing object is updated in place (not duplicated) - -## Non-Functional Requirements - -- **Performance:** Catalog page list and search MUST use OpenRegister's standard object-list endpoint (no bespoke N+1 status check per card on initial render); the live per-item status re-check (REQ-002 scenarios) is deferred to the detail-modal open, not the grid render. -- **Accessibility:** Category filter chips and the search field MUST carry accessible labels (WCAG 2.1 AA), consistent with existing `CnIndexPage` facet-filter usage elsewhere in the fleet. -- **Internationalization:** All catalog item labels, category names, and action labels MUST be translatable via the existing i18n mechanism; i18n keys MUST be English source strings (per fleet convention). - -## Acceptance Criteria - -- [ ] Catalog page renders as a manifest-v2 `type: "index"` + `viewMode: "cards"` page — no new `nextcloud-vue` component or schema. -- [ ] Every catalog card corresponds to a real, already-shipped adapter or seeded source (no fabricated entries). -- [ ] Enable/Instantiate action is gated by both the ADR-023 action matrix and existing OpenRegister data-layer authorization. -- [ ] Catalog materialization repair step is idempotent and re-runnable without duplication. - -## Notes - -Deferred: promoting PDOK/Digikoppeling/Berichtenbox/DSO into full `IntegrationRegistry` `IntegrationProvider`s (see design.md Trade-offs) — this capability consumes them via a lighter descriptor list instead. A community/remote template marketplace is explicitly out of scope (see proposal.md). diff --git a/openspec/changes/connector-catalog-ui/specs/openconnector-app-manifest/spec.md b/openspec/changes/connector-catalog-ui/specs/openconnector-app-manifest/spec.md deleted file mode 100644 index 8e86a6d97..000000000 --- a/openspec/changes/connector-catalog-ui/specs/openconnector-app-manifest/spec.md +++ /dev/null @@ -1,27 +0,0 @@ -# openconnector-app-manifest Specification (delta: connector-catalog-ui) - -## ADDED Requirements - -### Requirement: Manifest MUST declare a Catalog page and menu entry - -The manifest `pages` array MUST contain an entry with `id: "Catalog"`, `route: "/catalog"`, `type: "index"`, backed by `config.register: "openconnector"` and `config.schema: "catalog_item"`, with `config.viewMode: "cards"` and a `config.cardComponent` set. The manifest `menu` array MUST contain a corresponding entry (`id: "Catalog"`, `route: "Catalog"`) so the page is reachable from primary navigation. - -Notes: This requirement is scoped narrowly to the Catalog addition. It does not attempt to reconcile the base `openconnector-app-manifest` spec's existing drift against `src/manifest.json` at HEAD (missing `roadmap` type in the type enum, stale page count/list, a phantom `Import` page, `Settings` vs. `AppSettings` id mismatch, flat-vs-grouped menu) — that drift predates this change and is out of scope here (see connector-catalog-ui proposal.md Out of Scope). - -#### Scenario: Catalog page entry is present and uses the cards index pattern -- GIVEN the manifest file is loaded -- WHEN inspecting the page with id `"Catalog"` -- THEN its `type` field MUST be `"index"` -- AND `config.viewMode` MUST be `"cards"` -- AND `config.register` MUST be `"openconnector"` and `config.schema` MUST be `"catalog_item"` - -#### Scenario: Catalog menu entry is present and routes to the Catalog page -- GIVEN the manifest file is loaded -- WHEN inspecting `manifest.menu` (including nested `children` arrays, per the existing grouped-nav structure) -- THEN an entry with id `"Catalog"` MUST exist -- AND its `route` MUST equal `"Catalog"`, matching the `pages[].id` of the Catalog page entry - -#### Scenario: Catalog page does not require a new manifest page type -- GIVEN the manifest schema's `pages[].type` enum -- WHEN validating the Catalog page entry against it -- THEN validation succeeds using the existing `"index"` type — no new type value is introduced by this change diff --git a/openspec/changes/connector-catalog-ui/tasks.md b/openspec/changes/connector-catalog-ui/tasks.md index 94d62230d..31a96ac01 100644 --- a/openspec/changes/connector-catalog-ui/tasks.md +++ b/openspec/changes/connector-catalog-ui/tasks.md @@ -1,161 +1,13 @@ -# Tasks: connector-catalog-ui - -## Implementation Tasks - -### Task 1: Seed the three new actions into the existing ADR-023 matrix -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **files**: `lib/actions.seed.json` -- **acceptance_criteria**: - - GIVEN the existing `lib/actions.seed.json` (38 actions, `.` convention — `source.test`, `job.run`, `pdok.suggest`) WHEN `catalog.instantiate`, `configuration.export`, `configuration.import` are appended, each `["admin"]` THEN the existing `lib/Repair/InitializeActions.php` applies them on its next run and the existing admin Action authorization panel (`ActionMatrixController`) lists all three - - No new auth service or controller is created — the existing `lib/Service/ActionAuthService.php::requireAction()` is reused unchanged by Tasks 5, 8, 9 and 10 -- [ ] Implement -- [ ] Test - -### Task 2: catalog_item schema fragment -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **files**: `lib/Settings/register.d/catalog-item-schema.json` -- **acceptance_criteria**: - - GIVEN the fragment is shipped WHEN `occ upgrade` runs THEN `catalog_item` appears as a schema under the `openconnector` register with fields `name`, `description`, `category`, `kind`, `mechanism`, `flagKey`, `sourceTemplateSlug`, `standards`, `icon` -- [ ] Implement -- [ ] Test - -### Task 3: CatalogRegistryService — assemble catalog entries from existing sources -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-a-single-php-side-adapter-metadata-registry-is-the-source-of-truth-for-catalog-entries-req-003` -- **files**: `lib/Service/CatalogRegistryService.php` -- **acceptance_criteria**: - - GIVEN OpenRegister's `IntegrationRegistry` has 4 registered providers WHEN `CatalogRegistryService::collect()` runs THEN it returns 4 descriptor entries sourced from that registry, plus static entries for PDOK/Digikoppeling/Berichtenbox/DSO, plus one entry per `register.d/*-source.json` seed fragment found - - GIVEN a fifth provider is registered into `IntegrationRegistry` WHEN `collect()` runs again THEN a 5th entry appears with no code change to the static descriptor list -- [ ] Implement -- [ ] Test - -### Task 4: MaterializeCatalogItems repair step -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-a-single-php-side-adapter-metadata-registry-is-the-source-of-truth-for-catalog-entries-req-003` -- **files**: `lib/Repair/MaterializeCatalogItems.php`, `lib/AppInfo/Application.php` -- **acceptance_criteria**: - - GIVEN `CatalogRegistryService::collect()` returns N entries WHEN the repair step runs THEN N `catalog_item` objects exist, keyed by stable `kind:slug` - - GIVEN the repair step runs a second time with no underlying change WHEN it completes THEN the object count is unchanged (idempotent upsert, not duplicate creation) -- [ ] Implement -- [ ] Test - -### Task 5: CatalogController — status + instantiate endpoints -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **files**: `lib/Controller/CatalogController.php`, `appinfo/routes.php` -- **acceptance_criteria**: - - GIVEN a flag-gated catalog item WHEN `GET /api/catalog/items/{id}/status` is called THEN the response reflects the live `IConfig` value for its `flagKey` - - GIVEN an operator with `catalog.instantiate` permission WHEN `POST /api/catalog/items/{id}/instantiate` is called on a flag-gated item THEN the app-config flag is enabled - - GIVEN an operator with `catalog.instantiate` permission WHEN the same endpoint is called on a mock-seeded/template item not yet instantiated THEN a new Source object is created - - GIVEN a user without `catalog.instantiate` permission WHEN either endpoint is called THEN the request is rejected before any write - - GIVEN an operator with `catalog.instantiate` permission but not a Nextcloud admin WHEN instantiate is called THEN the underlying Source write is still rejected by the `source` schema's admin-only OpenRegister authorization -- [ ] Implement -- [ ] Test - -### Task 6: Catalog manifest page + card component -- **spec_ref**: `openspec/specs/openconnector-app-manifest/spec.md#requirement-manifest-must-declare-a-catalog-page-and-menu-entry` -- **files**: `src/manifest.json`, `src/components/CatalogItemCard.vue`, `src/store/catalog.js` -- **acceptance_criteria**: - - GIVEN the manifest is loaded WHEN inspecting the `Catalog` page entry THEN `type` is `"index"`, `config.viewMode` is `"cards"`, `config.register`/`config.schema` are `"openconnector"`/`"catalog_item"` - - GIVEN the manifest is loaded WHEN inspecting `menu` THEN a `Catalog` entry routes to the `Catalog` page id - - GIVEN the Catalog page is open WHEN the category filter is applied THEN only matching cards render (uses `CnIndexPage`'s existing `filters` config, no new component logic beyond the card itself) -- [ ] Implement -- [ ] Test - -### Task 7: Catalog detail modal (Enable / Instantiate) -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **files**: `src/dialogs/CatalogItemDetailDialog.vue` -- **acceptance_criteria**: - - GIVEN a card is clicked WHEN the detail dialog opens THEN it shows description, standards, and a live status re-check via `GET /api/catalog/items/{id}/status` - - GIVEN the item is dormant WHEN the operator clicks the primary action THEN the correct endpoint (Enable vs Instantiate) is called based on `mechanism` -- [ ] Implement -- [ ] Test - -### Task 8: ConfigurationController — export -- **spec_ref**: `openspec/specs/configuration-export-import/spec.md#requirement-req-006-export-a-configuration-from-the-ui` -- **files**: `lib/Controller/ConfigurationController.php`, `appinfo/routes.php` -- **acceptance_criteria**: - - GIVEN a configuration group with a Source containing credentials WHEN `POST /api/configurations/{id}/export` is called by an authorized operator THEN a redacted JSON file is returned matching the existing `ConfigurationService::exportConfiguration()` output unchanged - - GIVEN a user without `configuration.export` permission WHEN the endpoint is called THEN the request is rejected -- [ ] Implement -- [ ] Test - -### Task 9: ConfigurationController — import preview (non-mutating) -- **spec_ref**: `openspec/specs/configuration-export-import/spec.md#requirement-req-007-preview-an-import-before-writing-anything` -- **files**: `lib/Controller/ConfigurationController.php`, `lib/Service/ConfigurationImportPreviewService.php` -- **acceptance_criteria**: - - GIVEN an OAS document with one existing-slug and one new-slug Source WHEN `POST /api/configurations/import/preview` is called THEN the response correctly classifies each under `updates`/`creates` and no object is written - - GIVEN an OAS document with a Rule referencing an unresolvable Source slug WHEN previewed THEN `unresolvedReferences` lists it -- [ ] Implement -- [ ] Test - -### Task 10: ConfigurationController — confirmed import -- **spec_ref**: `openspec/specs/configuration-export-import/spec.md#requirement-req-008-import-requires-explicit-confirmation-after-preview` -- **files**: `lib/Controller/ConfigurationController.php` -- **acceptance_criteria**: - - GIVEN `confirmed` is omitted or false WHEN `POST /api/configurations/import` is called THEN the response is HTTP 400 and nothing is written - - GIVEN `confirmed: true` WHEN called THEN the system delegates unchanged to `ConfigurationService::importConfiguration()` and returns what was created/updated -- [ ] Implement -- [ ] Test - -### Task 11: Credential re-entry flagging in import response -- **spec_ref**: `openspec/specs/configuration-export-import/spec.md#requirement-req-009-imported-sources-with-redacted-credentials-are-flagged-for-re-entry` -- **files**: `lib/Service/ConfigurationImportPreviewService.php` -- **acceptance_criteria**: - - GIVEN an imported Source document with no credential fields WHEN import completes THEN the response's `credentialsNeedingReentry` lists that Source's slug and missing field names -- [ ] Implement -- [ ] Test - -### Task 12: Configuration import/export UI page + preview dialog -- **spec_ref**: `openspec/specs/configuration-export-import/spec.md#requirement-req-006-export-a-configuration-from-the-ui` -- **files**: `src/manifest.json`, `src/dialogs/ImportPreviewDialog.vue` -- **acceptance_criteria**: - - GIVEN an operator uploads an OAS document WHEN the preview dialog opens THEN it shows creates/updates/collisions/unresolved-references, and any unresolved reference blocks confirmation until acknowledged - - GIVEN the operator confirms WHEN the import completes THEN a post-import summary shows `credentialsNeedingReentry` with links to each Source's edit form -- [ ] Implement -- [ ] Test - -### Task 13: PHPUnit — CatalogRegistryService + import preview diff logic -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-a-single-php-side-adapter-metadata-registry-is-the-source-of-truth-for-catalog-entries-req-003` -- **files**: `tests/Unit/Service/CatalogRegistryServiceTest.php`, `tests/Unit/Service/ConfigurationImportPreviewServiceTest.php` -- **acceptance_criteria**: - - GIVEN mocked `IntegrationRegistry` + seed fragments WHEN `collect()` is unit-tested THEN entry count and shape are asserted - - GIVEN a fixture OAS document with a known create/update/collision/unresolved mix WHEN the preview service is unit-tested THEN each category is asserted -- [ ] Implement -- [ ] Test - -### Task 14: vitest — catalog Pinia store -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **files**: `src/store/catalog.spec.js` -- **acceptance_criteria**: - - GIVEN the catalog store is loaded WHEN filtering by category or search term THEN the store's filtered getter returns the expected subset -- [ ] Implement -- [ ] Test - -### Task 15: Playwright e2e — catalog browse + import flow -- **spec_ref**: `openspec/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **files**: `tests/playwright/catalog-browse.spec.js`, `tests/playwright/configuration-import.spec.js` -- **acceptance_criteria**: - - GIVEN a logged-in admin WHEN they navigate to `/catalog`, filter by category, open a detail modal, and instantiate a seeded source THEN the new Source appears in the Sources list - - GIVEN an admin exports a configuration, then re-imports the downloaded file WHEN the preview dialog appears THEN it shows the expected update classification and completing the import re-flags credential re-entry -- [ ] Implement -- [ ] Test - -## Verification -- [ ] All tasks checked off -- [ ] `openspec validate` passes -- [ ] Manual testing against acceptance criteria -- [ ] Code review against spec requirements - -## Tests (company-wide ADR-009) - -- [ ] PHPUnit unit tests for new/changed business logic (`tests/Unit/`) -- [ ] Newman/Postman tests for new/changed API endpoints -- [ ] Browser tests (Playwright MCP) for UI changes -- [ ] All tests pass (`composer test`, `newman run`) - -## Documentation (company-wide ADR-010) - -- [ ] Feature documentation updated in `docs/` -- [ ] Screenshot captured and committed to `docs/images/` - -## i18n (company-wide hydra ADR-007) - -- [ ] Dutch (`nl_NL`) and English (`en_US`) translation strings added for Catalog page labels, category names, status badges, and import-preview dialog text +# Tasks: connector-catalog-ui (superseded) + +The original 15-task / 41-checkbox list was removed with the 2026-09-02 +retirement (see proposal.md for the disposition; the list survives in +`archive/2026-07-14-connector-catalog-ui/tasks.md`, where 32/41 boxes are +checked with per-task evidence, and in git history). The residual +live-instance verification (running the authored Playwright specs, Newman, +a screenshot) is listed there with per-box reasons and belongs to a +verification-pack-style follow-up. + +`contract.md` and `design.md` stay in this directory as reference targets +for `appinfo/routes.php` and `lib/Settings/register.d/catalog-item-schema.json`. +There is nothing to implement from this change directly. diff --git a/openspec/changes/connector-catalog-ui/test-plan.md b/openspec/changes/connector-catalog-ui/test-plan.md deleted file mode 100644 index 23a473f48..000000000 --- a/openspec/changes/connector-catalog-ui/test-plan.md +++ /dev/null @@ -1,180 +0,0 @@ -# Test Plan: connector-catalog-ui - -## Test Cases - -### TC-1: Catalog lists items grouped by category -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **type**: functional -- **persona**: Noor (municipal CISO / functional admin — the operator who onboards new connectors) -- **preconditions**: `catalog_item` objects materialised for PDOK WMS, BRP HaalCentraal, S3 adapter -- **steps**: navigate to `/catalog`; apply the "Government registers" category filter -- **expected result**: only the BRP HaalCentraal card remains visible -- **test command**: `/test-functional` - -### TC-2: Status badge distinguishes flag-gated dormant vs mock-seeded available -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **type**: functional -- **persona**: Noor -- **preconditions**: `pdok.feature_flag` unset (default off); BRP source `isEnabled: true`, `configuration.mock: true` -- **steps**: open `/catalog`; inspect the PDOK WMS and BRP HaalCentraal cards -- **expected result**: PDOK WMS badge reads "dormant"; BRP HaalCentraal badge reads "available" -- **test command**: `/test-functional` - -### TC-3: Search narrows the catalog grid -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-lists-adapters-seeded-source-templates-and-configuration-templates-with-category-filter-and-status-badges-req-001` -- **type**: functional -- **preconditions**: catalog populated with ≥3 items, only one matching "brp" -- **steps**: type "brp" into the catalog search field -- **expected result**: only the matching item(s) remain visible -- **test command**: `/test-functional` - -### TC-4: Enable action flips a feature flag (flag-gated item) -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **type**: functional -- **persona**: Noor -- **preconditions**: operator has `catalog.instantiate` action permission; PDOK WMS dormant -- **steps**: open PDOK WMS detail modal; click "Enable" -- **expected result**: `pdok.feature_flag` becomes enabled; status badge updates to "available" on next check -- **test command**: `/test-functional` - -### TC-5: Instantiate action creates a Source from a seeded template -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **type**: functional -- **persona**: Noor -- **preconditions**: operator has `catalog.instantiate` permission; a not-yet-instantiated source-template item exists -- **steps**: open its detail modal; click "Instantiate" -- **expected result**: a new Source object appears in the Sources list -- **test command**: `/test-functional` - -### TC-6: catalog.instantiate action denial blocks the write (API) -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **type**: security -- **preconditions**: a non-admin user whose groups are NOT mapped to `catalog.instantiate` (admins always pass the existing `ActionAuthService::requireAction()` break-glass) -- **steps**: `POST /api/catalog/items/{id}/instantiate` as that user -- **expected result**: request rejected (403 via `OCSForbiddenException` from the existing `ActionAuthService`), no Source or app-config write occurs -- **test command**: `/test-security` - -### TC-7: Action-matrix pass but data-layer admin-only lock still blocks a non-admin -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-catalog-detail-modal-offers-an-authorized-enable-or-instantiate-action-req-002` -- **type**: security -- **preconditions**: a non-admin user IS mapped to `catalog.instantiate` in the action matrix -- **steps**: `POST /api/catalog/items/{id}/instantiate` (source-template item) as that user -- **expected result**: the underlying Source create is rejected by OpenRegister's admin-only `source` schema authorization, independent of the action-matrix pass -- **test command**: `/test-security` - -### TC-8: Catalog materialises new IntegrationRegistry providers with no frontend change -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-a-single-php-side-adapter-metadata-registry-is-the-source-of-truth-for-catalog-entries-req-003` -- **type**: api -- **preconditions**: a 5th `IntegrationProvider` registered into `IntegrationRegistry` -- **steps**: run the `MaterializeCatalogItems` repair step; call `GET /apps/openregister/api/objects/integriq/catalog_item` -- **expected result**: a 5th `catalog_item` object exists -- **test command**: `/test-api` - -### TC-9: Materialization is idempotent across repeated runs -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#requirement-a-single-php-side-adapter-metadata-registry-is-the-source-of-truth-for-catalog-entries-req-003` -- **type**: api -- **preconditions**: `catalog_item` objects already materialised once -- **steps**: run the repair step a second time with no underlying change -- **expected result**: object count unchanged, no duplicates -- **test command**: `/test-api` - -### TC-10: Export from UI produces a redacted file -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-006-export-a-configuration-from-the-ui` -- **type**: functional -- **persona**: Noor -- **preconditions**: a configuration group with a Source carrying `apikey`/`secret`; operator has `configuration.export` permission -- **steps**: open the Configuration UI page for that group; click "Export" -- **expected result**: downloaded JSON file contains no `apikey`, `secret`, or other REQ-005-redacted field -- **test command**: `/test-functional` - -### TC-11: configuration.export action denial blocks export (API) -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-006-export-a-configuration-from-the-ui` -- **type**: security -- **preconditions**: non-admin user not mapped to `configuration.export` (admins always pass the existing `ActionAuthService` break-glass) -- **steps**: `POST /api/configurations/{id}/export` -- **expected result**: 403, no file produced -- **test command**: `/test-security` - -### TC-12: Import preview classifies creates/updates/collisions without writing -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-007-preview-an-import-before-writing-anything` -- **type**: api -- **preconditions**: OAS document with one existing-slug Source and one new-slug Source -- **steps**: `POST /api/configurations/import/preview` -- **expected result**: response correctly classifies each; no object is created or modified -- **test command**: `/test-api` - -### TC-13: Preview surfaces an unresolvable slug reference as a blocking warning -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-007-preview-an-import-before-writing-anything` -- **type**: functional -- **preconditions**: OAS document with a Rule referencing a non-existent Source slug -- **steps**: upload the document in the import UI; observe the preview dialog -- **expected result**: `unresolvedReferences` is shown; the confirm button is disabled until the operator explicitly acknowledges the warning -- **test command**: `/test-functional` - -### TC-14: Import without confirmation is rejected -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-008-import-requires-explicit-confirmation-after-preview` -- **type**: api -- **preconditions**: valid OAS document -- **steps**: `POST /api/configurations/import` with `confirmed` omitted -- **expected result**: HTTP 400, nothing written -- **test command**: `/test-api` - -### TC-15: Confirmed import writes via the existing unchanged pipeline -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-008-import-requires-explicit-confirmation-after-preview` -- **type**: api -- **preconditions**: valid OAS document -- **steps**: `POST /api/configurations/import` with `confirmed: true` -- **expected result**: response reflects actual creates/updates; entities appear in their respective index pages -- **test command**: `/test-api` - -### TC-16: Imported Source with stripped credentials is flagged for re-entry -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/configuration-export-import/spec.md#requirement-req-009-imported-sources-with-redacted-credentials-are-flagged-for-re-entry` -- **type**: functional -- **persona**: Noor -- **preconditions**: import document containing a Source with no credential fields (post-REQ-005 export) -- **steps**: confirm the import; view the post-import summary -- **expected result**: `credentialsNeedingReentry` lists the Source and its missing fields, with a link to its edit form; the created Source itself has no credential values -- **test command**: `/test-functional` - -### TC-17: Catalog page manifest conformance -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/openconnector-app-manifest/spec.md#requirement-manifest-must-declare-a-catalog-page-and-menu-entry` -- **type**: regression -- **preconditions**: `src/manifest.json` updated -- **steps**: run `check:manifest` (existing `validateManifest` script per `openconnector-app-manifest` spec) -- **expected result**: validation passes with zero errors; Catalog page/menu entries present as specified -- **test command**: `/test-regression` - -### TC-18: Existing pages/routes unaffected (regression) -- **spec_ref**: `openspec/changes/connector-catalog-ui/proposal.md#impact` -- **type**: regression -- **preconditions**: full manifest with Catalog added -- **steps**: spot-check Sources, Endpoints, Jobs, Synchronizations pages and their existing actions -- **expected result**: no behavioural change to any pre-existing page -- **test command**: `/test-regression` - -### TC-19: Accessibility of catalog search/filter controls -- **spec_ref**: `openspec/changes/connector-catalog-ui/specs/connector-catalog/spec.md#non-functional-requirements` -- **type**: accessibility -- **preconditions**: Catalog page rendered -- **steps**: run automated WCAG 2.1 AA check against the search field and category filter chips -- **expected result**: no critical/serious violations; controls carry accessible labels -- **test command**: `/test-accessibility` - -## Coverage Summary - -| Requirement | Covered by | -|---|---| -| connector-catalog REQ-001 (list + filter + status badges) | TC-1, TC-2, TC-3, TC-19 | -| connector-catalog REQ-002 (Enable/Instantiate authorized action) | TC-4, TC-5, TC-6, TC-7 | -| connector-catalog REQ-003 (single registry source of truth) | TC-8, TC-9 | -| configuration-export-import REQ-006 (export from UI) | TC-10, TC-11 | -| configuration-export-import REQ-007 (import preview) | TC-12, TC-13 | -| configuration-export-import REQ-008 (confirmation required) | TC-14, TC-15 | -| configuration-export-import REQ-009 (credential re-entry flagging) | TC-16 | -| openconnector-app-manifest (Catalog page + menu entry) | TC-17, TC-18 | - -## Out of Scope - -- Load/performance testing of the O(all entities) export cost — pre-existing, documented limitation (`configuration-export-import` REQ-001 Notes), not changed or newly tested by this change. -- Testing a remote/community template marketplace — explicitly out of scope per proposal.md. -- Full environments/credential-rebinding promotion flow — deferred per proposal.md until `source-broker-credentials` lands. diff --git a/openspec/changes/environments-and-promotion/design.md b/openspec/changes/environments-and-promotion/design.md deleted file mode 100644 index d01e062f7..000000000 --- a/openspec/changes/environments-and-promotion/design.md +++ /dev/null @@ -1,349 +0,0 @@ -# Design: environments-and-promotion - -## Architecture Overview -Promotion is orchestrated entirely from the SOURCE instance. There is no new -inbound surface on the target beyond the two endpoints -`configuration-export-import` already routes (`POST -/api/configurations/import/preview`, `POST /api/configurations/import`). - -``` -[Operator on instance A] - | - v -Environments & Promotion UI (manifest-v2 page, instance A) - | - v -PromotionController (instance A) - | - v -PromotionService (instance A) - |-- 1. ConfigurationService::exportConfiguration() [unchanged, local] - |-- 2. scan export for credentialRef placeholders [new, in-process] - |-- 3. apply operator-supplied credentialBindings [new, in-process] - |-- 4. CallService::call($environmentB.sourceRef, 'POST', - | '/api/configurations/import/preview', ...) [reused dispatch] - |-- 5. merge target's preview + local credentialRef bucket -> UI - |-- 6. (on confirm) CallService::call($environmentB.sourceRef, 'POST', - | '/api/configurations/import', {confirmed:true, ...}) - |-- 7. write promotion_audit object [new, OR object] - | - v - [instance B's own ConfigurationController/ConfigurationService, - unchanged, receives the call exactly like any other API client] -``` - -`environment` objects never grant instance A code access to instance B's -database, filesystem, or OpenRegister directly — every cross-instance -interaction is an ordinary authenticated HTTP call through the SAME pipeline -a Source-to-external-API call already uses. - -## Decisions - -### Decision 1: Environment metadata is an OpenRegister object, not IAppConfig -**Chosen: new `environment` schema in the existing `openconnector` register -(OR object).** - -Rationale: -- ADR-008 / `openconnector-direct-or-usage` establish OpenRegister as the - required persistence layer for every Integriq entity; app-local - reimplementation of storage (which `IAppConfig` effectively is here — a - flat key/value store) is the pattern this change must NOT introduce. -- `IAppConfig` stores scalar key→value pairs per app; it has no native - concept of "many named rows with structured fields," no per-row RBAC, no - audit trail, and nothing analogous to OR's slug/relation `$ref` wiring - that the promotion UI needs (`environment.sourceRef` → `source` object). - Representing N named environments would require hand-rolled JSON-blob - serialisation into a single config key — exactly the kind of - app-local-reimplementation ADR-008 forbids. -- OR objects give environments a slug (consistent with every other - Integriq entity — sources, endpoints, jobs — for free), standard - CRUD via `ObjectService`, and RBAC through the same schema-lock mechanism - already used for `source` (admin-only writes, `99-source-lockdown.json` - precedent). -- Alternative considered and rejected: `IAppConfig` for a single - "environments list" JSON blob. Rejected because it duplicates OR's own - object storage, cannot be individually RBAC'd per environment, and breaks - the "every entity is persisted as an OpenRegister object" rule with no - compensating benefit — there is no performance or simplicity win over an - OR object for what is, structurally, a small list of named records. - -`environment` fields: `name`, `slug`, `role` (`source` | `target` | `both`), -`sourceRef` (uuid of a `source`-schema object describing how to reach that -environment's API), `description`. No credential material is stored on the -`environment` object itself — it lives exactly where every other Source -credential lives, behind the referenced Source's -`configuration.authentication.credentialRef`. - -### Decision 2: An environment's connectivity is a `source` object, not new plumbing -**Chosen: `environment.sourceRef` points at an ordinary `source`-schema -object (`type: "api"`), dispatched via the existing `CallService::call()`.** - -Rationale (see discovery.md): `CallService::call()` is the only outbound -HTTP path in the codebase and is hard-bound to a `source`-schema -`ObjectEntity`. `BrokeredCallService` already layers `credentialRef` proxy -resolution on top of exactly that shape. Wrapping connectivity in a `source` -object means promotion dispatch is a completely ordinary Source call: -CallLog auditing, retry policy, rate limiting, and REQ-005-style redaction -on any promotion CallLog all apply unchanged. The alternative — a new -`EnvironmentClientService` with its own Guzzle client and its own -credential-broker integration — would duplicate `BrokeredCallService` -end-to-end for no behavioural gain, violating "build ON the existing -services, do NOT fork them." - -Trade-off accepted: promotion calls appear in the Logs UI as ordinary Source -calls against a Source most operators won't otherwise interact with -directly. `promotion_audit` stores the dispatched CallLog id(s) as a -cross-reference so an operator can pivot from the audit entry to the raw -CallLog when diagnosing a failed promotion (see Decision 4). - -### Decision 3: `credentialRef` re-binding is explicit, client-computed, and never resolves a secret -**Chosen: `PromotionService` scans the exported OAS document for -`{"credentialRef": {...}}` placeholders (same shape `BrokeredCallService:: -isPlaceholder()` detects) and returns them as a new -`credentialRefsNeedingRebind` preview bucket, alongside the target's own -REQ-007 preview response. The document sent to the target is REWRITTEN -in-process to substitute each flagged `credentialRef` with an -operator-supplied replacement (`{"credentialId": "..."}` or -`{"credentialName": "..."}` valid in the TARGET environment) before the -confirmed import call — never resolved to plaintext at any point.** - -Rationale: -- `credentialRef` is, by design (`BrokeredCallService`), never resolved to a - plaintext secret anywhere except inside the broker's own constrained - proxy/injection call at actual dispatch time. Promotion must preserve that - invariant: `PromotionService` only ever reads/writes the reference SHAPE - (`credentialId`/`credentialName` strings), never a secret value. This is - the literal meaning of "re-binding, not copying." -- Rewriting happens BEFORE the document leaves instance A and is sent to - instance B — not as a post-import fixup on B — so the diff preview - (Decision 4) reflects exactly what will be written, and an unconfirmed - promotion never leaves a Source on B with a dangling reference. -- Validating that an operator-supplied replacement resolves on B happens by - delegating to B: the rewritten document is what gets sent to B's own - `/api/configurations/import/preview`; if the referenced credential doesn't - exist on B, that surfaces as a normal Source-auth failure the first time - B calls that Source — the SAME failure mode as any other misconfigured - Source, deliberately not re-invented as a new validation path. (B's - `CredentialBrokerService` is the only component that can authoritatively - answer "does this credentialId/Name exist and resolve for this owner" — - `PromotionService` on A has no visibility into B's broker state and must - not guess.) -- Alternative considered: auto-resolve by `credentialName` match only (skip - operator involvement when names match across environments). Rejected as - the default because a same-named credential on B is not guaranteed to be - the operator's intent (naming collisions, different owners) — silently - auto-binding cross-environment credentials is a lateral-movement risk. - Left as an OPT-IN convenience: the UI may pre-fill a rebind suggestion - when a `credentialName` (not `credentialId`) reference already matches a - visible name, but the operator must explicitly confirm it — never - automatic. -- Alternative considered: resolve the secret on A and re-inject it as an - embedded auth field on B. Rejected outright — this is exactly the - "copied secrets" anti-pattern the brief and `BrokeredCallService`'s "NO - fallback to embedded authentication under any circumstance" rule forbid. - -### Decision 4: Diff preview reuses the target's existing REQ-007 endpoint verbatim -**Chosen: `PromotionService` calls the target's unmodified `POST -/api/configurations/import/preview` remotely (via `CallService::call()`, -Decision 2) and merges its response with the local -`credentialRefsNeedingRebind` bucket (Decision 3). No new diff/classification -algorithm is written.** - -Rationale: `ConfigurationImportPreviewService` already computes creates/ -updates/collisions/unresolvedReferences/credentialsNeedingReentry against -whatever environment it runs in. Since promotion's target IS a different -environment, the only correct place to run that classification is ON the -target — a locally-computed diff on A would be comparing against A's own -data, not B's. Invoking B's already-tested, already-routed endpoint over -HTTP is both simpler and correct; re-implementing the same classification -logic locally (as if it read B's data over some new API) would duplicate -`ConfigurationImportPreviewService` for zero benefit and risk drift between -the two copies. - -## API Design - -### `POST /api/environments` -Create an `environment` object. Admin-only (ADR-023 `environment.manage`, -seeded `["admin"]`), plus OR data-layer authorization on the `environment` -schema. - -**Request:** -```json -{ "name": "Production", "slug": "production", "role": "target", "sourceRef": "", "description": "..." } -``` -**Response:** -```json -{ "id": "...", "uuid": "...", "slug": "production", "name": "Production", "role": "target", "sourceRef": "" } -``` - -### `GET /api/environments` -List registered environments. `environment.manage` action gate. - -### `POST /api/promotions/preview` -Non-mutating. Computes the merged diff preview (Decision 4) without writing -anything on A or B. `environment.promote` action gate. - -**Request:** -```json -{ - "configurationId": "cfg-1", - "targetEnvironmentSlug": "production", - "credentialBindings": [ - { "sourceSlug": "my-api-source", "field": "configuration.authentication.credentialRef", "credentialName": "prod-api-key" } - ] -} -``` -**Response:** -```json -{ - "creates": [], "updates": [], "collisions": [], - "unresolvedReferences": [], - "credentialsNeedingReentry": [], - "credentialRefsNeedingRebind": [ - { "type": "source", "slug": "my-api-source", "field": "configuration.authentication.credentialRef", "rebound": true } - ] -} -``` - -### `POST /api/promotions` -Confirmed promotion. Requires `confirmed: true` (mirrors REQ-008); rejects -with 400 otherwise. `environment.promote` action gate. Delegates the actual -write to the target's own `/api/configurations/import` (unchanged), then -writes a `promotion_audit` object. - -**Request:** same shape as `/api/promotions/preview` plus `"confirmed": true`. -**Response:** -```json -{ - "auditId": "...", - "written": { "sources": ["my-api-source"], "endpoints": ["..."] }, - "callLogId": "..." -} -``` - -## Database Changes -Two new OpenRegister schemas added to `lib/Settings/integriq_register.json` -(REQ-A-001/REQ-A-005 conventions from `openconnector-register-schema`): -- `environment` (mutable config schema — `appendOnly: false`, `immutable: false`): - `name`, `slug`, `role` (enum `source`|`target`|`both`), `sourceRef` (UUID, - `$ref` to `source`), `description`. -- `promotion_audit` (log schema — `appendOnly: true`, `immutable: true`, - carries `x-openregister-archival` retention matching the existing log - schemas' convention): `actorUid`, `configurationId`, `fromEnvironmentSlug`, - `toEnvironmentSlug`, `startedAt`, `completedAt`, `outcome` - (`success`|`failed`|`rejected`), `previewSummary` (counts only — no - entity payloads, no credential values), `credentialRebindCount`, - `callLogId` (cross-reference to the `CallLog` created by the underlying - `CallService::call()` dispatch, per Decision 2). - -## Nextcloud Integration -- Controllers: `PromotionController` (Controller → Service → Mapper, ADR-008). -- Services: `PromotionService` (new), reusing `ConfigurationService`, - `ConfigurationImportPreviewService`'s response SHAPE (not its code — the - actual preview call happens on the target), `CallService`, - `BrokeredCallService` (transitively, via `CallService::call()`), - `ActionAuthService`. -- Mappers/Entities: none new — `environment` and `promotion_audit` are plain - OR objects via `ObjectService`, consistent with every other Integriq - schema (no bespoke Doctrine mapper). -- Events/Hooks: none new. - -## Security Considerations -- Both new action keys (`environment.manage`, `environment.promote`) are - seeded `["admin"]` in `lib/actions.seed.json`, following the existing - `.` convention (`configuration.export`, `catalog.instantiate`). -- No secret ever crosses the promotion call: exported Sources are - REQ-005-redacted exactly as today, and `credentialRef` values are - references only (Decision 3) — `PromotionService` cannot read a plaintext - secret because it never calls the broker's resolution methods, only - `ConfigurationService`/`CallService`/`BrokeredCallService`'s existing, - unmodified entry points. -- `promotion_audit.previewSummary` stores counts and slugs only, never - entity payloads or credential values, mirroring - `BrokeredCallService::logOwnerRefusal()`'s "guard name + identity only, - never secret material" logging discipline. -- The underlying entity writes on the TARGET still pass through that - target's own OpenRegister data-layer authorization unchanged (e.g. - Source writes remain admin-only there too) — promotion does not grant any - new authority on B beyond what the environment Source's credential - already carries. -- CSRF: promotion is triggered from the UI via the standard Nextcloud - session + CSRF token flow; `#[NoCSRFRequired]` is NOT used on - `PromotionController` (unlike `ConfigurationController`'s export/import, - which accept file uploads/API-style calls) since promotion is - UI-initiated only in this change's scope. - -## NL Design System -Environments & Promotion page is a manifest-v2 `type: "index"` list page -(environment CRUD) plus a promote flow reachable from a configuration -group's existing actions. The promote flow's confirmation step is its own -`NcModal` file under `src/modals/PromotePreviewModal.vue` (never inlined — -hydra modal-isolation gate). Environment/target select uses `NcSelect` with -`inputLabel` set (hydra nc-input-labels gate). All strings ENGLISH per -project i18n convention. - -## File Structure -``` -lib/ - Controller/ - PromotionController.php - Service/ - PromotionService.php - Settings/ - integriq_register.json (add environment, promotion_audit schemas) - actions.seed.json (add environment.manage, environment.promote) -appinfo/ - routes.php (add /api/environments*, /api/promotions*) -src/ - modals/ - PromotePreviewModal.vue - views/ - EnvironmentsPromotion.vue -tests/ - Unit/Service/PromotionServiceTest.php - Integration/PromotionIntegrationTest.php -``` - -## Seed Data - -### Schema: `environment` -| Field | Object 1 | Object 2 | -|-------|----------|----------| -| slug | `local` | `acceptance` | -| name | Local | Acceptance | -| role | source | target | -| sourceRef | *(seeded `source` object, type: api, location: `https://acceptance.example.org`)* | *(same convention)* | -| description | This instance | Acceptance environment for pre-production promotion | - -**Related items per object:** none (no files/notes/tasks/contacts — environments -are configuration metadata, not content objects). - -### Schema: `promotion_audit` -No seed rows — append-only log schema, populated only by real promotions -(consistent with `call_log`/`job_log`, which also ship with zero seed rows). - -## Trade-offs -- Reusing `CallService::call()` for promotion dispatch means promotion - inherits Source-call semantics (retry, rate limit, CallLog) that were - designed for arbitrary external APIs, not specifically for - instance-to-instance Integriq calls — accepted because the - alternative (new dispatch code) duplicates a large, already-hardened - pipeline for a narrower use case. -- Diff preview requires a live round-trip to the target environment before - every promotion attempt (no offline/cached diff) — accepted because a - stale local diff could show a false "no collisions" and silently - overwrite something created on B after the last preview. -- `credentialRefsNeedingRebind` is computed by scanning the export JSON - client-side rather than by extending `ConfigurationImportPreviewService` - with this concern — accepted (per discovery.md) because REQ-007's preview - is scoped to slug/id resolution, not credential-broker semantics, and - bolting broker-awareness onto that service would blur its single - responsibility. - -## Open Questions -- Should `environment.role` (`source`/`target`/`both`) be enforced at - promotion time (reject promoting FROM an environment whose local record - has `role: target` only), or is it purely descriptive/UI-filtering? - Deferred to tasks.md implementation; default behaviour treats `role` as - UI-filtering only (any environment can technically be promoted to/from, - matching how Sources aren't role-locked today either). diff --git a/openspec/changes/environments-and-promotion/discovery.md b/openspec/changes/environments-and-promotion/discovery.md deleted file mode 100644 index 6e44bb55b..000000000 --- a/openspec/changes/environments-and-promotion/discovery.md +++ /dev/null @@ -1,103 +0,0 @@ -# Discovery: environments-and-promotion - -## Question -The context brief calls for "promote = export from environment A, import into -environment B" and "credential re-binding via the credential broker." Two -things needed verifying against HEAD before design could proceed: (1) does -Integriq have ANY existing outbound-call capability suitable for -reaching a *different* Integriq instance's API, or would promotion -require a brand-new HTTP client; and (2) does the existing export/import -pipeline already do anything with `credentialRef`-shaped values, or is -re-binding entirely new ground. - -## Approach Taken -Read `lib/Service/ConfigurationService.php`, `lib/Service/ -ConfigurationImportPreviewService.php`, `lib/Service/ConfigurationHandlers/ -SourceHandler.php`, `lib/Service/Security/SensitiveFieldRegistry.php`, -`lib/Service/BrokeredCallService.php`, `lib/Service/CallService.php` -(`call()` signature), `lib/Controller/ConfigurationController.php`, and -`appinfo/routes.php` at HEAD. Cross-checked against the `configuration-export-import` -spec (status: done) and the `source-broker-credentials` change referenced -from `BrokeredCallService`'s docblocks. - -## Findings -- **Import/preview endpoints already exist and are routed**: `POST - /api/configurations/import/preview` (REQ-007) and `POST - /api/configurations/import` (REQ-008) are live, gated by - `ActionAuthService::requireAction()` with `configuration.import`. A remote - instance can already be pushed a configuration document by any caller that - can authenticate to it — promotion does not need a new import/diff - algorithm, only a way to REACH that endpoint on the target. -- **`CallService::call(ObjectEntity $source, ...)` is the only outbound HTTP - path in the app**, and it is hard-bound to a `source`-schema `ObjectEntity` - (`$sourceData = $source->getObject()`, reads `location`, `configuration`, - drives CallLog/retry/rate-limit/redaction). There is no generic - "make an authenticated HTTP call" utility outside the Source abstraction. - `BrokeredCallService` layers `credentialRef` proxying and app-side - injection on top of exactly this same Source-shaped configuration - (`configuration.authentication.credentialRef`). -- **Consequence**: the cheapest, most reuse-faithful way to model "reach - environment B's API" is to make each `environment` object point at an - ordinary `source`-schema object (`type: "api"`, `location` = the target's - base URL, `configuration.authentication.credentialRef` = the credential - used to authenticate to it). Dispatching a promotion call then becomes a - normal `CallService::call()` invocation — CallLog auditing, retry, - rate-limiting, and `BrokeredCallService`'s broker-backed credential - resolution all apply with zero new code. Inventing a parallel - "EnvironmentClientService" with its own Guzzle client would duplicate all - of that. -- **`credentialRef` is NOT touched by the existing export/import pipeline**: - grepped `lib/Service/ConfigurationHandlers/*.php` and `SensitiveFieldRegistry.php` - for `credentialRef`/`credentialId`/`credentialName` — zero matches. - `SensitiveFieldRegistry::SECRET_NAME_PATTERN` does not match `credentialId` - or `credentialName` as key names (no substring overlap with `token|key| - secret|password|...|auth|...`), so a Source's `credentialRef` placeholder - survives export completely unredacted and unresolved — it is exported - verbatim as `{"credentialRef": {"credentialId": ""}}` (or - `credentialName`). REQ-004's id↔slug translation also does not touch it - (its vocabulary is `targetId`/`sourceId`/`inputMapping`/`outputMapping`/ - `rules[]`/nested `Id` keys — not `authentication`). This means a - naive promotion today would carry a source-environment-specific - credential UUID straight into the target environment, where it almost - certainly does not resolve to any credential (broker credentials are - per-instance/per-owner) — silently breaking the Source on first use rather - than failing loudly at promotion time. -- **REQ-009's `credentialsNeedingReentry` is a different, narrower thing**: - it flags the top-level fields `SourceHandler::export()` strips outright - (`apikey`, `secret`, `username`, `password`, `jwt`, ...) — it says nothing - about a `credentialRef` placeholder, because that field isn't stripped at - all (it's not in `SourceHandler`'s `unset()` list and doesn't match - `SensitiveFieldRegistry`). This change needs its own, - additional classification bucket for `credentialRef` re-binding; it cannot - reuse REQ-009's bucket as-is. - -## Recommendation -Model `environment` as OR-object metadata that WRAPS an existing `source` -object rather than inventing new connectivity plumbing, and reuse -`CallService::call()` (with its existing `BrokeredCallService` credentialRef -proxy path) to dispatch promotion's remote preview/import calls. Reuse the -target's existing `/api/configurations/import/preview` and `/api/ -configurations/import` endpoints unchanged. Add a dedicated, -promotion-specific `credentialRefsNeedingRebind` preview bucket — computed -client-side in the new `PromotionService` by scanning the exported document -for `credentialRef` placeholders (same detection logic as -`BrokeredCallService::containsPlaceholder()`/`isPlaceholder()`, applied to -already-exported JSON rather than a live source object) — since neither -REQ-004's translation nor REQ-009's reentry flag covers this case. - -## Risks Uncovered -- If the target environment's `source`-schema object (the one an - `environment` wraps) is itself misconfigured (wrong `location`, expired - credentialRef), promotion calls fail the same way any broken Source call - fails today (synthetic 409/403/502 CallLog) — acceptable and already - well-tested behaviour, not a new failure mode to design around. -- `CallService::call()` persists a `CallLog` for every promotion dispatch. - This is desirable (existing infra for free) but means promotion preview - and import calls are visible in the Logs UI as regular Source calls, not - labelled as "promotion" calls unless the `promotion_audit` object - cross-references the CallLog id — design.md should decide whether to - store that cross-reference. - -## Next Steps -Proceed to design.md and specs with the environment-wraps-a-Source model and -the promotion-specific credentialRef rebind bucket as settled decisions. diff --git a/openspec/changes/environments-and-promotion/migration.md b/openspec/changes/environments-and-promotion/migration.md deleted file mode 100644 index bfe456b67..000000000 --- a/openspec/changes/environments-and-promotion/migration.md +++ /dev/null @@ -1,99 +0,0 @@ -# Migration: environments-and-promotion - -## Current State -The `openconnector` OpenRegister register (`lib/Settings/integriq_register.json`, -per `openconnector-register-schema`) declares 15 schemas (`source`, -`endpoint`, `mapping`, `rule`, `job`, `synchronization`, the 4 log schemas, -etc.) but has no `environment` or `promotion_audit` schema. There are no -`oc_openconnector_environment*` or `oc_openconnector_promotion*` SQL tables -— every Integriq entity is a generic OpenRegister object, not a -bespoke table, so this change adds NO new PostgreSQL tables or columns of -its own. - -## Target State -Two new schemas exist inside the `openconnector` register: -- `environment` (mutable config schema): `name`, `slug`, `role` - (`source`|`target`|`both`), `sourceRef` (UUID `$ref` → `source`), - `description`. -- `promotion_audit` (append-only, immutable log schema, matching - `call_log`/`job_log`'s `appendOnly: true`/`immutable: true` + - `x-openregister-archival` retention convention): `actorUid`, - `configurationId`, `fromEnvironmentSlug`, `toEnvironmentSlug`, - `startedAt`, `completedAt`, `outcome`, `previewSummary`, - `credentialRebindCount`, `callLogId`. - -Both are added via a per-change register fragment -(`lib/Settings/register.d/environments-and-promotion.json`, an OpenAPI -`components.schemas` fragment), per ADR-037's "each change adds its own -`.json` instead of editing `integriq_register.json`" rule -(`lib/Settings/register.d/README.md`) — the same mechanism every other -recent Integriq schema addition uses (e.g. -`register.d/eudi-wallet-credential-issuance.json`). - -## Migration Class -**This change does NOT introduce a Nextcloud `IMigrationStep`/`changeSchema()` -class.** Integriq's schema additions are NOT applied via the standard -Nextcloud DB-migration framework — verified against HEAD -(`lib/Repair/InitializeRegister.php`): the register descriptor + its -`register.d/*.json` fragments are imported into OpenRegister via OR's own -`ConfigurationService::importFromApp()`, invoked by the `InitializeRegister` -`IRepairStep` wired in `appinfo/info.xml` under both `` and -``. This exists specifically because a `postSchemaChange()` -Nextcloud migration runs before peer apps' (OpenRegister's) autoloaders are -guaranteed available on a fresh `occ app:enable`, whereas an `IRepairStep` -runs after all enabled apps are bootstrapped. - -``` -No Version*.php migration class. Schema delivery mechanism: -File: lib/Settings/register.d/environments-and-promotion.json (new) -Repair step: lib/Repair/InitializeRegister.php (existing, unmodified — already - wired to import every register.d/*.json fragment; requires no code change, - only the new fragment file) -Idempotency: OR's importFromApp() short-circuits on the descriptor's `version` - field, exactly as it does today for the other 15 schemas. -``` - -## Migration Steps -1. Add `lib/Settings/register.d/environments-and-promotion.json` declaring - the `environment` and `promotion_audit` schemas (OpenAPI - `components.schemas` + `x-openregister` annotations), following the - existing fragment format (see any file already under `register.d/`). -2. No change to `InitializeRegister.php` — it already merges every - `register.d/*.json` fragment into the register descriptor before calling - `importFromApp()`. -3. On `occ app:enable integriq` (fresh install) or `occ upgrade` - (existing install), the repair step runs automatically and creates the - two new schemas inside the existing `openconnector` register — no - separate register is created. -4. Seed data (design.md's Seed Data section: two `environment` objects, - `local` and `acceptance`, each referencing a seeded `source` object) is - created the same way existing seed objects are — via the app's existing - seed-loading path (`lib/sources.seed.json`-style convention), not a - migration step. - -## Data Impact -Zero rows affected on existing schemas — this migration is purely additive -(two new, initially-empty schemas). No existing `source`, `endpoint`, -`mapping`, `rule`, `job`, or `synchronization` object is read, written, or -reshaped. Safe to run on live data; `promotion_audit` and `environment` -start empty (aside from seed data) on every install. - -## Rollback Procedure -Remove `lib/Settings/register.d/environments-and-promotion.json` and -redeploy. The two schemas remain declared inside OpenRegister (OR does not -retroactively delete schemas on a descriptor rollback) but become -unreachable from the Integriq UI/API once the routes and controller -are also rolled back (this change is deployed as one unit — see -proposal.md's Rollback Strategy). No SQL rollback is needed since no SQL -schema changed. - -## Validation -- `occ app:enable integriq` on a fresh instance completes without - error and `InitializeRegister`'s repair-step log line reports the - `environment` and `promotion_audit` schemas among the imported set. -- `GET /api/environments` (new) returns the two seeded environments - (`local`, `acceptance`) with `HTTP 200` on a fresh install. -- Creating a `promotion_audit` object directly via the OpenRegister object - API and then attempting to `PUT`/`DELETE` it fails with OR's - `appendOnly`/`immutable` enforcement, identically to an existing - `call_log` object. diff --git a/openspec/changes/environments-and-promotion/proposal.md b/openspec/changes/environments-and-promotion/proposal.md index cf3065fe7..10953d364 100644 --- a/openspec/changes/environments-and-promotion/proposal.md +++ b/openspec/changes/environments-and-promotion/proposal.md @@ -1,153 +1,41 @@ -# Proposal: environments-and-promotion - -## Summary -This change adds first-class named environments (e.g. staging, production) and a -promotion workflow to Integriq: promoting a configuration group means -exporting it from the local instance via the existing `ConfigurationService` -and pushing it into a registered target environment's existing import -endpoints, with a pre-promotion diff preview, explicit credential re-binding -(never secret copying) via the OpenRegister credential broker, and an -append-only promotion audit log. It builds entirely on the already-merged -configuration export/import substrate (slug translation, credential -redaction) and the credential broker (`source-broker-credentials`) — nothing -in either is forked. - -## Motivation -n8n gates environment promotion behind its paid Enterprise tier; Workato -sells this as "Recipe Lifecycle Management." Integriq already has the -hard parts — slug-referenced export/import, credential redaction, an import -preview endpoint, and a `credentialRef`-based credential broker — but no -concept of a *named* target environment, no automated push between -environments, no pre-promotion diff, and no audit trail of who promoted what, -from where, to where, and when. Shipping this open under EUPL is a -Common-Ground procurement wedge: government customers evaluating n8n/Workato -alternatives can get environment promotion without an enterprise license. -Codeberg issue #155. - -## Affected Projects -- [x] Project: `integriq` — new `environment` and `promotion_audit` - OpenRegister schemas, `PromotionService`, `PromotionController` + routes, - new `environment.manage` / `environment.promote` ADR-023 action keys, and - an Environments & Promotion manifest-v2 UI page. - -## Scope - -### In Scope -1. An `environment` OpenRegister object schema (name, slug, role, and a - `sourceRef` pointing at an existing `source`-schema object of - `type: "api"` that describes how to reach that environment's Integriq - API — reusing the Source schema's existing `location` + - `configuration.authentication.credentialRef` shape instead of inventing a - new connection-descriptor format). -2. A `PromotionService` that: (a) calls the existing, unmodified - `ConfigurationService::exportConfiguration()` locally; (b) dispatches the - exported document to the target environment's existing, unmodified - `POST /api/configurations/import/preview` and `POST - /api/configurations/import` endpoints (REQ-007/REQ-008) via the existing - `CallService::call()` outbound pipeline, using the target environment's - `sourceRef` Source — so retry, rate-limiting, CallLog auditing, and - `credentialRef` broker resolution for reaching the target are all reused - unchanged, not reimplemented. -3. Explicit credential re-binding: any `Source` in the exported document whose - `configuration.authentication` carries a `credentialRef` placeholder is - surfaced by the preview as needing an operator-supplied re-binding - (`credentialId`/`credentialName` valid in the TARGET environment's broker) - before the promotion is confirmed. `credentialRef` values are never - resolved to plaintext and never copied between environments — only the - reference is rewritten. -4. A diff preview step before promotion, reusing the target environment's - existing import-preview response (creates/updates/collisions/unresolved - references/credentials-needing-reentry) plus a promotion-specific - `credentialRefsNeedingRebind` bucket computed client-side from the - exported document. -5. An append-only, immutable `promotion_audit` OpenRegister object schema - (who, configuration id, from-environment, to-environment, timestamp, - preview summary, outcome) written after every promotion attempt, - following the same `appendOnly`/`immutable` convention as the existing - `call_log`/`job_log` schemas. -6. An Environments & Promotion manifest-v2 UI page: environment CRUD list and - a promote flow (select configuration group → select target environment → - review diff + credential rebind prompts → confirm). -7. Unit tests for environment metadata and credential-rebind resolution; - integration tests exporting from environment A and importing into - environment B, asserting `credentialRef`s are re-bound, not copied as - secrets. - -### Out of Scope -- Git-backed configuration storage / GitOps workflows — a follow-up change. -- Automatic, unattended promotion (e.g. on a schedule or CI trigger) — this - change is operator-confirmed only, matching REQ-008's existing - confirmation requirement. -- Multi-hop promotion chains (A→B→C in one operation) — one promotion is - always a single source→target pair. - -## Approach -Reuse, don't fork. Environment connectivity is modelled as an existing -`source`-schema object so the existing `CallService`/`BrokeredCallService` -dispatch pipeline (auth, retry, CallLog, redaction) carries promotion traffic -without new HTTP client code. The diff preview is the existing target-side -`/api/configurations/import/preview` endpoint, invoked remotely instead of -in-process — no new diff algorithm. Credential re-binding is a thin -preprocessing/postprocessing layer in `PromotionService` around the -unmodified export/import pipeline: it never touches `ConfigurationHandlers` -or `SensitiveFieldRegistry`. See design.md for the full architecture and the -credential-rebinding decision. - -## New Dependencies -None — reuses `ConfigurationService`, `ConfigurationImportPreviewService`, -`CallService`, `BrokeredCallService`, and OpenRegister's -`CredentialBrokerService`, all already present. - -## Impact -- New: `lib/Service/PromotionService.php`, `lib/Controller/PromotionController.php`, - `lib/Settings/integriq_register.json` additions (`environment`, - `promotion_audit` schemas), `lib/actions.seed.json` additions, `appinfo/routes.php` - additions, a new manifest-v2 page + Vue components under `src/`. -- Unchanged: `ConfigurationService`, `ConfigurationHandlers/*`, - `ConfigurationImportPreviewService`, `SensitiveFieldRegistry`, - `BrokeredCallService`, `CallService`. - -## Cross-Project Dependencies -Depends on OpenRegister's `CredentialBrokerService` (already a hard runtime -dependency per `openconnector-direct-or-usage`) for resolving a target -environment's connection credential and for validating operator-supplied -credential re-bindings. No other apps consume this change. - -## Risks - -### Risk 1: Target environment API version skew -**Severity:** Medium — **Mitigation:** `PromotionService` calls the target's -`/api/configurations/import/preview` and `/api/configurations/import` -endpoints exactly as documented in `configuration-export-import` (REQ-007/ -REQ-008); a target running an older Integriq without those routes -returns 404, surfaced to the operator as a promotion failure with an -actionable message, not a silent partial write. - -### Risk 2: Operator promotes with an unresolved credentialRef -**Severity:** Medium — **Mitigation:** the diff preview's -`credentialRefsNeedingRebind` bucket is a blocking warning; `import` on the -target still enforces REQ-008's `confirmed: true` gate, and an unrebound -`credentialRef` that does not resolve on the target simply fails at the -target's own Source-auth guard (`BrokeredCallService`) the first time that -Source is used — never at promotion time with a leaked secret, because no -secret ever transits the promotion call. - -### Risk 3: Promotion audit log grows unbounded -**Severity:** Low — **Mitigation:** `promotion_audit` follows the existing -log-schema retention convention (`x-openregister-archival`), matching -`call_log`/`job_log`. - -## Rollback Strategy -The new schemas, service, controller, routes, and UI page are additive. To -roll back, remove the routes and hide the manifest page; the `environment` -and `promotion_audit` OpenRegister objects remain harmless, inert data. -`ConfigurationService` and `CredentialBrokerService` are never modified, so -rollback carries zero risk to existing export/import or brokered-call -functionality. - -## Open Questions -- Should a promotion be retryable/resumable if the target import partially - succeeds (e.g. sources written, endpoints fail)? Deferred to design.md; - current default follows `importConfiguration()`'s existing per-type - best-effort behaviour (unchanged), recorded as a known limitation in the - audit entry rather than solved with new rollback machinery. +--- +kind: spec-only +depends_on: [] +--- + +# Proposal: environments-and-promotion (superseded — retired 2026-09-02) + +This directory double-counted a change that had already shipped. The +environments and promotion feature was implemented and archived on +2026-07-15 (`archive/2026-07-15-environments-and-promotion`, 25/37 tasks +checked with per-task evidence), yet this live copy was resurrected at +0/35: the openconnector→integriq rename applied to the prose, the evidence +notes stripped, every box reset. The machinery exists at HEAD: +`lib/Controller/EnvironmentController.php`, +`lib/Controller/PromotionController.php`, +`lib/Service/EnvironmentService.php`, `lib/Service/PromotionService.php`, +the `lib/Settings/register.d/environments-and-promotion.json` fragment, the +manifest-declared Environments page +(`src/manifest.d/environments-and-promotion.json`, page id +`src-environments`), `src/modals/PromotePreviewModal.vue`, and the +environment/promotion routes in `appinfo/routes.php`. + +No live `@spec` tags point into this directory. + +## Disposition of the original scope + +| Original scope | Where it went | +| --- | --- | +| `environment` schema, promotion service with preview/confirm, environment + promotion controllers and routes, Environments page (manifest fragment), promote-preview modal | **Already shipped and archived**: `archive/2026-07-15-environments-and-promotion` (25/37 boxes checked), code at HEAD | +| Residual verification: live `occ` install/fragment-merge run, Newman for `/api/environments*` and `/api/promotions*`, browser tests, feature docs, screenshot | Open, and honestly unticked in the archived twin (no live instance in that session; each open box carries its reason). Same shape as `approvals-verification-pack`; pick up in a verification pass, not by resurrecting this change | + +## Sequencing + +Nothing remains to implement from this change directly. The residual +live-instance verification and docs belong to a verification-pack-style +follow-up. + +## Archival + +This directory is retired in place (not moved or renamed) to keep the diff +reviewable; archive it via the normal flow at the next sweep. diff --git a/openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md b/openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md deleted file mode 100644 index a80d59cd2..000000000 --- a/openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md +++ /dev/null @@ -1,51 +0,0 @@ -# configuration-export-import Specification (delta: environments-and-promotion) - -## ADDED Requirements - -### Requirement: credentialRef authentication placeholders pass through export and import unresolved and untranslated (REQ-010) - -The system SHALL export and import a Source's -`configuration.authentication.credentialRef` placeholder (the reference -shape `{"credentialId": ""}` or `{"credentialName": ""}` used by -the credential broker per `http-call-engine`'s brokered-dispatch -requirements) byte-for-byte unchanged: `SensitiveFieldRegistry::redactArray()` -SHALL NOT redact the `credentialId` or `credentialName` leaf keys (neither -matches `SECRET_NAME_PATTERN` nor `EXACT_MATCH_NAMES`), and REQ-004's -id↔slug translation SHALL NOT rewrite them (its reference-field vocabulary — -`targetId`/`sourceId`/`inputMapping`/`outputMapping`/`rules[]`/nested -`Id` keys — does not include `authentication`). An exported document's -`credentialRef` therefore always carries the SOURCE environment's own -credential id or name verbatim; it is the responsibility of any consumer -that moves the document between environments (see the -`environments-and-promotion` capability) to re-bind it before or during -import into a different environment — `ConfigurationService` and its -handlers themselves perform no environment-awareness or rebinding. - -Notes: `ConfigurationImportPreviewService::missingCredentialFields()` -(REQ-009) checks only the fixed `CREDENTIAL_FIELDS` list -(`apikey`/`secret`/`username`/`password`/`jwt`/`authorizationHeader`/ -`authenticationConfig`) and has no awareness of `credentialRef` — a -credentialRef-authenticated Source, which never had any of those fields to -begin with, is therefore always reported as "needs re-entry" for all of -them even though nothing was stripped from it. This is a pre-existing, -narrow imprecision in REQ-009's classification (harmless: the operator -re-checks a Source that in fact needs no re-entry) and is not changed by -this requirement; it is recorded here because `environments-and-promotion` -introduces the correctly-scoped `credentialRefsNeedingRebind` classification -specifically to avoid relying on REQ-009 for this case. - -#### Scenario: A Source's credentialRef is not redacted on export -- GIVEN a Source whose `configuration` contains `{"authentication": {"credentialRef": {"credentialId": "550e8400-e29b-41d4-a716-446655440000"}}}` -- WHEN the Source is exported via `SourceHandler::export()` -- THEN the exported `configuration.authentication.credentialRef.credentialId` value is unchanged (`550e8400-e29b-41d4-a716-446655440000`), not `***REDACTED***` - -#### Scenario: Importing a credentialRef that does not resolve on the target does not block the write -- GIVEN an OAS document containing a Source with `configuration.authentication.credentialRef.credentialId` set to a UUID that does not correspond to any credential broker entry on the importing environment -- WHEN the document is imported via `importConfiguration()` -- THEN the Source object is created or updated exactly as REQ-003 describes, with the `credentialRef` value written verbatim -- AND no exception is thrown at import time — the dangling reference only surfaces later, when that Source is actually dispatched and `BrokeredCallService` fails to resolve the credential - -#### Scenario: credentialRef translation is absent from the id/slug mapping vocabulary -- GIVEN a Source whose `configuration.authentication.credentialRef.credentialName` is set to `"prod-api-key"` -- WHEN the Source is exported and then imported into an environment where a `source`-type or `register`/`schema` slug map entry happens to also be named `"prod-api-key"` -- THEN the `credentialRef.credentialName` value is NOT rewritten by REQ-004's translation (it is not a member of the translated field set), and remains the literal string `"prod-api-key"` on both export and import diff --git a/openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md b/openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md deleted file mode 100644 index bcba724f1..000000000 --- a/openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md +++ /dev/null @@ -1,231 +0,0 @@ ---- -status: planned ---- - -# environments-and-promotion Specification - -**Status**: planned -**Scope**: integriq -**OpenSpec changes**: -- environments-and-promotion - -## Purpose - -Integriq's `configuration-export-import` capability already moves a -coherent set of Sources/Endpoints/Mappings/Rules/Jobs/Synchronizations -between instances via a slug-referenced, credential-redacted OAS document, -and `source-broker-credentials` already lets a Source authenticate through a -`credentialRef` resolved by the OpenRegister credential broker instead of an -embedded secret. Neither capability names *environments* as first-class -objects, neither automates the export-then-import round trip between two -named environments, and neither surfaces a diff before writing or an audit -trail of who promoted what, from where, to where, and when. This capability -adds named `environment` metadata, a `PromotionService` that reuses -`ConfigurationService`, `ConfigurationImportPreviewService`'s existing -routed endpoints, and `CallService`/`BrokeredCallService`'s existing -dispatch pipeline to push a configuration group from one environment into -another, with mandatory diff preview, explicit credential re-binding -(references only, never secrets), and an append-only promotion audit log. -See `configuration-export-import` (REQ-001–REQ-010) and -`source-broker-credentials`'s `http-call-engine` delta for the underlying, -unmodified primitives this capability builds on. - -## ADDED Requirements - -### Requirement: Named environments are OpenRegister objects that wrap an existing Source for connectivity (REQ-001) - -The system SHALL persist named environments as `environment`-schema -OpenRegister objects in the `openconnector` register (`name`, `slug`, -`role` of `source`, `target`, or `both`, `description`, `sourceRef`). The -system SHALL NOT store environment connectivity as a new credential format, -a new HTTP client configuration, or an `IAppConfig` value: `sourceRef` -SHALL reference an existing `source`-schema object (`type: "api"`) whose -`location` and `configuration.authentication.credentialRef` describe how to -reach that environment's Integriq API, so that dispatching a call to an -environment reuses `CallService::call()` and, when the referenced Source -carries a `credentialRef`, `BrokeredCallService`'s existing broker -resolution — unchanged and unforked. - -#### Scenario: Creating an environment requires an existing Source reference -- GIVEN an operator with the `environment.manage` action permission -- WHEN they create an `environment` object with `slug: "acceptance"`, `role: "target"`, and `sourceRef` pointing at an existing `type: "api"` Source -- THEN the `environment` object is created in the `openconnector` register -- AND no new credential or connection material is stored on the `environment` object itself - -#### Scenario: An environment without a resolvable sourceRef cannot be used as a promotion target -- GIVEN an `environment` object whose `sourceRef` no longer resolves to an existing Source -- WHEN an operator attempts to preview or confirm a promotion to that environment -- THEN the request is rejected with an actionable error naming the missing `sourceRef` -- AND no export or remote call is attempted - -### Requirement: Promotion exports locally, unchanged, and dispatches to the target's existing import endpoints (REQ-002) - -The system SHALL implement promotion as: (1) calling the existing, unmodified -`ConfigurationService::exportConfiguration()` on the local instance to -produce the OAS document for the requested configuration id; (2) dispatching -that document to the target environment's own, unmodified `POST -/api/configurations/import/preview` (preview) or `POST -/api/configurations/import` (confirmed) endpoint via `CallService::call()` -against the target environment's `sourceRef` Source. The system SHALL NOT -reimplement export, slug translation, or redaction logic inside the -promotion path — `ConfigurationService` and its handlers remain the single -source of truth for both. - -#### Scenario: Promotion reuses the unmodified export pipeline -- GIVEN a configuration group `cfg-1` containing one Source and one Endpoint -- WHEN an operator promotes `cfg-1` from the local environment to a registered target environment -- THEN the system calls `ConfigurationService::exportConfiguration('cfg-1')` unchanged to build the document -- AND the redaction and slug-translation behaviour documented in `configuration-export-import` (REQ-001–REQ-005) applies identically to a promotion export as to a manual UI export - -#### Scenario: Promotion dispatch reuses CallService against the target's environment Source -- GIVEN a target environment whose `sourceRef` Source has `location: "https://acceptance.example.org"` -- WHEN a promotion is confirmed -- THEN the system dispatches the import call via `CallService::call()` using that Source -- AND the resulting `CallLog` is created exactly as it would be for any other Source call against that Source - -### Requirement: Diff preview merges the target's existing preview response with a credential-rebind classification (REQ-003) - -The system SHALL, before any promotion write occurs, retrieve a preview by -calling the target environment's existing `POST -/api/configurations/import/preview` endpoint (`configuration-export-import` -REQ-007, unmodified) with the exported document, and SHALL merge that -response with a `credentialRefsNeedingRebind` array computed locally by -scanning the exported document for `{"credentialRef": {...}}` placeholders -(REQ-004 below). The system SHALL NOT compute creates/updates/collisions or -unresolved slug references itself — that classification SHALL always come -from the target environment's own preview response, since only the target -knows its own object state. - -#### Scenario: Preview reflects the target's own creates/updates/collisions classification -- GIVEN a Source in the export document whose slug already exists on the target environment, and a second Source whose slug does not -- WHEN the promotion preview is requested -- THEN the response's `updates` array contains the first Source and `creates` contains the second, exactly as the target's own `/api/configurations/import/preview` response would classify them - -#### Scenario: Preview is required before a promotion can be confirmed -- GIVEN a valid configuration id and target environment -- WHEN an operator attempts to confirm a promotion without having first retrieved a preview in the same request flow -- THEN the system still computes the preview internally as part of the confirm call (mirroring REQ-005's `confirmed: true` requirement) before dispatching the write, so a promotion can never write without an equivalent preview having been computed - -### Requirement: credentialRef placeholders are re-bound per target environment, never resolved to a secret (REQ-004) - -The system SHALL detect every `{"credentialRef": {"credentialId": ...}}` or -`{"credentialRef": {"credentialName": ...}}` placeholder inside a promoted -Source's `configuration.authentication` (the same shape -`BrokeredCallService::isPlaceholder()` detects) and SHALL list each one under -the preview's `credentialRefsNeedingRebind` array, naming the Source slug and -field path. The system SHALL rewrite a flagged placeholder in the outgoing -document only when the operator supplies an explicit replacement reference -(`credentialId` or `credentialName` valid on the target) as part of the -promotion request; an un-rebound placeholder SHALL be sent to the target -verbatim (carrying the source environment's own reference), never silently -dropped or defaulted. The system SHALL NOT, at any point during promotion, -call any credential-broker method that returns a plaintext secret value — -re-binding SHALL operate on reference strings only. - -#### Scenario: A Source's credentialRef is flagged for rebinding -- GIVEN a Source in the export document with `configuration.authentication.credentialRef.credentialId` set to a UUID from the source environment's credential broker -- WHEN the promotion preview is computed -- THEN the response's `credentialRefsNeedingRebind` array contains that Source's slug and the field `configuration.authentication.credentialRef` - -#### Scenario: An operator-supplied rebinding replaces the reference before the target ever sees the original -- GIVEN the promotion request includes `credentialBindings: [{"sourceSlug": "my-api-source", "credentialName": "prod-api-key"}]` for a flagged Source -- WHEN the promotion is confirmed -- THEN the document dispatched to the target environment's import endpoint contains `configuration.authentication.credentialRef.credentialName = "prod-api-key"` for that Source, not the original source-environment credentialId -- AND at no point does the system read or transmit the plaintext secret behind either reference - -#### Scenario: An un-rebound credentialRef is sent verbatim, not resolved or dropped -- GIVEN a flagged Source with no corresponding entry in the promotion request's `credentialBindings` -- WHEN the promotion is confirmed -- THEN the document dispatched to the target contains the original, unmodified `credentialRef` value -- AND the target's own Source-auth guard (not the promotion path) is what eventually fails when that Source is eventually called against a credential that does not exist on the target - -### Requirement: Promotion requires explicit confirmation and the same action-matrix authorization as export/import (REQ-005) - -The system SHALL require `confirmed: true` on the confirm request and SHALL -reject the request with HTTP 400 if absent, mirroring -`configuration-export-import` REQ-008. Both the preview and confirm -promotion endpoints SHALL be gated by `ActionAuthService::requireAction()` -with a new `environment.promote` action key seeded `["admin"]` in -`lib/actions.seed.json`; environment CRUD endpoints SHALL be gated by a -separate `environment.manage` action key, also seeded `["admin"]`. - -#### Scenario: Promotion without confirmation is rejected -- GIVEN a valid configuration id and target environment -- WHEN the confirm endpoint is called with `confirmed` omitted or `false` -- THEN the response is HTTP 400 -- AND no export is dispatched to the target and no `promotion_audit` object is written - -#### Scenario: A user without the environment.promote action permission cannot promote -- GIVEN a non-admin user whose groups are not mapped to `environment.promote` in the action matrix -- WHEN that user calls the promotion preview or confirm endpoint -- THEN the request is rejected with `OCSForbiddenException` before any export or remote call occurs -- @e2e exclude API-level action-matrix denial — covered by PHPUnit `PromotionControllerTest::testPromoteDeniedForUnmappedNonAdmin` - -### Requirement: Every promotion attempt is recorded in an append-only promotion audit log (REQ-006) - -The system SHALL write one `promotion_audit` OpenRegister object per -confirmed promotion attempt (success or failure), recording the acting -user, the configuration id, the source and target environment slugs, start -and completion timestamps, the outcome, a preview summary (counts and -slugs only — never entity payloads or credential values), the number of -credential rebindings applied, and the id of the `CallLog` created by the -underlying dispatch. The `promotion_audit` schema SHALL be declared -`appendOnly: true` and `immutable: true`, following the same convention as -the existing `call_log`/`job_log` schemas. - -#### Scenario: A successful promotion is audited -- GIVEN a confirmed promotion of `cfg-1` from `local` to `acceptance` that writes two Sources and one Endpoint -- WHEN the promotion completes -- THEN a `promotion_audit` object is created with `outcome: "success"`, `fromEnvironmentSlug: "local"`, `toEnvironmentSlug: "acceptance"`, and a `previewSummary` reflecting the two creates/updates -- AND the object contains no credential values or full entity payloads - -#### Scenario: A failed promotion is still audited -- GIVEN a confirmed promotion whose dispatch to the target fails (e.g. the target returns 404 because it runs an older Integriq without the import routes) -- WHEN the promotion attempt completes -- THEN a `promotion_audit` object is created with `outcome: "failed"` and a message identifying the failure -- AND no partial `written` summary is fabricated — only what the target actually confirmed, if anything, is recorded - -#### Scenario: promotion_audit objects cannot be edited or deleted after creation -- GIVEN an existing `promotion_audit` object -- WHEN any caller attempts to update or delete it via the OpenRegister object API -- THEN the write is rejected by OpenRegister's `appendOnly`/`immutable` schema enforcement, identically to how `call_log`/`job_log` objects are protected today - -## Non-Functional Requirements - -- **Performance:** A promotion preview SHALL complete within the same order - of magnitude as a manual export (REQ-001's documented O(all entities of - each type) cost) plus one additional network round trip to the target - environment; no new O(n²) behaviour is introduced. -- **Accessibility:** The Environments & Promotion UI page's environment and - target selects MUST use `NcSelect` with an explicit `inputLabel` (WCAG - 2.1 AA 1.3.1/4.1.2), and the promotion confirmation flow MUST live in its - own `NcModal` file, never inlined in a parent component. -- **Internationalization:** All UI strings and API error messages MUST be - in English, matching the project-wide i18n-keys-English convention. - -## Acceptance Criteria - -- [ ] `environment` objects can be created, listed, and reference an - existing `source`-schema object via `sourceRef` -- [ ] A promotion preview merges the target's own REQ-007 response with a - `credentialRefsNeedingRebind` bucket computed locally -- [ ] A `credentialRef` placeholder is never resolved to a plaintext secret - anywhere in the promotion path -- [ ] A confirmed promotion without `confirmed: true` is rejected with 400 -- [ ] Every confirmed promotion attempt (success or failure) produces exactly - one immutable `promotion_audit` object - -## Notes - -- This capability intentionally does NOT modify `ConfigurationService`, - `ConfigurationHandlers/*`, `ConfigurationImportPreviewService`, - `SensitiveFieldRegistry`, `CallService`, or `BrokeredCallService` — see - design.md Decisions 1-4 for the reuse rationale. -- `configuration-export-import` REQ-009's `credentialsNeedingReentry` - bucket and this capability's `credentialRefsNeedingRebind` bucket are - deliberately distinct: REQ-009 covers top-level fields - `SourceHandler::export()` strips outright; this capability covers - `credentialRef` reference placeholders, which are never stripped (see the - `configuration-export-import` delta in this change, REQ-010). -- Git-backed configuration storage / GitOps is explicitly out of scope - (proposal.md) and may build on `environment` metadata in a future change. diff --git a/openspec/changes/environments-and-promotion/tasks.md b/openspec/changes/environments-and-promotion/tasks.md index d7e3f2c9c..7473fae33 100644 --- a/openspec/changes/environments-and-promotion/tasks.md +++ b/openspec/changes/environments-and-promotion/tasks.md @@ -1,135 +1,9 @@ -# Tasks: environments-and-promotion - -## Implementation Tasks - -### Task 1: Declare the environment and promotion_audit schemas via a register.d fragment -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-named-environments-are-openregister-objects-that-wrap-an-existing-source-for-connectivity-req-001` -- **files**: `lib/Settings/register.d/environments-and-promotion.json` -- **acceptance_criteria**: - - GIVEN a fresh `occ app:enable integriq` WHEN `InitializeRegister` runs THEN the `environment` and `promotion_audit` schemas exist in the `openconnector` register - - GIVEN the descriptor fragment WHEN inspected THEN `promotion_audit` declares `appendOnly: true` and `immutable: true`, matching `call_log`/`job_log` -- [ ] Implement -- [ ] Test - -### Task 2: Seed local + acceptance environment objects and their connectivity Sources -- **spec_ref**: `openspec/changes/environments-and-promotion/design.md#seed-data` -- **files**: `lib/environments.seed.json` (new, following `lib/sources.seed.json` convention) -- **acceptance_criteria**: - - GIVEN a fresh install WHEN seed data loads THEN `local` and `acceptance` `environment` objects exist, each with a `sourceRef` pointing at a seeded `type: api` Source -- [ ] Implement -- [ ] Test - -### Task 3: Environment CRUD service, controller, routes, and action keys -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-named-environments-are-openregister-objects-that-wrap-an-existing-source-for-connectivity-req-001` -- **files**: `lib/Service/EnvironmentService.php`, `lib/Controller/EnvironmentController.php`, `appinfo/routes.php`, `lib/actions.seed.json` -- **acceptance_criteria**: - - GIVEN an operator with `environment.manage` WHEN they `POST /api/environments` with a valid `sourceRef` THEN the object is created - - GIVEN an `environment` whose `sourceRef` does not resolve WHEN it is used as a promotion target THEN the request is rejected with an actionable error naming the missing `sourceRef` - - GIVEN a non-admin user without `environment.manage` WHEN they call any environment endpoint THEN `OCSForbiddenException` is returned -- [ ] Implement -- [ ] Test - -### Task 4: PromotionService — local export + credentialRef placeholder scan -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-exports-locally-unchanged-and-dispatches-to-the-targets-existing-import-endpoints-req-002` -- **files**: `lib/Service/PromotionService.php` -- **acceptance_criteria**: - - GIVEN a configuration id WHEN `PromotionService::export()` runs THEN it calls `ConfigurationService::exportConfiguration()` unchanged and returns its document verbatim - - GIVEN an exported document containing a Source with `configuration.authentication.credentialRef` WHEN scanned THEN each placeholder is detected using the same shape `BrokeredCallService::isPlaceholder()` checks -- [ ] Implement -- [ ] Test - -### Task 5: PromotionService — credential rebinding rewrite (reference-only, never plaintext) -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-credentialref-placeholders-are-re-bound-per-target-environment-never-resolved-to-a-secret-req-004` -- **files**: `lib/Service/PromotionService.php` -- **acceptance_criteria**: - - GIVEN a `credentialBindings` entry for a flagged Source WHEN the document is rewritten THEN the target document's `credentialRef` is replaced with the supplied `credentialId`/`credentialName`, never resolved to plaintext - - GIVEN no `credentialBindings` entry for a flagged Source WHEN the document is rewritten THEN the original `credentialRef` is sent verbatim, not dropped or defaulted - - GIVEN this task's code WHEN reviewed THEN it never calls `CredentialBrokerService::resolveInjectable()` or any method that returns a plaintext secret -- [ ] Implement -- [ ] Test - -### Task 6: PromotionService — remote dispatch via CallService against the target's sourceRef -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-exports-locally-unchanged-and-dispatches-to-the-targets-existing-import-endpoints-req-002` -- **files**: `lib/Service/PromotionService.php` -- **acceptance_criteria**: - - GIVEN a target environment's `sourceRef` Source WHEN a preview or confirm call is dispatched THEN it goes through `CallService::call()` against that Source, unmodified - - GIVEN the dispatch WHEN it completes THEN a `CallLog` is created exactly as for any other Source call -- [ ] Implement -- [ ] Test - -### Task 7: PromotionService — merge target preview response with credentialRefsNeedingRebind -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-diff-preview-merges-the-targets-existing-preview-response-with-a-credential-rebind-classification-req-003` -- **files**: `lib/Service/PromotionService.php` -- **acceptance_criteria**: - - GIVEN a target's `/api/configurations/import/preview` response WHEN merged THEN `creates`/`updates`/`collisions`/`unresolvedReferences`/`credentialsNeedingReentry` are passed through unchanged from the target - - GIVEN flagged `credentialRef` placeholders WHEN merged THEN they appear under a `credentialRefsNeedingRebind` array not present in the target's own response -- [ ] Implement -- [ ] Test - -### Task 8: PromotionController — preview and confirm endpoints with confirmation + action gates -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-requires-explicit-confirmation-and-the-same-action-matrix-authorization-as-exportimport-req-005` -- **files**: `lib/Controller/PromotionController.php`, `appinfo/routes.php`, `lib/actions.seed.json` -- **acceptance_criteria**: - - GIVEN `POST /api/promotions` without `confirmed: true` WHEN called THEN HTTP 400 is returned and nothing is dispatched - - GIVEN a user without `environment.promote` WHEN they call preview or confirm THEN `OCSForbiddenException` is returned before any export or remote call -- [ ] Implement -- [ ] Test - -### Task 9: promotion_audit — write append-only audit object after every promotion attempt -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-every-promotion-attempt-is-recorded-in-an-append-only-promotion-audit-log-req-006` -- **files**: `lib/Service/PromotionService.php`, `lib/Controller/PromotionController.php` -- **acceptance_criteria**: - - GIVEN a successful promotion WHEN it completes THEN a `promotion_audit` object is written with `outcome: "success"`, counts-only `previewSummary`, and the dispatch `CallLog` id - - GIVEN a failed promotion (e.g. target returns 404) WHEN the attempt completes THEN a `promotion_audit` object is written with `outcome: "failed"` and no fabricated `written` summary - - GIVEN an existing `promotion_audit` object WHEN a PUT/DELETE is attempted via the OR object API THEN it is rejected by `appendOnly`/`immutable` enforcement -- [ ] Implement -- [ ] Test - -### Task 10: Formalize the credentialRef pass-through contract on configuration-export-import -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md#requirement-credentialref-authentication-placeholders-pass-through-export-and-import-unresolved-and-untranslated-req-010` -- **files**: `tests/Unit/Service/ConfigurationHandlers/SourceHandlerTest.php` (extend), `tests/Unit/Service/ConfigurationServiceTest.php` (extend) -- **acceptance_criteria**: - - GIVEN a Source with a `credentialRef` placeholder WHEN exported THEN the placeholder is byte-for-byte unchanged in the output (regression test pinning existing, previously-undocumented behaviour) - - GIVEN an OAS document with a non-resolving `credentialRef` WHEN imported THEN the write succeeds and the reference is stored verbatim -- [ ] Implement -- [ ] Test - -### Task 11: Environments & Promotion manifest-v2 UI page -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#non-functional-requirements` -- **files**: `src/views/EnvironmentsPromotion.vue`, manifest-v2 page config (per `openconnector-app-manifest`) -- **acceptance_criteria**: - - GIVEN an operator opens the Environments & Promotion page THEN registered environments are listed with CRUD actions - - GIVEN the environment/target select fields THEN each `NcSelect` carries an explicit `inputLabel` -- [ ] Implement -- [ ] Test - -### Task 12: Promote flow — diff preview + credential rebind prompts + confirm, in its own modal -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-diff-preview-merges-the-targets-existing-preview-response-with-a-credential-rebind-classification-req-003` -- **files**: `src/modals/PromotePreviewModal.vue` -- **acceptance_criteria**: - - GIVEN an operator selects a configuration group and target environment WHEN they open the promote flow THEN the diff preview (creates/updates/collisions/credentialRefsNeedingRebind) renders before any confirm button is enabled - - GIVEN the modal markup WHEN inspected THEN it lives entirely in `src/modals/PromotePreviewModal.vue`, never inlined in a parent component -- [ ] Implement -- [ ] Test - -## Verification -- [ ] All tasks checked off -- [ ] `openspec validate` passes -- [ ] Manual testing against acceptance criteria -- [ ] Code review against spec requirements - -## Tests (company-wide ADR-009) - -- [ ] PHPUnit unit tests for new/changed business logic (`tests/Unit/`) — `PromotionServiceTest`, `EnvironmentServiceTest`, credentialRef scan/rebind cases -- [ ] Newman/Postman tests for new/changed API endpoints — `/api/environments*`, `/api/promotions*` -- [ ] Browser tests (Playwright MCP) for UI changes — environment CRUD, promote flow diff preview + confirm -- [ ] All tests pass (`composer test`, `newman run`) - -## Documentation (company-wide ADR-010) - -- [ ] Feature documentation updated in `docs/` — Environments & Promotion page, promotion workflow, credential rebinding -- [ ] Screenshot captured and committed to `docs/images/` - -## i18n (company-wide hydra ADR-007) - -- [ ] Dutch (`nl_NL`) and English (`en_US`) translation strings added for the new UI page, promote flow, and error messages +# Tasks: environments-and-promotion (superseded) + +The original 12-task / 35-checkbox list was removed with the 2026-09-02 +retirement (see proposal.md for the disposition; the list survives in +`archive/2026-07-15-environments-and-promotion/tasks.md`, where 25/37 boxes +are checked with per-task evidence, and in git history). The residual +live-instance verification and docs work is listed there with per-box +reasons and belongs to a verification-pack-style follow-up. There is +nothing to implement from this change directly. diff --git a/openspec/changes/environments-and-promotion/test-plan.md b/openspec/changes/environments-and-promotion/test-plan.md deleted file mode 100644 index 0308b2725..000000000 --- a/openspec/changes/environments-and-promotion/test-plan.md +++ /dev/null @@ -1,163 +0,0 @@ -# Test Plan: environments-and-promotion - -## Test Cases - -### TC-1: Creating an environment requires an existing Source reference -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-named-environments-are-openregister-objects-that-wrap-an-existing-source-for-connectivity-req-001` -- **type**: api -- **preconditions**: An admin session; an existing `type: api` Source object -- **steps**: `POST /api/environments` with `slug: "acceptance"`, `role: "target"`, `sourceRef` = the existing Source's uuid -- **expected result**: HTTP 200/201; the `environment` object is created; no new credential material is stored on it -- **test command**: /test-api - -### TC-2: An environment without a resolvable sourceRef cannot be used as a promotion target -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-named-environments-are-openregister-objects-that-wrap-an-existing-source-for-connectivity-req-001` -- **type**: api -- **preconditions**: An `environment` object whose `sourceRef` uuid has since been deleted -- **steps**: `POST /api/promotions/preview` targeting that environment -- **expected result**: Actionable error naming the missing `sourceRef`; no export or remote call attempted -- **test command**: /test-api - -### TC-3: Promotion reuses the unmodified export pipeline -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-exports-locally-unchanged-and-dispatches-to-the-targets-existing-import-endpoints-req-002` -- **type**: api -- **preconditions**: A configuration group `cfg-1` with one Source (apikey set) and one Endpoint -- **steps**: `POST /api/promotions/preview` for `cfg-1` -- **expected result**: The document underlying the preview reflects REQ-001–REQ-005 export/redaction/slug-translation exactly as a manual `/api/configurations/{id}/export` call would produce -- **test command**: /test-api - -### TC-4: Promotion dispatch reuses CallService against the target's environment Source -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-exports-locally-unchanged-and-dispatches-to-the-targets-existing-import-endpoints-req-002` -- **type**: integration -- **preconditions**: Two Integriq instances (A, B) reachable from each other; `environment` object on A pointing at a Source describing B's API -- **steps**: Confirm a promotion from A to B -- **expected result**: A `CallLog` is created on A for the dispatch, identical in shape to any other Source call's CallLog -- **test command**: /test-api - -### TC-5: Preview reflects the target's own creates/updates/collisions classification -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-diff-preview-merges-the-targets-existing-preview-response-with-a-credential-rebind-classification-req-003` -- **type**: api -- **preconditions**: Target environment already has a Source whose slug matches one Source in the export document; a second Source's slug is new -- **steps**: `POST /api/promotions/preview` -- **expected result**: `updates` contains the first Source, `creates` contains the second — sourced from the target's own `/api/configurations/import/preview` response -- **test command**: /test-api - -### TC-6: Preview is computed internally before every confirmed promotion -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-diff-preview-merges-the-targets-existing-preview-response-with-a-credential-rebind-classification-req-003` -- **type**: api -- **preconditions**: Valid configuration id and target environment -- **steps**: `POST /api/promotions` with `confirmed: true` directly, without a separate prior preview call -- **expected result**: The confirm call still computes an equivalent preview internally before dispatching the write (mirrors REQ-008's `import()` behaviour) -- **test command**: /test-api - -### TC-7: A Source's credentialRef is flagged for rebinding -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-credentialref-placeholders-are-re-bound-per-target-environment-never-resolved-to-a-secret-req-004` -- **type**: api -- **preconditions**: Configuration group containing a Source with `configuration.authentication.credentialRef.credentialId` set -- **steps**: `POST /api/promotions/preview` -- **expected result**: `credentialRefsNeedingRebind` contains that Source's slug and field path -- **test command**: /test-api - -### TC-8: Operator-supplied rebinding replaces the reference before the target sees the original (integration, credentialRef re-bind not secret copy) -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-credentialref-placeholders-are-re-bound-per-target-environment-never-resolved-to-a-secret-req-004` -- **type**: api -- **preconditions**: Flagged Source; `credentialBindings` supplying a target-valid `credentialName` -- **steps**: `POST /api/promotions` with `confirmed: true` and the `credentialBindings` entry; capture the outbound document (test double on the dispatch layer) -- **expected result**: The dispatched document's `credentialRef.credentialName` equals the supplied replacement, not the source environment's original `credentialId`; no plaintext secret appears anywhere in the request/response/log -- **test command**: /test-api - -### TC-9: An un-rebound credentialRef is sent verbatim, not resolved or dropped -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-credentialref-placeholders-are-re-bound-per-target-environment-never-resolved-to-a-secret-req-004` -- **type**: api -- **preconditions**: Flagged Source with no `credentialBindings` entry -- **steps**: Confirm the promotion -- **expected result**: The dispatched document's `credentialRef` is byte-for-byte the original; the eventual failure (if any) surfaces only when the target later calls that Source, not during promotion -- **test command**: /test-api - -### TC-10: Promotion without confirmation is rejected -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-requires-explicit-confirmation-and-the-same-action-matrix-authorization-as-exportimport-req-005` -- **type**: api -- **preconditions**: Valid configuration id and target environment -- **steps**: `POST /api/promotions` with `confirmed` omitted or `false` -- **expected result**: HTTP 400; no dispatch; no `promotion_audit` object written -- **test command**: /test-api - -### TC-11: A user without environment.promote cannot promote -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-promotion-requires-explicit-confirmation-and-the-same-action-matrix-authorization-as-exportimport-req-005` -- **type**: security -- **preconditions**: Non-admin user whose groups are unmapped to `environment.promote` -- **steps**: Call preview and confirm endpoints -- **expected result**: `OCSForbiddenException` before any export or remote call -- **test command**: /test-api -- **@e2e exclude**: API-level action-matrix denial has no browser surface — covered by PHPUnit `PromotionControllerTest::testPromoteDeniedForUnmappedNonAdmin` - -### TC-12: A successful promotion is audited -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-every-promotion-attempt-is-recorded-in-an-append-only-promotion-audit-log-req-006` -- **type**: api -- **preconditions**: A confirmable promotion writing two Sources and one Endpoint -- **steps**: Confirm the promotion -- **expected result**: A `promotion_audit` object exists with `outcome: "success"`, correct `fromEnvironmentSlug`/`toEnvironmentSlug`, a counts-only `previewSummary`, and no credential values or full entity payloads -- **test command**: /test-api - -### TC-13: A failed promotion is still audited -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-every-promotion-attempt-is-recorded-in-an-append-only-promotion-audit-log-req-006` -- **type**: api -- **preconditions**: Target environment simulated to return 404 (older Integriq without import routes) -- **steps**: Confirm the promotion -- **expected result**: A `promotion_audit` object exists with `outcome: "failed"` and a message identifying the failure; no fabricated `written` summary -- **test command**: /test-api - -### TC-14: promotion_audit objects cannot be edited or deleted after creation -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-every-promotion-attempt-is-recorded-in-an-append-only-promotion-audit-log-req-006` -- **type**: api -- **preconditions**: An existing `promotion_audit` object -- **steps**: Attempt `PUT`/`DELETE` on it via the OpenRegister object API -- **expected result**: Rejected by `appendOnly`/`immutable` enforcement, matching `call_log`/`job_log` behaviour -- **test command**: /test-api - -### TC-15: A Source's credentialRef is not redacted on export (configuration-export-import delta) -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md#requirement-credentialref-authentication-placeholders-pass-through-export-and-import-unresolved-and-untranslated-req-010` -- **type**: regression -- **preconditions**: A Source with `configuration.authentication.credentialRef.credentialId` set -- **steps**: Export the Source via `SourceHandler::export()` -- **expected result**: The `credentialRef.credentialId` value is unchanged, not `***REDACTED***` -- **test command**: /test-functional (PHPUnit, no browser surface) - -### TC-16: Importing a non-resolving credentialRef does not block the write (configuration-export-import delta) -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/configuration-export-import/spec.md#requirement-credentialref-authentication-placeholders-pass-through-export-and-import-unresolved-and-untranslated-req-010` -- **type**: regression -- **preconditions**: OAS document with a Source whose `credentialRef.credentialId` does not exist on the importing environment -- **steps**: `importConfiguration()` -- **expected result**: The Source object is written with the reference verbatim; no exception at import time -- **test command**: /test-functional (PHPUnit, no browser surface) - -### TC-17: Environments & Promotion page lists environments with accessible selects -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#non-functional-requirements` -- **type**: accessibility -- **preconditions**: At least two seeded environments -- **steps**: Open the Environments & Promotion page; inspect the target-environment `NcSelect` -- **expected result**: Environments render as a list/table; the select carries an explicit `inputLabel` -- **test command**: /test-accessibility - -### TC-18: Promote flow shows diff preview before confirm is enabled -- **spec_ref**: `openspec/changes/environments-and-promotion/specs/environments-and-promotion/spec.md#requirement-diff-preview-merges-the-targets-existing-preview-response-with-a-credential-rebind-classification-req-003` -- **type**: functional -- **preconditions**: A configuration group and a target environment with at least one collision/create -- **steps**: Open the promote flow, select configuration + target -- **expected result**: `PromotePreviewModal` renders creates/updates/collisions and `credentialRefsNeedingRebind`; Confirm is disabled until the preview has loaded -- **test command**: /test-functional - -## Coverage Summary -- REQ-001 (named environments wrap a Source): TC-1, TC-2 — covered -- REQ-002 (promotion reuses export + dispatch): TC-3, TC-4 — covered -- REQ-003 (diff preview reuse + merge): TC-5, TC-6, TC-18 — covered -- REQ-004 (credentialRef re-binding, never secret copy): TC-7, TC-8, TC-9 — covered -- REQ-005 (confirmation + action-matrix authorization): TC-10, TC-11 — covered -- REQ-006 (promotion audit log): TC-12, TC-13, TC-14 — covered -- configuration-export-import REQ-010 (credentialRef pass-through contract): TC-15, TC-16 — covered -- Non-Functional (accessibility, i18n): TC-17 — covered (i18n verified via code review of translation keys, no dedicated TC — all strings added under `l10n/en.json`/`l10n/nl.json` per existing convention) - -## Out of Scope -- Git-backed configuration storage / GitOps (proposal.md Out of Scope) — no test cases. -- Unattended/scheduled promotion — this change is operator-confirmed only; no test cases for automated triggers. -- Multi-hop promotion chains (A→B→C) — not supported; no test cases. diff --git a/openspec/changes/execution-trace-observability/proposal.md b/openspec/changes/execution-trace-observability/proposal.md index c44fd1276..90b07fea9 100644 --- a/openspec/changes/execution-trace-observability/proposal.md +++ b/openspec/changes/execution-trace-observability/proposal.md @@ -1,62 +1,47 @@ -# Proposal: execution-trace-observability - -## Summary -Integriq today records call/job/sync activity as independent per-entity logs (`call_log`, `synchronization_log`, `event_message`) with no shared identifier tying one request's rule → mapping → synchronization → outbound-call path together, and no way to re-run a failed execution. This change mints an execution id at every entry point (endpoint call, job run, event delivery, manual sync), threads it through the existing pipeline, persists an ordered per-execution timeline as a new `execution_trace` OpenRegister object (redacted via the existing `SensitiveFieldRegistry`), adds a Traces UI (manifest v2 list+detail) and a Prometheus counter, and adds dry-run/force replay of a traced failure. - -## Motivation -n8n shipped a per-step execution trace + replay debugging engine (June 2026); this is now a baseline expectation for integration-platform observability and a named competitive gap (Specter insight #1267, Codeberg issue #154). Operators debugging a failed sync today must correlate rows across three separate log schemas by timestamp and source/synchronization id, with no persisted step-by-step view and no one-click re-run. This change closes that gap without introducing distributed tracing (OpenTelemetry export is explicitly out of scope) or a new persistence layer — it is built entirely on existing Integriq/OpenRegister primitives (register.d fragments, FlowToken, SensitiveFieldRegistry, AppHost `tableCount` metrics, dead-letter replay dispatch). - -## Affected Projects -- [ ] Project: `integriq` — mints/propagates an execution id through `EndpointService`/`FlowToken`, `RuleService`, `SynchronizationService`, and `CallService`; adds an `execution_trace` register.d fragment schema; adds `ExecutionTraceService` (Controller→Service→Mapper, ADR-008) for trace assembly, persistence, and replay; adds a Traces manifest v2 page; adds an AppHost `tableCount` Prometheus counter. - -## Scope - -### In Scope -1. An execution id (`traceId`, distinct from the pre-existing, unrelated `correlationId` used by the case-handoff intake engine) minted once per entry point — endpoint call (`EndpointService::handleRequest`), job run, event delivery (`EventService::attemptDelivery`), manual synchronization run — and propagated through rule pipeline → mapping → synchronization → outbound `CallService` calls, so every log/call produced within one logical execution can be joined by `traceId`. -2. A per-execution timeline of ordered steps (type, order, duration, status, input/output snapshot) persisted as one `execution_trace` OpenRegister object per execution. Snapshots reuse `FlowToken`'s existing 8-slot shape where the pipeline already captures request/response/sync-input/sync-output state, and MUST be redacted via the existing `SensitiveFieldRegistry` before persistence — no new redaction logic. -3. Trace persistence as a `register.d` fragment schema (`execution_trace`) with retention modeled on `call_log`'s `x-openregister-archival` pattern. -4. Failed-execution replay: re-run a traced entry point with the same input. Dry-run by default (produces a preview trace, no writes); an explicit `force` flag performs the real write. Reuses the two existing replay dispatch points (`EventService::attemptDelivery`'s action.kind dispatch, `SynchronizationService::replaySynchronizationItem`) rather than inventing a third redispatch mechanism — both are extended with a dry-run parameter that does not exist today. -5. A Traces UI (manifest v2 `"type": "logs"` list page over `execution_trace`, following the `call_log`/`SourceLogs` precedent, plus a detail timeline view; any `NcSelect` filter carries `inputLabel`) and a `traces_total` Prometheus counter added as an AppHost `tableCount` descriptor in `src/manifest.json`, alongside the existing 9 descriptors. -6. Unit tests for trace-id propagation and redaction-in-snapshot; one integration test proving a single endpoint call produces a trace spanning rule → mapping → call. - -### Out of Scope -Distributed tracing across apps (OpenTelemetry export, W3C traceparent propagation to OpenRegister/other Conduction apps) — noted as a follow-up; this change is Integriq-internal correlation only. - -## Approach -Mint the `traceId` at each of the four entry points and carry it as a lightweight `ExecutionTraceContext` value object passed alongside `FlowToken` (not added as a 9th `FlowToken` constructor parameter — `FlowToken` has two existing zero-arg-then-rehydrate call sites that a required id param would break; see design.md Decision 1). Each pipeline stage (rule, mapping step, synchronization item, outbound call) appends one ordered step to an in-memory trace buffer; `ExecutionTraceService` persists the assembled buffer as one `execution_trace` object at the end of the execution (success, short-circuit, or exception). Redaction is applied per-step at snapshot-build time by calling `SensitiveFieldRegistry::redactArray()` directly (not through `CallService`'s local reimplementation — see design.md Decision 3, which flags that asymmetry as pre-existing debt this change does not need to fix but must not copy). Replay re-invokes the existing dead-letter dispatch points with a new dry-run parameter, producing a new `execution_trace` linked to the original via a `replayOf` field rather than mutating the original trace. - -## New Dependencies -None. - -## Impact -- `lib/Service/EndpointService.php` — mint/propagate `traceId` at `handleRequest()`/`doHandleRequest()`, emit rule-step trace entries from `processRules()`. -- `lib/Service/Helper/FlowToken.php` — unchanged (no new constructor param; see design.md). -- `lib/Service/RuleService.php` — emit trace entries for custom rule dispatch. -- `lib/Service/SynchronizationService.php` — propagate `traceId` into `processSynchronizationObject()`/`replaySynchronizationItem()`; add dry-run parameter. -- `lib/Service/CallService.php` — accept/forward `traceId` into `buildAndPersistCallLog()`; no change to existing redaction logic (REQ-006 in `http-call-engine` is unaffected). -- `lib/Service/EventService.php` — propagate `traceId` into `attemptDelivery()`; add dry-run parameter to the replay path. -- `lib/Service/ExecutionTraceService.php` (new) — assembly, persistence, retrieval, replay orchestration. -- `lib/Controller/ExecutionTracesController.php` (new) — list/detail/replay HTTP surface. -- `lib/Settings/register.d/execution-trace-observability.json` (new) — `execution_trace` schema fragment. -- `src/manifest.json` — new `Traces`/`TraceDetail` pages, new `traces_total` observability descriptor. -- `src/views/ExecutionTrace/*.vue` (new) — list + detail Vue components. - -## Cross-Project Dependencies -None — self-contained within `integriq`. No OpenRegister core change is required; the fragment mechanism (ADR-037) and `SensitiveFieldRegistry`/AppHost engine are consumed as-is. - -## Risks - -### Risk 1: Snapshot volume/PII exposure if redaction is skipped on a new code path -**Severity:** High — **Mitigation:** every snapshot-producing step MUST call `SensitiveFieldRegistry::redactArray()` before the step is appended to the trace buffer (never after persistence); the integration test in scope item 6 asserts no plaintext secret survives in a persisted `execution_trace`, mirroring `http-call-engine` REQ-006's existing test pattern. - -### Risk 2: Replay-without-dry-run causing duplicate writes -**Severity:** Medium — **Mitigation:** dry-run is the explicit default at both the controller and service layer (force requires an explicit boolean, never inferred), and dry-run replays never call the underlying write path (`processSynchronizationObject`'s persistence branch, `deliverMessage`) — see design.md Decision 4. - -### Risk 3: Trace-buffer memory growth on pipelines with many steps or large payloads -**Severity:** Low — **Mitigation:** snapshots follow the same size posture as existing `call_log`/`synchronization_log` bodies (no new truncation policy introduced or required beyond what those schemas already accept); flagged as a follow-up if pipelines with very large mapped result sets prove to be a problem in practice. - -## Rollback Strategy -The change is additive: a new register.d fragment (removable by deleting the file — no destructive migration, per ADR-037's version-gated re-import), a new service/controller pair, and threading of an optional `traceId`/`ExecutionTraceContext` parameter through existing methods with safe defaults (`null` disables trace-step emission, preserving current behavior byte-for-byte). Reverting is: remove the fragment file, remove the new controller route registrations and manifest pages, and drop the (default-`null`) trace parameters from the touched method signatures. No existing schema, log shape, or call path is modified. - -## Open Questions -- Should `execution_trace` supersede the currently-unused `sessionId`/`synchronization` correlation fields already present but unpopulated on `call_log` (see design.md), or leave them as separate, still-dead surface for a later cleanup change? Deferred to design.md Decision 5; recommend filing a follow-up issue rather than blocking this change on a `call_log` schema edit. +--- +kind: spec-only +depends_on: [] +--- + +# Proposal: execution-trace-observability (superseded — retired 2026-09-02) + +This directory double-counted a change that had already shipped. Execution +trace observability was implemented and archived on 2026-07-16 +(`archive/2026-07-16-execution-trace-observability`, 27/42 tasks checked +with per-task evidence), yet this live copy was resurrected at 0/41: the +openconnector→integriq rename applied to the prose, the evidence notes +stripped, every box reset. The machinery exists at HEAD: +`lib/Service/ExecutionTraceService.php`, +`lib/Service/Helper/ExecutionTraceContext.php`, +`lib/Controller/ExecutionTracesController.php` with its routes, the +`execution_trace` schema in +`lib/Settings/register.d/execution-trace-observability.json`, trace +propagation through `CallService`/`EndpointService`/`EventService`/ +`JobService`/`SynchronizationService`, and the UI (manifest pages `Traces` +/ `TraceDetail`, `src/views/ExecutionTrace/TraceDetailPage.vue`, +`TraceTimelineWidget.vue`). + +`lib/Settings/register.d/execution-trace-observability.json` references +this directory's `design.md` (Decision 2), so that file stays exactly where +it is as a reference target. The other artifacts (test plan, spec deltas) +are removed; they survive verbatim in the archived twin and in git history. + +## Disposition of the original scope + +| Original scope | Where it went | +| --- | --- | +| `execution_trace` schema, traceId minting + propagation across the rule → mapping → synchronization → call chain, traces controller + routes, timeline UI, Prometheus descriptor, Newman folder authored | **Already shipped and archived**: `archive/2026-07-16-execution-trace-observability` (27/42 boxes checked), code at HEAD | +| Residual verification with substance beyond a live-instance pass: the job-entryPoint replay has **no no-write test mode** (`executeJob()`'s `$forceRun` only bypasses the schedule gate — see the disclosed deviation in `ExecutionTraceService::replayJob()`'s docblock), the suspend→resume trace round trip is unwired-tested, the Traces UI has never been rendered, the trace metric has never been scraped, and the docs page is missing | Open, and honestly unticked in the archived twin with per-box reasons. This is the largest genuine residual of the twelve retirements; it deserves its own verification-pack-style successor (shaped like `approvals-verification-pack`) when the observability track is next picked up — author it then, not here | + +## Sequencing + +Nothing remains to implement from this change directly. The residual +verification above is real but not in flight; author a successor change +when the observability track resumes. + +## Archival + +This directory is retired in place (not moved or renamed): `design.md` is +referenced from the register fragment, and a rename would break that +pointer and detonate every diff-scoped gate. Archive it via the normal flow +only after that comment is repointed. diff --git a/openspec/changes/execution-trace-observability/specs/execution-trace/spec.md b/openspec/changes/execution-trace-observability/specs/execution-trace/spec.md deleted file mode 100644 index ef4eb8bf4..000000000 --- a/openspec/changes/execution-trace-observability/specs/execution-trace/spec.md +++ /dev/null @@ -1,366 +0,0 @@ -# execution-trace Specification - -**Status**: planned -**Scope**: integriq -**OpenSpec changes**: -- `execution-trace-observability` _(in progress)_ - -## Purpose - -Joins one logical Integriq execution — an inbound endpoint call, a cron -job run, a CloudEvent delivery, or a manual synchronization run — under a -single minted id, propagated through the rule pipeline, mapping, -synchronization, and outbound `CallService` dispatch, and persists an -ordered per-step timeline as one `execution_trace` OpenRegister object per -execution. Snapshots are redacted via the existing `SensitiveFieldRegistry` -before persistence. Failed executions can be replayed (dry-run by default, -explicit force for a real write) by reusing the dead-letter and test-mode -machinery that already exists in `dead-letter-replay`, -`synchronization-engine` REQ-011, and `job-management` REQ-JOB-002. See -`design.md` Decisions 1-5 for the propagation, schema, redaction, and -replay mechanics. - -## ADDED Requirements - -### Requirement: Execution id minted at every entry point and propagated through the pipeline (REQ-001) - -The system MUST mint a `traceId` (UUIDv4) at each of the four execution -entry points — `EndpointService::handleRequest()`, a cron-triggered job run -(`JobService::executeJob()`), a CloudEvent delivery attempt -(`EventService::attemptDelivery()`), and a manual synchronization run -(`SynchronizationService::synchronize()`) — before any downstream work -begins, and MUST carry it as an `ExecutionTraceContext` value object passed -alongside the existing `FlowToken` (never as a new `FlowToken` constructor -parameter; see `design.md` Decision 1) through the rule pipeline -(`EndpointService::processRules()`), synchronization item processing -(`SynchronizationService::processSynchronizationObject()`), and outbound -dispatch (`CallService::call()`). Every step recorded during one execution -(REQ-002) MUST carry the same `traceId`. When no `ExecutionTraceContext` is -supplied (e.g. `SourcesController::test()`'s ad-hoc outbound call, or any -other call path not originating from one of the four entry points), no -`traceId` is minted and no trace is recorded — this MUST NOT change -existing behaviour for untraced call paths. - -@e2e exclude backend correlation-id propagation — covered by PHPUnit - -#### Scenario: an endpoint call mints one traceId shared by every downstream step - -- **GIVEN** an endpoint with a `mapping` rule (before) and a `save_object` - rule (before) that dispatches one outbound `CallService` call via a - `synchronization` rule -- **WHEN** a request reaches `EndpointService::handleRequest()` -- **THEN** a single `traceId` is minted before `processRules()` runs -- **AND** the rule step, the mapping step, and the outbound-call step - recorded for this request all carry that same `traceId` - -#### Scenario: an ad-hoc source test call outside any entry point produces no trace - -- **GIVEN** an admin calls `SourcesController::test()` directly -- **WHEN** `CallService::call()` dispatches the test request -- **THEN** no `ExecutionTraceContext` is present -- **AND** no `execution_trace` object is created for that call - -#### Notes - -- `traceId` is a distinct concept from the pre-existing `correlationId` used - by the case-handoff intake engine (`OpenFormulierenIntakeService`, - `DsoIngestService`) — the two are unrelated and MUST NOT be conflated. - -### Requirement: Ordered per-execution step timeline (REQ-002) - -For each execution carrying an `ExecutionTraceContext`, the system MUST -append one ordered `Step` (`order`, `type` ∈ `rule|mapping|synchronization| -call`, `name`, `timing`, `status`, `durationMs`, `startedAt`, redacted -`input`/`output`) to the context's in-memory buffer for: every rule -evaluated by `processRules()` (including skipped rules, per `rule-pipeline` -REQ-RULE-001's skip semantics — skipped rules MUST still produce a step with -`status: 'skipped'`), every mapping application, every synchronization item -processed, and every outbound `CallService::call()` dispatch. Steps MUST -retain the pipeline's actual execution order (`order` matches the sequence -observed, not the rule's configured `order` field alone, since mapping and -call steps interleave between rule steps). - -@e2e exclude backend step assembly — covered by PHPUnit - -#### Scenario: a skipped rule still produces a step - -- **GIVEN** a rule whose JSON-Logic `conditions` evaluate to false -- **WHEN** the pipeline reaches it during a traced execution -- **THEN** a step with `status: 'skipped'` is appended, matching - `rule-pipeline` REQ-RULE-001's existing skip behaviour (no data mutation) - -#### Scenario: step order reflects real execution sequence - -- **GIVEN** a pipeline that runs rule A (order 10), then dispatches an - outbound call from within rule A, then runs rule B (order 20) -- **WHEN** the trace is assembled -- **THEN** the steps appear in the sequence [rule A, call, rule B], not - grouped by type - -### Requirement: Snapshot redaction before any step is buffered (REQ-003) - -Every step's `input`/`output` snapshot MUST be redacted via -`SensitiveFieldRegistry::redactArray()` (never a new or duplicated -redaction implementation) before it is appended to the `ExecutionTraceContext` -buffer. For the `call` step type specifically, the system MUST reuse the -already-redacted `request`/`response` array `CallService::buildResponseData()` -produces for `call_log` persistence (per `http-call-engine` REQ-006) rather -than deriving a second, independent redaction of the same data — see -`design.md` Decision 3. - -@e2e exclude backend redaction — covered by PHPUnit; integration scenario -below is the cross-layer contract test - -#### Scenario: a redacted rule-step snapshot never contains a plaintext secret - -- **GIVEN** an `authentication` rule step whose amended `FlowToken` request - slot carries an `Authorization` header -- **WHEN** the step is appended to the trace buffer -- **THEN** the persisted step's `input.headers.authorization` value is - `***REDACTED***` - -#### Scenario: the call step's snapshot matches the call_log's redacted request/response byte-for-byte - -- **GIVEN** a traced execution whose rule pipeline dispatches one outbound - `CallService::call()` to a source configured with a `client_secret` - form parameter -- **WHEN** the execution completes and both the `call_log` and the - `execution_trace` are persisted -- **THEN** the trace's `call` step `output` equals the `call_log.request`/ - `call_log.response` redacted shape exactly — no divergence, no duplicate - redaction pass - -### Requirement: Trace persistence as one execution_trace object per execution (REQ-004) - -The system MUST persist the assembled `ExecutionTraceContext` as exactly one -`execution_trace` OpenRegister object (register/schema `openconnector` / -`execution_trace`, register.d fragment per `design.md` Decision 2) when the -execution completes — on success, on pipeline short-circuit (e.g. an `error` -rule or approval suspension), or on an uncaught exception (`rule-pipeline` -REQ-RULE-001's HTTP 500 path) — using the minted `traceId` as the object's -own id. Persistence MUST be a single create for every entry point EXCEPT the -approval-suspend/resume continuation (`EndpointService::resumeFromApproval()`), -where the system MUST update the SAME `execution_trace` object (matched by -`traceId`, carried in the rehydrated `ApprovalService::rehydrateFlowToken()` -context) to append the `after`-phase steps rather than create a second, -disconnected trace for the same logical execution. - -@e2e exclude backend persistence orchestration — covered by PHPUnit - -#### Scenario: a successful execution persists exactly one trace - -- **GIVEN** a traced endpoint call that completes successfully -- **WHEN** the response is returned -- **THEN** exactly one `execution_trace` object exists with `status: - 'success'` and every step recorded during the request - -#### Scenario: an approval-suspended execution's resume appends to the same trace - -- **GIVEN** a `before`-phase `approval` rule suspends a traced execution - (`approval-workflow` REQ-001), producing a trace with `status: 'running'` - and the `before`-phase steps -- **WHEN** an approver later approves and `EndpointService::resumeFromApproval()` - runs the remaining `after`-phase rules -- **THEN** the SAME `execution_trace` object (same `traceId`) is updated - with the `after`-phase steps appended and `status` set to its final value -- **AND** no second `execution_trace` object is created for this execution - -#### Scenario: an uncaught rule exception still produces a completed trace - -- **GIVEN** a rule that throws during a traced execution -- **WHEN** the pipeline surfaces the HTTP 500 (`rule-pipeline` REQ-RULE-001) -- **THEN** the `execution_trace` is persisted with `status: 'failed'` and an - `error` object carrying the endpoint name, rule name, rule type, and error - message — the same fields the HTTP 500 body already carries - -### Requirement: Dry-run replay performs no writes (REQ-005) - -`POST /api/execution-traces/{id}/replay` MUST default to dry-run -(`force` absent or `false`) and MUST NOT perform any write with an external -or persisted side-effect for the replayed execution: for a `sync`-entryPoint -trace it MUST invoke `SynchronizationService::replaySynchronizationItem()` -with `isTest: true` (reusing `synchronization-engine` REQ-011's existing -no-write guarantee); for a `job`-entryPoint trace it MUST invoke -`JobService::executeJob()`'s existing test mode (`job-management` -REQ-JOB-002); for an `event`-entryPoint trace of `action.kind: webhook` it -MUST resolve and return the request that would be dispatched WITHOUT -invoking the network call; for an `endpoint`-entryPoint trace it MUST run -`processRules()` with `dryRun: true` (`rule-pipeline` REQ-RULE-010), -suppressing every write-shaped rule's side effect. Every dry-run replay MUST -create a NEW `execution_trace` with `isReplay: true`, `dryRun: true`, and -`replayOf` set to the original trace's id — it MUST NOT mutate the original -trace. - -@e2e exclude backend replay orchestration — covered by PHPUnit - -#### Scenario: a dry-run replay of a failed sync-entryPoint trace makes no writes - -- **GIVEN** a `failed` `execution_trace` with `entryPoint: 'sync'` -- **WHEN** an admin calls replay with no `force` flag -- **THEN** `SynchronizationService::replaySynchronizationItem()` is invoked - with `isTest: true` -- **AND** no `synchronization_contract` or target object is created or - updated -- **AND** a new `execution_trace` is persisted with `isReplay: true, - dryRun: true, replayOf: ''` - -#### Scenario: a dry-run replay of a webhook event-entryPoint trace never dispatches - -- **GIVEN** an `execution_trace` with `entryPoint: 'event'` whose - subscription resolves to `action.kind: 'webhook'` -- **WHEN** an admin calls replay with no `force` flag -- **THEN** the resolved outbound request (URL, method, headers) is returned - in the response -- **AND** no HTTP request is dispatched to the sink - -#### Scenario: a dry-run replay of an endpoint-entryPoint trace skips write rules - -- **GIVEN** an `execution_trace` with `entryPoint: 'endpoint'` whose original - execution ran a `mapping` rule then a `save_object` rule -- **WHEN** an admin calls replay with no `force` flag -- **THEN** the `mapping` rule executes for real and produces a real step -- **AND** the `save_object` rule does NOT persist an object; its step is - recorded with `status: 'skipped_dry_run'` - -### Requirement: Forced replay reuses the original entry point's real dispatch path (REQ-006) - -`POST /api/execution-traces/{id}/replay` with `force: true` MUST perform a -real write using the SAME dispatch mechanism the original execution would -have used, never a bespoke re-implementation: `sync`-entryPoint traces -dispatch via `SynchronizationService::replaySynchronizationItem(isTest: -false)`; `job`-entryPoint traces dispatch via `JobService::executeJob()` -with test mode off; `event`-entryPoint traces dispatch via the existing -`EventService::attemptDelivery()` / `dead-letter-replay` REQ-DLR-003 path -unchanged; `endpoint`-entryPoint traces dispatch via `processRules()` with -`dryRun: false` (ordinary execution). A forced replay MUST NEVER read -outbound-call credentials from the stored (redacted) trace snapshot — -Source-level authentication MUST be re-resolved live by `CallService` from -the Source object exactly as in the original execution, matching the -existing `sync_item_dead_letter.payload` pattern where the stored payload is -business data, never a credential. Every forced replay MUST create a new -`execution_trace` with `isReplay: true`, `dryRun: false`, `replayOf` set to -the original trace's id. - -@e2e exclude backend replay orchestration — covered by PHPUnit - -#### Scenario: a forced replay of a failed sync-entryPoint trace writes for real - -- **GIVEN** a `failed` `execution_trace` with `entryPoint: 'sync'` whose - original mapping bug has since been corrected -- **WHEN** an admin calls replay with `force: true` -- **THEN** `SynchronizationService::replaySynchronizationItem()` is invoked - with `isTest: false` -- **AND** the corresponding `synchronization_contract` is created/updated as - if the item had succeeded on first processing -- **AND** a new `execution_trace` is persisted with `isReplay: true, - dryRun: false, replayOf: ''` - -#### Scenario: forced replay resolves live credentials, never the redacted snapshot - -- **GIVEN** an `execution_trace` whose `call` step snapshot carries - `***REDACTED***` in place of the original Source's `Authorization` header -- **WHEN** an admin calls replay with `force: true` -- **THEN** the replayed outbound call carries the Source's current live - credential (resolved fresh by `CallService`), never the literal string - `***REDACTED***` - -### Requirement: Traces UI — typed list and detail timeline (REQ-007) - -The app's manifest MUST expose a `Traces` page (`"type": "logs"`, following -the `SourceLogs`/`EndpointLogs`/`CloudEventLogs` precedent, config -`{register: 'integriq', schema: 'execution_trace'}`) listing traces -with filters for `entryPoint`, `status`, and time range, and a `TraceDetail` -view rendering the ordered step timeline (type, duration, status per step, -with redacted input/output expandable per step) plus a "Replay" action -(dry-run by default, an explicit confirmation step required before a forced -replay). Every `NcSelect` filter control MUST carry an `inputLabel` prop -(never a bare `