diff --git a/docs/INDEX.md b/docs/INDEX.md index f6287a1a3..f48d9695e 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -53,6 +53,10 @@ Provider-specific setup that needed writing down: credential Secret is shaped differently from every other provider's, and the three test layers that cover a provider CI cannot reach. +**Before you re-raise a claim:** [`misconceptions.md`](misconceptions.md) collects the beliefs the +project held between July and October 2026 that turned out to be false, most of them about applying +Git back to the cluster, each with what is true and where it was corrected. + **Before you name anything:** [`definitions.md`](definitions.md) — one name per concept (resource collection, watch, branch worker, author attribution, refusal, folder), the two clusters and the two namespace directions, and the nine rules that decide a CRD field, condition type, reason, enum value or printer column. It is @@ -130,12 +134,11 @@ decided), **design, decided** (decision made, not built), **partly built**, **bu **deferred** (parked, kept as a decision record). The label is the first thing in the page, so you never have to read a proposal to find out it already shipped. -### Open (32 pages) +### Open (27 pages) | Doc | Open question | |---|---| | [`branch-worker-event-model.md`](design/branch-worker-event-model.md) | **proposal; its in-process fixes are built in #412 and #413.** Publication retry, Git operation deadlines, intake pause and resume, and publication status are done. Worker ownership, FIFO inputs, save and deadline semantics, and the transition boundary for later durability, which is next. Journal storage and rollout belong to the HA plan. | -| [`gittarget-state-of-affairs.md`](design/gittarget-state-of-affairs.md) | **snapshot, 2026-10-04.** What is built for `GitTarget` and the branch worker, which page owns which decision, the review findings and how each was closed, and what is left elsewhere. Read it first to find the right page on this topic. | | [`gittarget-branch-worker-pending-writes.md`](design/gittarget-branch-worker-pending-writes.md) | **built in #413.** Ordered pending writes, one materializer and one retry schedule; a safe admission boundary for the watch; Git call deadlines; intake pause and resume; publication status. Holds the recovery contract for an outage, replay limits, and the acceptance scenarios. | | [`event-pipeline-overview.md`](design/event-pipeline-overview.md) | **snapshot, 2026-10-04.** Diagrams of the pipeline as built: watch streams and their replay/resume rules, the three meanings of "replay", the branch worker's inputs, the pending writes' lifecycle, retry and intake, a save end to end, what is an event today, and an honest list of what is still missing. | | [`gittarget-parent-branch.md`](design/gittarget-parent-branch.md) | **merged in #407.** `spec.parentBranch`: a target stays on standby until a write needs a branch, then creates its write branch from the parent's tip as checked at publication. The contract as built, the defect it started from, prior art and naming. Open: parent per `GitProvider.status.branches` entry. | @@ -145,17 +148,13 @@ so you never have to read a proposal to find out it already shipped. | [`gittarget-red-status-plan.md`](design/gittarget-red-status-plan.md) | **design, partly built.** The smaller alternative to Git write preflight: make a `GitTarget` turn red and explain why no commit was made, without moving Git checks into the admission webhook and without adding a second state source. Keeps the existing kstatus trio and domain conditions, ranks the most likely operator-facing failures, and has now shipped the first status-transition slice: refusal messages republish when details change, scoped refusals cannot be cleared by unrelated scope success, `BlockMessage()` carries stable issue kind/path/actor detail, and the unsupported-folder e2e covers Git-side break and recovery. | | [`open-asks-priority.md`](design/open-asks-priority.md) | **the work queue.** Merges three overlapping backlogs — the downstream consumer asks, the API-surface block left unbuilt by the status and configuration-model review, and the config-surface proposal (B1–B6) — into one ordered queue under four stated tests, and says where we deliberately do **not** do what was asked. The standing caveat narrowed once the layout model reversed: a Tier 2 entry belongs to postponed [#294](https://github.com/ConfigButler/gitops-reverser/issues/294) only if it breaks a `GitTarget` field, and everything else is independently schedulable. Makes one design call against what was asked: **delete Option C sibling inference** rather than ship an off-switch for it, because it let a human's edit to the repository change operator behavior with nothing in status recording the move. That deletion has shipped, and "what the deletion taught" records what building it found. **F9 is Tier 1**: the only item whose answer is unknown rather than whose work is unscheduled, and it gates planning the enum work | | [`proving-the-metric-surface.md`](design/proving-the-metric-surface.md) | **design, partly built.** What identity an operator needs to locate a failure, and what an e2e check has to do to be worth trusting. Written after a metric value that was declared, switched on, surfaced as a condition reason and counted turned out to have no producer at all, and the check added to catch that class then skipped whenever the counter read zero — so deleting every recording call left the suite green. Two rules follow: an assertion must not take its trigger from the metric under test, and no enum value may be reachable only through an injected fake. Carries a cardinality policy that separates bounded configuration identity from per-request identity, a table of which instrument families should own `source_cluster` directly rather than by join, and the PromQL reasoning for `increase()` versus `max_over_time` (neither applies mechanically: one cannot see an event before the first sample, the other cannot reconstruct totals across a reset). Steps 1–4 are built; step 5, diagnostics export, the `e2e_run` scrape label and a pinned-seed regression are not | -| [`placement-visibility-and-declared-defaults.md`](design/placement-visibility-and-declared-defaults.md) | **design, half built.** The three questions the inference deletion left, decided and then built in two goes. PR #291 shipped the deletion and none of the eight items queued behind it; the residue was filed as [#295](https://github.com/ConfigButler/gitops-reverser/issues/295) — **which shipped in 0.42.1 via [#319](https://github.com/ConfigButler/gitops-reverser/pull/319) and is what reversed the layout model** — and [#296](https://github.com/ConfigButler/gitops-reverser/issues/296), whose largest half then shipped in [#326](https://github.com/ConfigButler/gitops-reverser/pull/326): `status.layout` became `status.placement`, and the ambiguous render root came with it as a refusal rather than a third counter outcome. The `declared` metric value has since split into `by_type`/`default` too, which was the one row with an external consumer because it changes a label a dashboard may select on. What still stands and is still unbuilt is the rest of the legibility set: the canonical path as a template constant, and `{kindLower}`, tracked on [#339](https://github.com/ConfigButler/gitops-reverser/issues/339) now that #296 is closed; the Event on the GitTarget was dropped. Its Question 2, a CRD default for `placement.default`, is **re-opened rather than superseded** and now lives as an open question in [`layout/model.md`](layout/model.md); the argument against it is legibility, since #319 removed the correctness objection | -| [`gittarget-api-wave.md`](design/gittarget-api-wave.md) | **partly built**, filed as [#294](https://github.com/ConfigButler/gitops-reverser/issues/294). One breaking wave on `GitTarget`, of which steps 2 to 7 have **shipped**: `spec.suspend`, `status.placement` and the reconcile-request annotation in [#326](https://github.com/ConfigButler/gitops-reverser/pull/326), and B4's `commitWindow`/`commit.message` move plus the source-scope deletion in [#330](https://github.com/ConfigButler/gitops-reverser/pull/330). Organizing principle: **the folder is described on the GitTarget, the connection describes only the connection** — and this is where that became a struct boundary rather than a sentence, since grouping a field is free only in a release that is already breaking. `spec.mode` and `GitTarget.spec.interval` are both **dropped**, with re-open triggers, and `status.lastHandledReconcileAt` was dropped in the build for a stated reason. Step 8, the riders, is closed: the `CommitRequest` lifecycle hole shipped in [#400](https://github.com/ConfigButler/gitops-reverser/pull/400), the missing source-cluster provider message in [#258](https://github.com/ConfigButler/gitops-reverser/pull/258), and an asserted `CommitRequest.spec.author` is **declined** because authorship is established, never asserted ([#339](https://github.com/ConfigButler/gitops-reverser/issues/339)); #294 is closed. The reference collapse and the `scope` removal shipped in [#333](https://github.com/ConfigButler/gitops-reverser/pull/333); the `TooManyStreams` cap and the Event on a changed resolution were dropped. Also records that staying `v1alpha3` on loud rejections is a **one-consumer countdown** | | [`target-watch-plan.md`](design/target-watch-plan.md) | **partly built.** The diff is built and applied; removal semantics are not. The companion to [`watch-manager-ownership.md`](design/watch-manager-ownership.md): the ownership page says WHO applies a plan, this one says WHAT a plan is and what changing it may touch. A resource collection — group, resource, namespace, deliberately no served version — is the one identity the watch stream, the render-fidelity scope and the mark-and-sweep boundary all agree on, because a key that does not round-trip to the scope it sweeps under is the class of error that deletes user data. The plan is diffed into `keep`/`start`/`restart`/`stop` and applied per collection, so adding one WatchRule stops re-initializing every unrelated collection into a queue shared with other tenants; a `restart` is a served-version change, which is why the version is spec DATA rather than identity. Readiness and the fidelity revision are per scope, so a KEPT collection holds the result its own initialization produced rather than being asked to prove itself again over an unrelated edit. `stop` never touches files — removal is a Git-side sweep under the target's existing `spec.prune.mode`, not a watch-layer delete. "Cut at the producer" is the accepted consequence: nothing fences the queue, so a deselected collection may leave a short tail of writes, bounded by the queue and converged afterwards. Still open: the `stop` classification wants a settled `TypeRemoved` from `typeset` (see TODO), and removal on INTENT is undecided. | -| [`collection-terminology-rename.md`](design/collection-terminology-rename.md) | **step 1, shipped.** Replaced cells with resource collections and removed per-rule `operations` from both rule kinds. Rules observe every object event; GitTarget prune mode controls removal. Records the rename census (760 Go identifiers, 265 documentation occurrences), the operation-filter inventory, API/Helm migration, compatibility boundaries, and test cases. | | [`watches-labels-simplification.md`](design/watches-labels-simplification.md) | **step 2, implemented.** After the collection rename and operation-filter removal, native per-rule `objectSelector` (#146). Kubernetes is authoritative: `Always` compares selected identities with all managed Git identities in the type/namespace scope, without comparing Git labels. Full support includes supported Kustomize layouts from the first release, plus label-exit attribution (now selector-agnostic, see the attribution spec) and ordered producer replacement. | | [`label-selection-follow-ups.md`](design/label-selection-follow-ups.md) | **design, open.** What the label-selection review rounds left open, with the evidence for each: real deletions in a selected collection wait the full 3s attribution grace (the watch cannot tell them from a label exit, measured), the resync fence that stops overlapping collections from coalescing (kept on purpose), same-second overlap ties, and one store rebuild per object coming back. Also lists what was fixed on the branch | | [`attribution-removal-wait-options.md`](design/attribution-removal-wait-options.md) | a removal now waits for evidence about the DELETION rather than accepting the object's last write, which stopped it naming whoever last edited the object as the author of a deletion they did not perform. Enumerates the eight situations a resolution can be in and shows the cost is concentrated in exactly one: a removal for which no delete fact will ever arrive (a graceful pod delete, a status-only removal, a type the audit policy skips) spends the whole grace to return the answer it had at t=0, measured at ~3.1s against ~70ms when evidence is present. Prices five options against that, and recommends a per-route watermark — stop waiting once the fact stream has demonstrably moved past this event — over a second timeout flag whose right value lives in the API server's config rather than ours. Open: the decision, and how common the case is outside the e2e suite | -| [`concurrent-author-windows.md`](design/concurrent-author-windows.md) | **proposal, not built.** A branch worker holds one commit window at a time, so a second person's edit closes the first person's window and splits their change into two commits. Proposes a window per GitTarget and author, open side by side, with one rule that keeps every path's history in order: a write to a path another open window holds closes that window first (`path_conflict`). Branch-wide closes (foreign commit, apply, shutdown, buffer limit) close every window. Commit order becomes close order. A second save of one author stays refused (`SaveInProgress`). Open: what `buffer_limit` closes, windows of several GitTargets side by side, a cap on open windows, and a save cut short by a path conflict | -| [`commitrequest-save-wait-options.md`](design/commitrequest-save-wait-options.md) | **design, decided** — a five-phase plan, reported from a live cluster: a `CommitRequest` can expire before the write it exists to publish arrives, because the close delay defaulted to `0` while the write is still held head-of-line waiting for an audit fact, so the save resolves `Ready=True`/`NoWindowInGrace` and the edit commits seconds later under the target's `liveTemplate`. The decision is to repair the default (`spec.closeDelay`, now a duration string defaulting to `"2s"`, **shipped as phase 0**) and then let a request **name the write it is waiting for**, reusing the `(uid, resourceVersion)` identity the attribution join already keys on, including its settled `ExactCapable` rule. Records why the fact index cannot BE the wait (it sits upstream of the watch event, so a fact-keyed wait fires before the window opens) and what to take from it anyway — the register-then-check waiter discipline, and a best-effort retrospective oracle bounded by a TUNABLE TTL. Twelve situations, and the page is deliberate about the two it cannot fix cheaply: row 11 (a saturated queue is rejected in the WATCH path and never reaches the worker, so no worker-side wait can diagnose it) is an accepted, named gap rather than a promise, and row 10 is fixed only by a hook in the fidelity-drop branch that marks a wait terminally UNSATISFIABLE — satisfying before that gate would arm a finalize for a write that never commits. Carries a **preconditions chapter** for saves on today's build (audit policy first, then delay sizing, fidelity warm-up, save-button debounce, and assert on the commit rather than on `Ready`). `CommitterActivity` is deferred, not recommended: by its own arrival-not-existence rule it cannot satisfy the already-arrived case. Open: situation 12 (one actor, two saves, one window — blocks phase 2), whether the fact-index lookup earns its coupling, and how an unsatisfied wait is graded | +| [`concurrent-author-windows.md`](design/concurrent-author-windows.md) | **proposal, not built.** A branch worker holds one commit window at a time, so a second person's edit closes the first person's window and splits their change into two commits. Proposes a window per GitTarget and author, open side by side only when the writes' conflict footprints (the Git files they may change, kept separate from the per-object coalescing key) are proven disjoint; anything unproven stays serial. Every close trigger keeps today's exceptions (a push hint and an apply flush leave a save's window alone), windows close in opening order, and commit order becomes close order. Tested against an independent reference model on every commit, and footprint proofs tied to a tree revision and recomputed when it moves, and shipped in five stages behind a startup flag. Follow-ups in `window-field-ownership-and-apply-hold.md` | +| [`window-field-ownership-and-apply-hold.md`](design/window-field-ownership-and-apply-hold.md) | **proposal, not built.** Two separate follow-ups to concurrent author windows. Part 1: a window owns the fields it changed, the way an SSA field manager does, so two authors on one object split by field. It reopens the decided `manifestedit-field-ownership-spike.md` and argues the use case it asks for; every field-owned commit writes only owned fields, and convergence is a separate, explicitly attributed repair that compares Git with a desired view (withheld applies undone), never the live object; experimental until that is accepted. Part 2: an incoming Git commit that is provably disjoint from the open windows (rendered, against the last applied revision) holds the apply instead of closing them, with an absolute deadline of its own; it needs a fetch on a busy branch and brings back a bounded form of the wait S1 removed. The next candidate if measurements justify it. Lists each part's potential problems | | [`commit-timing-surface.md`](design/commit-timing-surface.md) | **built.** One configuration surface for commit windows and saves, an in-place `v1alpha3` break with one tested upgrade procedure: only writes open a window; a `window` block (`idleTimeout`, `maxDuration`, target default `1m`) on both kinds, which a `CommitRequest` replaces whole for the window it attaches to; `attach: CurrentOrNext \| Next` with `attachTimeout`; opt-in `whenNothingToCommit: CommitEmpty` that keeps the cause in the reason. Precise clock and zero rules. Supersedes the save-wait timer model. Open: the target's `maxDuration` value | -| [`watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) | the target three-layer watch model — **needs a human call before building** | | [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | **built.** the canonical metrics document, rewritten 2026-09 and shipped in the same wave. Reads the product as two ingestion paths — object state through watch, authorship through audit — joining at the resolver, and gives each stage one counter with a bounded `outcome` so the flow and its loss paths are drawable. Records what three review rounds corrected, including two reversals worth not re-litigating: the fact-loss counters must NOT be merged (an eviction counts a fact, a trim gap an occurrence, a decode error a whole batch), and a cumulative counter cannot replace a current count, which is why `status.streams` and `status.retainedDocuments` stayed. The instrument inventory lives in [`interpreting-metrics.md`](interpreting-metrics.md), not here. Open: the Grafana JSON and alert rules | | [`reconcile-triggering.md`](design/reconcile-triggering.md) | which controllers still fail to wake up | | [`applier-echoes.md`](design/applier-echoes.md) | **built 2026-10-20, released as one change.** The applier's own write, back through the watch, is mirrored and re-planned onto a newer tip on replay, reverting a Git-side push. Two ideas close every way a stale picture can reach a commit: a write that changes nothing does not exist, and a pending write carries the actions decided on its tree, not a picture. The applier's write in the apply window is not mirrored (F1); a write is upserts and deletes decided once, so a replay never sweeps again and a save keeps its objects (F2); a resync never decides from a snapshot collected while Git was ahead or across an apply, by an apply epoch (F3); another applier's object is not mirrored into a coordinated folder (F4); flush, then apply (S1); `CatchUp` only to adopt API creates and to converge, for Flux and Argo CD alike, and no apply before every initial snapshot is published (S2). Carries the collisions left open, the promises, the code plan, thirteen e2e scenarios, and the twelve decisions taken while building (among them: siblings do not count for F3, a 5-minute hold, initial snapshots exempt from F3, the prefetch removed, no heal resync, and the flush covering the sibling workers) | @@ -164,7 +163,6 @@ so you never have to read a proposal to find out it already shipped. | [`multi-source-audit-ingress-hardening.md`](design/multi-source-audit-ingress-hardening.md) | how independent sources authenticate to a named audit route, when annotation routing is trustworthy, and how multi-provider ingestion remains fair | | [`release-image-reuse-plan.md`](design/release-image-reuse-plan.md) | PRs 2–5 unstarted | | [`e2e-coverage-gaps-and-improvements-plan.md`](design/e2e-coverage-gaps-and-improvements-plan.md) | tests A/B/C still proposals | -| [`e2e-finish-plan.md`](design/e2e-finish-plan.md) | remaining e2e harness work | | [`sensitive-resource-diagnostics-follow-up.md`](design/sensitive-resource-diagnostics-follow-up.md) | deferred diagnostics | | [`e2e-git-server-choice.md`](design/e2e-git-server-choice.md) | stay on Gitea or move to Forgejo — the `_csrf` pin is fixable in place on both, so the migration is now a preference call, not a fix; also why we adopt no SDK either way | @@ -234,7 +232,7 @@ Four more ideas sit beside them. ## History — [`finished/`](finished/) -Twenty-four shipped plans and closed investigations. **Nothing here binds.** Read one +Fourteen shipped plans and closed investigations. **Nothing here binds.** Read one only when you want to know *why* something is the way it is; the answer to *what it is* always lives in `spec/`. diff --git a/docs/TODO.md b/docs/TODO.md index 80b41eff9..8e5891468 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -102,8 +102,7 @@ Their argument for each is in [gitops-api-open-asks.md](future/gitops-api-open-a watch it took over. Redis becomes required in this mode. [ha-gittarget-distribution-plan.md](future/ha-gittarget-distribution-plan.md). - [ ] Re-assess unifying the pending-write kinds - ([idea-unify-pending-write-kinds.md](future/idea-unify-pending-write-kinds.md), now four kinds, - not two). Deferred. + (grouped window, resync and request record, since #450 removed the atomic kind). Deferred. - [ ] Improve queue and worker observability, with more high-load tests. ## Watch plane @@ -177,12 +176,10 @@ Their argument for each is in [gitops-api-open-asks.md](future/gitops-api-open-a ## Docs and vocabulary - [ ] **Stale design-doc headers**, one docs PR: [open-asks-priority.md](design/open-asks-priority.md) - (#339 closed, #23 fixed), [gittarget-api-wave.md](design/gittarget-api-wave.md) (riders shipped), + (#339 closed, #23 fixed), [git-write-preflight.md](design/git-write-preflight.md) (superseded by the red-status plan), - [commitrequest-save-wait-options.md](design/commitrequest-save-wait-options.md) (written against - `closeDelay`), [gittarget-red-status-plan.md](design/gittarget-red-status-plan.md) (steps 5 and 6 - built), [watch-and-catalog-architecture.md](design/watch-and-catalog-architecture.md) - (superseded), [target-watch-plan.md](design/target-watch-plan.md) (queue depth is 1000, not 100), + [gittarget-red-status-plan.md](design/gittarget-red-status-plan.md) (steps 5 and 6 + built), [target-watch-plan.md](design/target-watch-plan.md) (queue depth is 1000, not 100), and the go-git version in [azure-devops-multi-ack.md](design/azure-devops-multi-ack.md) and [gittarget-parent-empty-repository.md](design/gittarget-parent-empty-repository.md). - [ ] **Wrong claims in design bodies**: [support-contract.md](design/support-boundary/support-contract.md) diff --git a/docs/architecture.md b/docs/architecture.md index 438712e1d..7769a22d9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1326,8 +1326,7 @@ peer of any named-namespace collection on the same type, never a replacement for Desired state comes from one **raw watch per `(GitTarget, GVR, scope)`**. Each event is sanitized, diffed against current Git content, and applied. There is no separate per-type object store to reconstruct, -because Git already holds current state. This section is the authoritative contract; the -[watch-first design record](finished/watch-first-ingestion-architecture.md) is historical context. +because Git already holds current state. This section is the authoritative contract. - **Manager**: [internal/watch/manager.go](../internal/watch/manager.go) - **Watch / replay / sweep**: [internal/watch/target_watch.go](../internal/watch/target_watch.go) @@ -2035,8 +2034,6 @@ Deeper dives live under [docs/design/](design/): **Ingestion, attribution, and reconcile:** -- [Watch-first ingestion design record](finished/watch-first-ingestion-architecture.md): historical context - for the current watch-only object-state model and optional audit attribution. - [Attribution: how a commit gets its author](spec/attribution.md): the two halves, the tier ladder, and the wait. - [Attribution facts as a stream, not a keyspace](finished/attribution-fact-stream.md): the shipped @@ -2054,7 +2051,6 @@ Deeper dives live under [docs/design/](design/): - [The support contract: what we edit, what we refuse, and why](design/support-boundary/support-contract.md) - [The applier claim](design/apply-trigger.md) - [Git credentials interop](finished/git-credentials-interop.md) -- [SOPS/age key management](finished/sops-repo-bootstrap-and-key-management-architecture.md) - [Commit signing](commit-signing.md) - [Interpreting metrics](interpreting-metrics.md) diff --git a/docs/components.md b/docs/components.md index ec157ac10..c3875e884 100644 --- a/docs/components.md +++ b/docs/components.md @@ -270,9 +270,5 @@ Three gates can stop a type before it reaches a stream: - [`architecture.md`](architecture.md) for how the pieces work together, starting at Ground Rules and Mental Model. -- [`design/watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) for the - target three-layer watch model: collections, the confidence model, and the managed projection. - [`design/target-watch-plan.md`](design/target-watch-plan.md) for the implementable plan that replaces wholesale stream replacement with an incremental diff. -- [`design/data-plane-triggering.md`](design/data-plane-triggering.md) for why that refactor is - needed, starting from the production failure that prompted it. diff --git a/docs/design/branch-worker-event-model.md b/docs/design/branch-worker-event-model.md index 9fdd013f8..b4974e5c2 100644 --- a/docs/design/branch-worker-event-model.md +++ b/docs/design/branch-worker-event-model.md @@ -574,7 +574,7 @@ The detailed design records continue to own adjacent policy: project them. A sibling scope's success cannot clear another scope's refusal. - [Configuration freshness](gittarget-configuration-freshness.md): watch-plan application and publication are separate observations. This design does not require new public freshness fields. -- [API boundaries](gittarget-api-wave.md) and +- API boundaries and [push notifications](apply-trigger.md): a remote-change notification requests work through the owner. It does not become another checkout writer or prove that Git changes reached the source cluster. diff --git a/docs/design/collection-terminology-rename.md b/docs/design/collection-terminology-rename.md deleted file mode 100644 index 781ac47d6..000000000 --- a/docs/design/collection-terminology-rename.md +++ /dev/null @@ -1,588 +0,0 @@ -# Replace cells with collections and remove event filters - -> **Implemented; step 1 of 2.** This is now a historical record of the migration; the code and -> the current documents are the reference. [Label selection and pruning](watches-labels-simplification.md) -> is step 2. -> Index: [`../INDEX.md`](../INDEX.md). Audited: 2026-09-28, source at `69836e70`. -> Rename counts describe tracked working-tree files before adding this plan and its cross-references; -> the existing documentation edits are included. The operation-filter audit is separate, against -> unchanged source at `02e4cd93`. Recount both before implementation. - -**Remove the cell concept from active code and documentation.** Call the selected object set a -resource collection and the process observing it a watch. This is the vocabulary already chosen -in [Definitions](../definitions.md#read-side). The object set remains necessary; its project-specific -name and duplicate vocabulary disappear. - -**Remove `spec.rules[].operations` from both WatchRule and ClusterWatchRule.** Rules select resource -collections, whose complete lifecycle the operator observes. `GitTarget.spec.prune.mode` decides -whether an observed removal deletes a Git document. There is no second switch to suppress creates, -updates, or deletions within a selected collection. - -Kubernetes remains the source of truth and Git follows it. The collection rename preserves behavior; -removing operation filters is an explicit breaking API and behavior change in this same first step. -Step 2 adds server-side selectors, identity-based `Always` pruning for supported Kustomize layouts, -and the required attribution and producer-ordering changes. - -## Result and scope - -The resulting code reads: - -```go -type CollectionKey struct { - Group string - Resource string - Namespace string -} - -// Existing structures, with their fields renamed: -// targetWatchPlan.Collections map[types.CollectionKey]watchSpec -// ResyncScope.Collection types.CollectionKey -// Event.SourceCollection types.CollectionKey -// ResyncRequest.SourceCollection types.CollectionKey -``` - -A collection still belongs to one GitTarget/source-cluster context. Its key remains versionless; -served version stays in the watch specification, which loses its operation filter. An empty -namespace retains its existing cluster-wide meaning. An all-namespace collection remains distinct -from a named namespace collection, including when they overlap. Zero-valued provenance remains unclaimed. - -Make a direct rename throughout the dependency graph. Do not leave `type CellKey = CollectionKey`, -deprecated constructors, duplicate fields, or a wrapper carrying the old vocabulary. Keep -`ResyncScope`: it describes the Git comparison boundary, which step 2 must distinguish from a -label-filtered collection. The rename adds neither selectors nor new state. - -## Remove the rule operation filter - -### Contract and reason - -The current filter lives on -[`ResourceRule.Operations`](../../api/v1alpha3/watchrule_types.go) and -[`ClusterResourceRule.Operations`](../../api/v1alpha3/clusterwatchrule_types.go). It is applied by -[`routeLiveTargetWatchEvent`](../../internal/watch/target_watch.go) after the API server delivers -the event. Initial snapshots already include the full collection. Consequently, a create-only rule -can still write existing objects during initialization, and excluding `DELETE` does not stop -`Always` from pruning an absent identity during a snapshot. Removing the field makes the rule -describe the object set consistently across initialization and live observation. - -After this step, every selected collection processes `ADDED`, `MODIFIED`, and `DELETED` events. -Keep bookmark/error handling, content deduplication, sanitization, attribution, and write gates. -Observing every object event does not require a Git commit for an unchanged object. Keep the -existing mapping from a terminating object's `MODIFIED` event to deletion intent. - -| GitTarget prune mode | Observed removal | Absence from a complete snapshot | -|---|---|---| -| `Never` | Retain Git document | Retain Git document | -| `OnEvent` (default) | Remove Git document | Retain Git document | -| `Always` | Remove Git document | Remove Git document | - -All modes continue accepting creates and updates. There is no create-only or update-only mirror, -replacement event filter, or implicit `operations: ["*"]` field. This also gives step 2 one rule -for label exits: observe the exit, then apply prune policy. - -### Additional source and configuration inventory - -This focused scan counts Go **identifier tokens**, using the scanner described below and exact -spellings in this table. It finds **316 occurrences in 23 Go files**: 94 in 10 production files, -214 in 12 test files, and eight in the generated deepcopy file. These are review/migration counts; -some identifiers move or survive in a different role. They overlap the cell census and must not -be added to it. Lowercase locals such as `ops`, comments, and string renderings need the source -walkthrough too. Two unrelated `Operations` tokens in Git commit counters are excluded. - -| Identifier(s) | Occurrences | Action | -|---|---:|---| -| `Operations` | 57 | Remove rule, compiled-selector, and watch-spec fields and references | -| `OperationSet`, `NamespaceOps` | 47 / 29 | Remove operation sets; retain namespace membership | -| `matchesOperations`, `operationsString`, `operationSpec`, `renderTargetWatchSpecs` | 3 / 3 / 2 / 3 | Delete filter matching and rendering | -| `watchOutcomeOperationFiltered` | 4 | Delete obsolete ingest outcome and its tests | -| `OperationType` | 59 | Remove API/filter uses; move the remaining event classification into `internal/types` | -| `OperationCreate`, `OperationUpdate`, `OperationDelete`, `OperationAll` | 55 / 23 / 19 / 12 | Retain the three event tags internally; delete wildcard selection | - -The configuration inventory adds **two CRD properties**, **two generated deepcopy blocks**, and -**one Helm values-schema property**. There are **17 `operations:` entries in 10 active example, -values, or documentation files**, excluding the two CRD properties: - -- [`config/samples/`](../../config/samples/): two examples, one entry each. -- [`charts/gitops-reverser/values.yaml`](../../charts/gitops-reverser/values.yaml): one starter-rule entry; - remove the matching property from [`values.schema.json`](../../charts/gitops-reverser/values.schema.json). -- [`docs/configuration.md`](../configuration.md): two examples, plus its two generated settings rows. -- [`test/e2e/setup/demo-only/`](../../test/e2e/setup/demo-only/): two example entries. -- [`test/e2e/templates/`](../../test/e2e/templates/): ten entries across four rule templates - (`demo/clusterwatchrule-demo`, `demo/watchrule-all`, `manager/clusterwatchrule-crd`, - and `restart/watchrule-wildcard`, all `.tmpl`). - -Kubernetes admission webhook `rules[].operations` fields in `config/webhook/` and chart webhook -templates remain required for their own API. Do not remove them with a global YAML replacement. -Keep audit verbs, `git.Event.Operation`, commit-operation counts, and mutation classification. -The synthetic CRD-field parser fixture in `hack/crdfields/main_test.go` is also unrelated. - -### Implementation changes - -1. **API and packaging.** Remove both rule fields and the API's operation enum. Keep the event - classification type and three concrete tags in `internal/types`, with their existing string - values, and update audit/watch consumers. Regenerate deepcopy, CRDs, chart CRD copies through - `task helm-sync`, and the configuration settings index through `task settings-index`. -2. **Rule compilation.** Remove the slice, cloning, and matching check from - [`compiledSelector`](../../internal/rulestore/store.go), including the operation parameters on - `GetMatchingRules`, `GetMatchingClusterRules`, and their helpers. Those public matcher methods - currently have test callers only; preserve their namespace/type contract while simplifying them. -3. **Selection and planning.** In [`watched_type_table.go`](../../internal/watch/watched_type_table.go), - replace `NamespaceOps` with `NamespaceScopes map[string]struct{}` and remove `watchSelection.ops`. - Deduplicate namespace selections and served versions without any operation union. Keep named - and all-namespace collections distinct; this change does not resolve their ordering limitation. -4. **Running watches.** In [`target_watch.go`](../../internal/watch/target_watch.go), use a declared - watch-key set instead of `map[targetWatchKey]OperationSet`. Remove `targetWatchStream.ops`, - `runningTargetWatch.spec`, the operation-rendering maps/helpers, and the live `Match` gate. - Keep `watchSpec{Version: ...}` for plan comparison; version changes still restart a collection. - Update startup arguments, plan descriptions, sorted-key helpers, and their callers together. -5. **Fingerprints and diagnostics.** Remove operation components from the two rule fingerprints in - [`watched_type_resolver.go`](../../internal/watch/watched_type_resolver.go) and from the effective - plan fingerprint in [`owner_observability.go`](../../internal/watch/owner_observability.go). - Update the in-memory hashes while preserving durable cursor keys. Drop operation suffixes - from plan logs. Retire `watch_events_total{outcome="operation_filtered"}`; retain the metric name, other - outcomes, and the exactly-once event census. Update its telemetry comments and documentation. - -The internal tag move must preserve [`VerbToOperation`](../../internal/auditutil/objectref.go), -[`IdentityFromAuditEvent`](../../internal/auditutil/identity.go), deletion body preference, -deletion-intent mapping, and author lookup. Do not remove audit mutation filtering or change the -grace policy as a side effect of deleting a user-configurable rule filter. - -### Upgrade contract - -Keep `v1alpha3` and remove the field directly, consistent with the project's pre-1.0 migration -approach. Update live rules, checked-in manifests, and Helm values; no alias or compatibility toggle -survives. A former subset such as `[CREATE]` now observes updates and removals too. If Git documents -must survive removals, set the target's prune mode to `Never` before removing the old field or -upgrading. This policy applies to the whole GitTarget; mixed per-rule retention has no automatic -equivalent. Create-only/update-only behavior has no replacement. - -Do not promise every stale manifest is rejected. Kubernetes -[field validation](https://kubernetes.io/docs/reference/using-api/api-concepts/#field-validation) -rejects unknown fields in `Strict` mode; `Warn` and `Ignore` can accept the request while -[pruning the unknown field](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#field-pruning). -The Helm values schema rejects the removed starter-rule key. Document and test these paths so a -leftover field cannot be mistaken for active deletion protection. - -Preserve the first-session snapshot in [`runTargetWatch`](../../internal/watch/target_watch.go), -even with a stored cursor, so current objects converge after upgrade. A snapshot cannot recover -the historical deletion events a previous filter discarded: `OnEvent` retains those old orphans; -`Always` can remove them through its existing sweep. This change does not retroactively fabricate -deletions or their authors. Document the API removal and retired metric outcome in `UPGRADING.md`. - -## Rename inventory and counting method - -The audit scans `git ls-files`, reading working-tree contents. Go files are tokenized with -`go/scanner`; identifier occurrences are separated from comment and string occurrences. Other -tracked text is scanned for the standalone word or an identifier component `cell`/`cells`, -including camel case and underscores. Counts are occurrences, not matching lines or distinct -symbols. Thus `Cell types.CellKey` contributes two identifiers. `cancelled` contributes none. - -The broad census has **1,452 occurrences in 93 files**. Manual classification finds: - -| Population | Files | Identifier occurrences | Comment occurrences | String/text occurrences | Total | -|---|---:|---:|---:|---:|---:| -| Collection-related production Go | 27 | 396 | 204 | 25 | 625 | -| Collection-related Go tests | 27 | 364 | 74 | 52 | 490 | -| Active documentation/reference cleanup | 18 | n/a | n/a | 265 | 265 | -| **Rename work** | **72** | **760** | **278** | **342** | **1,380** | - -The remaining 72 occurrences are 33 unrelated Go occurrences, 22 unrelated non-Go occurrences, -and 17 historical references. File categories can overlap: an index or investigation can contain -both current terminology and historical quotes. The 760 Go identifiers use **49 distinct -spellings**, including **24 test function names**. The core declarations are two types, 12 -production functions/methods, and eight struct fields; three test helpers also need renaming. - -There are **four file moves**, **24 production log-field occurrences across six key spellings**, -and **one production error-message occurrence**. No collection-related `cell` occurs in a -serialized CRD field, generated CRD/chart schema, metric name/label, or Redis cursor key segment. -The API source does contain five comment occurrences, listed below. - -### Identifier map - -Counts include declarations and references, including tests, but exclude comments and strings. -Identical local spellings in different scopes share a row; these are not symbol-resolution counts. - -| Existing spelling | Replacement | Occurrences | -|---|---|---:| -| `CellKey` | `CollectionKey` | 213 | -| `CellKeyFor` | `CollectionKeyFor` | 50 | -| `Cell` | `Collection` | 70 | -| `Cells` | `Collections` | 17 | -| `cellSpec` | `watchSpec` | 38 | -| `SourceCell` | `SourceCollection` | 16 | -| `sourceCell` | `sourceCollection` | 13 | -| `sourceCellForLog` | `sourceCollectionForLog` | 11 | -| `sourceCellForEvents` | `sourceCollectionForEvents` | 4 | -| `refusalCell` | `refusalCollection` | 3 | -| `RefusedCell` | `RefusedCollection` | 7 | -| `RefusedCellSet` | `RefusedCollectionSet` | 6 | -| `CellSet` | `CollectionSet` | 4 | -| `cellsForWatchKeys` | `collectionsForWatchKeys` | 5 | -| `deduplicateCells` | `deduplicateCollections` | 4 | -| `keysByCell` | `keysByCollection` | 3 | -| `sortCells` | `sortCollections` | 6 | -| `describeCells` | `describeCollections` | 4 | -| `cell` | `collection` | 180 | -| `cells` | `collections` | 48 | -| `byCell` | `byCollection` | 2 | -| `gotCell` | `gotCollection` | 2 | -| `planCell` | `planCollection` | 18 | -| `configMapCell` | `configMapCollection` | 8 | -| `cellForTest` | `collectionForTest` | 4 | -| 24 test function names containing `Cell`/`Cells` | Replace the collection component in each name | 24 | - -`Cell` covers both `targetWatchKey.Cell()` and the fields on `ResyncScope` and `gitPathRefusal`. -`sourceCell` covers both `targetWatchStream.sourceCell()` and `WriteRequest.sourceCell()`. -Rename their actual declarations and callers together; avoid substring replacement across all Go. - -### File moves - -| Existing file | New path | -|---|---| -| `internal/types/cell.go` | [`internal/types/collection.go`](../../internal/types/collection.go) | -| `internal/types/cell_test.go` | [`internal/types/collection_test.go`](../../internal/types/collection_test.go) | -| `internal/git/source_cell.go` | [`internal/git/source_collection.go`](../../internal/git/source_collection.go) | -| `internal/git/source_cell_test.go` | [`internal/git/source_collection_test.go`](../../internal/git/source_collection_test.go) | - -Keep the source-provenance rationale with its renamed helper. Update Markdown links and plain -path references in Go comments, including `watchrule_types.go`, `watched_type_resolver.go`, and -`internal/git/types.go`. New paths above are planned paths; their links become valid after the move. - -## Walk through the affected behavior - -### Identity, planning, and watch lifecycle - -Start with the key and constructor in `internal/types`. Preserve equality, comparability, -`String()` output, namespace matching, and version omission byte for byte. Then update -[`target_watch_plan.go`](../../internal/watch/target_watch_plan.go): `Collections` maps keys to -`watchSpec`; `Keep`, `Start`, `Restart`, and `Stop` retain their meaning and ordering. Operation-only -differences disappear from that specification; served-version changes and forced restarts remain. - -In [`target_watch.go`](../../internal/watch/target_watch.go), rename key conversion and provenance -helpers and update every keyed map, loop variable, and cancellation comment. Adding an unrelated -rule still preserves running watches; changing served version remains a restart of the same -collection. This phase does not change cancellation timing or fix the known replacement race. - -[`stream_readiness.go`](../../internal/watch/stream_readiness.go), -[`watch_plane_state.go`](../../internal/watch/watch_plane_state.go), and namespace planning must -use the same key. Preserve map-copy isolation and the weakest-state roll-up per resource type. -A type observed in three namespaces still counts as one type in public status. - -### Git writes, resyncs, and refusals - -Rename provenance in `Event`, `ResyncRequest`, write aggregation, queue logging, and -[`refusal_reapply.go`](../../internal/git/refusal_reapply.go). Preserve these distinct cases: - -- A single producer supplies its collection; mixed-producer or whole-target work remains unclaimed. -- `ResyncRequest.refusalCollection()` continues deriving its key from the resync scope. It must - not be replaced by diagnostic `SourceCollection`, which answers a different question. -- A successful collection clears only that collection's write refusal and queued refusal commit. - Whole-target scan refusals retain their separate meaning and precedence. -- A nil `ResyncScope` continues covering the whole target. `CollectionKey{}` as provenance means - unclaimed; it must not become an all-resources wildcard in `Matches`. - -[`markResyncTailForWriteLocked`](../../internal/git/branch_worker.go) still matches target and -structural resource identity, including payload-free deletes. Provenance remains diagnostic; -no consumer-side watch-plan filter or generation fence is introduced. - -### Readiness, render fidelity, and retention - -Update both halves of the render-fidelity gate and all acceptance/retention maps. Scope revisions -must remain attached to the watch incarnation that produced each report. Kept watches retain -their result; stale canceled reports remain rejected. Renaming `CellSet`/`RefusedCellSet` must -preserve the distinction between a scoped refusal and a target-wide refusal. - -Retention continues reporting documents the prune policy kept outside the selected snapshot. -Counts, revision checks, and the existing overlap-counting limitation remain as implemented. -Neither retention nor readiness changes units in this phase. - -## Compatibility surfaces - -| Surface | Step 1 contract | -|---|---| -| Internal Go types, fields, helpers, and tests | Direct rename; no compatibility aliases | -| `cell` log key, 13 occurrences | `collection` | -| `sourceCell` log key, 7 occurrences | `sourceCollection` | -| `keepCells`, `startCells`, `restartCells`, `stopCells`, one each | Corresponding `*Collections` keys | -| Duplicate-collection error in `targetWatchPlanFor`, one occurrence | Replace the word `cell`; retain detection and values | -| `CollectionKey.String()` and `ResyncScope.String()` | Preserve rendered values; coalescing/refusal keys use them | -| CRD fields, condition types/reasons, printer columns | Remove only the two `rules[].operations` properties; preserve the remaining contract | -| Metrics and their labels/values | Retire `watch_events_total`'s `operation_filtered` outcome; preserve remaining definitions | -| Redis cursor and author keys, TTLs | Preserve bytes and versions | -| Plan fingerprints and log descriptions | Remove operation components; preserve collection/version identity | -| Git paths, placement, encryption, prune policy | Preserve behavior; every live object event now reaches the normal processing path | - -The six log keys change in place. Document the mapping in `docs/UPGRADING.md` when implementing, -using present tense as required by the repository. Mention the change for log-query consumers; -do not invent duplicate old/new fields as a compatibility layer. - -[`watchCursorKey`](../../internal/queue/redis_store.go) already encodes target UID, group/resource, -and namespace under `:watch:v1:`. It contains no cell segment. Preserve its golden strings in -[`key_prefix_test.go`](../../internal/queue/key_prefix_test.go), including the arbitrary -`cell-a:tenant-7` user prefix. That sample is not collection identity. Step 2 owns selector-aware -cursor identity and any resulting storage-version decision. - -The five rename occurrences in [`watchrule_types.go`](../../api/v1alpha3/watchrule_types.go) are -rationale or helper comments. Keep them outside field doc comments and Kubebuilder marker blocks. -Use the repository's description-stripped schema comparison for comment-only edits. For the full -step, allow exactly the two `rules[].operations` property removals, including their item enums; -all remaining validation, resource scope, defaults, required fields, and serialization must match. - -### Adjacent vocabulary reviewed, outside this rename - -These counts are Go identifier occurrences in tracked files, including tests. They are separate -from the cell census. They identify the cost of a broader migration without folding it into this -prerequisite. - -| Existing identifier(s) | Occurrences | Disposition | -|---|---:|---| -| `targetWatchStream`, `targetWatchStreams`, `startTargetWatchStreams` | 18 / 6 / 3 | Keep in step 1; managed-watch naming can be changed independently | -| `targetWatchReplayAndStream`, `foldTargetReplayEvent`, `enqueueReplayResync` | 6 / 5 / 3 | Keep identifiers; prose uses initialization/initial events | -| `StreamState`, `StreamStateReplaying` | 10 / 15 | Keep; changing public state/reason vocabulary needs its own complete map | -| `targetStreamStatus`, `StreamSummary`, `StreamsStatus` | 30 / 34 / 14 | Keep status vocabulary and type-count semantics | -| `watchCursorKey` | 9 | Keep; resume-resourceVersion storage remains unchanged | -| `resyncPlan` | 2 | Keep the existing Git snapshot-reconciliation helper | - -A watch stream, Redis fact stream, and table cell are valid terms. The acceptance criterion is -removal of the project-specific collection synonym, not removal of those words from every file. -In particular, renaming `status.streams` to `collections` would misdescribe its current type count. - -## Documentation work - -Update all 18 current documents in the inventory below, including diagrams, section titles, -code examples, and references to types or test names. Keep release/commit/branch quotes literal. -The plan itself necessarily records the old spelling for migration; it is not an active alias. - -- In Definitions, remove the transitional `CellKey`/`SourceCell` explanation when the code lands. - Define the collection directly and point broader status vocabulary to its unchanged contract. -- Update architecture, components, commit-message scope descriptions, and metrics explanations. - A metric still describes the same observation after its explanatory prose says collection. -- Rename the heading “What a cell leaving means” in `target-watch-plan.md` and fix its inbound - anchor in `typeset-owns-discovery-grace.md`, plus prose citations in `internal/typeset/lifecycle.go`, - `docs/TODO.md`, `data-plane-triggering.md`, and `watch-manager-ownership.md`. -- Review `watch-and-catalog-architecture.md` and `target-watch-plan.md` as older design records. - Updating vocabulary does not adopt their old removal proposals or add missing implementations. -- Rebase step 2's source references and examples onto `CollectionKey`, `SourceCollection`, and - `watchSpec`. Keep its decisions about cluster authority and full `Always` support intact. -- Remove operation-filter instructions from `configuration.md`, `architecture.md`, `components.md`, - and the older watch-plan/source-scope designs. Their namespace isolation argument still applies. - Update `spec/gittarget-isolation-on-rule-change.md` to describe duplicate collection selection - without operation unions, and `spec/watch-event-ordering-and-attribution-grace.md` to remove the - filter before attribution. Update `interpreting-metrics.md` and `design/metrics-observability-plan.md` - for the retired ingest outcome. Cover the changed rule contract in `README.md` and `UPGRADING.md`. - -Historical exceptions: the one CHANGELOG entry, 13 occurrences in the two `docs/finished/` records, -and three literal branch/deleted-symbol references in `watch-plane-status-convergence-failures.md`. -Preserve their historical spelling. If a historical link points at a moved source file, update -its destination or pin the historical source; do not leave a broken relative link. - -Unrelated exclusions are deliberate: table/layout cells in `hack/crdfields`, -`hack/gitops-layouts-baseline.sh`, Markdown lint/style examples, the Git request ledger, manifest -comparison tables, census metric buckets, the push-conflict matrix, and sparse expansion matrices. -The sample Redis prefix is also unrelated. These account for the 55 unrelated occurrences. - -## Implementation sequence - -1. **Capture the baseline.** Recount the working tree, record the commit and any existing changes, - and classify new hits. Confirm the desired `CollectionKey`/`watchSpec` names do not collide. -2. **Rename the Go graph and four files together.** Start with types and fields, then planner, - watch/queue producers, Git consumers, gates, and tests. Keep the declarations and every caller - in one reviewable change. Use symbol-aware edits; no temporary alias layer is needed. -3. **Remove operation selection across the same graph.** Apply the API, planner, matcher, live - routing, internal-tag, metric, and packaging changes above. Remove filter-specific tests while - preserving their independent namespace, version, ownership, and copy-isolation assertions. -4. **Update diagnostics and documentation.** Apply the six log-key mappings, the error message, - prose, paths, headings, and affected test names. Cover the deliberate behavior/API change and - diagnostic changes in the upgrade entry. The original rename counts are a baseline; some of - those sites disappear with the filter removal instead of being renamed. -5. **Inspect generated output and run the checks below.** Verify the exact schema delta and new - behavior, as well as the preserved invariants. No selector implementation belongs here. -6. **Land step 1 independently.** Its scope is terminology plus removal of event selection. - Rebase the label-selection plan onto it before beginning step 2. - -No Go renames are performed by creating this document. Implementation and its full validation -suite are the next task. - -### Validation for the implementation - -Run `task fmt`, `task generate`, `task manifests`, `task helm-sync`, `task settings-index`, and -`task vet`, then the required `task lint` and `task test`. Validate Helm rendering/schema and CRD -installation with the changed rule shapes. Preserve any coverage-baseline increase. Verify Docker -with `docker info` before running `task test-e2e`; run e2e sequentially as the repository requires. -Because the change spans watch plumbing, follow the repository's local-e2e-before-push rule if publishing. - -Reuse existing tests for the rename; add focused regressions for operation-filter removal. -Inspect coverage of these preserved behaviors: - -| Invariant | Existing evidence to preserve | -|---|---| -| Versionless identity, namespace matching, stable display | `cell_test.go` (renamed), `target_watch_plan_test.go` | -| One watch per collection across served versions; separate namespace scopes | `target_watch_test.go`, `source_namespace_stream_summary_test.go` | -| Unrelated edits keep watches/readiness; forced changes restart the expected set | `target_watch_plan_test.go`, `target_watch_test.go` | -| Nil/zero/mixed provenance and diagnostic rendering | `source_cell_test.go` (renamed), `git_path_refusal_test.go` | -| Scoped snapshots protect sibling namespaces and preserve prune modes | `resync_scope_test.go` in watch and Git packages | -| FIFO and payload-free delete fences prevent unsafe resync coalescing | `internal/git/resync_push_test.go` | -| A recovered collection cannot clear another's refusal | `refusal_observation_test.go`, `git_path_acceptance_scan_test.go` | -| Stale revisions cannot reopen fidelity or restore retired retention counts | `render_fidelity_gate_test.go`, `retention_rollup_test.go` | -| Cursor bytes and prefixes are stable | `internal/queue/key_prefix_test.go`, `redis_store_test.go` | -| Type-count status unchanged; exactly two API properties removed | Stream-summary/controller tests and generated-schema comparison | - -Required operation-removal cases: - -| Case | Expected result / test location | -|---|---| -| Both rule kinds omit `operations` | Valid CRs; selected collections observe creates, updates, and deletes | -| Old manifests still send the field | Raw requests with `fieldValidation=Strict` fail; `Warn`/`Ignore` drop it, with all-event behavior; cover both CRDs in envtest | -| Helm starter rule still supplies the key | Values-schema validation rejects it; default and corrected examples render and install | -| Live create, changed update, physical delete, terminating `MODIFIED` | Normal mapping and routing in `target_watch_test.go`; no operation-selection gate | -| Delete with each prune mode | `OnEvent`/`Always` remove; `Never` retains; snapshot-only absence still removes only under `Always` | -| Create, unchanged update, delete, recreate under the same name | No-op update suppression survives; delete clears cached content so recreation is written | -| Duplicate rules and several served versions | One collection per namespace scope; deterministic version choice; no operation-union state | -| Named and all-namespace selection coexist | Both scopes remain; removing one rule preserves the other and its readiness | -| Rule-store snapshot isolation | Remaining group/version/resource/namespace slices still cannot mutate stored state | -| A version edit or forced recheck | Restarts the expected collections; replace the old operation-edit restart test with a version edit | -| New watch incarnation with a saved cursor | Starts with a complete snapshot; `OnEvent` does not invent deletions missed by old filters | -| Audit create, PATCH/UPDATE, delete/deletecollection | Same tag strings, body priority, and attribution behavior after the type moves to `internal/types` | -| Delivered watch events and ingest metrics | Exactly one existing applicable outcome each; none emits `operation_filtered` | - -### Completion criteria - -- No collection-related old identifier, comment, live log key, or active explanatory term remains - in the audited source graph. No `CellKey` alias or duplicate old/new representation survives. -- Four files are moved, source/doc links resolve, and surviving affected test names use collection. -- The only old spellings remaining are the explicit unrelated/historical exceptions and migration - documentation. Raw `rg cell` is not a valid zero-occurrence gate because it also matches - `cancelled` and unrelated table cells. -- Both rule operation fields and their schema, starter-config, matcher, planner, and live-filter - machinery are gone. Internal operation tags remain; no replacement selection switch exists. -- Outside the two removed API properties and retired metric outcome, serialized contracts and - metric definitions are unchanged. Redis keys, prune policy, collection equality/matching, - ordering, revision behavior, and status units remain intact. All object events are processed; - formerly filtered live events can now change Git. Diagnostics match the documented table. -- Required checks pass and step 2 references the renamed interfaces. Label support is still unbuilt. -- Remove this plan's line from [`.docs-lint-scope`](../../.docs-lint-scope). It is gated only while - the plan is active; once this step ships, the plan is a historical record and leaves the prose gate. - -## Handoff to step 2 - -| Concern | After step 1 | Label-selection implementation | -|---|---|---| -| Collection identity | Group/resource/namespace | Add canonical selector consistently | -| Watch specification | Served version only; all object events observed | Preserve the same separation from identity; no operation filter | -| Rule contract | Resources, groups, versions, namespace scope | Add `objectSelector`; prune mode still controls removals | -| Provenance | `SourceCollection`, diagnostic | Carry selector-aware identity; retain raw attribution evidence | -| Git sweep predicate | Structural `ResyncScope.Matches` | Keep structural ownership; compare with the selected snapshot | -| Queue coalescing | Existing identifier-based ordering fence | Include selector identity; preserve the structural write fence | -| Watch replacement | Existing cancellation behavior | Add the producer handoff specified by step 2 | -| Redis resume position | Existing `:watch:v1:` bytes | Prevent resume under a different selector; decide format migration there | -| Pruning/Kustomize/attribution | Existing behavior | Full `Always`, shared semantic removal, and filtered-removal attribution | - -Do not make `CollectionKey.Matches` evaluate labels when step 2 adds a selector. The object set -observed by a watch and the structural scope of Git objects owned by its mirror answer different -questions. Step 1 gives them accurate names; step 2 establishes the new membership behavior. - -## Per-file audit - -This is the baseline inventory, not a claim that implementation has happened. `IDs` counts Go -identifier occurrences; `comments` and `strings` count matched words/components inside those tokens. - -### Go files - -| File | IDs | Comments | Strings | Total | -|---|---:|---:|---:|---:| -| [`api/v1alpha3/watchrule_types.go`](../../api/v1alpha3/watchrule_types.go) | 0 | 5 | 0 | 5 | -| [`internal/authz/source_namespace.go`](../../internal/authz/source_namespace.go) | 0 | 3 | 0 | 3 | -| [`internal/authz/source_namespace_test.go`](../../internal/authz/source_namespace_test.go) | 0 | 1 | 3 | 4 | -| [`internal/git/branch_worker.go`](../../internal/git/branch_worker.go) | 14 | 7 | 5 | 26 | -| [`internal/git/commit.go`](../../internal/git/commit.go) | 4 | 0 | 0 | 4 | -| [`internal/git/git_path_refusal.go`](../../internal/git/git_path_refusal.go) | 7 | 2 | 1 | 10 | -| [`internal/git/git_path_refusal_test.go`](../../internal/git/git_path_refusal_test.go) | 16 | 0 | 0 | 16 | -| [`internal/git/refusal_observation_test.go`](../../internal/git/refusal_observation_test.go) | 9 | 9 | 3 | 21 | -| [`internal/git/refusal_reapply.go`](../../internal/git/refusal_reapply.go) | 23 | 14 | 1 | 38 | -| [`internal/git/refusal_reapply_test.go`](../../internal/git/refusal_reapply_test.go) | 2 | 2 | 0 | 4 | -| [`internal/git/render_fidelity_gate.go`](../../internal/git/render_fidelity_gate.go) | 13 | 6 | 0 | 19 | -| [`internal/git/render_fidelity_gate_test.go`](../../internal/git/render_fidelity_gate_test.go) | 16 | 7 | 8 | 31 | -| [`internal/git/resync_flush.go`](../../internal/git/resync_flush.go) | 2 | 3 | 0 | 5 | -| [`internal/git/resync_push_test.go`](../../internal/git/resync_push_test.go) | 8 | 0 | 1 | 9 | -| [`internal/git/resync_scope_test.go`](../../internal/git/resync_scope_test.go) | 2 | 1 | 1 | 4 | -| `internal/git/source_cell.go` | 13 | 17 | 0 | 30 | -| `internal/git/source_cell_test.go` | 19 | 0 | 4 | 23 | -| [`internal/git/types.go`](../../internal/git/types.go) | 17 | 19 | 0 | 36 | -| [`internal/git/write_boundary_precondition_test.go`](../../internal/git/write_boundary_precondition_test.go) | 1 | 0 | 0 | 1 | -| [`internal/rulestore/store.go`](../../internal/rulestore/store.go) | 0 | 1 | 0 | 1 | -| [`internal/telemetry/exporter.go`](../../internal/telemetry/exporter.go) | 0 | 1 | 0 | 1 | -| `internal/types/cell.go` | 6 | 11 | 0 | 17 | -| `internal/types/cell_test.go` | 10 | 3 | 2 | 15 | -| [`internal/typeset/lifecycle.go`](../../internal/typeset/lifecycle.go) | 0 | 2 | 0 | 2 | -| [`internal/watch/config_plane_split_review_fixes_test.go`](../../internal/watch/config_plane_split_review_fixes_test.go) | 1 | 0 | 0 | 1 | -| [`internal/watch/event_router.go`](../../internal/watch/event_router.go) | 22 | 0 | 6 | 28 | -| [`internal/watch/event_router_test.go`](../../internal/watch/event_router_test.go) | 8 | 1 | 0 | 9 | -| [`internal/watch/git_path_acceptance.go`](../../internal/watch/git_path_acceptance.go) | 29 | 9 | 0 | 38 | -| [`internal/watch/git_path_acceptance_scan_test.go`](../../internal/watch/git_path_acceptance_scan_test.go) | 14 | 2 | 2 | 18 | -| [`internal/watch/git_path_acceptance_test.go`](../../internal/watch/git_path_acceptance_test.go) | 53 | 4 | 1 | 58 | -| [`internal/watch/gitpath_events.go`](../../internal/watch/gitpath_events.go) | 0 | 2 | 0 | 2 | -| [`internal/watch/manager.go`](../../internal/watch/manager.go) | 3 | 1 | 0 | 4 | -| [`internal/watch/owner_observability.go`](../../internal/watch/owner_observability.go) | 0 | 1 | 0 | 1 | -| [`internal/watch/owner_test.go`](../../internal/watch/owner_test.go) | 4 | 3 | 0 | 7 | -| [`internal/watch/render_fidelity_gate.go`](../../internal/watch/render_fidelity_gate.go) | 19 | 4 | 2 | 25 | -| [`internal/watch/resync_scope_test.go`](../../internal/watch/resync_scope_test.go) | 1 | 0 | 0 | 1 | -| [`internal/watch/retention_rollup.go`](../../internal/watch/retention_rollup.go) | 12 | 12 | 5 | 29 | -| [`internal/watch/retention_rollup_test.go`](../../internal/watch/retention_rollup_test.go) | 40 | 8 | 4 | 52 | -| [`internal/watch/source_namespace_planning_test.go`](../../internal/watch/source_namespace_planning_test.go) | 0 | 3 | 0 | 3 | -| [`internal/watch/source_namespace_stream_summary_test.go`](../../internal/watch/source_namespace_stream_summary_test.go) | 1 | 3 | 1 | 5 | -| [`internal/watch/source_namespace_test.go`](../../internal/watch/source_namespace_test.go) | 1 | 1 | 0 | 2 | -| [`internal/watch/stream_readiness.go`](../../internal/watch/stream_readiness.go) | 49 | 7 | 0 | 56 | -| [`internal/watch/stream_readiness_test.go`](../../internal/watch/stream_readiness_test.go) | 24 | 1 | 1 | 26 | -| [`internal/watch/target_watch.go`](../../internal/watch/target_watch.go) | 93 | 42 | 0 | 135 | -| [`internal/watch/target_watch_plan.go`](../../internal/watch/target_watch_plan.go) | 64 | 23 | 5 | 92 | -| [`internal/watch/target_watch_plan_test.go`](../../internal/watch/target_watch_plan_test.go) | 120 | 6 | 14 | 140 | -| [`internal/watch/target_watch_session_end_test.go`](../../internal/watch/target_watch_session_end_test.go) | 1 | 2 | 1 | 4 | -| [`internal/watch/target_watch_test.go`](../../internal/watch/target_watch_test.go) | 13 | 13 | 6 | 32 | -| [`internal/watch/watch_event_metrics.go`](../../internal/watch/watch_event_metrics.go) | 0 | 2 | 0 | 2 | -| [`internal/watch/watch_event_metrics_test.go`](../../internal/watch/watch_event_metrics_test.go) | 0 | 1 | 0 | 1 | -| [`internal/watch/watch_plane_state.go`](../../internal/watch/watch_plane_state.go) | 5 | 5 | 0 | 10 | -| [`internal/watch/watched_type_resolver.go`](../../internal/watch/watched_type_resolver.go) | 1 | 5 | 0 | 6 | -| [`test/e2e/audit_route_attribution_e2e_test.go`](../../test/e2e/audit_route_attribution_e2e_test.go) | 0 | 2 | 0 | 2 | -| [`test/e2e/refusal_reapply_e2e_test.go`](../../test/e2e/refusal_reapply_e2e_test.go) | 0 | 1 | 0 | 1 | - -### Current documents - -The counts omit the unrelated matrix cell in `docs/INDEX.md` and the three literal historical -references in the status investigation. Other historical and unrelated documents are excluded. - -| Document | Occurrences to review | -|---|---:| -| [`docs/INDEX.md`](../INDEX.md) | 5 | -| [`docs/TODO.md`](../TODO.md) | 4 | -| [`docs/UPGRADING.md`](../UPGRADING.md) | 2 | -| [`docs/architecture.md`](../architecture.md) | 6 | -| [`docs/commit-messages.md`](../commit-messages.md) | 1 | -| [`docs/components.md`](../components.md) | 9 | -| [`docs/definitions.md`](../definitions.md) | 6 | -| [`docs/design/data-plane-triggering.md`](../design/data-plane-triggering.md) | 14 | -| [`docs/design/gittarget-configuration-freshness.md`](../design/gittarget-configuration-freshness.md) | 1 | -| [`docs/design/source-scope-simplification.md`](../design/source-scope-simplification.md) | 15 | -| [`docs/design/target-watch-plan.md`](../design/target-watch-plan.md) | 96 | -| [`docs/design/watch-and-catalog-architecture.md`](../design/watch-and-catalog-architecture.md) | 76 | -| [`docs/design/watch-manager-ownership.md`](../design/watch-manager-ownership.md) | 11 | -| [`docs/design/watch-plane-status-convergence-failures.md`](../design/watch-plane-status-convergence-failures.md) | 12 | -| [`docs/design/watches-labels-simplification.md`](../design/watches-labels-simplification.md) | 3 | -| [`docs/interpreting-metrics.md`](../interpreting-metrics.md) | 1 | -| [`docs/spec/typeset-owns-discovery-grace.md`](../spec/typeset-owns-discovery-grace.md) | 1 | -| [`docs/spec/watch-event-ordering-and-attribution-grace.md`](../spec/watch-event-ordering-and-attribution-grace.md) | 2 | - -### Reproduce the census - -Start from tracked files so ignored `docs/local-only/`, external checkouts, module caches, and -build artifacts do not inflate the plan. This component expression avoids `cancelled`: - -```bash -git grep -n -E 'Cell(s)?([^a-z]|$)|(^|[^A-Za-z0-9])(cell|cells|CELL|CELLS)([A-Z0-9_]|[^A-Za-z0-9_]|$)' -- '*.go' '*.md' -git ls-files '*cell*' '*Cell*' -``` - -That command is for finding candidates; it prints matching lines, so its line count is not the -occurrence count above. To reproduce the precise totals, enumerate `git ls-files -z`, scan Go -with `go/scanner` using `scanner.ScanComments`, and count matching components separately in -`token.IDENT`, `token.COMMENT`, and `token.STRING`. Scan other text using the same component rule, -then apply the unrelated/historical exclusions documented above. Recount on the implementation -base; this plan and new cross-references intentionally add migration mentions after the baseline. diff --git a/docs/design/commit-timing-surface.md b/docs/design/commit-timing-surface.md index d9ee7ccae..ad51c32ce 100644 --- a/docs/design/commit-timing-surface.md +++ b/docs/design/commit-timing-surface.md @@ -31,7 +31,7 @@ > ``` > > It replaces `CommitRequest.spec.closeDelay` and the string `GitTarget.spec.commit.window`, and it -> supersedes the timer model in [`commitrequest-save-wait-options.md`](commitrequest-save-wait-options.md#the-timer-model). +> supersedes the timer model in `commitrequest-save-wait-options.md` (removed). ## Why diff --git a/docs/design/commitrequest-save-wait-options.md b/docs/design/commitrequest-save-wait-options.md deleted file mode 100644 index bf33a53b1..000000000 --- a/docs/design/commitrequest-save-wait-options.md +++ /dev/null @@ -1,513 +0,0 @@ -# What a save should wait for - -> **design, decided**: a plan to execute, not an open question. Index: [`../INDEX.md`](../INDEX.md) -> -> The short version: `spec.closeDelay` is a timer standing in for a happens-before the API -> never expresses. A request names a `GitTarget` and has no way to name the write it means to -> capture, so the integrator is asked to guess a duration that covers a latency belonging to the -> cluster's audit configuration. Its default, `0`, is shorter than the minimum possible -> window-open latency, so the primary use case cannot work unmodified. -> -> The decision: repair the default, then give a request a way to **name the write it is waiting -> for**, using the `(uid, resourceVersion)` identity the attribution join already keys on. Five -> phases, each shippable alone. The one invariant none of them may weaken: a `CommitRequest` -> finalizes its own submitter's work and nobody else's. - -## The report this comes from - -An integrator ran 0.46.0 against a live cluster and wired the product's "save now" path: a -participant edits a custom resource, the backend creates a `CommitRequest` so the edit reaches Git -while the operator is standing in front of it. It had never worked, and nothing reported a failure. -Every request resolved `Ready=True` with reason `NoWindowInGrace`, whose message reads "no matching -open commit window was collected within the grace; nothing was pending to save". - -Something was pending. It had not arrived yet. The controller log for one save, in order: - -```text -10:47:32 admission.validate-operator-types recorded command author at admission -10:47:32 worker-manager.branch-worker CommitRequest registered with worker closeDelaySeconds=0 -10:47:32 worker-manager.branch-worker CommitRequest resolved outcome=NoOpenWindow sha="" -10:47:32 CommitRequestReconciler CommitRequest finalized reason=NoWindowInGrace age=153ms -10:47:32 worker-manager.branch-worker Opening commit window <-- after -10:47:37 worker-manager.branch-worker Finalizing open commit window reason=timer attachedCR=false -10:47:40 git commit created message="chore(demo2): 1 change from demo:CgZzaW1vbjQSCXJvb20tcGFzcw" -``` - -The request was created, attributed, evaluated, and finalized before the write it exists to publish -reached the worker. The edit still committed, five seconds later on the window's own timer, under -the target's `liveTemplate` instead of the sentence the participant typed. From the cluster's point -of view nothing failed. From the operator's, the save button did nothing. - -The reporting integrator has since set a two-second close delay client-side, which moves the failure -from "always" to "whenever the cluster is slower than two seconds". That is a workaround at one -call site, not a fix: the operator still ships a default that cannot work. - -## The shared-object save shape - -The reported case is not only a single-user latency problem. A public interactive editor can put -many distinct people in front of one shared resource. Each person edits the same object, the form is -fed by a watch, and stale forms are handled in the application before the save is accepted. The -operator can rely on three facts: - -- every participant has a distinct Kubernetes identity, name, and email address; -- every accepted write returns the object's current `uid` and `resourceVersion`; -- the caller cannot know where that write will land in the watch stream relative to other accepted - writes. - -That shape is general. It is any collaborative editor that writes one API object and wants the Git -history to remain legible: "Sanne changed this value because ..." rather than "some later window -closed with the wrong message". - -```mermaid -sequenceDiagram - participant UserA as User A - participant UserB as User B - participant API as Kubernetes API - participant Watch as Watch stream - participant Worker as Branch worker - - UserA->>API: PATCH shared object - API-->>UserA: 200 OK, uid, resourceVersion=41 - UserA->>API: Create CommitRequest(message, wait for rv 41) - UserB->>API: PATCH shared object - API-->>UserB: 200 OK, uid, resourceVersion=42 - UserB->>API: Create CommitRequest(message, wait for rv 42) - API-->>Watch: Event rv 41, author User A - API-->>Watch: Event rv 42, author User B - Watch-->>Worker: Attributed events, maybe delayed by audit batching - Worker-->>Worker: Match each request to its named write -``` - -The unique identities keep one participant's request from claiming another's open window. They are -not a causal token by themselves: "the next activity from this author" is safe across people but -still approximate for one person who submits twice, retries after a network hiccup, or has an -earlier same-author window already open. The `resourceVersion` from the accepted write is the -smallest token the UI already has that can name the save it wants Git to show. - -## What happens today - -Four mechanisms decide the outcome, and the failure is structural rather than a tuning miss. - -**The deadline is stamped from receipt.** `handleAttachCommitRequest` in -[`commit_request_attach_loop.go`](../../internal/git/commit_request_attach_loop.go) sets -`finalizeAt = time.Now().Add(closeDelay)` on first registration, and idempotent re-sends keep -that first value. With the field omitted the deadline is the present instant, so the request is due -on the next pass of the worker's event loop. - -**The request is evaluated on the fast path.** Authorship comes from the validating admission -webhook, resolved synchronously and present-or-never: the record is written before the object is -visible, so `CommitRequestReconciler` attaches the instant it first sees the object. - -**The write travels the slow path.** A watch event is held head-of-line inside the watch goroutine -by [`author_resolver.go`](../../internal/watch/author_resolver.go) until a matching audit fact -arrives or `--author-attribution-grace` (3s) expires. On the reporting cluster the API server posts -audit in batches bounded by `--audit-webhook-batch-max-wait=1s`, so the floor on window-open latency -is bounded below by that batch plus the join. - -**Attach is author-bound.** `matchesWindow` in -[`commit_request_attach.go`](../../internal/git/commit_request_attach.go) admits a window only when -the author strings are equal, the two sides agree on whether an actor is named, and the `GitTarget` -identity matches. - -So the request is evaluated against a window that is still in flight, and the two paths are -asymmetric by design. A `0` deadline is not tight. It is shorter than the smallest latency the -architecture can produce. - -## The invariant, and why it is the first test - -**A `CommitRequest` finalizes the work of the actor that submitted it, and never anyone else's.** -There are no exceptions, and no phase below may introduce one. It has two halves, both enforced by -`matchesWindow` today: - -- a request with a named submitter claims only that actor's named window; -- a request with no named submitter claims only a window that names no actor. - -The second half is what keeps an unattributed request from picking up a named person's edits when -admission capture is off or missed. The comparison stays on `AttributionOutcome.NamesActor()` rather -than on enum equality, because the window's outcome and the request's are produced by two -independently configured subsystems, and the code comment on `matchesWindow` records what breaks in -both deployments when that is got wrong. - -**A satisfied wait does not imply an attach.** This is the consequence that matters most, and the -one easiest to state wrongly. A wait asks "has the write I named been observed"; attach asks "may I -claim this window". They are different questions with different keys, and both must be answered. A -request can wait successfully and still land on `WindowMismatch` — see situation 7. - -## Requirements - -**R1. The committer binding is absolute.** No phase may widen what a request may **attach to**. -A wait may be keyed on anything; attachment stays keyed on `matchesWindow` and nothing weaker. - -**R2. A save must not be able to expire before the write it exists to publish can arrive.** The -configuration that produces this must be impossible to reach by omission. A default that cannot work -is a defect regardless of what the reference text says about it. - -**R3. The bound belongs to the cluster.** The latency being covered is the API server's audit -batching plus the attribution join. An integrator cannot read either from where they stand, so -asking them for a number in seconds asks for a number they cannot derive. - -**R4. No silent nothing-happened.** "You asked before your write arrived", "nothing was pending", -and "your write was already committed under a generated message" are three different events. Today -they are one `Ready=True` with one reason and one message. Every terminal state must be -discriminating enough for a first-time integrator to tell which one they are in. - -**R4a. Shared-object saves keep per-actor intent legible.** When several distinct actors edit the -same resource, their messages must not become a race against watch delivery. A request with enough -information to name the accepted write should be able to wait for that write, and a weaker request -must say what approximation it is making. - -**R5. No transactional promise.** A request does not reserve a window, cannot rename a finalized -commit, and must not suspend the `GitTarget`'s own flush triggers. A causal wait edges toward that -line, so each phase says where the line still is. - -**R6. Compatible by construction, except the broken default.** `CommitRequest.spec` is immutable, so -capability arrives as new optional fields. Apart from an explicit default repair for -`closeDelay`, a request that sets none of the new fields behaves exactly as it does today. - -**R7. Do not tune against the attribution grace.** Sizing a delay against the 3s grace is the -obvious mistake and the wrong bound. When the grace expires with no fact, the event still ships, as -an unresolved window that names no actor, and a named submitter can never claim it (R1). The honest -outcome there is `WindowMismatch`, and no amount of waiting changes it. The delay only ever needs to -cover audit-fact arrival. - -**R8. Bounded cost on the worker.** Waits are serviced on the branch worker's single event-loop -goroutine, which also does the Git work. Per-wake cost must stay proportional to the small number of -pending requests, with a cap on anything a caller can put in a list. - -**R9. Legible while it waits.** What a request is waiting for has to be readable from the object -while it is still waiting. Reconstructing it afterwards from controller logs is too late to help. - -## The situations - -`W` is the write the requester wants published. Today's column is 0.46.0. - -| # | Situation | Today | After the plan | Phase | -|---|---|---|---|---| -| 1 | `W` is already in a claimable open window when the request registers | attaches, then waits out the whole delay | satisfied at once from the open window, then the collect delay | 2 | -| 2 | `W` arrives during the delay | works: this is what `closeDelay: "2s"` buys | unchanged | — | -| 3 | `W` arrives after the delay | `NoWindowInGrace`, `Ready=True`; `W` commits later under `liveTemplate` | wait for `W`, then finalize | 2 | -| 4 | `W` arrived and its window closed before the request registered | `NoWindowInGrace`, indistinguishable from row 5 | **when a usable attribution fact exists**, the index says whether the cluster ever saw `W`, which splits 4 from 5; otherwise unchanged until the ledger. Naming the commit always needs the ledger | 2 (partial) / later | -| 5 | Nothing was pending | `NoWindowInGrace`, correct | unchanged | — | -| 6 | `W` is suppressed by live-content dedup (sanitized content did not change) | nothing opens; `NoWindowInGrace` | `AlreadyPresent`, for the right reason | 3 | -| 7 | Attribution missed, so the window names no actor and the request names one | `WindowMismatch` | unchanged (R7). The wait **is** satisfied and the attach still fails — the terminal message must say both | 2 | -| 8 | Another committer holds the window | `WindowMismatch`, their window untouched | unchanged, permanently (R1) | — | -| 9 | Several distinct actors edit one resource and each sends a save message | safe from cross-author attach, but delay-shaped and unable to prove which object version a message means | each request binds to the accepted write it names | 2 | -| 10 | Render fidelity is not established for the target | live events dropped at `normalWritesAllowed`; `NoWindowInGrace` | the drop happens **inside** the worker, so it can mark a wait naming `W` terminally unsatisfiable, with its own reason | 2, needs the hook named below | -| 11 | The worker queue was saturated (`ErrFinalizeQueueFull`) | the event is dropped; `NoWindowInGrace` | **not fixed by phase 2.** The drop happens upstream in the watch path and never reaches the worker, so the wait times out saying only that `W` was not observed | see the cost below | -| 12 | One actor saves twice against the same object inside one window | the second write subsumes the first at the same path; the window carries one message, so the second request gets no window and its message is lost | both waits satisfy, but one window carries one message: the second needs its own outcome | 2, open | - -Rows 3 and 4 are the reported defect. Row 1 is a latency win from the same change. Rows 7 and 8 are -correct today and constrain every phase. Rows 10 to 12 are the ones the first draft of this page -missed; 12 is still an open question. - -## Why the attribution fact index is not the answer - -The obvious shortcut is to satisfy a wait from the attribution fact index, which is already keyed on -exactly the identity a named write would use — `FactQuery{AuditRoute, GroupResource, UID, -ResourceVersion}` in [`fact_index.go`](../../internal/queue/fact_index.go) — and already has a -register-then-check waiter registry in [`fact_waiters.go`](../../internal/queue/fact_waiters.go). - -It cannot be the satisfaction signal, for three reasons. - -**It is early, not late.** A fact says the API server audited a write. The watch event *blocks -waiting for that fact*. Fact arrival is strictly upstream of the write reaching the worker, so a -fact-keyed wait fires before the window opens — the reported defect, relocated one layer up. In the -trace above the fact must have landed at or before `10:47:32`, which is when the request had -already resolved. - -**It is blind to everything between audit and Git.** It cannot see live-content dedup (row 6 gets -*worse*: the write is audited, so the wait satisfies, then finds no window), the render-fidelity -gate (row 10), queue saturation (row 11), whether this `GitTarget` watches the type, or which -`GitTarget` a write routed to — the index is deliberately one per process, not one per target. - -**It is empty in two supported deployments.** With `--author-attribution=false` there are no facts -at all, and the wire contract refuses a fact that names nobody -([`author_fact.go`](../../internal/queue/author_fact.go)), so a write the audit policy skips is -indistinguishable from one that never happened. - -**What to take from it anyway.** Three things, and they remove most of the "new vocabulary" cost: - -1. `ExactCapable` and `AttributionExact` have already settled this repo's `resourceVersion` policy — - exact `uid`+`resourceVersion` is the match, and a removal's `resourceVersion` is *not* the one the - write produced. The wait must cite that rule, not invent a second one. -2. `factWaiterRegistry`'s register-then-check discipline is the same race the worker-side wait has - (register before scanning the open window, or a write landing in the gap is lost). Mirror it. -3. The **default** fact TTL is 10 minutes (`DefaultAttributionFactTTL`, tunable per install via - `--author-attribution-ttl`), comfortably longer than any wait timeout at its default, so the index - can serve as a retrospective oracle for one question: did the cluster ever see the write you - named? That splits row 4 from row 5 for the cost of a lookup, with no new structure. - - It is **best-effort by construction**, and the plan must not promise more. The lookup answers only - when attribution is on, the audit policy covers the type, and the fact has not aged out — and an - aged-out fact is indistinguishable from one that never arrived, because the index keeps no - tombstone. An operator who lowers the TTL below `waitFor.timeoutSeconds` silently narrows the - window in which the split works. Validation should refuse a `timeoutSeconds` above the configured - TTL rather than let the two drift apart unnoticed; where the lookup cannot answer, row 4 stays - folded into row 5 until the ledger exists. - -## The timer model - -> **Superseded** by [`commit-timing-surface.md`](commit-timing-surface.md), which keeps the idea -> below (one job per field) and gives the fields their final names: `window.attachTimeout` is the -> give-up bound, and `window.maxDuration` and `window.idleTimeout` bound the collection from the -> attach. `waitFor` keeps only the named write. - -Give each field one job, rather than reinterpreting `closeDelay` as a ceiling (R6): - -- `waitFor.timeoutSeconds` is the give-up bound, anchored at receipt. This is where audit-fact - latency is covered, and it is the number a cluster's installer can reason about. -- `closeDelay` keeps its meaning and is measured from **satisfaction**: once the awaited - arrival lands, collect the rest of its burst for this long, then finalize. -- With no `waitFor`, `finalizeAt = receipt + closeDelay`, exactly as today. - -Idempotent re-sends keep the first receipt anchor, and once satisfied, the first satisfaction time. - -Note what this does and does not buy. Rows 2 and 3 get faster and more reliable, because the window -closes a short collect after the write lands rather than after a delay sized for the worst case. -Row 1 is unchanged in wall-clock terms: satisfaction is immediate, so it still finalizes at -receipt + `closeDelay`. The win in row 1 is correctness of the *reason*, not latency. - -## What a save needs today - -Only phase 0 has shipped. Until the rest does, a `CommitRequest` save can be made to work -reliably, but only by satisfying outside the operator what the API cannot yet express. -These are the preconditions, in the order they bite. The first is a blocker; the rest are races you -can widen your way out of. - -**1. The audit policy must cover the type being saved.** If audit emits nothing for the object's -group and resource, or `--author-attribution` is off, the window names no actor while the request -names one, and `matchesWindow` refuses it: `WindowMismatch`, every time, permanently. No value of -`closeDelay` changes this, because nothing is ever going to arrive that the request could -claim (R7). Verify this against the cluster the save actually runs on before tuning anything else. - -**2. Set `closeDelay` generously, and do not size it against the 3s grace.** Phase 0 moved -the default from `"0s"` to `"2s"`, so an omitted field is no longer the broken case; the sizing below is -what a cluster slower than the reference configuration still needs. The number has -to cover audit-fact arrival: the API server's `--audit-webhook-batch-max-wait` plus the attribution -join. At the reference configuration of `1s` that is roughly 1 to 1.5 seconds in practice, so -`"2s"` leaves under a second of headroom and `"4s"` to `"5s"` leaves a margin that survives a -loaded or distant -cluster. The cost of overshooting is that the commit lands a few seconds later; the cost of -undershooting is a silent no-op. They are not symmetric. - -**3. Let the target reach render fidelity before the first save.** Live events are dropped while -render fidelity is not established (situation 10), so the first save after an operator restart is -the one most likely to vanish. Give the target time to settle, or spend one throwaway save on it. - -**4. Do not let one actor save twice against the same object in quick succession.** A window carries -at most one `CommitRequest`, and a second write to the same path subsumes the first inside the open -window, so the second message is lost while its change still commits under the first request's -message — or under `liveTemplate`. This is situation 12, and it is the failure a save button with no -debounce produces on the first double-click. Disable the control for the length of the collect -window. - -**5. Know what else writes the object.** A `/status`-only update is safe: it sanitizes to identical -content and is dropped by live-content dedup before it reaches the worker, which is why that dedup -exists. A controller that adds a finalizer, or stamps a label that survives sanitize, is not safe — -it arrives under a different author and forces an identity-change finalize that splits the collect -window out from under the request. - -**6. Assert on the commit, not on the condition.** Every failure above resolves `Ready=True`. An -integration that checks the `CommitRequest` went green is checking the one thing that is true in -both the working and the broken case; check that the commit message is the one that was typed. - -Requirement R2 exists because points 2 and 3 are load-bearing and undiscoverable, and R4 exists -because point 6 is currently necessary advice. A build that still needs this chapter has not -finished the plan. - -## The plan - -Five phases. Each is shippable alone and leaves the system correct. - -### Phase 0 — repair the default — **shipped** - -Make `CommitRequest.spec.closeDelay` a `*metav1.Duration` with `+kubebuilder:default="2s"`. -(Shipped first as `closeDelaySeconds`, an int; renamed when every duration in the API became a Go -duration string behind the same pattern.) - -A pointer rather than a bare value, decided: a schema default on a bare field means a typed Go -client can no longer express "finalize immediately", because its zero value is not serialized. More -importantly, the cluster-level default flag below is **blocked** on the pointer — once stored, an -omitted field and an explicit `0` are the same value, so nothing downstream can tell them apart. -The type is `v1alpha3` with one known consumer; this is the cheapest it will ever be. - -- `api/v1alpha3/commitrequest_types.go`, then `task generate` and `task manifests` -- nil handling at the controller seam (`commitrequest_controller.go`, where the field is copied onto - `AttachCommitRequest`) -- docs: `configuration.md`, `architecture.md`, `spec/commitrequest-design.md`, an `UPGRADING.md` note - -**Half a day.** Fixes R2. Leaves rows 1, 4, 6 untouched, and moves row 3 from "always" to "when the -cluster is slower than two seconds" — an improvement in reliability, not a change in kind. - -### Phase 1 — carry write identity on the event - -[`sanitize`](../../internal/sanitize/sanitize.go) deletes `uid` and `resourceVersion` before the -event is built in [`target_watch.go`](../../internal/watch/target_watch.go), so an event carries no -version identity downstream. Add them to `git.Event` as watch-observed fields, populated on the -live path only, stamped in `targetWatchGitEvent` where both values are in hand — the author resolver -already reads both at exactly that point. - -No behaviour change; nothing reads the fields yet. - -**Half a day to a day.** Unlocks phases 2 and 3. - -### Phase 2 — the named-write wait - -Add `spec.waitFor`: one named resource, and a timeout. - -```yaml -spec: - gitTargetRef: {name: demo2} - waitFor: - timeoutSeconds: 10 - resources: - - apiVersion: examples.configbutler.ai/v1 - kind: CoffeeConfig - name: demo-coffee - uid: 3f0a... # optional - resourceVersion: "2577872" # optional - closeDelay: "2s" -``` - -The caller already holds these values: they are in the response to the `PATCH` it makes one line -earlier. The list is capped at one resource for now and stays a list because the same shape later -serves multi-object form saves. - -**Satisfaction** is: the named write observed by this worker, either already present in the open -window at registration (row 1) or arriving afterwards (rows 2, 3). Match on `uid` when supplied — -a mismatch is unsatisfiable immediately, since the object was recreated — then on exact -`resourceVersion`, per `AttributionExact`. Register the wait before scanning the open window. - -**Where the row-10 hook lives.** The render-fidelity gate drops a live event *inside* the worker -loop, in `handleQueueItem` before any window code, so the worker does see the write. The hook goes in -that drop branch: it must mark any pending wait naming `W` **terminally unsatisfiable** with its own -reason, not satisfy it. Satisfying before the gate would be the worse bug — the request would arm its -finalize for a write that is never going to be committed, and then report a timeout for the wrong -reason. This is the one place where "observed" and "will reach Git" come apart on purpose. - -**Attach is unchanged.** Satisfaction arms the finalize; `matchesWindow` still decides what may be -claimed (R1). A satisfied wait that cannot attach reports `WindowMismatch` with a message saying the -write was observed but the window named a different actor — row 7. - -**Validation stays in the schema.** The commands webhook captures the submitter and always allows, -so the list is bounded by CRD schema and CEL: a cap on length (R8), kind and name required, namespace -defaulting to the request's own. No admission check that the `GitTarget` watches the named type — -rules change independently of requests, so an inability to satisfy belongs in the timeout message -rather than in a rejection. - -**Where R5 still holds.** The request waits for its write to be observed. It does not reserve the -window, cannot rename a commit that already exists, and does not delay the target's own flush. The -promise is bounded to "I will not give up before my write arrives". - -Also in this phase: the terminal-state work R4 asks for. New reasons and bounded -`gitopsreverser_commit_requests_total{outcome}` values for an unsatisfied wait, separating "the -cluster never saw the write you named" (fact-index lookup, row 4 vs 5), "the target was not -accepting writes" (row 10), and "nothing was pending" (row 5). - -- `api/v1alpha3`, CRD, CEL cap -- `commitrequest_controller.go` plumb; `commitrequest_finalize.go` reasons and messages -- `pendingCommitRequest` satisfaction state and the two-timer model in `commit_request_attach_loop.go` -- `status.waitingFor` on the object while it waits (R9) -- docs: `spec/commitrequest-design.md`, `spec/status-conditions-guide.md`, `interpreting-metrics.md`, - `configuration.md`, `architecture.md` -- unit tests plus a timing-sensitive e2e in `test/e2e/commit_request_e2e_test.go` - -**Three to four days.** Fixes rows 1, 2, 3, 7, 9 outright, and row 10 via the hook above. Splits -row 4 from row 5 only where a usable fact exists. Does **not** fix row 11. - -**Row 11 is a named cost, not an oversight.** A saturated queue is rejected in the watch path — -`RouteToGitTargetEventStream` returns an error and the event is counted as -`watchOutcomeRouteFailed` in [`target_watch.go`](../../internal/watch/target_watch.go) — so the write -never reaches the worker and no worker-side wait can distinguish it from a write that never -happened. Reporting it honestly needs a route-failure signal plumbed from the watch layer into the -`CommitRequest` outcome machinery, which is its own piece of work across a seam nothing else -crosses. The decision for phase 2 is to **accept the gap and say so in the timeout message** — "`W` -was not observed" is true — and to file the plumbing separately rather than let the plan imply a -diagnosis it cannot make. The drop is already counted at the metric, so an operator investigating a -run of unsatisfied waits has the signal; the individual request does not. - -### Phase 3 — observed-write work item - -`skipUnchangedLiveUpdate` drops a write whose sanitized content is unchanged before it reaches the -worker, so a named wait on such a write runs to timeout on a write that was observed and correctly -decided to be a no-op (row 6). - -**Do not route this as an event.** An observation-only `git.Event` carries no `Object`, and — because -the dedup check runs *before* `attachAuthor` — no author either. It would either open a commit window -with an empty author or trip `canAppend` and force an identity-change finalize of the live window, -cutting the very window the request is waiting for. It would also be dropped outright by the -`normalWritesAllowed` fidelity gate. - -Add a fourth arm to `WorkItem` instead — `Observed *ObservedWrite`, beside `Attach` and `Resync` in -[`types.go`](../../internal/git/types.go) — dispatched at the top of `handleQueueItem` before any -window code. Same FIFO, same ordering guarantee, no contact with the write path. - -**One day.** Fixes row 6: `AlreadyPresent` for the right reason. - -### Phase 4 — say when a request asked too early - -The worker knows the difference between "you asked too early" and "nothing was pending": in the -reported trace it opened a matching window microseconds after resolving `NoOpenWindow`. Keep a -short-lived record of recently expired requests, and when a window opens that would have matched one, -say so. - -The terminal status is already written by then, so the reason cannot change retroactively. What is -available is a log line, a counter, and one further reconcile that emits an Event on the object. The -first two are cheap and certain; the third needs the reconciler's terminal short-circuit -(`stampFirstSightConditions` returns early once `Ready` exists) to be re-enterable without -re-running a finalize. - -This is *not* a mirror of `sawForeignWindow`, which is sticky-on-every-pass and deliberately skips -already-due requests. This record fires after expiry and needs its own retention keyed by what the -request would have matched. - -**One day.** Diagnostics only; it is not a substitute for phase 2, because a late Event does not put -the participant's message on the commit. - -### Not scheduled - -- **An observation ledger on the worker**, so a request whose named write was collected and flushed - before it registered can say the write reached Git under a generated message (row 4, fully). Most - of a phase on its own; the fact-index lookup in phase 2 covers the diagnostic half. -- **Multiple named resources** in one `waitFor`, for multi-object form saves. Wait for a consumer. -- **`waitFor.mode: CommitterActivity`** — wait for the first sign of activity from the committer, - received after registration. Lower friction for callers that cannot name a write, and safe by R1 - however coarse it is, because the binding bounds the blast radius to the requester's own work. - Deferred because the shared-object case no longer needs it, and because by its own "arrival, not - existence" rule it cannot satisfy row 1 — where the write has already arrived — so it would - *regress* the case phase 2 fixes outright. -- **`--commit-request-default-close-delay`**, a cluster-level default set by whoever installed the - operator and does know how their API server is configured. Deriving it inside the operator is not - possible: half the input is the API server's `--audit-webhook-batch-max-wait`, which we cannot - read, and the half we do hold is the wrong bound by R7. Unblocked by phase 0's pointer. - -## Open questions - -- **Situation 12.** One actor, two saves, one window: both waits satisfy, the second write subsumes - the first at the same path, and a window carries one message. Does the second request get a - distinct outcome ("your write is in a commit carrying another request's message"), or does it - defer to the next window and risk a timeout? Needs a decision before phase 2 is built. -- **Terminal grading for an unsatisfied wait.** A request whose named write never arrived asserted a - precondition that did not hold. Grading it `Ready=True` keeps "a save with nothing pending is a - success" intact but hides a real failure; grading it `Ready=False` requires `Stalled=True` as well, - because automation is documented to stop on `Ready=True` or `Stalled=True` and a bare - `Ready=False` would hang it forever. -- **Whether the fact-index lookup in phase 2 is worth its coupling.** It is the only thing that - splits row 4 from row 5 before the ledger exists, but it makes the `git` package read a `queue` - structure it does not otherwise touch, and it answers only best-effort (attribution on, type - audited, fact still inside TTL). A reasonable alternative is to skip it entirely in phase 2 and - let the ledger answer row 4 properly later, rather than ship a split that works on some installs. -- **Whether row 11's route-failure plumbing is worth doing at all**, or whether the metric is the - right place to leave it. It is the only situation in the table that phase 2 knowingly does not - improve. - -## Related - -- [`../spec/commitrequest-design.md`](../spec/commitrequest-design.md), the current contract. The - code depends on it, so each phase changes that page in the same commit. -- `commitrequest-admission-authorship.md`, since removed with the admission webhook: where the - submitter came from and why the lookup was present-or-never. -- [`../spec/watch-event-ordering-and-attribution-grace.md`](../spec/watch-event-ordering-and-attribution-grace.md), - the head-of-line wait this page proposes to wait behind rather than shorten. -- [`attribution-removal-wait-options.md`](attribution-removal-wait-options.md), the same shape of - question on the attribution side: what evidence is worth waiting for, and for how long. diff --git a/docs/design/concurrent-author-windows.md b/docs/design/concurrent-author-windows.md index 0ebd89f7a..159ef3536 100644 --- a/docs/design/concurrent-author-windows.md +++ b/docs/design/concurrent-author-windows.md @@ -2,8 +2,15 @@ Status: **proposal, not built.** Raised by gitops-api's review of the source-side save (PR #456): with saves created straight from a browser, a second person's edit splits the first person's change -into two commits. This page sets out the change that would fix that, what it costs, and what is -still open. +into two commits. This page sets out the change that fixes that, the rules it must keep, how to test +it, and the stages to ship it in. + +It splits authors by **destination**: two people editing different objects in different files get +one commit each. Two follow-ups that go further (splitting one object by field, and letting an +incoming Git commit leave windows open) carry different correctness risks (and the first reopens the decided +[field-ownership spec](../spec/manifestedit-field-ownership-spike.md)), so they live on their own +page, [window-field-ownership-and-apply-hold.md](window-field-ownership-and-apply-hold.md). +Neither is needed for the benefit this page delivers. ## The problem @@ -18,21 +25,27 @@ sequenceDiagram participant B as Bob participant W as Branch worker A->>W: edit cm/a - Note over W: window 1 opens: Alice + Note over W: window 1 opens (Alice) B->>W: edit cm/b - Note over W: window 1 closes (identity_change), window 2 opens: Bob + Note over W: window 1 closes (identity_change) + Note over W: commit 1 by Alice: cm/a, generated message + Note over W: window 2 opens (Bob) A->>W: edit cm/c - Note over W: window 2 closes, window 3 opens: Alice + Note over W: window 2 closes (identity_change) + Note over W: commit 2 by Bob: cm/b, generated message + Note over W: window 3 opens (Alice) A->>W: CommitRequest "fix(shop): cart limit" - Note over W: attaches to window 3 only + Note over W: the request attaches to window 3 + Note over W: window 3 closes on its timers + Note over W: commit 3 by Alice: cm/c, her message ``` Alice's change is three files and one save. Git gets two commits for her: `cm/a` under the generated -message, `cm/c` under hers. Nothing fails; the history is just wrong about what belongs together. +message, `cm/c` under hers. Nothing fails; the history is wrong about what belongs together. gitops-api avoids it today by serializing every save and its edits behind one lock in its server -(`commitGate`). A client that creates `RemoteSaveRequest`s from the browser has no such lock, so the -split comes back. +(`commitGate`). A client that creates `CommitRequest`s on a source cluster from the browser has no +such lock, so the split comes back. Two saves by **one** author on one `GitTarget` are a different problem, and it is already handled. A window carries one message, so the second save is refused with `SaveInProgress` while the first @@ -42,93 +55,279 @@ how different authors interleave. ## The proposal Keep a window per **(GitTarget, author, attribution class)** instead of one per worker. A write joins -the open window with its identity, or opens one. Writes by others no longer close it. Each window -keeps its own timers, its own attached request and its own message. +the open window with its identity, or opens one. Writes by others no longer close it, **when the +worker can prove the two writes land in different places in Git** (see +[conflict footprint](#a-write-has-a-conflict-footprint)). Each window keeps its own timers, its own +attached request and its own message. Commits stay whole-object writes, as today. ```mermaid sequenceDiagram participant A as Alice participant B as Bob participant W as Branch worker + Note over W: cm/a, cm/b and cm/c each sit alone in their own file A->>W: edit cm/a - Note over W: Alice's window opens + Note over W: Alice's window opens, footprint {cm/a} B->>W: edit cm/b - Note over W: Bob's window opens; Alice's stays open + Note over W: footprint {cm/b} is disjoint from {cm/a} + Note over W: Bob's window opens, Alice's stays open A->>W: edit cm/c - Note over W: joins Alice's window - A->>W: CommitRequest - Note over W: Alice's window: cm/a + cm/c, her message + Note over W: joins Alice's window, footprint {cm/a, cm/c} + A->>W: CommitRequest "fix(shop): cart limit" + Note over W: the request attaches to Alice's window + Note over W: Bob's window closes (idle_timeout) + Note over W: commit 1 by Bob: cm/b, generated message + Note over W: Alice's window closes on the request's timers + Note over W: commit 2 by Alice: cm/a and cm/c, her message ``` Writes that name no author keep sharing one window, as today: they have no author to tell apart. +The loop itself does not change shape: one event-loop goroutine, and the ordered pending-write +pipeline after it. Closing a window still produces one pending write; only the number of windows +that can be open in front of the pipeline changes. + ## What has to stay true -### One path, one order +### A write has a conflict footprint Today the single window gives a simple guarantee: commits land in the order their writes arrived, -so the last write to a path is the one Git ends with. With several open windows, that breaks unless +so the last write to a file is the one Git ends with. With several open windows, that breaks unless something keeps it: 1. Alice edits `cm/x` (her window). 2. Bob edits `cm/x` after her (his window). 3. Bob's window closes first, then Alice's: Git ends with Alice's **older** `cm/x`. -**Rule: a write to a path another open window holds closes that window first.** Bob's edit of `cm/x` -closes Alice's window, so her commit lands before his. The close reason is new, `path_conflict`. -The path is the window's own key (`windowPathKey`: the Git path, after placement), so a delete, a -rename to the same document, and a namespaced and a cluster-scoped object that map to one file are -all caught. Two authors editing the same object are therefore split exactly as today; two editing -different objects are not. +**Rule: a write whose footprint is not proven disjoint from every other open window's closes those +windows first.** Bob's edit of `cm/x` closes Alice's window, so her commit lands before his. The +close reason is new, `path_conflict`. + +The footprint is **the set of Git files a write may change**. It is a different thing from the key a +window coalesces events by, and the two must stay separate: + +- **The coalescing key** is `windowPathKey`: the `GitTarget`'s path joined with the file name + `generateFilePath` would give the object (`internal/git/open_window.go`). It is a per-object key, + and it stays exactly that. Replacing it with a physical file name would be wrong in the other + direction: several objects in one YAML file would then replace each other in `pathToEvent`. +- **The footprint** is where the write lands, which the generated name only guesses. The write plan + (`planEventsToWorktree`, `writeBatch`) resolves the destination when the window is committed: + an existing document wherever it already lives, a file shared with other objects, a generator + source such as a `configMapGenerator` literal in `kustomization.yaml`, and the files a removal + touches. + +The footprint has to be known when a write **arrives**, before any plan has run, so the first +release proves independence only where it is cheap and certain, and serializes everywhere else. +A write is **proven independent** only when all of these hold: + +- the `GitTarget` folder that holds the object is governed by no kustomization, and `useKustomize` + is off, so the write registers nothing in a `kustomization.yaml` and creates none (a new file is + always added to the kustomization governing it, and `useKustomize` creates one on the first new + document); +- the object's destination is the built-in path, with no `placement.byType` or `placement.default` + template for its type: a template can bundle several objects into one file (`all.yaml`); +- the destination file holds only this object's document, or does not exist yet. The built-in + path has no API version, so two versions of one object share a file: they count as one object. + +| Write | Footprint | Can open a window beside others | +| --- | --- | --- | +| Update or delete, proven independent, of a document the index places alone in its own file | that file | yes | +| Create, proven independent, with no document for the object yet | the built-in path | yes, unless another open window's footprint holds that path | +| An object in a file with other documents | unknown | no | +| A create or delete that registers or unregisters a file in a kustomization, or creates one | unknown | no | +| A create under a placement template | unknown | no | +| An object under a kustomization, a generator source, or any layout the worker renders | unknown | no | +| An object the index cannot place | unknown | no | + +"No" closes every other open window first, which is today's behavior, reported as +`identity_change` as it is now. The rule never guesses: unknown is treated as overlapping +everything. Shared files and kustomize layouts are the obvious next candidates once the base change +is measured; each needs its own proof. + +### A footprint proof is tied to a revision + +A proof is made against the worker's placement index, and the index changes under open windows: +the worker's own commits add and remove files, a replay onto a foreign commit brings in another +tree, and a `GitTarget` spec change (placement templates, `useKustomize`) changes where a create +goes. A proof that was true at arrival can be false when the window is committed. + +So every footprint records the **revision** it was proven against: the tree the index was built +from, plus the `GitTarget` generation. The invariant: + +> A write may join or open a window beside others only while every open window's footprint is +> proven against the current revision. + +When the revision changes (an own commit, a replay, a fetched foreign commit, a `GitTarget` +generation change), the worker recomputes every open window's footprint against the new index +**before** the next write is admitted beside others. A footprint that is now unknown, or that now +overlaps another window's, closes the windows involved, oldest first, with `path_conflict`. + +A last check runs at materialization: the files the write plan touched (`writeBatch.touched`) must +be inside the window's footprint. A plan that touched more is a footprint bug. The worker cannot +undo the order of what is already decided, so it logs the paths, counts it in a metric, and falls +back to one window for the rest of its life. The reference model treats it as a test failure. + +### Every close trigger keeps its exceptions -### Everything that closes "the window" closes all of them +Today's close triggers are about the branch, not about one author. They do not all close +unconditionally, and the first release must keep each exception. In particular, a window carrying +a `CommitRequest` is protected from two of them, and a blanket "close everything" would let an +unrelated push split a save that is protected today. -The other close reasons are about the branch, not about one author, and each must close every open -window, oldest first: +| Trigger | Which windows close | Exception that stays | +| --- | --- | --- | +| `foreign_commit` (`closeWindowForPushHint`) | every window **without** an attached `CommitRequest` | a window carrying a request stays open for its save | +| `apply_due` (`flushed`) | every window, when the flush begins and takes the flush mark | the flush does not begin while any request is registered (`writeBusy`) | +| `resync_before_apply` | every window | none (as today) | +| `shutdown` | every window | none | +| `buffer_limit` | every window | none; closing only the largest is an open question | +| `path_conflict` | the windows whose footprint the new write overlaps | none | +| `identity_change` | every other window | stays for writes of another `GitTarget` and for unproven footprints | +| `idle_timeout`, `max_duration`, `attach_next` | that window only | none | -| Reason | Why all of them | -| --- | --- | -| `foreign_commit` | the branch moved under every window | -| `apply_due`, `resync_before_apply` | an apply must not run with unpublished edits from anyone | -| `shutdown` | nothing survives the loop | -| `buffer_limit` | the cap is the worker's memory, not one author's | +Why the first two exist matters for every later change to them: -`buffer_limit` could instead close only the largest or oldest window. Closing all is simpler and the -cap is rarely reached; this is an open question below. +- **`foreign_commit` is how a busy branch learns about a push.** A push notice is a hint, and with a + window open the worker reads nothing from the remote first (the refresher skips a busy branch, + `noteBusyPushNotice`). Closing the windows and publishing is what discovers the move: the push is + rejected, the worker fetches and replays onto the foreign commit, and an apply becomes due. Without + the close, the push would be found only when a window closed on its own timers. +- **`apply_due` is the flush** ([S1 in applier-echoes.md](applier-echoes.md#s1-flush-then-apply)). + It closes the windows, takes the flush mark, and publishes every write up to the mark. An edit + decided after the mark does not hold the apply back: the applier's apply window records it, so an + apply that overwrites it is withheld and repaired. That replaced the old "quiet branch" wait, which + starved the apply for as long as someone kept editing. Several windows must not bring that wait + back. + +**Order:** when one trigger closes several windows, they close in opening order, oldest first. Two +windows opened in the same wake are ordered by a sequence number taken when each opened, never by +map iteration or by time, so the order is deterministic and testable. ### Commit order is close order Commits land in the order their windows close, not the order their first writes arrived. With Alice -open since 10:00 and Bob since 10:01, Bob's commit can come first. Together with the path rule above, -no path ever goes back in time, but `git log` no longer reads as one timeline of edits. That is the -point of the change, and it needs saying in `docs/configuration.md` when it ships. +open since 10:00 and Bob since 10:01, Bob's commit can come first. Together with the footprint rule +above, no file ever goes back in time, but `git log` no longer reads as one timeline of edits. That +is the point of the change, and it needs saying in `docs/configuration.md` when it ships. ### Saves - A save attaches to the open window of its identity, by lookup instead of comparison with the one window. `matchesWindow` keeps its rules. -- `WindowMismatch` becomes rare: it meant "only another author's window was open". With every author - allowed a window, it remains only for the window a save may not claim: an unattributed one, without - `SavedBy` or `TakeOver`. +- `WindowMismatch` loses most of its meaning: it said "only another author's window was open", and + with a window per author another author's window no longer stands in a save's way. A save that + finds no window of its own reports `NoWindow`. - `SaveInProgress` is unchanged: it is already per author and `GitTarget`. -- The takeover (`TakeOver`, `SavedBy`) claims the unattributed window, which stays one window. +- Each request settles exactly once, whichever trigger closes its window. + +## How to test it: a reference model + +"Git equals the cluster at the end" is too weak a check for this change. A later write can repair a +temporary revert or a wrongly credited commit, so the end state looks right while the history was +wrong in between. And it is the wrong check in places: an incoming Git edit that is not applied yet +is supposed to survive in Git, and a no-op edit is supposed to produce no commit. + +So the test is a **reference model**: a small, independent implementation of the rules on this page, +fed the same random sequence of inputs as the worker, with the worker's output checked against it. + +**Inputs:** accepted writes (with author, `GitTarget`, object, footprint), deletes and recreates, +undos (a write that puts back an earlier value), no-op writes, saves (including `attach_next` and a +zero `maxDuration`), foreign commits, `GitTarget` spec changes, applies, resyncs, failed +publications and replays, buffer pressure, shutdown, and clock steps from a controllable clock. + +**Checked on every commit the worker makes:** + +- its author and message are the ones the model says the closing window has; +- the files it changes are inside that window's footprint, proven against the revision current when + it was committed; +- **an older accepted write never lands after a newer conflicting one.** Two writes conflict when + their footprints overlap; acceptance order is the order the worker accepted them. The check is + about the order of writes, never about content: an undo that puts back an earlier value is a newer + write, and it is correct for Git to end with that value; +- a save's request settles exactly once, with the outcome the model expects. + +**Checked after every step, not only at the end:** the retained-byte charge equals the model's +expected charge for what is open and pending at that moment. A charge that is right at the end can +still have been too low in between, and an undercharge lets the worker pass its cap. + +**Checked when the worker is drained:** Git's tree is the model's tree, and the retained-byte +charge is zero. + +The model and the controllable clock come **first**, in stage 1, while the worker still has one +window, so they are proven against today's behavior before they are used to judge the new one. + +## Stages + +1. **Extract window management without changing behavior.** Move the window bookkeeping + (`openWindow`, its timers, attach and finalize) behind one type that holds a set of windows, with + one window allowed. Add the controllable clock, the opening sequence number, and the reference + model. Every existing test passes unchanged. +2. **Allow several windows for proven-independent destinations.** Writes keep whole-object commits + and apply behavior stays as it is. Writes of another `GitTarget` keep closing other windows + (`identity_change`), and every layout in the footprint table's "no" rows stays serial. +3. **Test the failure boundaries.** Both close orders, conflicting writes, delete and recreate, + `attach_next`, zero-duration windows, resync, shutdown, failed publication and replay, and buffer + pressure, each against the reference model. +4. **Ship behind a startup flag**, off by default at first. The flag is read once, at start; changing + it takes a restart, and the restart drains every open window through the normal shutdown path + (opening order, close reason `shutdown`), so there is no live transition to get right. Measure concurrent-window + counts, forced splits by reason, save outcomes, and memory before turning it on by default. +5. **Reconsider the follow-ups** in + [window-field-ownership-and-apply-hold.md](window-field-ownership-and-apply-hold.md), using those + measurements. Field ownership and the apply hold are separate changes, each with its own + state-machine review. ## What it costs -- **The loop's state.** `openWindow` becomes a map with an opening order, the close timer becomes the - earliest of all windows' deadlines, and every path that reads `l.openWindow` today (attach, - finalize, the apply window, resync cut-off, metrics) has to say which window, or all. +- **The loop's state.** The single `openWindow` becomes a set with an opening order, the close timer + becomes the earliest of all windows' deadlines, and every path that reads `l.openWindow` today + (attach, finalize, the apply window, resync cut-off, metrics) has to say which window, or all. + Stage 1 does this refactor on its own, with no behavior change. +- **Footprints at arrival.** Deciding a footprint needs the worker's placement index of the + `GitTarget` subtree when a write arrives, not only when it plans a commit. - **Memory.** One window per active author. The byte cap stays global, so the bound does not change. - **Tests.** The commit-window, attach and apply-window suites assume one window. Most of them keep - their meaning with one author; the interleaving cases need new ones, the path-conflict rule first. + their meaning with one author; the interleaving cases are new, and the reference model is new. - **Metrics.** `git_commit_windows_total` gains `close_reason="path_conflict"`. A gauge of open - windows would show how often authors overlap at all. + windows shows how often authors overlap at all. + +## Potential problems + +Most failures here are **silent**: Git gets a plausible commit with the wrong content or the wrong +author, and nothing reports an error. That is why every unknown is treated as overlapping, and why +the reference model checks every commit. + +- **P1. A footprint that is too small.** A write lands somewhere its footprint did not name (an + existing document in an unexpected file, a generator source, a removal that edits a shared file), + and two windows commit to one file in the wrong order. *Mitigation:* footprints are proven or + unknown, never guessed; the table's "no" rows stay serial; the reference model includes shared + files and generator layouts. +- **P2. A close trigger loses its exception.** A push hint or an apply flush that closes a window + carrying a save splits a save that is protected today. *Mitigation:* the trigger table above is the + specification, and each exception has a test in stage 3. +- **P3. Nondeterministic order.** Windows that close in the same wake in map order give a different + history run to run, and a flaky test. *Mitigation:* the opening sequence number. +- **P4. A request settles twice, or never.** With several windows, a request can be seen by two close + paths, or by none. *Mitigation:* the reference model checks exactly-once settling on every run. +- **P5. Retained bytes drift.** Charges taken when a window opens or a message attaches, released on + a different window's close, leave the cap wrong. *Mitigation:* the model checks the charge after + every step, as well as once drained. +- **P6. A stale footprint proof.** A proof made against one tree is used after an own commit, a + replay or a `GitTarget` change moved the index. *Mitigation:* proofs carry their revision, are + recomputed before the next concurrent admission, and the materialization check falls back to one + window if a plan ever touches a file outside its footprint. +- **P7. The flush starts waiting again.** A change that makes `apply_due` wait for windows to close + on their own reintroduces the starvation S1 removed. *Mitigation:* the flush closes every window + at once, as today; only the save exception holds it. +- **P8. `git log` stops being a timeline.** *Mitigation:* document it with the close-order note in + `docs/configuration.md`. ## Alternatives 1. **Keep one window, and let clients serialize.** What gitops-api does today. Correct, and costs a - server on the save path, which is what `RemoteSaveRequest` set out to remove. -2. **Per-author windows without the path rule.** Simpler, and wrong: a path can go back in time + server on the save path, which is what creating `CommitRequest`s on the source cluster set out to + remove. +2. **Per-author windows without the footprint rule.** Simpler, and wrong: a file can go back in time with no error. 3. **Merge concurrent windows into one commit with several authors.** Git has one author per commit; `Co-authored-by` trailers would blur exactly the attribution the product exists for. @@ -136,10 +335,11 @@ point of the change, and it needs saying in `docs/configuration.md` when it ship ## Open questions 1. Does `buffer_limit` close every window, or only the one that crossed it? -2. Should `identity_change` remain, for a write of another `GitTarget` on the same branch, or do - windows of different `GitTarget`s also stay open side by side? The proposal says side by side; - the per-`GitTarget` encryption and bootstrap that made one window per target necessary are per - window already. +2. After stage 4, do windows of different `GitTarget`s on one branch also stay open side by side? + The per-`GitTarget` encryption and bootstrap that made one window per target necessary are per + window already; the first release keeps them serial anyway. 3. Is a cap on open windows needed beyond the byte cap, such as one per `GitTarget`? 4. When a save's own window is closed early by `path_conflict`, it commits with the save's message and less than the person meant to save. Is that acceptable, or should the save report it? +5. How much of the proof can the worker's placement index answer at arrival without a plan, and + what does that lookup and the recomputation after each revision change cost? diff --git a/docs/design/data-plane-triggering.md b/docs/design/data-plane-triggering.md deleted file mode 100644 index 6b6b5633c..000000000 --- a/docs/design/data-plane-triggering.md +++ /dev/null @@ -1,289 +0,0 @@ -# Data-plane triggering: why one config change replays everything - -> **design**: background, open. Index: [`../INDEX.md`](../INDEX.md) -> Date: 2026-08-25. -> Related: [`target-watch-plan.md`](target-watch-plan.md), -> [`reconcile-triggering.md`](reconcile-triggering.md), -> [`watch-and-catalog-architecture.md`](watch-and-catalog-architecture.md), -> [`../spec/reconcile-via-watchlist-mark-and-sweep.md`](../spec/reconcile-via-watchlist-mark-and-sweep.md), -> [`../spec/type-lifecycle-events-and-wobble-settling.md`](../spec/type-lifecycle-events-and-wobble-settling.md) - -**The short version.** Adding a single WatchRule tears down and replays every -watch stream a GitTarget has. Under load that floods a queue which is shared with -other tenants. The fix is to reconcile only the collections that changed, and it lives -in [TargetWatchPlan](target-watch-plan.md). This page is the background: what -went wrong, how the pipeline works, and the one ordering rule any fix has to -respect. - -Read [Watch and catalog architecture](watch-and-catalog-architecture.md) for the -model, [TargetWatchPlan](target-watch-plan.md) for what is being built, and this -page for why it is worth building. Where this page disagrees with either of -those, they win. - -[`reconcile-triggering.md`](./reconcile-triggering.md) asks how the **control -plane** wakes up. This asks the same question one layer down, about the **data -plane**. - -## 1. What happened - -CI run 32848418546 dropped **595 resync requests in 16 seconds**, all for a -single GitTarget whose namespace was being deleted. - -A branch worker's `eventQueue` is a 100-slot FIFO shared by **every GitTarget on -one provider and branch**. One target saturated it, so unrelated GitTargets on -the same branch waited on resyncs that never entered the queue, and their -WatchRules never reached `Ready`. Five e2e specs failed this way, which is why it -read as flakiness rather than as one bug. - -```mermaid -flowchart LR - subgraph Deleted["GitTarget being deleted"] - S1["target watch: stream A"] - S2["target watch: stream B"] - S3["target watch: stream …N"] - end - subgraph Healthy["Unrelated GitTargets"] - H1["resync for target X"] - H2["resync for target Y"] - end - S1 --> Q - S2 --> Q - S3 --> Q - H1 -.->|"dropped"| Q - H2 -.->|"dropped"| Q - Q["eventQueue (100 slots)
shared per provider + branch"] - Q --> W["branch worker"] - - classDef bad stroke-dasharray: 4 3; - class H1,H2 bad; -``` - -Each stream of the deleted target looped `replay → enqueue resync → fail → -reconnect → replay`, minting a new payload every two seconds, for a target that -could never come back. - -[#312](https://github.com/ConfigButler/gitops-reverser/pull/312) fixed it twice -over: a deleted GitTarget is now terminal for its watches, so the loop stops; and -queued resyncs coalesce by `(GitTarget, scope)`, so queue depth is bounded by the -number of distinct scopes rather than by the request rate. - -That stopped the bleeding. It did not address the shape underneath, which the -next section is about. - -## 2. How the pipeline works today - -Two independent inputs converge on one FIFO. - -```mermaid -flowchart TB - subgraph Cluster["Source cluster"] - API["kube-apiserver"] - end - - API -->|"audit webhook"| AUD["audit ingest"] - AUD --> FS["per-type audit fact stream
(Redis)"] - - API -->|"watch"| TW["target watch
one goroutine per
(GitTarget, collection)"] - - TW --> SAN["sanitize + followability
+ no-op suppression"] - SAN --> GR["attribution grace window
(head-of-line, per event)"] - FS -.->|"who made this change"| GR - - GR --> ER["event router"] - TW -->|"replay / LIST snapshot"| ER - - ER --> Q["branch worker eventQueue
FIFO, 100 slots
shared per provider + branch"] - Q --> CW["commit window
one (author, GitTarget) at a time"] - CW --> C["local commit"] - C --> P["push (5s cooldown)"] -``` - -Three properties carry the rest of this page: - -- **Object state comes from watch, never from polling.** A watch that is lost and - re-established replays with `sendInitialEvents=true` and runs a - **mark-and-sweep**: everything replayed is marked, and at the - `initial-events-end` bookmark any managed Git file left unmarked is deleted, - because its object is gone. That sweep is the only thing that reconciles a - delete which happened while no watch was running. -- **Attribution comes from somewhere else.** The audit webhook feeds a fact - stream, and the grace window waits on it to name the author. An event therefore - cannot be recomputed: the author of a past change exists only there. -- **Everything funnels through one FIFO per branch**, so same-object order is - strictly preserved and the commit window groups by `(author, GitTarget)`. - -[`../architecture.md`](../architecture.md) has the full treatment, under "State -ingestion and not losing deletes", "Watch event ordering", and "Mark and sweep -resync". - -### 2.1 The stream set was declarative already; only the diff was missing - -This is the finding that reframed everything else, and it is now built: -[target-watch-plan.md](target-watch-plan.md) carries the design and the state. -`targetWatchSpecs(table)` already computed the desired stream set as a map keyed -by collection, and what was missing was a **per-collection** diff. The comparison -was whole-set equality, so a declaration was either untouched or replaced -wholesale: - -```mermaid -flowchart TB - C["add one WatchRule
(configmaps @ team-d)"] --> D{"specs equal?"} - D -->|"yes"| K["keep everything running"] - D -->|"no"| X["cancel the whole set"] - X --> R1["restart: configmaps@team-a
replay + resync"] - X --> R2["restart: secrets@team-b
replay + resync"] - X --> R3["restart: deployments@team-c
replay + resync"] - X --> R4["start: configmaps@team-d
replay + resync"] - - R1 --> Q["shared FIFO"] - R2 --> Q - R3 --> Q - R4 --> Q - - classDef want stroke-width:3px; - class R4 want; -``` - -Only the bold stream is new work. The other three re-replay state that never -changed, which is the storm shape from §1 reached by ordinary configuration -change instead of by deletion. - -The cause is structural: the render-fidelity **epoch is per-target**. One epoch -covers every scope, so a scope that resumed from its cursor instead of replaying -would stay pending in the new epoch forever. Making readiness per collection is -what unlocks the diff. - -One subtlety worth carrying forward: when a rule **moves**, both scopes are -invalidated. Editing a rule's `sourceNamespace` from `team-b` to `team-c` -produces a `stop` for the old collection and a `start` for the new one. What -happens to the departing collection's documents is decided by the cause table in -[TargetWatchPlan](target-watch-plan.md), "What a collection leaving means". - -## 3. The queue is doing two unrelated jobs - -`WorkItem` carries three shapes, and they do not share a nature. - -```mermaid -flowchart TD - WI["WorkItem"] --> R["Request — WriteRequest"] - WI --> A["Attach — CommitRequest"] - WI --> RS["Resync — ResyncRequest"] - - R --> RE["Edge-triggered
an observation of history"] - A --> AE["Edge-triggered
a control message"] - RS --> RL["Level-triggered
a statement about current state"] - - RE --> RN["cannot be recomputed"] - AE --> AN["cannot be recomputed"] - RL --> RLN["can always be recomputed"] -``` - -**A write is an observation of something that happened.** "alice deleted this -ConfigMap" cannot be recomputed from cluster state: the object is gone, and the -author exists only in the attribution fact stream. Order and attribution both -matter, and both are destroyed by collapsing a write to "something changed". This -half stays a log. - -**A resync is a statement about current state**, gathered by listing the live -cluster. Throwing one away loses nothing, because the next gather reconstructs it -fresher. The cluster is the source of truth. - -## 4. Why a resync carries both a snapshot and a position - -This is the part that is easy to get wrong, and the reason the design looks -heavier than it first seems. - -A resync is not applied into a vacuum. Live events for the same scope arrive -while it is in flight, and the snapshot it carries was gathered at some revision. -FIFO position is what reconciles the two. The LIST fallback makes the ordering -explicit: buffer the watch without applying it, LIST a snapshot, enqueue the -resync, record the cursor, and only then release the buffered events. - -```mermaid -sequenceDiagram - participant API as apiserver - participant TW as target watch - participant Q as eventQueue - participant W as branch worker - - TW->>API: open watch (buffer, do not apply) - API-->>TW: event rv 101 - API-->>TW: event rv 102 - TW->>API: LIST snapshot - API-->>TW: state @ rv 100 - TW->>Q: resync (desired @ rv 100) - TW->>Q: buffered event rv 101 - TW->>Q: buffered event rv 102 - Q->>W: snapshot first - Q->>W: then the deltas on top - Note over W: state ends at rv 102 -``` - -Invert those and the snapshot lands after edits it does not contain, and -mark-and-sweep reverts them. The events are not merely redundant. They are -unapplyable out of order, because an event may already be included in the -snapshot, so its value is defined only relative to the snapshot boundary. - -**So a FIFO position is not an implementation detail that can be discarded for -free.** It is the boundary that makes "snapshot, then deltas" well defined, and -any redesign has to solve that again. It is why -[TargetWatchPlan](target-watch-plan.md) keeps a coalescing fence, and why a -proposal to replace the resync payload with a bare dirty set was dropped. - -## 5. Where we are - -The plan lives in [TargetWatchPlan](target-watch-plan.md), which is the single -place to read it from. In outline: diff the plan by collection and log it, then -apply the diff so unrelated replays stop. The semantics of removal come after, -deliberately. - -That plan applies its decisions by starting and canceling streams, and nothing -filters the queue. A branch worker serves one GitProvider and branch rather than -one GitTarget, so its queue is shared across tenants and must not hold per-tenant -configuration. A short tail of writes from a collection that has been deselected -is therefore accepted: it is bounded by the queue, and the files it touches are -retained anyway. - -Shipped so far: the storm source is fixed and resyncs coalesce -([#312](https://github.com/ConfigButler/gitops-reverser/pull/312)); the ordering -hazard that coalescing introduced is fenced; collection identity is settled, -with one stream per collection and the producing collection stamped on queued -work; and the diff is built and applied. Removal semantics are not. - -The writes stay ordered pending writes throughout. Only the resync half of the queue is in -question, and the answer is to make the stream set incremental rather than to -replace the queue. - -## 6. Related work - -**The layer above.** [Reconcile triggering](reconcile-triggering.md) asks the -same question for the control plane. - -**How ingestion works.** [Architecture](../architecture.md) is the reference. -[Watch-first ingestion](../finished/watch-first-ingestion-architecture.md) is how -the model was arrived at, and -[Reconcile via WatchList and mark-and-sweep](../spec/reconcile-via-watchlist-mark-and-sweep.md) -is the mechanism §4 depends on. -[Watch event ordering under the attribution grace window](../spec/watch-event-ordering-and-attribution-grace.md) -has the worked examples. - -**Scopes and streams.** A sweep is bounded by the exact slice its snapshot was -gathered over, and a cluster-wide scope is a **peer** of a named namespace rather -than a replacement for it: collapsing the two widened the named rule's stream to -every namespace its credential could read. - -**Deletion and retention.** `spec.prune.mode` -([`../configuration.md`](../configuration.md), deletion policy) already draws the -line: an observed DELETE is evidence, a mark-and-sweep drop is an inference, and -only the second is gated. `status.retention` makes a suppressed sweep visible. - -**Types and identity.** -[Type lifecycle events and wobble settling](../spec/type-lifecycle-events-and-wobble-settling.md) -gives `RemovalGrace`, which the plan consumes rather than re-decides. -[Typeset owns discovery grace](../spec/typeset-owns-discovery-grace.md) is where -that grace lives, [Type followability](../spec/type-followability.md) defines -what may be mirrored at all, and -[the GVK/GVR mapping layer](../spec/gvk-gvr-mapping-layer.md) is the identity -resolution the plan's collection turns on. -[Unsupported folder refusal](../spec/unsupported-folder-refusal-plan.md) decides -what a folder may contain before any of this runs. diff --git a/docs/design/e2e-finish-plan.md b/docs/design/e2e-finish-plan.md deleted file mode 100644 index bb372c0d8..000000000 --- a/docs/design/e2e-finish-plan.md +++ /dev/null @@ -1,96 +0,0 @@ -# E2E Finish Plan - -> **design**: open, not yet built. Index: [`../INDEX.md`](../INDEX.md) - -This is the one active plan for the remaining e2e harness work. - -For current debugging and reuse behavior, see -[`../../test/e2e/E2E_DEBUGGING.md`](../../test/e2e/E2E_DEBUGGING.md). - -## Current Baseline - -The harness is already in a much better place than the older plans imply: - -- `BeforeSuite` owns shared cluster and install preparation -- repo-using e2e files create file-local repo fixtures through `SetupRepo(...)` -- repo bootstrap is now owned directly by the e2e Go helpers -- signing covers generated keys and BYOK -- signing is verified three ways: - - `ssh-keygen -Y verify` - - `git verify-commit` - - Gitea commit verification after the `verify_ssh` flow - -Older migration and follow-up documents have been folded into the main e2e docs or removed. - -## Open Work - -### 1. Final harness polish after the direct Go bootstrap shift - -The big shell-to-Go seam is now closed. The remaining work is cleanup and -consolidation: - -- trim stale docs and comments that still describe the deleted Task and shell repo-bootstrap path -- decide whether the current Go bootstrap helper should stay in the main - `test/e2e` package or be extracted into a smaller e2e support package -- review whether any now-unused historical migration notes should be archived or - deleted -- keep the persisted artifact set intentionally small unless a new stamp buys clear debugging value - -Acceptance: - -- the active docs describe the direct Go-owned repo bootstrap accurately -- no active e2e path depends on the deleted repo-bootstrap Task targets -- no new file-based stamp is introduced without a concrete consumer - -## Current Direction - -The preferred direction is now explicit: - -- keep repo bootstrap owned in Go -- keep signing keys memory/Secret-backed -- keep transport SSH keys memory-first as well -- persist only the artifacts that are directly useful to tests and humans - -In the current shape that means: - -- the local checkout remains a stable artifact -- the generated `secrets.yaml` remains a stable artifact -- intermediate values such as transport keys, tokens, and webhook bookkeeping - stay in memory unless there is a strong reason to persist them - -## Why This Simplifies The Harness - -- fewer stamp files with overlapping meaning -- less hidden coupling to filenames and shell pipelines -- fewer subprocesses and fewer write-file/read-file-back cycles -- clearer ownership boundaries between typed Gitea calls, Kubernetes lookups, - and local checkout setup -- easier future parallelism because less mutable setup state is written to disk - -## Explicit Non-Goals - -This plan does not require: - -- rebuilding every remaining shell helper immediately -- immediate `ginkgo -p` safety for the whole package -- redesigning the whole Task-based cluster bootstrap -- restoring deleted repo-bootstrap Task compatibility for manual use - -## Validation - -Minimum focused checks while working: - -```bash -go test -timeout 15m ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=signing -``` - -Required wrap-up: - -```bash -task lint -task test -docker info -task test-e2e -task test-e2e-quickstart-manifest -task test-e2e-quickstart-helm -``` diff --git a/docs/design/gittarget-api-wave.md b/docs/design/gittarget-api-wave.md deleted file mode 100644 index e0669abdf..000000000 --- a/docs/design/gittarget-api-wave.md +++ /dev/null @@ -1,342 +0,0 @@ -# The wave after placement left it - -> **partly built**: a sequencing proposal. Steps 2 to 7 have all **shipped** — `spec.suspend`, -> `status.placement` and the reconcile-request annotation in -> [#326](https://github.com/ConfigButler/gitops-reverser/pull/326), B4 and the source-scope deletion -> in [#330](https://github.com/ConfigButler/gitops-reverser/pull/330) on 2026-09-01, and the -> reference collapse in [#333](https://github.com/ConfigButler/gitops-reverser/pull/333). **Step 8, -> the riders, is the only unbuilt member**, trimmed under this page's own rule to three items. Step 5 -> (an Event on a changed resolution) and the `TooManyStreams` cap were **dropped**, not deferred. -> [#294](https://github.com/ConfigButler/gitops-reverser/issues/294) is closed; the riders are -> tracked on [#339](https://github.com/ConfigButler/gitops-reverser/issues/339). -> Index: [`../INDEX.md`](../INDEX.md) -> Date: 2026-08-28 (originally 2026-07-30). -> -> **The API-server behaviour this wave measured is written up separately**, in -> [`../facts/crd-upgrade-strategies.md`](../facts/crd-upgrade-strategies.md): the two strategies for -> removing a field, a decision matrix keyed on whether pruning fails open or closed, and why a -> conversion webhook could have automated only two of this wave's five changes. -> -> **The step-1 envtest has run.** A status update onto a stored object whose spec no longer -> validates is ACCEPTED — measured on 1.31 with `CRDValidationRatcheting` on and off, and on the -> version this module builds against. The status subresource does not re-validate spec, so the -> fallback below was not needed and the loud-rejection pattern is safe here. Pinned by -> `internal/controller/stored_superseded_value_status_test.go`. -> -> This document was written to sequence one breaking wave whose centrepiece was `spec.layout`. -> [`model.md`](../layout/model.md) has since reversed: the path template stays and gains two additive fields, -> so the placement work is not breaking at all. The wave lost its largest member. What is left is a -> batch, and this page is honest about being one. - -## What is in it - -| Member | Object | Why breaking | -|---|---|---| -| **B4**: `commitWindow` and `commit.message` move off the connection | `GitProvider` → `GitTarget` | fields change object | -| The source-scope deletion ([`source-scope-simplification.md`](source-scope-simplification.md)) | `GitTarget`, `ClusterProvider`, `WatchRule` | one removal, two renames, one redefinition | -| The riders: ~~**#5** asserted `CommitRequest.spec.author`~~ (**declined**, see step 8), the `CommitRequest` lifecycle hole, the `default` `ClusterProvider` message; `meta.LocalObjectReference` for our six reference shapes shipped in #333 | various | shape changes | - -Additive, and therefore **not** wave members even though they are discussed here: `spec.suspend`, -`status.placement` and the post-scan pass, the reconcile-request annotation, and the two placement -fields. They ship whenever they are ready. Tier 1 in -[`open-asks-priority.md`](open-asks-priority.md) — the removal-wait decision, #15's -condition — is untouched and should not wait for this. - -## Why still one wave - -The consumer pins us three ways (image, Go module, `require` line), so each breaking release costs a -coordinated bump. That argues for batching, and with placement gone it is most of what is left. - -The stronger half survives in reduced form: several of these items are the same design decision seen -from different angles, and building them separately means deciding it several times, inconsistently. - -> **The folder is described on the GitTarget. The connection describes only the connection.** - -`spec.suspend` says whether we write to the folder; `commitWindow` and `commit.message` say how those -writes are batched and phrased, and today they live on `GitProvider`, which is the connection (B4, -and the config-model review's "GitProvider is doing three jobs"). Shipping B4 alone would still -assert half the principle, which is the reason to keep the rest together. - -## Where the fields live - -The principle above has been a sentence in a document for two drafts. This wave is the moment it -becomes a struct boundary, because grouping a field costs nothing extra in a release that is -breaking anyway, and costs a bump in every release that is not. - -`GitTargetSpec` today is flat: `gitProviderRef`, `branch`, `path`, `encryption`, `placement`, -`clusterProviderRef`, `allowedSourceNamespaces`, `prune`. Queued on top of it were `suspend`, -`serializeNamespace`, `useKustomize`, `commitWindow` and `commit.message` — five more members on a -spec that already flattens six orthogonal axes. Left alone, this object accumulates faster than it -sheds, and moving `commit.message` off `GitProvider` for exactly that reason while doing it just -relocates the problem one hop. - -Two groupings, and one deliberate exception: - -- **`useKustomize` nests under `spec.placement`**, where an earlier draft of - [`model.md`](../layout/model.md) had it at the top level beside it. `spec.placement` is an existing - optional struct, so nesting is **still purely additive**, and it is free only before the field - exists. The test it passes is retroactivity: like every other placement field, it only ever affects - a **new** document. -- **`serializeNamespace` stays at the top level**, and that is not an exception to the grouping rule - but the same rule applied. It governs the bytes of every write, updates included, and the identity - a managed document is located by; nesting it inside a struct documented as "new files only" would - have hidden a field that rewrites existing documents. Group by blast radius, not by topic. -- **`commitWindow` and `commit.message` land as `spec.commit`**, not as two top-level fields: - `spec.commit.window` and `spec.commit.message.template`. The move is breaking either way, so the - grouping is free, and `GitProvider.spec.commit` is the shape they already have. -- **`spec.suspend` stays top-level.** It is the object-level switch, it is where every Flux user - reaches for it, and a `spec.write` wrapper invented to hold it plus `prune` plus `commit` would be - a category we made up rather than one the API already has. `prune` is its own struct already. - -After the wave the spec reads as named axes rather than a list: the immutable destination -(`gitProviderRef`/`branch`/`path`), `encryption`, `clusterProviderRef`, `placement`, `commit`, `prune`, -and two object-level switches — `suspend` and `serializeNamespace`. That is the test for the next -field too: a new member joins an axis, or names a new one, or is genuinely object-level — and if it -can do none of the three it is probably not a `GitTarget` field. - -## The interactions that change the design - -These are the reasons to combine, as opposed to merely batch. Each one changes what gets built. - -### 1. `spec.suspend` is a panic knob, and adoption is a scratch branch - -The review's central complaint is that *this controller writes to a Git repository and there is no -way to make it stop that is not deleting the object.* One field that stops the writes without -deleting the target or unpicking its `WatchRule` objects answers that, and that is the whole -justification — which is why **`spec.mode: Observe|Write` (B1) was dropped**: mode bought only the -difference between a temporary pause and a declared permanent read-only posture, a distinction in -intent rather than in behavior. - -`suspend` is not how a user previews a target. That is done by pointing a `GitTarget` at a scratch -branch and reading the commits it makes — real bytes, real registrations, real deletes, in a -reviewable diff -([`../layout/model.md`](../layout/model.md#previewing-a-target-point-it-at-a-scratch-branch)). - -The one deviation is stated rather than hidden: **`suspend` keeps observing.** Flux's `suspend` stops -reconciliation altogether; ours stops *writes* and keeps scanning, because a valve that stopped -looking as well as writing would freeze `status.placement` at whatever the folder looked like the -moment someone panicked — exactly when a stale answer costs the most. That belongs in the field's -documentation, in one sentence, because it is the only place we differ from a convention a Flux user -brings with them. - -**Re-open trigger for `mode`**: someone who needs a target that can never write, as a property of the -object rather than a switch a colleague can flip. - -### 2. `GitTarget.spec.interval` is dropped, because we watch - -Flux polls because **a Git remote cannot be watched**, which is why `GitProvider.spec.interval` stays -and is the honest one of the pair. A `GitTarget`'s inputs are API objects and we are already -streaming them, so every input that can change what a scan would conclude arrives as an event. - -The name would also have misled: in Flux, `interval` is the drift-correction cadence, and it would -have driven neither of the two passes a Flux user pictures — the observation pass (reads this -target's folder, never writes, never decides deletions) or the resync mark-and-sweep (reads the API -via the streaming list's initial-events snapshot, and is the only thing qualified to infer a -deletion; [`event_router.go`](../../internal/watch/event_router.go), -[`reconcile-via-watchlist-mark-and-sweep.md`](../spec/reconcile-via-watchlist-mark-and-sweep.md)). - -What is left uncovered is one case: a repository whose folder was changed **by someone else** while -our target wrote nothing. The reconcile-request annotation refreshes that on demand. A stale -`resolvedAtCommit` on an idle target is a legible cost; a periodic scan on every target is not. - -**Re-open trigger**: a user who needs an idle target's `status.placement` to track a repository other -people edit, for whom the annotation is not enough. Then it is a scan cadence, named for scanning. - -### 3. `spec.suspend` is a precondition for `useKustomize: true` - -Creating a root writes a `kustomization.yaml` the user did not author. That raises the stakes on the -review's central complaint: *this controller writes to a Git repository and there is no way to make -it stop that is not deleting the object.* So `suspend` is not a rider here, it is a precondition — -and it must stop bootstrap creation specifically, not only resource writes. - -### 4. The Events question is closed: no Event - -[`open-asks-priority.md`](open-asks-priority.md) left open whether a fall-back to canonical -should raise a `corev1.Event` on the GitTarget. Decided against on 2026-09-04 -([#339](https://github.com/ConfigButler/gitops-reverser/issues/339)). The metric already names the -target and the exact `byType` key that is missing, and `status.placement` is the -durable record a bug report contains. A third surface for the same fact would be one more thing to -keep consistent, for a notification that is deduplicated and expires within the hour anyway. - -### 5. The source-scope deletion is the only member that shrinks the API - -It belongs here for the ordinary reason first: it removes a field from `GitTarget`, and B4 already -breaks `GitTarget` in the same release, so shipping them apart costs the consumer two bumps for one -object. The better reason is the review surface. Every other member adds a field; this one deletes -4,569 lines, a three-valued verdict, and the only cross-cluster read in the authorization path. A -wave that is otherwise all addition is easier to justify when the object comes out simpler than it -went in. - -Two interactions worth naming, because they change what gets built rather than merely when: - -- **The `SelfSubjectAccessReview` pass is the same shape as the dry run.** Both answer "tell me what - you would do before you do it". They are separate conditions on separate objects and neither - depends on the other, so the SAR work stays additive and can ship after the wave — but whoever - writes the second should read the first. -- **The `TooManyStreams` cap is dropped by the `*` decision** - ([definition of record](source-scope-simplification.md#sourcenamespace--needs-its-own-decision)), - which removes the fan-out the cap was queued for. Explicit enumeration is bounded by whoever - writes the rules; a cap on it would be a guard against a list a human typed. It was never in code. - -### 6. Two facts kept from arguments that dissolved - -`spec.layout` immutability and the namespace-agreement rule were both settled by the reversal: -neither field exists. Two facts from underneath them survive and will be reached for again. - -- **`spec.placement` is mutable and stays mutable.** Existing files never move, so a template change - affects only what is written afterwards and a folder can hold documents placed under two templates. - Match-first identity keeps finding and editing those in place, so this is worth saying plainly - rather than fixing. -- **`GitTarget` has no finalizer**, so deleting one leaves the folder untouched and re-creating it at - the same path re-adopts every document by identity. Recreating a target costs status and a moment - of mirroring, not data — which is what made immutability affordable for `path` and unaffordable for - `prune`, and is the right test for any future field on this object. - -## The GitTarget after the wave - -```yaml -apiVersion: configbutler.ai/v1alpha3 -kind: GitTarget -metadata: - name: prod - annotations: - reconcile.configbutler.ai/requestedAt: "2026-07-30T09:14:22Z" -spec: - # --- the connection: unchanged, and now only the connection --- - gitProviderRef: - name: platform - branch: main - path: clusters/prod - # allowedSourceNamespaces is gone: the provider's credential bounds what can be read, - # and ClusterProvider.accessFrom bounds who may wield it. - - # --- what the documents look like: ADDITIVE, not part of the wave --- - serializeNamespace: false # every write, updates included - placement: - byType: - v1/secrets: "secrets/{name}{sensitiveSuffix}" - useKustomize: true # the created kustomization carries namespace: - - # --- whether we write, and how those writes are batched and phrased --- - suspend: false # the only stop-writes switch; a suspended target still scans - commit: # B4, was GitProvider.spec.push.commitWindow and .commit.message - window: 5s - message: - template: "chore(mirror): {{ .Summary }}" - prune: - mode: OnEvent -status: - observedGeneration: 4 - conditions: - - type: LayoutResolved - status: "True" - reason: SingleKustomization - message: "render root '.' governs new files" - observedGeneration: 4 - placement: - mode: KustomizeRoot - renderRoot: . - resolvedAtCommit: 9f3c1ab -``` - -**`status.lastHandledReconcileAt` was not built.** #326 tracks the last handled request in memory -instead, so a standing annotation forces one re-read rather than one per reconcile, and a restart -costs one extra re-check rather than a field every consumer has to understand -([`gittarget_reconcile_request.go`](../../internal/controller/gittarget_reconcile_request.go)). - -The placement fields are shown in place so the object reads as a whole, but they carry defaults equal -to today's behavior and are additive: only `spec.commit` and the riders make this release breaking. -The status stanza is specified where it is built, in -[`model.md`](../layout/model.md). - -## The envtest that has to run before any of this is planned - -One question is not in the wave, and its answer constrains the enum work, so settle it first. - -`ClusterWatchRule`'s `rules[].scope` enum is narrowed to `Cluster` only, deliberately keeping the -field so that re-applying a stored `Namespaced` value **fails** -([`clusterwatchrule_types.go`](../../api/v1alpha3/clusterwatchrule_types.go)). For CRDs the apiserver -validates the **whole object** against the OpenAPI schema on **status-subresource** updates too. If -that holds here, the controller cannot write `Stalled=True` onto an object whose stored -`spec.rules[].scope` is `Namespaced` — the status update is rejected 422, and the one object that -most needs to explain itself is the one that cannot. CRD Validation Ratcheting skips re-validation of -*unchanged* fields and is beta and default-on from 1.30, GA in 1.33, so the exposure is older -clusters or the gate turned off — which is why the test has to name a version. - -**The test.** Create a `ClusterWatchRule` with `scope: Namespaced` through a client that bypasses the -enum (or against an older CRD), then attempt a status update, on the minimum supported Kubernetes -version. - -**The fallback if it fails.** Widen the enum back and rely on the compile-path refusal plus a loud -`Stalled` condition. Refusing at admission is nice, but not at the cost of being unable to report the -refusal. - -## Version strategy: stay `v1alpha3` - -A wave this size invites `v1alpha4`, and I would not take it. A new version means a conversion path, -and the honest options — a conversion webhook (a serving dependency plus a cert lifecycle) or `None` -conversion with a stored-version migration — both cost more than the problem. The convention the repo -already uses is a **loud rejection**: keep the removed field in the schema, refuse it with a message -naming the replacement, for one release. `ClusterWatchRule.spec.rules[].scope` set that precedent, -and a rename is the case it is kindest on: the user's next `apply` tells them the new spelling. - -**The assumption it rests on is one consumer, and that is a countdown rather than a constant.** Loud -rejection is cheap because a single coordinated bump can absorb it; a second consumer makes the same -wave cost two, on two schedules. Two things follow, and both belong on this repo's roadmap rather -than only on the consumer's: - -- **Each wave leaves residue.** A refused field stays in the schema to say "no, not that anymore". - That graveyard is a real cost to a newcomer reading the CRD, and it is paid per wave, so the - number of remaining waves on `v1alpha3` is finite. Sweeping the refusals is what `v1alpha4` should - be for, and it should be one version bump carrying the removals rather than a version bump per - change. - - **What the residue turned out to be, measured on `main` after #330: one field.** This wave itself - left none — `allowedSourceNamespaces` and the two relocated `GitProvider` fields were **removed - outright** rather than refused, which is what - [`../UPGRADING.md`](../UPGRADING.md) is a hand-written migration for and what - [`../facts/crd-upgrade-strategies.md`](../facts/crd-upgrade-strategies.md) prices. The one - survivor is `ClusterWatchRule.spec.rules[].scope`, whose own doc comment says "Removed one release - from now" and which has now stood through 0.40, 0.41 and 0.42. Its countdown has expired, so a - release that is breaking anyway is where it goes. -- **The coupling is a dependency, not a courtesy.** Staying on `v1alpha3` requires the consumer to - keep pace with this repo's release cadence. If it cannot, the choice is not "revisit" in the - abstract — it is a real API version with a real conversion path, and the time to notice is before - the wave, not during it. - -## Order inside the wave - -Dependencies first, then the things that only need the object to be breaking. - -1. **The `scope: Namespaced` envtest**, above. Not an API change; its answer constrains the enum - work. Do it before planning. -2. **`spec.suspend`.** Precondition for anything that creates files, and independently the review's - highest-value gap: a way to stop the writes that is not deleting the object. -3. **`status.placement`** plus the post-scan validation pass. Independent of step 2 — it explains a - write that already happened, rather than previewing one that has not. -4. **`requestedAt`.** On-demand refresh of step 3. Shipped without the - `lastHandledReconcileAt` echo, for the reason recorded above. -5. ~~**Events on a changed resolution**, over the existing recorder.~~ Dropped; see interaction 4. -6. **B4**, as `spec.commit`. Last of the principle items, and the one that makes the object coherent. -7. **The source-scope deletion.** Independent of every step above, so it can be written in parallel; - placed here because a deletion reviews better once the additions it is not entangled with are - settled. -8. **The riders, closed.** The `CommitRequest` lifecycle hole shipped in #400 and the `default` - `ClusterProvider` message in #258. `CommitRequest.spec.author` is **declined**: authorship is - established, never asserted - ([`attribution.md`](../spec/attribution.md#authorship-is-established-never-asserted)). The - `TooManyStreams` cap was dropped by step 7. - -Steps 2 to 5 are additive and need no bump. Steps 6 to 8 are one release; step 1 gates the planning; -step 8 can be trimmed if the wave gets too big to review, since nothing else depends on it. The -placement fields are absent from this list on purpose: their order is -[`model.md`](../layout/model.md)'s. - -## What this costs, stated plainly - -- **One coordinated consumer bump**, for `spec.commit`, the source-scope changes and the riders. -- **One `docs/UPGRADING.md` entry.** The placement work needs no migration entry at all. The `*` - paragraph is the one to write carefully: it is the only change here that keeps its spelling and - changes its meaning. -- **A capability genuinely lost**: source-side label selectors, priced in the source-scope document. - It is the only thing in this wave a user can do today and cannot do afterwards. diff --git a/docs/design/gittarget-branch-worker-pending-writes.md b/docs/design/gittarget-branch-worker-pending-writes.md index fe7ff6832..b9a79e50d 100644 --- a/docs/design/gittarget-branch-worker-pending-writes.md +++ b/docs/design/gittarget-branch-worker-pending-writes.md @@ -4,7 +4,7 @@ > below is built, 5a2 included; step 3 was split into 3a, 3b, and 3c after review. The page stays > here rather than in `finished/` because its recovery contract is still the reference for how a > branch behaves through an outage. The -> [source review](gittarget-state-of-affairs.md#review-findings-and-remaining-gaps) records how each +> source review records how each > finding was closed. This is the write-path part of the "transition boundary" in > [`branch-worker-event-model.md`](branch-worker-event-model.md#prepare-the-transition-boundary-for-later-durability), > limited to the write path. Persistence stays with the diff --git a/docs/design/gittarget-state-of-affairs.md b/docs/design/gittarget-state-of-affairs.md deleted file mode 100644 index e4e4a9840..000000000 --- a/docs/design/gittarget-state-of-affairs.md +++ /dev/null @@ -1,153 +0,0 @@ -# GitTarget and the branch worker: state of affairs - -> **Source review**, 2026-10-03 at `373bf8d7` on the step-4 branch, updated 2026-10-04 when #413 -> built the remaining steps. This page distinguishes implemented behavior from what is left. It -> reports source and test coverage inspected; it does not claim current PR/CI status. - -## Where the implementation stands - -Every step of the [branch-worker pending-writes plan](gittarget-branch-worker-pending-writes.md) is built. Decided -writes take one materialization path, retry uses one schedule, and a refused replay entry no longer -blocks unrelated retained work. On top of that: the watch's admission boundary, bounded Git calls, -an explicit publication pause, and a status that operators can read. - -Recovery stays inside the existing worker. It pauses intake for the affected branch when capacity -fills, keeps accepted work, retries with backoff, and wakes producers once a push lands. -Restarting the Pod loses in-memory decisions and save receipts. Redis persistence remains in the -[future HA plan](../future/ha-gittarget-distribution-plan.md). - -## What is built - -| Capability | Code | Evidence inspected | -|---|---|---| -| All write kinds become pending writes before materialization | [`pending_writes_loop.go`](../../internal/git/pending_writes_loop.go) | `decided_write_test.go`, `branch_worker_split_test.go` | -| One checkout materializer and explicit applied-prefix tracking | [`branch_worker.go`](../../internal/git/branch_worker.go) | `dirty_worktree_recovery_test.go`, `branch_worker_split_test.go` | -| One retry deadline, 10s doubling to 5m, including parent recovery | [`retry.go`](../../internal/git/retry.go), [`parent_recovery.go`](../../internal/git/parent_recovery.go) | `retry_test.go`, `parent_recovery_test.go` | -| Failed materialization retains decided writes and their saves | `pending_writes_loop.go` | `TestDecidedWrite_SurvivesAFailedRebuildAndLandsThroughTheRetry` | -| Every accepted item counts against the outage budget, and the pause holds until a push lands | [`intake.go`](../../internal/git/intake.go) | `intake_test.go`, `TestDecidedWrite_EmptySavesCountAgainstTheBudgetDuringAnOutage` | -| A resync during backoff spends no connection | `pending_writes_loop.go`, `retry.go` | `TestDecidedWrite_AResyncDuringBackoffWaitsForTheRetry` | -| A stream on a paused branch waits to be woken, graded `BranchIntakePaused` | [`target_watch.go`](../../internal/watch/target_watch.go) | `TestRunTargetWatch_APausedBranchWaitsForIntakeToReopen` | -| Every call to a Git server is bounded; a lost push reply is settled by evidence | [`network_bound.go`](../../internal/git/network_bound.go) | `network_bound_test.go` | -| One publication report feeds every `GitTarget` on the branch, its held saves, and the backlog metrics | [`publication.go`](../../internal/git/publication.go), [`gittarget_publication.go`](../../internal/controller/gittarget_publication.go) | `publication_test.go`, `gittarget_publication_test.go` | -| Replay refusal settles only the affected entry | `pending_writes_loop.go`, `branch_worker.go` | [`replay_refusal_test.go`](../../internal/git/replay_refusal_test.go) | -| Save outcomes follow remote publication; the controller cannot time out a held save | [`commit_request_attach_loop.go`](../../internal/git/commit_request_attach_loop.go), [`commitrequest_controller.go`](../../internal/controller/commitrequest_controller.go) | Held-request retry and controller safety-bound tests | -| Parent selection and compare-and-swap remain publication guards | [`git_atomic_push.go`](../../internal/git/git_atomic_push.go) | Parent-change tests and `TestGitRoundTripLedger` | -| New watch streams start with a fresh replay; reconnects resume a cursor only after the stream's own replay completed | [`target_watch.go`](../../internal/watch/target_watch.go) | `runTargetWatch`, `TestRunTargetWatch_ResumesOnlyAfterItsOwnReplayCompleted` | -| A live event changes the dedup baseline and the cursor only once the worker accepted it | [`target_watch.go`](../../internal/watch/target_watch.go) | `refused_admission_test.go`, `desired_state_change_filter_test.go` | -| Each stream owns its filter baselines, seeded from its accepted replay; one `GitTarget` holds no overlapping collections | [`desired_state_change_filter.go`](../../internal/watch/desired_state_change_filter.go), [`collection_overlap.go`](../../internal/watch/collection_overlap.go) | `desired_state_change_filter_test.go`, `collection_overlap_test.go` | - -The pending writes live in memory. They do not yet reconstruct windows, timers, or outcomes after -process loss, and `PendingWrite` contains live interfaces and process references. - -## Review findings and remaining gaps - -### 1. A refused UPDATE could be skipped on reconnect (fixed in step 5a) - -`routeLiveTargetWatchEvent` stored the sanitized-content hash before routing, so a refused UPDATE -redelivered by the cursor resume was skipped as unchanged and the cursor advanced past it. Step 5a -records the baseline only after the worker accepts the event, records no cursor for an event a -stopping stream never enqueued, and resumes a stream from a cursor only after its own replay -completed. The [step 5a section](gittarget-branch-worker-pending-writes.md#step-5a-make-refused-admission-safe-built) -lists the producer inventory and the limits left for 5b. - -### 2. The retention threshold was not a total memory bound (fixed in step 5b) - -`syncAdmission` closed intake when a retry was pending and `pendingWritesBytes` reached the branch -budget. That improved ordinary outage behavior, but: - -- Empty request records and refusal touches have zero `ByteSize`; repeated empty saves bypass it. -- The FIFO caps item count (default 1,000), not payload bytes. A snapshot can be large. -- The open window, deferred resyncs, registered requests, and upstream snapshot collection also - consume memory. An 8 MiB retained threshold does not cap these allocations. -- Admission can reopen when retained bytes fall below the threshold even if publication still - fails. There is no explicit backlog-settlement latch. - -Step 5b charges every item at enqueue, with a fixed per-item overhead, counts everything the loop -holds, latches the pause until a push lands, refuses an oversized snapshot during an outage with -both sizes in the message, and keeps lifecycle work outside the gate. Accepted decisions are never -evicted. The budget counts serialized bytes, not heap. - -### 3. Backoff did not suppress every failing Git connection (fixed in step 5b) - -`decide` exempts resyncs from the ordinary retry hold. A resync can fetch during publication -backoff unless the missing-parent hold stops it. The producer also reconnects after a fixed -two-second delay when admission refuses it. That can repeatedly gather snapshots which the -worker cannot accept. - -Producers now wait for the branch to reopen intake, and a resync during backoff is answered with -the failure the retry is waiting out and kept pending. The single retry schedule remains the -recovery driver. - -### 4. Git network calls had no operation budget (fixed in step 6) - -Ref listing and fetch ran without the caller's context, and push ran under a worker context with -no deadline. Measurement showed worse: go-git's SSH transport ignores the context past the dial. -Step 6 bounds every call (two minutes) and the whole push cycle (five minutes), closes an SSH -connection when its call's context ends, and settles a push whose reply was lost by finding the -remote at the commits it sent. The -[step 6 section](gittarget-branch-worker-pending-writes.md#step-6-deadlines-on-git-network-calls-built) has the -measurement and the limits left. - -### 5. Publication and saturation had no coherent operator state (fixed in step 5c) - -Parent recovery has status, and push failures have logs and a counter. General publication failure -has no shared worker observation projected to all its targets. A held save can say `WaitingForPush` -without explaining that the remote rejects pushes or intake has stopped. A pre-push rebuild failure -also misses the push-failure counter. Intentional producer pause currently looks like a watch error. - -Step 5c publishes one worker report and projects it onto every `GitTarget` on the branch as -`Ready=False`/`Progressing`, with a message that changes only when the cause or the pause does, and -onto a held save's `WaitingForPush` message. A paused stream is `BranchIntakePaused`, and gauges -describe the backlog and the next retry. Independent target refusals and source replay stay -visible; a successful push cannot clear them. - -### 6. Watch replay recovers a weaker contract than accepted-write replay - -Backpressure cannot stop Kubernetes mutations or retain watch history indefinitely. A cursor that -expires forces a fresh snapshot; intermediate versions, deleted short-lived objects, and their -authors cannot be reconstructed. With `prune.mode: onEvent`, an expired DELETE can leave a Git -object retained. A complete scoped snapshot may sweep it under `always`; recovery must not silently -change that policy. - -A new process already starts a fresh watch replay even with a stored cursor, so the old assertion -that every restart resumes past volatile work was inaccurate. Initial replay repairs current -content where policy permits. It does not restore accepted windows or save outcomes. A push that -lands immediately before a crash can still be followed by a duplicate empty save after restart. - -## What the refactor got right - -Keep the separation between deciding, materializing, and publishing. It lets a failed fetch keep a -save waiting for its answer without guessing what a later snapshot would have meant. Step 4's -refusal isolation also prevents one invalid folder from blocking another target on the same branch. - -The earlier `owedSnapshot` proposal (a placeholder that would request a fresh snapshot in place of -discarded work) and a separate parent retry timer are unnecessary for the chosen capacity policy. -Keep accepted entries and reject new admission explicitly. The next change should repair the -acceptance boundary, not replace the pending writes or introduce a generic workflow engine. - -## Recommended order - -1. **Step 5a (built):** fix producer deduplication after rejected admission and test cursor resume. -2. **Step 6 (built):** bound network calls and test stalls and lost push responses. -3. **Step 5b (built):** complete capacity accounting and per-worker pause/resume, including producer waits. -4. **Step 5c (built):** expose publication failure, intake pause, and recovery through status and metrics. -5. **Step 7 (built):** update runtime documentation and validate the final branch. - -What is left belongs elsewhere: recording timers, attaches, withdrawals, and refreshes as -transitions ([event model](branch-worker-event-model.md#prepare-the-transition-boundary-for-later-durability)), -and persistence and HA ([HA plan](../future/ha-gittarget-distribution-plan.md)). - -## Document ownership - -| Page | Role after this review | -|---|---| -| `gittarget-branch-worker-pending-writes.md` | The steps as built, the pause/resume contract, and the review findings | -| [`branch-worker-event-model.md`](branch-worker-event-model.md) | Transition semantics and the later boundary for reconstructing execution | -| [`push-cooldown.md`](push-cooldown.md) | Healthy publication cadence; the success cooldown remains unchanged | -| [`commitrequest-design.md`](../spec/commitrequest-design.md) | Current save contract, including retained versus unaccepted requests | -| `ha-gittarget-distribution-plan.md` | Future durable acceptance, publication evidence, storage budgets, and HA | - -Parent observation, empty-repository bootstrap policy, configuration freshness, and refusal state -across restart remain separate follow-ups. Two providers naming one remote still create separate -workers; only remote compare-and-swap coordinates their writes. These do not need to be solved to -make an in-process outage explicit and recoverable. diff --git a/docs/design/open-asks-priority.md b/docs/design/open-asks-priority.md index 6d2c3381b..abbaa346c 100644 --- a/docs/design/open-asks-priority.md +++ b/docs/design/open-asks-priority.md @@ -26,7 +26,7 @@ > on its own, and **the GitTarget work is deliberately not in it**. The standing caveat has since > narrowed, because [`../layout/model.md`](../layout/model.md) reversed and the placement work is no > longer breaking: **a Tier 2 entry belongs to the postponed wave only if it changes a `GitTarget` -> field in a breaking way**, and [`gittarget-api-wave.md`](gittarget-api-wave.md) is the one place +> field in a breaking way**, and `gittarget-api-wave.md` (removed) is the one place > that lists which those are. Everything else here — the placement fields, `suspend`, > `status.placement`, and every Tier 1 entry — is additive and independently schedulable, and should > not wait for a bump. @@ -41,7 +41,7 @@ > Three backlogs are open at once and they overlap: the downstream consumer asks (revision 11, > 2026-07-28, which is the revision that filed #23), the API-surface block left unbuilt by the > status and configuration-model review — now sequenced in -> [`gittarget-api-wave.md`](gittarget-api-wave.md) — and the config-surface proposal in +> `gittarget-api-wave.md` (removed) — and the config-surface proposal in > [`config-surface-for-a-structured-repository.md`](../future/config-surface-for-a-structured-repository.md) > (B1–B6). This page merges them into one queue and says where we deliberately do **not** do > what was asked. @@ -253,18 +253,18 @@ and is not independently schedulable. |---|---|---|---|---| | ~~15~~ | ~~A declared `auditRoute` with zero facts must say so~~ **SHIPPED** as the latched `AuditFactsReceived` condition and `FACTS` column on `ClusterProvider` | consumer | — | — | | n/a | Stop paying a full grace for a delete fact that will never arrive (F, then C) | [`attribution-removal-wait-options.md`](attribution-removal-wait-options.md) | **1** | — | -| F9 | The `scope: Namespaced` status-write envtest | maintainer review | **1** | outside the wave, and **gates its planning**: the answer decides whether the narrowed enum can be kept ([`gittarget-api-wave.md`](gittarget-api-wave.md)) | -| ~~n/a~~ | ~~A declared path in a kustomize subdirectory is never rendered; the identity gate rejects the versionless canonical path~~ **SHIPPED** in 0.42.1 | [`placement-visibility-and-declared-defaults.md`](placement-visibility-and-declared-defaults.md) | — | [#295](https://github.com/ConfigButler/gitops-reverser/issues/295), [#319](https://github.com/ConfigButler/gitops-reverser/pull/319) | +| F9 | The `scope: Namespaced` status-write envtest | maintainer review | **1** | outside the wave, and **gates its planning**: the answer decides whether the narrowed enum can be kept (`gittarget-api-wave.md` (removed)) | +| ~~n/a~~ | ~~A declared path in a kustomize subdirectory is never rendered; the identity gate rejects the versionless canonical path~~ **SHIPPED** in 0.42.1 | `placement-visibility-and-declared-defaults.md` (removed) | — | [#295](https://github.com/ConfigButler/gitops-reverser/issues/295), [#319](https://github.com/ConfigButler/gitops-reverser/pull/319) | | n/a | `useKustomize` and `serializeNamespace`: the two things a path template cannot say (`spec.layout` was reversed) | [`../layout/model.md`](../layout/model.md) | **2** | [#322](https://github.com/ConfigButler/gitops-reverser/issues/322), **not** breaking, so not the wave | -| F6 | `spec.suspend`, `GitProvider.spec.interval`, `requestedAt` (no `interval` on `GitTarget`, see [`gittarget-api-wave.md`](gittarget-api-wave.md)) | maintainer review | **2** | wave | +| F6 | `spec.suspend`, `GitProvider.spec.interval`, `requestedAt` (no `interval` on `GitTarget`, see `gittarget-api-wave.md` (removed)) | maintainer review | **2** | wave | | ~~5~~ | ~~`CommitRequest.spec.author`, SAR-guarded~~ **declined**: authorship is established, never asserted | consumer (#220) | — | [`attribution.md`](../spec/attribution.md#authorship-is-established-never-asserted) | | B4 | `commitWindow` / `commit.message` move to GitTarget | config surface | **2** | wave | -| ~~B1~~ | ~~`GitTarget.spec.mode: Observe\|Write`~~ **dropped**: `suspend` already stops the writes, and `mode` buys only a declared posture over a pause | config surface | — | [`gittarget-api-wave.md`](gittarget-api-wave.md) | +| ~~B1~~ | ~~`GitTarget.spec.mode: Observe\|Write`~~ **dropped**: `suspend` already stops the writes, and `mode` buys only a declared posture over a pause | config surface | — | `gittarget-api-wave.md` (removed) | | 6 | Movable destination via `status.observedDestination` | consumer (#220) | **refused** | the destination stays immutable; a folder moves by delete-and-recreate | | F10 | CommitRequest TTL / ownerRef + the `delete` verb | maintainer review | **2** | wave | | n/a | The blocking resolve is head-of-line on the shard goroutine | [`../spec/attribution.md`](../spec/attribution.md#the-wait) | **2** | — | | B2 | `GitTarget.status.placement` (was `status.layout`) | config surface | **3** | [#296](https://github.com/ConfigButler/gitops-reverser/issues/296) | -| n/a | The ambiguous render root, the `declared` metric split, `{kindLower}`, canonical-as-template | [`placement-visibility-and-declared-defaults.md`](placement-visibility-and-declared-defaults.md) | **3** | [#296](https://github.com/ConfigButler/gitops-reverser/issues/296) | +| n/a | The ambiguous render root, the `declared` metric split, `{kindLower}`, canonical-as-template | `placement-visibility-and-declared-defaults.md` (removed) | **3** | [#296](https://github.com/ConfigButler/gitops-reverser/issues/296) | | B6 | The `default` ClusterProvider not-found message | config surface | **3** | — | | n/a | An aggregated create carries no name and no body: accept it, or stop waiting for it | [`../spec/attribution.md`](../spec/attribution.md#what-the-shape-driven-rules-reach-and-what-they-do-not) | **3** | — | | n/a | Entry-size ceiling and per-type stream count under a few hundred watched types | [`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) | **3** | — | @@ -463,7 +463,7 @@ the page whose *answer is unknown* rather than whose work is unscheduled, and be downstream assumes an answer. If the apiserver validates the whole object on a status-subresource write, the narrowed `ClusterWatchRule` enum leaves the one object that most needs to explain itself unable to write its own `Stalled` condition. The test, the version it has to name and the fallback -are in [`gittarget-api-wave.md`](gittarget-api-wave.md). Run it before planning anything that +are in `gittarget-api-wave.md` (removed). Run it before planning anything that depends on it. ### Tier 2: the breaking wave, all at once, while `v1alpha3` @@ -652,5 +652,5 @@ defects. 9. Not letting a decided-but-unbuilt list read as imminent. The placement work is filed as [#295](https://github.com/ConfigButler/gitops-reverser/issues/295) (correctness) and [#296](https://github.com/ConfigButler/gitops-reverser/issues/296) (visibility); - [`placement-visibility-and-declared-defaults.md`](placement-visibility-and-declared-defaults.md) + `placement-visibility-and-declared-defaults.md` (removed) now says which of its eight items shipped, which is **none of them**. diff --git a/docs/design/placement-visibility-and-declared-defaults.md b/docs/design/placement-visibility-and-declared-defaults.md deleted file mode 100644 index 3c2744197..000000000 --- a/docs/design/placement-visibility-and-declared-defaults.md +++ /dev/null @@ -1,581 +0,0 @@ -# Placement, made visible: naming, a declared default, and `status.layout` - -> **design, decided**: half built. Index: [`../INDEX.md`](../INDEX.md) -> Date: 2026-07-30, reconciled 2026-09-03 against what has since shipped. -> -> **`status.layout` shipped as `status.placement`, and the ambiguous render root with it**, both in -> [#326](https://github.com/ConfigButler/gitops-reverser/pull/326) — so the two largest rows of the -> table below are done, and the `declared` metric value has since split into `by_type`/`default`. -> What is left is the rest of the legibility set: the canonical path as a template constant, -> `{kindLower}`, and the re-opened CRD default. -> -> **The decisions below stand. The build list does not.** An earlier revision of this page said -> "everything decided here lands in that same PR", and that turned out to be wrong: PR #291 shipped -> the sibling-inference deletion, the three placement counters, and the namespace-transformer safety -> fix, and **none of the eight items** this page had queued behind them. What is decided and unbuilt -> is now tracked as GitHub issues rather than asserted here as imminent; see -> [what is built and what is filed](#what-is-built-and-what-is-filed). -> -> Reading it in order matters, because the two halves have different standing. The **findings** are -> checkable facts about the tree and they are why the calls are what they are. The **calls** are still -> the calls. Question 2 (a CRD default for `placement.default`) has since been **reopened rather -> than superseded**. An earlier revision of this note said [`model.md`](../layout/model.md) had -> argued the path template was the wrong primitive to default at all; model.md has since reversed -> that. The template stays, #319's ancestor walk removed the correctness objection, and what is left -> is the legibility trade this page already priced — now an open question in -> [`model.md`](../layout/model.md#open-questions). - -Three questions came out of reviewing #291, and one of them (a CRD default for -`placement.default`) is a better idea than my first answer to it gave credit for. This document -states the findings behind each, the options, and the call. - -## The decisions, up front - -| Question | Call | Why in one line | -|---|---|---| -| Rename `source="canonical"` to `default`? | **No.** Keep `canonical`, split `declared` into `byType` and `default`, and fix the prose | `placement.default` is a *declared* template; reusing the word for the built-in path makes one name mean both a declaration and the absence of one | -| Default `placement.default` in the CRD? | **No**, on narrower grounds than the first draft claimed | A defaulted default consumes the slot ahead of the one step no template can express ("beside the folder's one kustomization"). Once F10 is fixed this is a legibility trade rather than a correctness wall | -| Publish the effective layout in status? | **Yes, now** | It gives the clarity the CRD default was reaching for, without freezing anything, and the seam it needs already exists | -| `{kindLower}` or a `toLower` function? | **`{kindLower}`** | A function syntax is a language; one variable answers the actual need | -| Is the kustomize root "still a little bit inference"? | **Yes, and it earns its keep anyway** | Its answer changes only when a `kustomization.yaml` changes, and ignoring it produces a file nothing renders. It has to be made as visible as everything else | -| A declared path in a subdirectory of a kustomize folder | **Bug, fix it**: walk up to the nearest kustomization | Today it is registered only when render-root scoping happens to be in force, so a `byType` entry into a subdirectory silently produces a file nothing renders (F10) | -| Two kustomize roots (ambiguous) | **Keep writing, make it loud** | The file is currently unreachable *and* uncounted. Refusing would protect nothing that mirroring does not | - -## Findings - -Each of these is checkable against the tree, and three of them changed a decision. - -**F1. The word "default" already means two things in our own docs.** `spec.placement.default` is a -user-declared catch-all. [`architecture.md`](../architecture.md) calls the built-in path "the -**built-in default** path"; the CRD comment calls the same thing "the built-in canonical path". One -of those has to go, whatever we do with the metric. - -**F2. `source="declared"` hides which declaration answered.** A `byType` hit and a `default` hit are -one series, so a catch-all quietly swallowing a type you meant to name explicitly looks identical to -a rule working as intended. For a metric whose job is "is a rule missing?", that is the wrong place -to lose resolution. - -**F3. A CRD default for `placement.default` fails our own validation today.** Not the decisive -objection (F9 is), and not unfixable (F4 is the fix), but it is where the idea stops today. -[`validateSecretSafety`](../../internal/controller/gittarget_placement_validation.go) rejects a -`default` that is not identity-complete unless the target also declares an identity-complete -`byType["v1/secrets"]` entry. And `IdentityCompletePlacementTemplate(tmpl, false)` requires -`{groupPath}`, `{version}` **and** `{resource}`. The built-in path is deliberately **versionless** -(see [`../facts/`](../INDEX.md) and the versionless-path decision), so the template we would default -to, - -```text -{namespace}/{groupPath}/{resource}/{name}{sensitiveSuffix} -``` - -is judged *not* identity-complete, and every GitTarget that did not also declare a Secret route -would go `Validated=False`. The CRD's default would be refused by the CRD's own gate. - -**F4. The `{version}` requirement is wrong on its own terms.** Two versions of one group/resource -are the *same object*, which is exactly why the built-in path dropped the version segment. A -template carrying scope, group, resource and name cannot collide two distinct identities, with or -without a version. So the requirement rejects safe templates: a user who writes the -versionless canonical layout by hand is told it is a bundling path. That is a bug independent of -anything else here, and fixing it is a precondition for F3 ever being reconsidered. - -**F5. Defaulting semantics, precisely, because a first draft of this document got them wrong.** -A CRD default is applied to the request object on create, update and patch, and is therefore -**persisted**. It is also applied **in memory when reading from etcd**, using the storage version's -schema, so the controller sees the value even for an object stored before the default existed. It is -**not** persisted by our own status writes: `GitTarget` has a status subresource -([gittarget_types.go](../../api/v1alpha3/gittarget_types.go)), so a status update writes only the -status stanza and leaves the stored spec alone. In a GitOps-managed cluster the answer is -nevertheless "yes, quickly", because the user's own Flux or Argo re-applies the object continuously -and each apply persists it. - -Two consequences follow, and **both are smaller than an earlier draft of this document claimed**. -Freezing the built-in path per target is arguably a feature rather than a cost: placement is already -create-time and non-retroactive, so a long-lived target keeping its established layout for new types -is the stability we already promise, and a value in the spec is visible and editable in a way a -built-in is not. "Indistinguishable from a declaration" is close to true, and an earlier draft's -escape from it was wrong: `metadata.managedFields` is field-management bookkeeping rather than -durable provenance, so it cannot separate "the user declared this" from "the schema supplied it". -What is left is spec bloat: every GitTarget carries a template string whether -its owner cares about layout or not. That alone would not decide anything. - -**F5b. Defaulting a map is a floor that vanishes when you stand on it.** A CRD default applies to an -**absent** field; it is not an additive base and is never re-merged per key. So a defaulted -`byType: {"v1/secrets": "..."}` is gone as soon as that field is present at all, which is the moment a -user writes any `byType` entry of their own. -Their unrelated ConfigMap entry would silently drop the Secret route, leaving a defaulted bundling -`default` unguarded, and the object would go `Validated=False` on an edit that had nothing to do with -Secrets. This is why "default the Secret route too" (the obvious repair for F3) is worse than -fixing F4. - -**F6. The data-plane to status seam already exists, and it enqueues.** -[`retention_rollup.go`](../../internal/watch/retention_rollup.go) is the pattern: -`MarkTargetRetention` records a fact from the write path into an epoch-scoped per-target roll-up and -calls `enqueueGitPathChange` **on a change only**, and the controller projects it in -`gitTargetRetentionStatus`. This retires the objection recorded in -[`open-asks-priority.md`](../design/open-asks-priority.md) that placement facts cannot reach the GitTarget -promptly because "a refusal recorded on the data plane does not enqueue the GitTarget". One does -already. `status.retention` is proof. - -**F7. `{kind}` renders capitalized and `{resource}` renders plural.** So neither gives -`deployment-simon.yaml`, and the spec's own "keep it small" rules out template functions ("no -template functions except safe path-segment sanitization"). - -**F8. Two supported kustomizations produce a file nothing renders, and nothing counts it.** -Ambiguity declines to the canonical path, which no root's `resources:` graph reaches. No entry is -attempted, so `placement_kustomization_entries_total` never sees it. It is the exact failure that -counter exists for, in the one case it cannot observe. - -**F9. A defaulted `default` would make the kustomize-root step unreachable.** This is the objection -that decides Question 2, and it is structural rather than a trade. -[`LocateNew`](../../internal/manifestanalyzer/placement.go) resolves in this order: - -```text -byType -> default -> kustomize root -> canonical -``` - -`resolveDeclared` returns on any non-empty `policy.Default`, so a default that is *always* set means -step 3 never runs. Every new file in a kustomize overlay would take the canonical path instead of -landing beside the `kustomization.yaml`, with no `resources:` entry: committed, looking mirrored, and -rendered by nothing. That is exactly the bug the kustomize-root step was added to prevent. - -The repairs are worse than the defect. Reordering so the render root beats `default` makes a user's -real declaration lose to the folder, which inverts the precedence the whole feature rests on. -And there is no sound way to distinguish a defaulted value from a declared one at read time. - -**F10. And that failure is reachable today, from a declaration, which makes it a live bug rather -than an argument about defaulting.** Review found this, and it is the most useful thing on the page. -[`governingKustomization`](../../internal/manifestanalyzer/placement.go) decides whether a new file -gets a `resources:` entry by looking in exactly two places: the kustomization in the file's **own** -directory, and (only when render-root scoping is in force) the write scope's root. So: - -| Folder shape | Declared path | Registered? | -|---|---|---| -| overlay reading a base (`writeScope != ""`) | `configmaps/app.yaml` | yes, in the overlay root, by accident of a branch added for another reason | -| self-contained folder that IS the kustomize root | `configmaps/app.yaml` | **no** | - -One line of user config reproduces it: a kustomize folder plus -`byType: {v1/configmaps: "configmaps/{name}.yaml"}`. The file lands in a directory with no -kustomization of its own, no entry is added, and kustomize never builds it. Because no entry is -attempted, `placement_kustomization_entries_total` cannot see it either. - -**The fix**: `governingKustomization` walks **up** from the resolved path to the nearest kustomization -within the write jail, replacing the own-directory check and the write-scope special case with one -rule. An ancestor walk cannot escape the jail by construction, the relative entry it produces is what -`appendKustomizationResource` already computes, and the already-listed check is path-based, so it -stays idempotent. A nearest ancestor that is *unsupported* still cannot be edited; that becomes a -counted `failed` instead of silence. - -**What this does to F9**: it weakens it, and the page should say so rather than keep an argument that -has been undercut. With the ancestor walk a defaulted `default` no longer produces unrendered files, -because the nested tree would be registered in the root and would build. What survives is narrower. -The built-in path is the wrong *layout* for a kustomize folder, and **no template can express "beside -the folder's one root"**, which is why that step is a step rather than a template. Defaulting -`default` consumes the slot ahead of a step that exists precisely because a template cannot say what -it says. That is a legibility and structure argument, not a correctness one, so the CRD default moves -from "no" to "a trade available to us". The recommendation does not change; its grounds do. - -## Question 1: `canonical` or `default`? - -The pull toward `default` is real: it is the word for "what happens when you say nothing", and -`placement.default` is the field a user reaches for. But that is precisely the collision (F1). If -the metric said `source="default"`, a reader could not tell whether their `spec.placement.default` -had matched or whether nothing had matched at all, and those are opposite situations with opposite -fixes. - -So: **`canonical` keeps the built-in path**, the prose stops calling it "the built-in default", and -the resolution gains the distinction that was missing (F2): - -| `source` | Means | -|---|---| -| `byType` | an exact type entry matched | -| `default` | the declared catch-all matched. Same word, same meaning as the CRD field | -| `kustomize_root` | the folder has exactly one supported kustomization, so the file went beside it | -| `canonical` | nothing above applied; the built-in path was used | - -`default` then means one thing everywhere: the user's catch-all. - -## Question 2: a CRD default for `placement.default` - -The idea: if the built-in path were the CRD's default for `placement.default`, every GitTarget would -show its own layout, "canonical" would stop being a hidden fourth mechanism, and `source` would -collapse to `byType` and `default`. That reasoning is sound, and for a reader of one object it **is** -clearer. This section was rewritten after review, because the first version of it led with the wrong -objection. - -### What is not the reason - -**Not "it fails validation" on its own (F3).** It does today, but the rule doing the rejecting is -itself wrong (F4): the versionless canonical template cannot collide two identities, and demanding -`{version}` of it contradicts the decision that removed the version segment. One line fixes it. - -**Not "the value gets persisted" (F5).** It does get persisted, on every spec-writing apply, and in a -GitOps-managed cluster that is continuous. But freezing the built-in path per target is arguably -*desirable*: placement is already create-time and non-retroactive, so a long-lived target keeping its -established layout for new types is the stability we already promise, and a string in the spec is -visible and editable where a built-in is neither. `metadata.managedFields` is not a way around -it either: field management is bookkeeping, not provenance. What is left is spec bloat, which decides -nothing. - -### Why not also default the Secret route? - -The obvious repair for F3 is to default `byType["v1/secrets"]` as well: narrowed to one type, it only -needs `{name}` plus scope to be identity-complete, `secretRouteComplete` goes true, and the -bundling-default check stops objecting. It does work, and it is still the wrong move, because -**defaulting a map is a floor that vanishes when you stand on it** (F5b). A user who writes any -`byType` entry of their own replaces the whole map, silently dropping the Secret route and leaving a -defaulted bundling `default` unguarded, so an edit about ConfigMaps flips the object to -`Validated=False`. Fixing F4 removes the need for the workaround entirely. - -### The objection that decides it - -**A defaulted `default` is never empty, so it shadows the kustomize-root step (F9).** `resolveDeclared` -returns on any non-empty declared template, and the render-root step runs after it, so defaulting the -field makes step 3 unreachable. - -The first version of this section claimed the consequence was unrendered files. **That claim does not -survive F10**: the same gap is reachable from a plain declaration today, it is a bug we are fixing, -and once `governingKustomization` walks up to the nearest kustomization a nested path is registered -and builds. So the honest form of the objection is smaller and structural: - -- a nested canonical tree registered inside an overlay **renders**, and no one would choose it over a - file beside the `kustomization.yaml`. Defaulting would make the worse layout the layout; -- **no template can express "beside the folder's one supported kustomization"**, which is why that - step exists as a step. A defaulted template consumes the slot in front of it, so the one part of - the ladder that cannot be written down as a path would become unreachable by construction. - -The repairs still invert something load-bearing. Putting the render root ahead of `default` makes a -user's real declaration lose to the folder. And there is no sound way to tell a defaulted value from a -declared one at read time, so "treat a declared default differently" is not implementable. - -### What we do instead - -Publish the **effective** placement in status. It answers the same question ("what will happen to a -new Deployment in this folder?") from a derived field, so it cannot fork from the code and improves -when the code improves. Crucially it can show the ladder **without collapsing it**: the declared half, -the render root the operator found, and what happened, all at once. A spec default can only -express the ladder by flattening it, which is what F9 is. - -**If we want the spec default later**, the order is written down: fix F4, pin the canonical template -against `ToGitPath()` byte-for-byte, and solve F9 first, because nothing else on this page matters -until a defaulted value can coexist with the render-root step. Status is worth having either way, so -nothing built now is wasted. - -## `status.layout`, renamed `status.placement` - -> **The field is `status.placement` now, and it is much smaller than the shape below.** -> [`model.md`](../layout/model.md) renamed it when `spec.layout` stopped existing, and is the -> specification of record. What it kept from this page is the *principle* — an observation, not a -> condition, bounded rather than per-type. What it dropped is most of the shape: the historical half -> (`placedResources`, `canonicalTypes`, `refusedResources`) went to metrics, `examples` went because -> a fabricated object at a fabricated path is not an answer, `declaredTypes` went because it counts a -> spec map the same GET returns, and `renderRootReason` became the `LayoutResolved` condition's -> reason. `status.placement` now publishes `mode`, `renderRoot`, `readOnlyBases`, -> `resolvedAtCommit` and `resolvedAt`, and nothing else. Read the shape below as the argument that -> produced it, not as the field. - -An **observation, not a condition**, in the sense -[`GitTargetStatus.Retention`](../../api/v1alpha3/gittarget_types.go) already establishes: nothing -here may fail a reconciliation or move a condition, because a folder with no declared placement is a -supported configuration and not a fault. It follows `status.streams`'s bounding rule too: counts and -a capped example list, never a per-type list, however many types a target watches. - -```yaml -status: - layout: - # what the operator understood about the FOLDER - renderRoot: overlays/production # empty unless exactly one supported kustomization - renderRootReason: SingleKustomization # SingleKustomization | Ambiguous | None - # where new files are decided FROM - effectiveFallbackSource: KustomizeRoot # DeclaredDefault | KustomizeRoot | Canonical - declaredTypes: 2 # count of placement.byType entries in force - # what actually happened, bounded - placedResources: 14 # resources placed since observedRevision, appends included - canonicalTypes: 1 # distinct types that took the built-in path - refusedResources: 0 # resources NOT mirrored (see reasons below) - examples: # <= 3, most recent first - - type: apps/v1/deployments - path: team-a/apps/deployments/api.yaml - source: Canonical # ByType | Default | KustomizeRoot | Canonical - observedRevision: 9f3c1ab # the scanned commit the current half came from - observedTime: "2026-07-30T09:14:22Z" -``` - -`renderRootReason` and `effectiveFallbackSource` are the two fields that make the operator's judgement -inspectable per target rather than only aggregated in a metric, and `effectiveFallbackSource` answers -"what happens to a type I have not named?" in one word. An earlier draft called it `defaultSource` -with values `BuiltInCanonical | Declared`, which was wrong: it read `BuiltInCanonical` for a folder -whose files were going to the kustomize root, contradicting the `source` on its own example. - -**Two halves with different freshness, which must not be confused.** `renderRoot`, -`renderRootReason`, `effectiveFallbackSource` and `declaredTypes` are the **current** answer, derived -from the last repository scan and stamped with `observedRevision`. `placedResources`, -`canonicalTypes`, `refusedResources` and `examples` are **historical**, accumulated since that -revision. Placement is sparse, so a stable target may place nothing for weeks: a status built only -from write events would be empty for exactly the folder someone is asking about. The current half has -to come from the scan rather than from a placement having happened. `examples` is what makes the -historical half actionable: a type plus the path it landed on is the `byType` line to write. - -### Case 1: greenfield, nothing declared - -A target this operator bootstrapped. No `spec.placement`, no kustomization. Canonical is not a -fallback here, it is the layout, and status says so without implying a problem. - -```yaml -status: - layout: - renderRootReason: None - effectiveFallbackSource: Canonical - declaredTypes: 0 - placedResources: 23 - canonicalTypes: 4 - refusedResources: 0 - examples: - - type: v1/configmaps - path: team-a/configmaps/app-config.yaml - source: Canonical - observedTime: "2026-07-30T09:14:22Z" -``` - -Nothing to do. `canonicalTypes: 4` with `renderRootReason: None` is a tidy canonical folder. - -### Case 2: a kustomize overlay, nothing declared - -The folder has one `kustomization.yaml`. Every new file lands beside it and joins its `resources:` -list, so nothing falls back and no declaration is needed. - -```yaml -status: - layout: - renderRoot: overlays/production - renderRootReason: SingleKustomization - effectiveFallbackSource: KustomizeRoot - declaredTypes: 0 - placedResources: 6 - canonicalTypes: 0 - refusedResources: 0 - examples: - - type: v1/configmaps - path: overlays/production/debug-toolbox.yaml - source: KustomizeRoot - observedTime: "2026-07-30T09:15:02Z" -``` - -This is the case a metric alone reads ambiguously. `canonicalTypes: 0` plus a named `renderRoot` says -the folder decided, and that the operator agrees with it. - -### Case 3: brownfield, one type still missing a rule - -The user declared their ConfigMap bundle. Deployments were not covered, and no kustomization governs -the folder, so they went to the built-in path. - -```yaml -status: - layout: - renderRootReason: None - effectiveFallbackSource: Canonical - declaredTypes: 1 - placedResources: 9 - canonicalTypes: 1 - refusedResources: 0 - examples: - - type: apps/v1/deployments - path: team-a/apps/deployments/api.yaml - source: Canonical - - type: v1/configmaps - path: all.yaml - source: ByType - observedTime: "2026-07-30T09:16:41Z" -``` - -The two examples side by side are the whole point: one type is doing what was asked, one is not -covered, and the fix reads straight off the field. - -```yaml -placement: - byType: - v1/configmaps: "all.yaml" - apps/v1/deployments: "{namespace}/deployments/{name}.yaml" # the missing line -``` - -### Case 4: two overlays under one GitTarget (F8, silent today) - -`spec.path` covers `overlays/staging` and `overlays/production`. Neither root can be assumed, so new -files take the canonical path, where **no kustomization reaches them**. They are committed, they -look mirrored, and nothing applies them. - -```yaml -status: - layout: - renderRootReason: Ambiguous - effectiveFallbackSource: Canonical - declaredTypes: 0 - placedResources: 3 - canonicalTypes: 2 - refusedResources: 0 - examples: - - type: v1/configmaps - path: team-a/configmaps/cache.yaml - source: Canonical - observedTime: "2026-07-30T09:18:03Z" -``` - -`Ambiguous` is the word that does not exist today. The fix is the user's to choose: split the -GitTarget per overlay, or declare a `byType`/`default` that points inside one of them. - -### Case 5: a refusal, from a type the static gate cannot see - -`placement.default` is a bundle, and an operator-configured sensitive type (via -`--additional-sensitive-resources`) resolves onto it. The `Validated` gate cannot catch this, because -it only knows about core Secrets, so the write path refuses it fail-safe. Today that is a log line -and a resync counter; here it is on the object. - -```yaml -status: - layout: - renderRootReason: None - effectiveFallbackSource: DeclaredDefault - declaredTypes: 0 - placedResources: 4 - canonicalTypes: 0 - refusedResources: 1 - refusedReason: SensitiveAppend # most recent, from the counter's closed set - examples: - - type: v1/configmaps - path: all.yaml - source: Default - observedTime: "2026-07-30T09:20:11Z" -``` - -`refusedResources: 1` is a resource **absent from the mirror**, which is the one number on this -field that is unambiguously a problem to fix. - -## Question 3: `{kindLower}` - -A variable, not a function. Once `|lower` exists, `|upper`, `|trim` and `|replace` are each one PR -away, every one of them is new validation surface, and every one is a new way to render an empty path -segment. The spec's "keep it small" section already forbids it, and one variable answers the need: - -```yaml -placement: - default: "{kindLower}-{name}{sensitiveSuffix}" # deployment-simon.yaml -``` - -`{kindLower}` is `strings.ToLower` of a value we already hold, so it needs no new inputs and cannot -be empty when `{kind}` is not. The alternative, `{resourceSingular}` from discovery's -`SingularName`, is what `kubectl` uses and is more authoritative in principle, but it has to be -threaded from the type registry, needs an empty-value fallback, and I could not construct a real type -where it differs from the lowercased Kind. Not worth the plumbing unless "identical to kubectl" is a -guarantee we want to state. - -**Say plainly in the docs** that this layout is not identity-complete: two namespaces holding a -Deployment named `simon` render the same path, and they would land as two documents in one file. It -is the right layout for a single-namespace folder and the wrong one for a fleet folder, and the -operator cannot tell the two apart without being told. - -## Is the kustomize root inference? - -Yes, in the sense that it reads the repository and its answer can change without any Kubernetes -object changing. It differs from Option C in two ways that decide whether it stays: - -| | Option C | kustomize root | -|---|---|---| -| Reads | where similar documents live, an aggregate over content | one structural fact: is there exactly one supported root | -| Changes when | any document is added or removed | a `kustomization.yaml` is added or removed | -| Cost of ignoring it | a file in an unusual place | a file **nothing renders** | - -The third row is why it stays; the first two are why it is safe enough to keep. But it has to be -made just as visible, and that is what `renderRoot` plus `renderRootReason` provide: the -judgement becomes a field an operator can read, instead of a behavior they infer from where files -appeared. - -For the ambiguous case (F8), whether to refuse the write or write it loudly is **open**, and an -earlier draft calling it settled was a mistake. "An unrendered file is a better record than no file" -is a policy claim, and the opposite is at least as strong: a committed manifest nothing applies -manufactures a false appearance of convergence, which is the failure class this project ranks first. -The layout model probably decides it, because refusing is principled once a user has *asserted* a -single-root folder and the folder disagrees. - -What is not open: the signal does not belong on `placement_kustomization_entries_total`. That counter -counts **attempts** to register an entry -([placement_metrics.go](../../internal/git/placement_metrics.go)), and this is the case where no -attempt is made. It needs its own value on the refusal counter (if we refuse) or its own series (if we -write), never a third outcome on a counter about attempts. - -## What is built, and what is filed - -PR #291 shipped three things, and they are the ones that did not depend on any decision on this page: -the sibling-inference deletion, the `placements_total` / `placement_refusals_total` / -`placement_kustomization_entries_total` counters, and the namespace-transformer safety fix (a -kustomization naming a namespace other than the resource's own no longer renders the document as a -different object). `source="canonical"` kept its name, which is this page's Question 1 answered by -doing nothing. - -Everything else below was **decided and unbuilt** when this page was written; two rows have since -shipped and are marked. The rest is filed as GitHub issues so it is legible without reading this -page, and the two correctness items are separated from the legibility ones -because they are worth different urgency: - -| Item | State | Why it is where it is | -|---|---|---| -| F10: register a declared path with the kustomization that governs it | **SHIPPED** in 0.42.1 (#319) | One `byType` line into a subdirectory silently produced a file nothing renders. The fix made registration an invariant, which is what reversed [`model.md`](../layout/model.md) | -| Drop the `{version}` requirement from `IdentityCompletePlacementTemplate` | **SHIPPED** in 0.42.1 (#319) | It contradicted the versionless-path decision, and it was what made any future spec default fail our own gate | -| `status.layout` | **SHIPPED** as `status.placement` in #326 | The durable half of "what did the operator understand about this folder". It waited for the layout model's vocabulary and shipped with it | -| Split `declared` into `byType` and `default`; unify the prose on "canonical" | **SHIPPED** as `by_type`/`default` | A catch-all quietly swallowing a type you meant to name looks identical to a rule working | -| `{kindLower}` | **SHIPPED** in #400 | Small, self-contained | -| Canonical path as a template constant | **SHIPPED** in #400 as `BuiltinPlacementTemplate` | Removes the hand-written duplication; what a future default would reuse | -| `renderRootReason: Ambiguous` | **SHIPPED** in #326 | Shipped with `status.placement`, and the layout model decided the policy: a folder covering two roots REFUSES the placement rather than writing it unrendered | -| A CRD default for `placement.default` | **re-opened** | The reversal took away the answer this row used to give. There is no `layout.kind`; the template stays, and now that the ancestor walk registers a defaulted path, the objection is legibility rather than correctness. It is an open question in [`model.md`](../layout/model.md) | - -The original build list follows, because each entry says *how* to build the thing and that is the part -an issue should not have to restate. - -Ordered by risk, smallest first. - -1. **Register a declared path with the kustomization that governs it (F10).** Replace - `governingKustomization`'s own-directory check plus write-scope special case with a walk up to the - nearest kustomization inside the write jail, so a `byType` or `default` path into a subdirectory of - a kustomize folder joins the root's `resources:` list instead of being committed outside every - render. A correctness fix, so it goes first, and the two cases stop differing by whether - render-root scoping happens to be in force. -2. **Metric values.** Split `declared` into `byType` and `default`; keep `canonical`. Unify the prose - on "canonical" and drop "built-in default" from [`architecture.md`](../architecture.md) and the - CRD comments (F1, F2). -3. **`{kindLower}`**, plus the single-namespace recipe and its identity caveat in - [`configuration.md`](../configuration.md) (F7). -4. **Drop the `{version}` requirement** from `IdentityCompletePlacementTemplate` for a non-narrowed - template, with a test for the versionless canonical shape being accepted. This is a bug fix on its - own terms and it unblocks any future spec default (F3, F4). -5. **Express the canonical path as a template constant** rendered through - `RenderPlacementTemplate`, and pin it byte-for-byte against `ResourceIdentifier.ToGitPath()` - across cluster-scoped, core, grouped and sensitive identities. Removes the hand-written - `canonicalPath` duplication and is what a future default would reuse. -6. **`status.layout`**, in the shape above, over the `MarkTargetRetention` seam: an epoch-scoped - per-target roll-up marked from the write path, enqueued on change, projected by the controller, - asserted by an envtest (F6). -7. **The ambiguous render root**: publish `renderRootReason: Ambiguous` now, and leave the - refuse-or-write policy to the layout model. Whatever it decides, the signal gets its own series - rather than a third outcome on a counter that counts entry *attempts* (F8). -8. **Docs**: the spec's resolution-ladder section, `configuration.md`, `interpreting-metrics.md` for - the new label values, and the `UPGRADING.md` entry extended with the metric-value split. - -**Not now, with the trigger written down:** a CRD default for `placement.default`. This was "not -now" when the page was written and it stays "not now", but for a weaker reason than the page -originally gave. An earlier revision said the layout model dissolved the question by replacing the -template with a defaultable `layout.kind`; the model has since reversed and the template stays, so -the question is live again. What changed in its favour is the shipped ancestor walk: a defaulted -path into a subdirectory is now registered with the kustomization that governs it, so F9's -shadowing objection is no longer a correctness wall. The freezing question (F5) is a trade we could -take. What is left is legibility, and that is materially weaker than the case this page refused. -`spec.expect.layout` stays out too, on the config-surface doc's own rule: publish the -observation before inventing the assertion. - -## Open questions - -- Should `status.layout.examples` prefer the **most recent** placements or the **most actionable** - ones (fall-backs and refusals first)? Most recent is simpler and cheaper; actionable is what - someone reading it wants. My inclination is actionable, capped at three, with fall-backs ranked - above successes. -- Does `refusedReason` belong on the field at all, or is the count plus the metric enough? It is the - one value here that can go stale in a way a count cannot. -- `{resourceSingular}` as a stated "identical to kubectl" guarantee: worth the discovery plumbing, or - is `{kindLower}` the end of it? diff --git a/docs/design/sensitive-resource-diagnostics-follow-up.md b/docs/design/sensitive-resource-diagnostics-follow-up.md index f68a4b528..4ed89274f 100644 --- a/docs/design/sensitive-resource-diagnostics-follow-up.md +++ b/docs/design/sensitive-resource-diagnostics-follow-up.md @@ -3,7 +3,7 @@ > **design**: open, not yet built. Index: [`../INDEX.md`](../INDEX.md) This follow-up builds on -[sensitive-resource-classification-plan.md](../finished/sensitive-resource-classification-plan.md). +sensitive-resource-classification-plan.md (removed). The first increment only adds explicit startup classification to the Git write path. This document keeps the diagnostics and policy work out of that small change. diff --git a/docs/design/source-scope-simplification.md b/docs/design/source-scope-simplification.md index 62c1babc7..491ac5e0e 100644 --- a/docs/design/source-scope-simplification.md +++ b/docs/design/source-scope-simplification.md @@ -184,7 +184,7 @@ hundred list calls at warm-up today, one per namespace, each with its own cursor schedule and its own share of apiserver watch cache. Cluster-wide makes it one of each, and the saving grows with the cluster, which is exactly the direction the enumeration got worse in. A stream-count cap had been queued to bound that fan-out; with the fan-out deleted, the cap was dropped -rather than resized ([`gittarget-api-wave.md`](gittarget-api-wave.md)). +rather than resized (`gittarget-api-wave.md` (removed)). ## Consequences, including two breaking semantic changes diff --git a/docs/design/support-boundary/README.md b/docs/design/support-boundary/README.md index f94595e58..79fb1ddaa 100644 --- a/docs/design/support-boundary/README.md +++ b/docs/design/support-boundary/README.md @@ -89,6 +89,6 @@ distinguishes those planned gates from the permanent construct boundary. cluster is opt-in, through `onRefusal: Reapply` and [the apply trigger](../apply-trigger.md); preflight itself promises no automatic rollback. - Historical implementation records: [image/replica edit-through](finished/images-and-replicas-edit-through.md) - and [higher-level KRM support](finished/higher-level-krm-documents.md). + and [higher-level KRM support](support-contract.md). - [Fixture corpus](../../../test/fixtures/gitops-layouts/) — executable evidence; its generated `support-today.md` is a scanner report, not the runtime contract. diff --git a/docs/design/support-boundary/finished/higher-level-krm-documents.md b/docs/design/support-boundary/finished/higher-level-krm-documents.md deleted file mode 100644 index 30cd4530b..000000000 --- a/docs/design/support-boundary/finished/higher-level-krm-documents.md +++ /dev/null @@ -1,162 +0,0 @@ -# Higher-level KRM objects as first-class documents - -> Status: shipped ([#203](https://github.com/ConfigButler/gitops-reverser/pull/203)) — -> corpus + e2e pins landed with this doc; **no operator code changed** (the guarantee -> already held; this work makes it load-bearing). Filed under `finished/`. -> Captured: 2026-07-07 -> Related: -> [../README.md](../README.md), -> [../kustomize-support-boundary.md](../kustomize-support-boundary.md), -> [images-and-replicas-edit-through.md](images-and-replicas-edit-through.md), -> [../../../future/idea-application-editing.md](../../../future/idea-application-editing.md), -> [../../../architecture.md](../../../architecture.md) - -## Premise: the pipeline is already kind-agnostic - -The promise is *install an app = add a KRM document; roll out a version = edit a -field*, and it must hold for the objects platform teams actually deploy: -Flux `HelmRelease`, Flux `Kustomization`, Argo CD `Application`, KRO resources, -and plain core objects. That already works, because **nothing in the write path -carries a per-kind allowlist**: - -- **Watch selection** is by `(group, version, resource)` resolved against the - *served* API surface — any served GVR (core, CRD, or aggregated) is watchable. - A `WatchRule`/`ClusterWatchRule` names plural resources and optional - `apiGroups`; an omitted group is resolved from discovery - ([../../../../api/v1alpha3/watchrule_types.go](../../../../api/v1alpha3/watchrule_types.go), - `ResourceRule` — "resolves the resource name across all served API groups"). - There is no set of "supported kinds." -- **The in-document editor** (`internal/git/manifestedit`) edits generic - `yaml.Node` trees keyed only by object identity - (apiVersion/kind/namespace/name). It has no notion of any specific kind - ([../../../../internal/git/manifestedit/DECISION.md](../../../../internal/git/manifestedit/DECISION.md)). -- **Placement** is GVR-driven: the cold-start default is - `{namespace}/{group}/{resource}/{name}.yaml`, and sibling inference / - `spec.placement` are equally generic - ([../../../architecture.md](../../../architecture.md), *What It Writes to Git*). - -The one kind-keyed *filter* in the pipeline is a small **exclusion** list of -noisy built-ins (`pods`, `events`, `endpoints`, `endpointslices`, `leases`, -`controllerrevisions`, `flowschemas`, `jobs`, …) — a deny list, never an allow -list ([../../../../internal/watch/resource_policy.go](../../../../internal/watch/resource_policy.go)). -`HelmRelease`, `Application`, and KRO kinds are not on it, so they are -default-allowed. The remaining kind-specific branches are **write-safety** and -**override** rules, and higher-level KRM falls through all of them: - -- Sensitive types (core `Secret` plus configured types) are encrypted before - they touch the worktree. A `HelmRelease`/`Application`/KRO instance is not - sensitive by default, so it writes plaintext like any document. -- The `/scale` subresource route and the kustomize - [`images:`/`replicas:` override route](images-and-replicas-edit-through.md) - apply to specific shapes. A control-plane CR hits neither and takes the - ordinary in-place patch path. -- `status` is stripped from every mirrored object and status-only churn is - deduped, so a controller reconciling the CR does not produce spurious commits - (`internal/sanitize/marshal.go`, `internal/watch/target_watch.go`). - -This kind-agnosticism is **already e2e-proven for a generic CRD**: the -icecream-order lifecycle spec installs an arbitrary CRD and pins -create→file / update→file / delete→file-removed with `status` never committed -([../../../../test/e2e/crd_lifecycle_e2e_test.go](../../../../test/e2e/crd_lifecycle_e2e_test.go)). -What this work adds is the same proof for a **named, real higher-level -control-plane type** (Flux `HelmRelease`) plus the un-installed kinds at the -unit level. - -So a Flux `HelmRelease`, a Flux `Kustomization`, an Argo CD `Application`, a KRO -`ResourceGraphDefinition` or its generated instance, and a plain `Deployment` -all mirror and edit **identically** — they are KRM documents, and editing KRM -documents is the whole surface. Higher-level tools come into scope as KRM, not -as special cases. - -## Why prove something that already works - -This is not a code feature; it is the **proof and the promise**. It matters for -two reasons: - -1. **Regression fence.** "Kind-agnostic" is an *emergent* property of several - independent components (watch resolution, the node editor, placement, - sensitivity classification, override routing). A future special-case — a new - override heuristic, a placement tweak, a sensitivity default — could quietly - break a control-plane CR with no test noticing. A corpus + e2e make the - guarantee load-bearing so a regression announces itself. -2. **Stating the boundary correctly.** The support contract is "we edit KRM - documents; we do not inflate Helm charts." Users conflate "Flux - `HelmRelease`" with "Helm." The user docs draw the line: the `HelmRelease` - *document* is first-class; `helmCharts:` *inflation* stays permanently - refused — see - [../kustomize-support-boundary.md](../kustomize-support-boundary.md). - -## What this pins - -### Coverage split - -| Kind | live e2e (mirror + edit) | unit corpus (round-trip + convergence) | -|---|---|---| -| Flux `HelmRelease` (`helm.toolkit.fluxcd.io/v2`) | ✅ CRD in base e2e cluster | ✅ | -| Argo CD `Application` (`argoproj.io/v1alpha1`) | — (CRD not in cluster) | ✅ | -| KRO instance, e.g. `PodInfoApp` (`kro.run` group) | — (KRO is demo-only) | ✅ | -| core (`ConfigMap`/`Deployment`) | already pinned (e2e labels `inplace-edit`, `new-file-placement`, `scale`) | already pinned | - -Flux CRDs (`helmreleases.helm.toolkit.fluxcd.io`, -`kustomizations.kustomize.toolkit.fluxcd.io`) are established during standard -e2e setup ([../../../../test/e2e/Taskfile-e2e.yml](../../../../test/e2e/Taskfile-e2e.yml), -`_flux-installed`), so the live pin uses a `HelmRelease`. The Argo and KRO -controllers are not in the base cluster; their kinds are pinned at the unit -level, where the analyzer/editor need only the document bytes — no controller, -no CRD. **That is exactly the point:** the pipeline never consults the -controller, only the document, so a static document is a faithful proof of the -mirror/edit behavior. - -### Live e2e pin — `test/e2e/helmrelease_mirror_edit_e2e_test.go` - -Pins the *roll out a new version* case — a chart version on a Flux -`HelmRelease`: - -1. A `WatchRule` selecting `helmreleases` → `GitTarget`; create a `HelmRelease` - live with `spec.chart.spec.version: X` (its source need not resolve — the - operator mirrors the *object*, not Flux's reconcile result). -2. Assert the operator commits the canonical file - `{path}/{ns}/helm.toolkit.fluxcd.io/helmreleases/{name}.yaml`. -3. Seed a hand-authored comment into the committed file (semantically inert, so - the operator leaves it alone). -4. Patch live `spec.chart.spec.version` → `Y`. -5. Assert the file shows `Y`, the comment survives, and `X` is gone — an - in-place, comment-preserving edit of a control-plane CR, identical to the - `ConfigMap` case ([inplace_edit_e2e_test.go](../../../../test/e2e/inplace_edit_e2e_test.go)). - -### Unit corpus — `internal/git/manifestedit/testdata/corpus` - -Add `helmrelease.yaml`, `argocd-application.yaml`, `kro-podinfoapp.yaml`. The -two existing globbed gates then automatically cover them: - -- `TestCorpusRoundTrip_ByteIdentical` — every document re-encodes byte-for-byte - (comment and formatting fidelity). -- `TestConvergence_Corpus` — perturb the projection with a label, prove the next - reconcile settles to a byte-stable no-op (a field edit converges). - -Each document carries a hand-authored comment and a realistic version/revision -field so the gates exercise the properties that matter for these kinds. - -## What this deliberately does not do - -- **No per-kind semantics.** A `HelmRelease`'s `chartRef`, an `Application`'s - `targetRevision`, a KRO field are ordinary scalars; the operator edits the - field a human pointed it at, it does not understand chart/app semantics. -- **No chart inflation.** `helmCharts:` in a kustomization stays refused; the - operator mirrors the `HelmRelease` *document*, it never renders a chart. -- **No new controllers in e2e.** Argo and KRO stay out of the base cluster; - adding their CRDs for a live pin is future work, wanted only if the unit-level - proof ever proves insufficient. -- **No placement changes.** New-file placement for these kinds takes the generic - placement path; no kind-specific placement was added. - -## Test plan - -- **Unit:** three corpus documents flow through the two globbed gates - (round-trip byte-identical, convergence perturb-then-settle) with no new test - code — the gates already glob `testdata/corpus/*.yaml`. -- **e2e:** `HelmRelease` mirror + chart-version-bump edit, comment-preserving, - under the `manager` + `higher-level-krm` labels. -- **User docs:** [../../../installing-apps-as-krm.md](../../../installing-apps-as-krm.md) - — "adding an app is adding a KRM document," with `HelmRelease`/`Application` - examples and the chart-inflation boundary. diff --git a/docs/design/support-boundary/helm-light-support-boundary.md b/docs/design/support-boundary/helm-light-support-boundary.md index 8055ffcf0..e2493a1b4 100644 --- a/docs/design/support-boundary/helm-light-support-boundary.md +++ b/docs/design/support-boundary/helm-light-support-boundary.md @@ -302,7 +302,7 @@ locally, no plugins, no network, in-memory filesystem as the jail. How helm-light surfaces on the API (`layout.kind`, or a renderer property of the structural kind) belongs to the GitTarget API wave -([`gittarget-api-wave.md`](../gittarget-api-wave.md)), not to this boundary. +(`gittarget-api-wave.md` (removed)), not to this boundary. ## Where the arguments live diff --git a/docs/design/support-boundary/repo-discovery-and-onboarding-scan.md b/docs/design/support-boundary/repo-discovery-and-onboarding-scan.md index 11cdb960e..dabc60001 100644 --- a/docs/design/support-boundary/repo-discovery-and-onboarding-scan.md +++ b/docs/design/support-boundary/repo-discovery-and-onboarding-scan.md @@ -13,7 +13,7 @@ > [README.md](README.md), > [kustomize-support-boundary.md](kustomize-support-boundary.md), > [finished/images-and-replicas-edit-through.md](finished/images-and-replicas-edit-through.md), -> [finished/higher-level-krm-documents.md](finished/higher-level-krm-documents.md), +> [support-contract.md](support-contract.md), > [../manifest/file-agnostic-placement.md](../../layout/new-file-placement-rules.md), > [../../finished/current-manifest-support-review.md](../../spec/current-manifest-support-review.md) @@ -434,7 +434,7 @@ folder reached only through a `patches:` entry. - **`WatchRule` proposal is optional (open question).** Discovery can suggest `WatchRule`s from the kinds present in a folder, but watch selection is about *which cluster resources flow* - ([higher-level KRM documents](finished/higher-level-krm-documents.md)), + ([higher-level KRM documents](support-contract.md)), orthogonal to the Git path — it may be cleaner to stop at `GitTarget` and leave watch scoping to the user. diff --git a/docs/design/support-boundary/values-file-projection.md b/docs/design/support-boundary/values-file-projection.md index dac12ac71..739d0be75 100644 --- a/docs/design/support-boundary/values-file-projection.md +++ b/docs/design/support-boundary/values-file-projection.md @@ -11,7 +11,7 @@ > [orchestrator-knowledge-boundary.md](orchestrator-knowledge-boundary.md), > [values-content-architecture.md](values-content-architecture.md), > [acceptance-precision.md](acceptance-precision.md), -> [finished/higher-level-krm-documents.md](finished/higher-level-krm-documents.md) +> [support-contract.md](support-contract.md) [expansion-boundary-and-corpus-organisation.md](expansion-boundary-and-corpus-organisation.md) names the free-standing Helm values file as *"the single highest-leverage thing available @@ -124,7 +124,7 @@ Why this is not chart inflation, and not a widening of the boundary: - **No renderer stands between the object and Git.** The file is plain YAML we own end to end. Round-tripping it is the same comment-preserving `yaml.Node` edit the operator already performs on every manifest — the pipeline is kind-agnostic, which is precisely - what [finished/higher-level-krm-documents.md](finished/higher-level-krm-documents.md) + what [support-contract.md](support-contract.md) proved when a `HelmRelease` needed no new code. - **Fan-in must be modelled separately.** The current analyzer records only a path set; it cannot report consumers or calculate values-file fan-in. A future `InputBinding` graph can refuse a diff --git a/docs/design/target-watch-plan.md b/docs/design/target-watch-plan.md index 86441d8e3..0c5c87e00 100644 --- a/docs/design/target-watch-plan.md +++ b/docs/design/target-watch-plan.md @@ -4,8 +4,7 @@ > and the deletion the diff unlocks has not been made. > Index: [`../INDEX.md`](../INDEX.md) > Date: 2026-08-26. -> Related: [`watch-and-catalog-architecture.md`](watch-and-catalog-architecture.md), -> [`data-plane-triggering.md`](data-plane-triggering.md), +> Related: [`watch-manager-ownership.md`](watch-manager-ownership.md), > [`../spec/type-lifecycle-events-and-wobble-settling.md`](../spec/type-lifecycle-events-and-wobble-settling.md) **The short version.** A GitTarget's watch set is replaced wholesale today, so @@ -13,11 +12,6 @@ changing one rule replays all of them. This plan diffs the set instead and acts only on the collections that changed. It applies that decision by starting and canceling streams, and deliberately leaves the branch worker's queue alone. -New to this area? Read -[Watch and catalog architecture](watch-and-catalog-architecture.md) for the -model and [Data-plane triggering](data-plane-triggering.md) for the failure that -prompted this, then come back here. - ## The problem A **resource collection**, or **collection**, is one `(group, resource, @@ -324,7 +318,7 @@ would be destroying a tenant's manifests during an outage. ### What this deletes from the plan - The managed projection, and the dependency on - [Watch and catalog architecture](watch-and-catalog-architecture.md) building it + Watch and catalog architecture building it first. - The held-out set for withdrawn types, and with it the need to resolve a Git file back to a type whose mapping discovery may no longer offer. diff --git a/docs/design/watch-and-catalog-architecture.md b/docs/design/watch-and-catalog-architecture.md deleted file mode 100644 index 2b5ec7c11..000000000 --- a/docs/design/watch-and-catalog-architecture.md +++ /dev/null @@ -1,508 +0,0 @@ -# Watch & Catalog Architecture — Requirements, Current State, Target Design - -> **design**: open, not yet built. Index: [`../INDEX.md`](../INDEX.md) - -Status: **vision / proposed** — supersedes the framing in -[watchrule-wildcard-support-plan.md](../spec/type-followability.md), -[watchrule-wildcard-and-resolution-semantics.md](../spec/type-followability.md), -and [rule-set-snapshot-discovery-lag-fix.md](../spec/typeset-owns-discovery-grace.md); -those become implementation details of the model described here. - -This document is in three parts: - -1. **Requirements** — how it *should* work, including requirements not yet - written down anywhere. -2. **Current architecture** — what exists today and where it falls short. -3. **Target architecture** — one clean model and how we get there. A large - refactor is acceptable; correctness and simplicity win over diff size. - ---- - -## Part 1 — Requirements: how it should work - -## 1.0 One sentence - -The application keeps a **live, authoritative model of the cluster's API -surface**, and every `GitTarget` mirrors a **deterministic projection** of that -surface (selected by its rules) into git, reacting to surface changes -**incrementally** and **never writing an uncertain or partial view**. - -## 1.1 The catalog is the single source of truth - -There must be exactly one component that knows "what does this cluster currently -serve, and can I trust that knowledge right now." Everything else -(resolution, informers, snapshots, status) asks the catalog; nothing else talks -to discovery or guesses. - -The catalog must: - -- **Maintain last-known-good.** A `GroupVersion` that blips out during an - apiserver rollout or CRD upgrade must not vanish from the model the instant - discovery hiccups. The catalog keeps the prior entries and marks the GV - *degraded*, not *gone*. -- **Distinguish the three reasons a resource can be "absent":** - 1. *Never served* / not yet served (stable). - 2. *Degraded* — discovery temporarily can't confirm it (transient). - 3. *Withdrawn* — confirmed removed from the surface (e.g. CRD uninstalled). - These three drive completely different downstream actions, so the catalog must - label them, not collapse them to "empty". -- **Debounce withdrawal.** A GV is only promoted from *degraded* to *withdrawn* - after a confirmation window / repeated clean discovery that still omits it. - This is what "abstracts away shortly-unavailable CRDs" means concretely. -- **Expose confidence.** A consumer can always ask "is the surface authoritative - enough to make a destructive (delete) decision right now?" -- **Emit deltas.** Consumers react to *changes* (GVR added / withdrawn / degraded - / recovered), not by recomputing the world each tick. - -## 1.2 Wildcards behave like Kubernetes policy APIs (non-negotiable) - -Skipping wildcards is **not** an option. `WatchRule`/`ClusterWatchRule` selectors -must follow the admission-webhook / RBAC / audit-policy idiom that users already -know: - -- `apiGroups: ["*"]` matches all groups; `resources: ["*"]` matches all - resources; `apiVersions: ["*"]` matches all versions. -- A `*` rule is a *live* selection: when a new CRD appears that matches it, the - target starts watching it automatically; when one is withdrawn, it stops. -- The selector is a **projection of the catalog**, evaluated continuously — not a - one-time literal expansion frozen at apply time. - -Decisions to lock (see open questions): the meaning of *empty* vs `*` (proposal: -empty `apiGroups` = "resolve to the single served group, else surface an error"; -`["*"]` = "all served groups" — a strict mode and a wildcard mode, both useful), -and that *subresources* (`pods/log`, `pods/*`) are **out of scope** for the -list/watch+mirror model and must not be advertised. - -## 1.3 Incremental, relaxed reconcile - -An active `GitTarget` must respond to a newly added or removed CRD by changing -**only the part that changed**: - -- New GVR matches a target's plan → start one informer for that GVR, seed **only - that GVR's objects** into the mirror, done. No re-snapshot of the rest of the - target. -- GVR withdrawn → stop that one informer; apply the configured withdrawal policy - to only that GVR's files. - -The unit of reconciliation is the **(GitTarget, GVR) collection**, not the whole -GitTarget. Rationale (the user's, and worth stating as a requirement): because -live audit/event coverage is high-quality, a full-target re-list is rarely needed -— the only thing a new GVR genuinely needs is its *initial seed*; steady state is -maintained by events. A full re-snapshot should be a **fallback**, not the -normal path. - -## 1.4 The authoritative-write invariant (the core safety property) - -The mirror must **never** be written from a partial or uncertain view, because a -resource that is merely *unconfirmed* is indistinguishable from a *deleted* one, -and writing the difference would mirror phantom deletions into git. - -Concretely: - -- A delete is only ever written for a collection whose source-of-truth was - **authoritatively observed** (a successful list of that GVR, or a real DELETE - event for that object) **or** whose removal was **intent-driven** (the rules no - longer select it — see 1.6). -- If a GVR's list fails, its GV is degraded, or the controller lacks permission, - that collection is **left untouched and retried** — an *inability to observe* must - never be read as a deletion. -- The dividing line is **intent vs observability**: a collection leaving the desired - set because of a rule/scope/policy change is an intent-driven mutation and may - delete; a collection that is merely unobservable right now must hold. -- This invariant is per-collection, so one flaky GVR cannot corrupt an entire target. - -## 1.5 Convergence, determinism, idempotency - -- **Deterministic projection.** A target's desired mirror is a pure function of - (its rules, the authoritative catalog surface, cluster object state). Same - inputs → same files. This is what makes the plan-hash and tests trustworthy. -- **Convergence.** After any transient disruption (discovery gap, list failure, - permission blip), the system returns to the correct mirror automatically within - a bounded time, with no operator action and no spurious churn commits. -- **Idempotency.** Re-running a reconcile with unchanged inputs produces no - commit. - -## 1.6 A collection can disappear for several reasons; the cause decides the action - -This is the most important semantic to get right. "Collection removed" is **not** one -case, and collapsing them all into one policy is wrong. A collection can leave a -target's desired set because of an **intent** change or because of an -**observability** change, and the mirror must treat those oppositely: - -> **The table below is superseded on two rows; the intent-versus-observability -> split above is what survives.** -> [target-watch-plan.md](target-watch-plan.md) is authoritative for the actions. -> A withdrawn type is **swept**, not kept, because a manifest whose CRD is gone -> takes its Kustomization or Application out of `Ready` rather than sitting -> inert, and `prune.mode: Never` already exists for anyone who wants an archive. -> Deselection by **intent** is an open decision, and today it **retains**, which -> is what [`../configuration.md`](../configuration.md) promises users. Read the -> `Delete` entries as the direction this page was heading, not as the contract. - -| Cause | Class | Mirror action | -|---|---|---| -| Rule deleted / narrowed so the collection is no longer selected | intent | **Delete** the collection's managed files — required to keep the mirror a deterministic projection of the rules. | -| Namespace / scope change | intent | Old collection deselected → delete its managed files; new collection selected → seed it. | -| GitTarget rewiring (provider/branch/path change) | intent | Projection moves: delete from the old path, seed at the new one. | -| Resource policy now excludes the kind | intent/config | Likely **delete** (open question 6) — it is a config-driven deselection, not an outage. | -| Surface withdrawal (CRD uninstalled) | observability | **Keep** files (proposed) and record withdrawal in status/log. Absence is an operator surface change, not GitOps intent; deleting hundreds of files on a CRD upgrade/reinstall would be destructive and noisy. | -| Object deleted while kind still served | observability (authoritative) | Real deletion, driven by DELETE event / authoritative re-list → **delete** the file. | -| GV degraded / list failed / permission denied | observability (non-authoritative) | **Hold** the collection — never delete (see 1.4). | - -The principle: **only intent-driven removals and authoritatively-observed -deletions mutate the mirror; non-authoritative absences hold.** The reconciler -must therefore know *why* a collection left the desired set, not merely *that* it did. - -This requires a notion of **managed projection** (see 1.7): the set of files a -given collection owns, so "delete the collection's files" is well-defined and scoped. - -## 1.7 Requirements you may be missing - -Beyond the four beliefs above, a complete design needs: - -- **Deterministic file ownership (managed projection).** ~~There must be a - deterministic mapping from a collection to the set of repo paths it owns.~~ - **Dropped.** [target-watch-plan.md](target-watch-plan.md) turns the question - around: rather than asking which files a collection owns, the sweep walks the - managed documents and asks of each whether it is still wanted. That needs no - projection and keeps file knowledge on the Git side, where the acceptance gate - and the retention policy already live. - -- **RBAC / least privilege.** "Watch everything" means the controller's - ServiceAccount must `list`/`watch` every selected GVR. A wildcard implies broad - RBAC. Required: either a documented broad ClusterRole, or per-GVR - permission-denied handling that degrades that *collection* gracefully (treat as - "cannot observe" → never delete, surface in status) instead of failing the - target. -- **Scale bounds & backpressure.** One informer per selected GVR across the whole - surface is real memory + apiserver load, and a CRD burst can cause a list - storm. Required: a soft cap with observability, and coalescing of surface - bursts (the existing single-slot refresh channel is a start). -- **Startup gating.** On cold start, do not snapshot (and certainly do not delete) - until the catalog has reached initial trust — otherwise an empty early surface - could wipe the mirror. -- **Observability / status conditions.** `WatchRule`/`ClusterWatchRule`/`GitTarget` - status must show: resolved GVR count (incl. wildcard expansion), degraded - surface, withdrawn types, permission failures, and unseeded/retrying collections. -- **Version selection.** A resource served at multiple versions must mirror - exactly one (preferred) to avoid duplicate files; define wildcard-version - behavior (proposal: still collapse to preferred unless a specific version is - named). -- **Resource policy under wildcard.** The default denylist - (pods/events/leases/…) still applies to `*`; define whether users can opt - excluded kinds back in. -- **Sensitive resources.** Wildcards will select Secrets and similar. This must - compose with the existing sensitive-resource classification work - ([sensitive-resource-classification-plan.md](../finished/sensitive-resource-classification-plan.md)). -- **Migration / backward compatibility.** Changing selector semantics changes - what *existing* rules watch. Required: no silent expansion of an existing user's - scope on upgrade — a migration story or explicit opt-in. -- **Causal ordering per collection.** A GVR's initial seed must be reconciled - before its live events are flushed, so the seed isn't overwritten or - double-counted. Today this is done per-target; it must hold per-collection. - ---- - -## Part 2 — Current architecture - -## 2.1 Components - -- **`APIResourceCatalog`** ([api_resource_catalog.go](../../internal/watch/api_resource_catalog.go)) - — since the typeset-grace relocation - ([typeset-owns-discovery-grace.md](../spec/typeset-owns-discovery-grace.md)): a thin - per-scan normalizer producing a `typeset.Scan` (entries + scanned/failed - group/versions + completeness) with only mechanical state (last scan as the - change fingerprint, `generation`, readiness). Degraded-GV retention and the - removal grace for omitted GVs both live in `typeset.Registry.UpdateFromScan`. -- **`RuleGVRResolver`** (rule_gvr_resolver.go) - — maps one rule selector to concrete GVRs via the catalog, returning - `ResolveMiss` values classified by reason. **Refuses wildcards** in - `preflightMisses`. -- **`Manager`** ([manager.go](../../internal/watch/manager.go)) — owns the - reconcile loop, the active informer set (`activeInformers` keyed by - `GVR → namespace → cancel`), GVR resolution for both the watch plan and the - snapshot, and snapshot emission. -- **Event surface triggers** ([manager_catalog.go:278-303](../../internal/watch/manager_catalog.go#L278-L303)) - — informers on CRDs and APIServices that coalesce changes into - `catalogRefreshCh`. -- **`EventRouter` / per-GitTarget streams / reconcilers** — translate informer - events and control events (`RequestRepoState`, `RequestClusterState`) into git - commits, with a RECONCILING→LIVE buffering handshake per target. - -## 2.2 Control flow today - -```text -discovery / CRD trigger / 30s tick - │ - ▼ -ReconcileForRuleChange (manager.go:740) - ├─ RefreshAPIResourceCatalog → catalog updated - ├─ computeRequestedGVRs → resolve ALL rules → desired GVR set - ├─ compareGVRs → added / removed (per GVR+namespace) - ├─ snapshotTargetsNeedingDelivery → per-target plan-hash diff - ├─ stopInformer(removed) - ├─ beginReconciliationForTargets → buffer live events - ├─ startInformersForGVRs(added) → seed via informer cache - ├─ emitSnapshotForRuleChange → RequestRepoState + RequestClusterState - │ └─ whole-target cluster list, ABORTS on any blocking miss/list error - └─ completeReconciliationForTargets → flush buffered events -``` - -> **Part 2 predates the watch-first rewrite and is a survey, not a current map.** The snapshot -> path it describes — `currentRuleSetSnapshots`, `emitSnapshotForRuleChange`, `RequestClusterState`, -> the `compareGVRs` informer diff — no longer exists: object state comes from target watches, one -> stream per collection, replaying under a scoped mark-and-sweep -> ([data-plane-triggering.md](data-plane-triggering.md), [target-watch-plan.md](target-watch-plan.md)). -> The REQUIREMENTS in Part 1 and the target model in Part 3 stand; read 2.3 and 2.4 as the reasoning -> that led here rather than as a description of the tree. - -## 2.3 What already works in our favor - -- **Dynamic surface reaction exists.** The CRD/APIService triggers + reconcile - loop already start/stop informers as the surface changes. Reacting to a new - GVR is the steady state, not a new capability. -- **Per-(GVR, namespace) informer diffing exists** (`compareGVRs`, - `findAddedGVRs`/`findRemovedGVRs`). -- **Degraded-aware catalog exists** — last-known-good is already preserved. -- **Per-target snapshot isolation exists** (plan-hash selection). - -## 2.4 Where it falls short (the gaps Part 3 must close) - -1. **Two resolution call sites, two truths.** The watch-plan path - (`currentRuleSetSnapshots`, manager.go:1248/1264) and the snapshot path - (manager.go:559-594) both resolve rules independently — duplicated logic that - has already drifted (the bug that birthed - [watchrule-gvr-resolution-plan.md](../spec/type-followability.md)). -2. **Snapshots are whole-target, not per-collection.** Adding one GVR re-lists and - re-diffs the *entire* target (`emitSnapshotForRuleChange` → - `RequestClusterState` over all GVRs). Violates requirement 1.3. -3. **One failure aborts the whole target.** A single GVR list failure or blocking - miss aborts the entire target snapshot (manager.go:596-628). Safe but coarse; - at wildcard scale it means a target may rarely snapshot. Violates the - *per-collection* form of requirement 1.4. -4. **Wildcards refused** at planning (requirement 1.2 unmet) — and the CRD field - docs advertise the opposite, so a `*` rule applies cleanly and silently - watches nothing. -5. **Degraded handling is bolted on, not central.** The discovery-lag fix - reintroduces "is this authoritative?" logic at the snapshot-selection layer - because the catalog doesn't expose confidence/deltas as first-class outputs. - Many `ResolveMiss` reasons + special cases = the exception sprawl the user - wants gone. -6. **Withdrawal semantics are emergent**, not decided (requirement 1.6). - ---- - -## Part 3 — Target architecture - -The strategy: make the **catalog** a proper live model that emits deltas and -confidence, make the **watch plan** a single deterministic projection consumed in -one place, and make **reconciliation per-collection and incremental**. This -removes the duplicate resolution, the whole-target snapshots, and most of the -miss/degraded special-casing in one move. - -## 3.1 Layer 1 — `ClusterSurface` (evolve the catalog) - -Promote `APIResourceCatalog` into the authoritative, observable -`ClusterSurface`: - -- **State per GVR:** `served | degraded | withdrawn`, plus the existing - allowed/listable/scope/preferred metadata. (`served`/`degraded` already exist; - add explicit `withdrawn` with a debounce before a degraded GV is declared - withdrawn.) -- **Per-collection confidence, not a global flag.** Expose - `AuthoritativeFor(GVR) Confidence` (and a startup-trust gate), **not** a global - `Authoritative() bool`. A single boolean would either block healthy targets - because one unrelated APIService is degraded, or get bypassed and become unsafe. - Confidence travels with each planned collection so destructive decisions are gated - per collection (1.4). Degradation is observed at GV granularity (discovery fails per - GroupVersion) and projected down to the GVRs in that GV — see open question 7 - on GV-vs-GVR granularity. -- **Current surface + generation is the source of truth; deltas are only - triggers.** Expose a queryable current surface snapshot (with its `generation`) - as the authority, and a `Subscribe() <-chan SurfaceDelta` (`{GVR, Change: - Added|Withdrawn|Degraded|Recovered}`) purely as an *optimization* that tells - consumers *when* and *roughly what* to re-evaluate. Consumers must establish - correctness by **re-diffing desired collections against active collections** - against the current surface, never by replaying deltas as if they were complete - — deltas can be missed across restart, backpressure, or subscription churn. - Deltas make reconvergence cheap; the re-diff makes it correct. -- **Absorbs transience centrally.** No consumer ever sees a flapping CRD as - add/remove churn; the surface only reports `Withdrawn` after the debounce. This - is requirement 1.1, in one place, deleting the need for scattered - `CatalogUnavailable`/`DiscoveryDegraded` handling elsewhere. - -## 3.2 Layer 2 — `WatchPlan` (one resolver, projection semantics) - -A single `WatchPlan` projects rules onto the surface, used by **both** informers -and snapshots (kills gap 2.4.1): - -- `Plan(rules, surface) → map[GitTarget]TargetPlan`, where a `TargetPlan` is a - set of **collections**: `{GVR, scope, namespaces}`. -- **Wildcards are projection, not expansion-at-apply** (requirement 1.2): `*` - selects all matching `served` collections from the surface *now*; the plan is - recomputed on every surface delta, so membership tracks the cluster live. -- **Empty vs `*`**: empty = strict unique-or-error; `*` = all. One small, explicit - rule — not a pile of miss reasons. -- **Resolution errors still exist and still matter.** Strict-empty ambiguity, - not-served, and disallowed are real errors the resolver must report. The - critical rule: an unresolved or errored selector must mark its collections - **unresolved/held**, never collapse them into an *empty desired set* — otherwise - a failed resolve looks like "the user wants nothing here" and the reconciler - deletes the projection (1.4). **Resolution failure ≠ empty desired state.** - What this design removes is the *duplicated, drifting* blocking-miss handling - spread across two resolution sites and the snapshot layer — not error handling - itself, which consolidates into one place with one meaning. - -## 3.3 Layer 3 — per-collection reconciler (incremental) - -Replace whole-target snapshotting with a **collection reconciler** driven by two delta -sources — rule changes and surface deltas — joined against current plans: - -```text -SurfaceDelta{GVR, …} ─┐ (triggers only) -RuleChange{target} ─┼─► RE-DIFF desired collections (WatchPlan over current -periodic tick ─┘ surface+generation) vs active collections + managed files - │ - ▼ - per-collection actions: - collection appeared → start informer + SEED ONLY THIS GVR (collection-scoped) → additions - collection removed → classify cause (1.6): - intent (rule/scope/policy) → stop informer + DELETE the - collection's managed files - surface withdrawal → stop informer + KEEP files, - record in status - non-authoritative (degraded/ - denied/list-failed) → HOLD: do not stop as removed, - do not delete, retry - collection unchanged → nothing -``` - -Two things this diagram makes explicit, both from review: - -- **Correctness is the re-diff, not the delta.** Desired collections are recomputed - from the `WatchPlan` over the *current* surface snapshot (and its generation) - and diffed against active collections and the managed-file set. Deltas and the - periodic tick only *schedule* a re-diff; missing a delta can never cause - divergence because the next re-diff is authoritative (§3.1). -- **Repo-side scoping must match.** Seeding/diffing/deleting a collection operates - only on that collection's managed files (1.7). `RequestRepoState` must be - answerable at collection granularity — list/diff just `{target, GVR, scope, - namespaces}` — or the per-collection path silently degrades back into - whole-target snapshotting. - -Properties this gives us for free: - -- **Incremental** (requirement 1.3): a new CRD seeds exactly one GVR's objects on - the targets that select it; everything else is untouched. -- **Per-collection safety** (requirement 1.4): seeding lists only that GVR; on - failure the collection stays `unseeded/retrying` and writes nothing — no other - collection, no deletions. The "partial view = phantom deletes" failure mode - becomes structurally impossible because deletes are scoped to a collection that - just listed successfully. -- **Convergence** (requirement 1.5): a failed/degraded collection is simply retried on - the next relevant delta or tick; no target-wide churn. -- **Full re-snapshot becomes the fallback**, triggered only by explicit - resync/repair, not routine surface changes. - -Per-collection causal ordering reuses the existing RECONCILING→LIVE buffering, -scoped to the collection (requirement 1.7). - -**Seed transport — streaming-list with a per-GVR fallback.** A collection seeds by -opening a single streaming-list watch (`SendInitialEvents=true`, -`ResourceVersionMatch=NotOlderThan`, `AllowWatchBookmarks=true`) rather than a -mega-LIST: the apiserver replays each existing object as a synthetic `ADDED` and -closes the initial set with a `k8s.io/initial-events-end` bookmark, whose -resourceVersion becomes the live resume cursor -([target_watch.go](../../internal/watch/target_watch.go)). That replay window *is* -the collection's seed — exactly the collection-scoped, one-watch-per-GVR shape -this model wants, with no re-list race that could miss an in-flight delete. - -This only holds where the serving apiserver honors `SendInitialEvents` and emits -the terminating bookmark. Standard kube-apiserver and -`genericregistry.Store`-backed resources do; hand-written `rest.Storage` in -aggregated servers may not, and the hazard is **per-GVR, not per-APIService** — -one aggregated binary can serve a conformant group beside a non-conformant one. So -the fallback is keyed by GVR and is structural, not a kill switch: when the -streaming option is unsupported the collection drops to a plain **LIST + buffered -WATCH** for that one GVR ([`watchListUnsupported` → -`targetWatchListAndStream`](../../internal/watch/target_watch.go)), yielding the -same seed-then-stream result while every other collection keeps the streaming -path. There is deliberately no binary-level "disable streaming-list" switch — the -per-GVR fallback is the emergency valve. - -Operationally this is an audit per unfamiliar cluster, not a one-time task: for -each non-local `APIService` a target watches, confirm it is -`genericregistry.Store`-based, handles `SendInitialEvents` in its `Watch` (grep -its source for `SendInitialEvents` / `k8s.io/initial-events-end`), or rides the -plain-LIST fallback. The runtime fallback means a missed audit degrades one collection -gracefully instead of hanging its seed. - -## 3.4 How the existing docs fold in - -- **Discovery-lag fix** → disappears as a special case: the surface never reports - a degraded GVR as withdrawn, so a transient gap can't drop a collection or erase a - baseline. The plan-hash machinery is replaced by collection-level delta diffing. -- **Wildcard semantics doc** → becomes the spec for `WatchPlan` projection. -- **Wildcard support plan** → its phases map onto building Layers 1–3; the - "snapshot robustness" risk (its Phase 2) is resolved structurally by per-collection - seeding rather than needing a bespoke list-failure policy. - -## 3.5 Suggested build order - -Each step is independently shippable and leaves the system correct. - -1. **Unify resolution** behind one `WatchPlan` used by both informer and snapshot - paths (removes drift; no behavior change). Pure refactor, fully unit-testable. -2. **Add surface state + deltas** to the catalog (`served/degraded/withdrawn`, - `Authoritative()`, `Subscribe()`), with withdrawal debounce. Internal; no - consumer change yet. -3. **Per-collection seeding** for *added* collections (incremental seed), keeping - whole-target snapshot as the fallback path. Closes requirement 1.3 for - additions. -4. **Per-collection withdrawal policy** (requirement 1.6) for removed collections. -5. **Enable wildcards** in `WatchPlan` projection (requirement 1.2) — now safe - because Layers 1–3 handle dynamic membership, per-collection safety, and scale. -6. **Fix the field docs + admission validation** to match (and add status - conditions, RBAC guidance, scale caps). - -## 3.6 Cleanliness check (the user's steer) - -This design *removes* code as much as it adds: one resolver instead of two; one -authority for "is the surface trustworthy" instead of miss-reason handling spread -across resolver, snapshot, and selection layers; structural per-collection safety -instead of the whole-target abort + discovery-lag patch. The number of distinct -"cases" a maintainer must hold in their head drops from "many `ResolveMiss` -reasons × two resolution sites × whole-target abort rules × discovery-lag -bookkeeping" to "a surface with three states (queried as the source of truth, -with deltas as cheap triggers), projected by one resolver into collections that each -reconcile independently and classify their own removal cause." - ---- - -## Open decisions (need a human call before building) - -1. **Empty vs `*`**: adopt the strict-`[]` / wildcard-`*` split, or make `[]` - also mean "all" to match RBAC exactly? (Proposal: strict `[]`, wildcard `*`.) -2. **Surface-withdrawal default**: keep files on CRD uninstall (proposed) or - delete them? Per-GitTarget override? (This is the *observability* class in 1.6.) -3. **RBAC model for wildcard**: ship a broad ClusterRole, or require users to - grant it and degrade collections gracefully on permission-denied? -4. **Migration**: how do existing rules behave on upgrade — frozen semantics, or - opt-in to the new projection? -5. **Scale cap**: hard limit on informer count per controller, or - observe-and-warn only? -6. **Resource-policy deselection** (ref 1.6): when the default denylist newly - excludes a kind that was being mirrored, delete its managed files (treat as a - config-driven intent change) or keep them (treat as observability)? -7. **`withdrawn` granularity** (ref 3.1): is withdrawal a GV-level state, a - GVR-level state, or both? Degradation is naturally GV-level (discovery fails - per GroupVersion) while deltas are GVR-level — the model must state how the two - compose rather than switching between them. -8. **Preferred-version change**: when a resource's preferred version changes, is - that `Withdrawn(old)+Added(new)` (risks a delete+reseed and file churn), an - `Updated` delta, or an explicit **migration** action (move/rewrite files, - never delete)? Affects whether a routine apiserver upgrade churns the mirror. -9. **Rule-deselection disposition** (ref 1.6): when a rule stops selecting a - collection, delete its managed files immediately (proposed — preserves the - deterministic mirror) or leave them as *unmanaged historical* files? If the - latter, define how unmanaged files are marked and ever reclaimed. diff --git a/docs/design/watch-manager-ownership.md b/docs/design/watch-manager-ownership.md index bd9c96792..efd67206d 100644 --- a/docs/design/watch-manager-ownership.md +++ b/docs/design/watch-manager-ownership.md @@ -4,9 +4,7 @@ > "What shipped" at the end records where the implementation departed from this page and why. > Index: [`../INDEX.md`](../INDEX.md) > Date: 2026-08-27. -> Related: [`target-watch-plan.md`](target-watch-plan.md), -> [`data-plane-triggering.md`](data-plane-triggering.md), -> [`watch-and-catalog-architecture.md`](watch-and-catalog-architecture.md) +> Related: [`target-watch-plan.md`](target-watch-plan.md) **The short version.** A rule edit is applied by the controller worker that observed it, synchronously, and it re-plans every GitTarget in the process rather than the one @@ -395,7 +393,7 @@ no reason to walk every resident table because one rule changed. Its filter goes with it, and that is worth naming separately: it re-plans only targets already present in `m.targetWatches`, which means **a target whose first declare never completed is never picked up again**. That property is a trap. It is the reason a stuck -GitTarget in [`e2e-declare-investigation.md`](../finished/e2e-declare-investigation.md) could +GitTarget in `e2e-declare-investigation.md` (removed) could never recover on its own. A dirty set marked on intent rather than on success does not have it. diff --git a/docs/design/watches-labels-simplification.md b/docs/design/watches-labels-simplification.md index 0d313263a..9bc661bb2 100644 --- a/docs/design/watches-labels-simplification.md +++ b/docs/design/watches-labels-simplification.md @@ -2,7 +2,7 @@ > **Implemented.** `rules[].objectSelector` shipped with all three prune modes, including `Always` > with supported Kustomize layouts, filtered-removal attribution, and ordered producer replacement. -> **Step 2 of 2.** Step 1, the [collection rename and event-filter removal](collection-terminology-rename.md), +> **Step 2 of 2.** Step 1, the collection rename and event-filter removal, > shipped first. This page is now the design record; the > [configuration reference](../configuration.md#selecting-objects-by-label-rulesobjectselector) > describes the behavior. diff --git a/docs/design/window-field-ownership-and-apply-hold.md b/docs/design/window-field-ownership-and-apply-hold.md new file mode 100644 index 000000000..b10ddf83a --- /dev/null +++ b/docs/design/window-field-ownership-and-apply-hold.md @@ -0,0 +1,478 @@ +# Design: field-owned windows, and holding an apply for an incoming commit + +Status: **proposal, not built.** [concurrent-author-windows.md](concurrent-author-windows.md) lets +two authors' windows stay open side by side when they write different files. This page sets out two +follow-ups that would reduce conflicts further, in this order: + +1. **Part 2, the apply hold, is the next candidate**, if the base change's measurements show forced + splits by incoming commits are common enough to pay for the latency it adds. It needs only that + page's stage 1 (the reference model and the controllable clock). +2. **Part 1, field ownership, stays experimental** until the repair's interaction with withheld + applies is worked through ([the gate](#convergence-is-a-separate-operation)), and it reopens a + decided spec. + +- [Field ownership](#part-1-a-window-owns-fields): two authors editing **one object** get one commit + each, when they changed different fields. +- [The apply hold](#part-2-an-incoming-commit-holds-the-apply): an incoming Git commit that touches + nothing an open window holds delays the apply, instead of closing the windows. + +They are **two separate changes**. Neither needs the other, and each needs its own state-machine +review. The apply hold does not even need concurrent windows: it works with today's single window. +Both carry correctness risks the base proposal does not, which is why they are kept apart from it. + +## Part 1: a window owns fields + +### Why destinations are coarse + +A window keeps the **whole latest object** for each object (`openWindow.pathToEvent`), and its +commit merges that whole object into the file. That is why the base proposal closes Alice's window +when Bob writes the same file: Bob's live snapshot of `cm/x` already contains Alice's edit. +Committed first, it would put her edit in Git under his name, and her later commit would write her +older copy back. + +Server-side apply solves the same problem inside the API server: each field manager owns a set of +fields, an apply sets only the fields its manager owns, and two managers conflict only on a field +both claim. A window can be a field manager over Git. + +### It reopens a decided spec + +[Field ownership: the API wins](../spec/manifestedit-field-ownership-spike.md) is decided, and it +lists partial ownership under "Explicitly rejected (do not build)". It says a comeback needs its own +proposal with a worked use case that bi-directional GitOps cannot serve. Part 1 is that proposal, and +it has to be accepted against the spec before anything is built: + +- **The use case:** two people editing one object at the same time get one commit each, under their + own names. Bi-directional GitOps cannot serve it, because it is about the history Git records. + The state Git ends with is the same either way. +- **What the spec rejected, and how part 1 differs.** The spec rejected a **declared** subset of + owned fields (a configured, permanent list) and ownership **driven by `managedFields`**. Part 1 has + neither: ownership lives only as long as one open window, is computed from the change itself, and + has no configuration surface. +- **What part 1 relaxes:** "documents are whole-object truth" no longer holds for **each commit**. + It still holds for the tree once the windows on an object have closed and the + [repair](#convergence-is-a-separate-operation) has run. That is the guarantee the spec's + convergence tests check, and part 1 keeps it, one repair later. + +If the spec's decision stands, part 1 is not built, and the base proposal's footprint rule stays the +answer for two authors on one object. + +### The rule + +- **A window owns the fields its writes changed**, per object, next to the latest event, until a + write of its own sets one back ([a field set back is released](#a-field-set-back-is-released)). + A write's changes come from comparing its object with the object's **baseline**: the last + live object the worker saw for it, across every window (see + [the baseline lifecycle](#the-baseline-lifecycle)). It must not be compared with Git, which lacks + the other windows' unpublished edits, so the comparison would credit them to this write. + `managedFields` does not provide the set either. Its managers are clients (`kubectl`, + `gitops-api`) rather than people, and it records who owns a field, which says nothing about what + one write changed. +- **Every commit of a field-owned window sets only the fields that window owns**, and leaves every + other field of the document as Git holds it, including when it is the last window to close on that + object. The values are the ones the window's own writes set + ([what a window stores](#what-a-window-stores)), not read back from its latest event. The editing + primitive exists: + `manifestedit.PartialDesired` and `manifestedit.OwnsAssignedPaths` + (`internal/git/manifestedit/fieldpatch.go`) do this for subresource audit patches, and keep the + drift guard and the formatting preservation of the whole-object merge. +- **`path_conflict` becomes `field_conflict` for field-owned writes.** A write closes another open + window only when it changes a field that window owns, or a parent or child of one. A removed field + counts as owned. + +```mermaid +sequenceDiagram + participant A as Alice + participant B as Bob + participant W as Branch worker + Note over W: live and Git: cm/x data.a = 0, data.b = 0 + A->>W: cm/x data.a = 1 + Note over W: baseline a = 0, b = 0, so Alice's write changed data.a + Note over W: Alice's window owns cm/x data.a + Note over W: retained event: a = 1, b = 0, new baseline + A->>W: CommitRequest "feat(shop): raise a" + Note over W: the request attaches, Alice's window now closes on its timers + B->>W: cm/x data.b = 2 + Note over W: baseline a = 1, b = 0, so Bob's write changed data.b + Note over W: no field in common with Alice's window + Note over W: Bob's window opens and owns cm/x data.b + Note over W: Bob's window closes (idle_timeout) + Note over W: commit 1 by Bob sets data.b = 2 only + Note over W: Git: data.a = 0, data.b = 2 + Note over W: Alice's window closes on the request's timers + Note over W: commit 2 by Alice sets data.a = 1 only, her message + Note over W: Git: data.a = 1, data.b = 2, equal to live +``` + +A last whole-object write would be wrong here. Alice's retained event is her snapshot, +`data.a = 1, data.b = 0`: written whole after Bob's commit, it restores `data.b = 0` and undoes his +work. Writing the latest live snapshot instead fixes that case but overwrites any field an incoming +Git commit changed meanwhile. So no field-owned commit ever writes a field it does not own. + +### What a window stores + +A window keeps no log of its writes. For each object it holds: + +| Kept | What it is | Used for | +| --- | --- | --- | +| **The latest event** | the whole object at the window's last write, as today (`openWindow.pathToEvent`) | the cases that [stay whole-object](#what-stays-whole-object) | +| **The owned fields** | for each field the window owns, its **start value** (the baseline's value when the window took the field) and its **own value** (what the window's latest write to that field set; a removal counts as a value) | the commit, `field_conflict`, and [releasing a field set back](#a-field-set-back-is-released) | + +A log of every write would add nothing. While a window owns a field, `field_conflict` keeps every +other write off it, so the window's last value for each field is all its commit needs. A window +commit records where the window ended, never the steps, as it does today. A log would only matter +for merging two windows' edits to one field, and part 1 never does that: it closes one of them. + +The own value is kept per field, rather than read back from the latest event, because the latest +event is the whole live object at the window's last write, and it can hold a value nobody in the +window chose: + +1. Alice sets `data.a = 1`. Her window owns `data.a`, start value `0`, own value `1`. +2. A late apply sets the live `data.a` back to `0`. The apply window withholds it, and the observed + baseline becomes `data.a = 0`. +3. Alice edits `data.b`. Her latest event holds `data.a = 0`, but compared with the baseline her + write changed only `data.b`, so `data.a` keeps its own value `1`. +4. Her commit sets `data.a = 1` and `data.b`. Read from the latest event, it would have set + `data.a = 0` under her name. + +The commit carries the owned fields as a field patch: an `Event` already holds either a whole +`Object` or a `FieldPatch`, and `manifestedit.PartialDesired` builds the patch from the +assignments. So the pending write holds the window's **change**, not the whole object, which is what +lets a replay onto a newer tip leave the newer tip's fields alone +([replays stop reverting Git](#side-effect-replays-stop-reverting-git)). + +### A field set back is released + +A window owns a field for the change it made, not for having touched it. When a window's write sets +an owned field back to its start value, the window **releases** the field: it no longer owns it, its +commit does not write it, and another author's later write to it is not a `field_conflict`. + +```mermaid +sequenceDiagram + participant A as Alice + participant B as Bob + participant W as Branch worker + Note over W: live and Git: cm/x data.a = 0, data.c = 0 + A->>W: cm/x data.a = 1 + Note over W: Alice's window owns data.a (start 0, own 1) + A->>W: cm/x data.c = 3 + Note over W: Alice's window owns data.a and data.c + A->>W: cm/x data.a = 0 + Note over W: data.a is back at its start value + Note over W: Alice's window releases data.a and keeps data.c + B->>W: cm/x data.a = 5 + Note over W: no field in common with Alice's window + Note over W: Bob's window opens and owns data.a (start 0, own 5) + Note over W: Bob's window closes (idle_timeout) + Note over W: commit 1 by Bob sets data.a = 5 only + Note over W: Alice's window closes + Note over W: commit 2 by Alice sets data.c = 3 only + Note over W: Git: data.a = 5, data.c = 3, equal to live +``` + +If a window owned every field it ever changed, Bob's write would close Alice's window +(`field_conflict`): a forced split for a change that no longer exists, and her commit would set +`data.a = 0`, a value Git already holds. + +Releasing is safe because it leaves the field exactly as if the window had never written it. While +Alice owned `data.a`, no other write changed it, so the start value is still the value the field +held before her window, and her commit writing nothing for it keeps that value. + +Two rules keep the release honest: + +- **Only the window's own write releases a field**, and only a field that write changed compared + with the baseline. Comparing the latest event with the start value instead would read the withheld + apply in [the example above](#what-a-window-stores) as Alice undoing her edit. +- **The release is per owned path.** Adding a label (`kubectl label cm/x tmp=1`, start value + absent) and removing it again (`kubectl label cm/x tmp-`) releases `metadata.labels.tmp`. An + object whose owned fields are all released leaves the window; a window left with nothing commits + nothing, and a save attached to it settles as a save with nothing to commit does today + (`whenNothingToCommit`). + +A whole-object entry (a create, a delete, a write with no baseline) has no start value per field and +is never released. + +### Convergence is a separate operation + +A whole-object write today also repairs whatever else differs between the live object and Git (a +field an earlier write left wrong, a stray `resourceVersion`). Field-owned commits do not, so Git can +drift from the live object in fields nobody owns. + +That repair becomes an operation of its own, with an explicit attribution (the operator's identity +and a message that says it is a repair), so no person is credited with it. It is a pending write like +any other, so it is ordered behind the commits before it. + +**The repair must not compare Git with the live object.** The live object is what the cluster runs, +and the cluster can run a value nobody wants: a late apply the apply window withheld leaves its +reverted value live until a `Repair` puts the edit back. Copying that into Git undoes the edit: + +1. Alice sets `data.a = 1`; her window owns `data.a`. +2. A late apply writes `data.a = 0`. The apply window withholds it (it is not mirrored), and a + `Repair` is pending for the applier. +3. Alice's window closes: her commit sets `data.a = 1`. +4. A repair that compares Git (`1`) with the live object (`0`) commits `data.a = 0`, and Alice's + edit is gone from Git as well as from the cluster. + +So the worker keeps two views of every object it tracks, and they serve different purposes: + +| View | What it is | Used for | +| --- | --- | --- | +| **Observed** | the last live object the worker saw, withheld writes included | the baseline: deciding which fields a write changed | +| **Desired** | the observed object, with every field a withheld write reverted set back to the last accepted value | what Git should hold, and what the repair compares Git with | + +The repair's policy, against the desired view: + +- it does not run for an object while its applier has a `Repair` pending that covers the object. + The `Repair` puts the cluster back first, and then the two views agree; +- a field an incoming Git commit changed since the applier's last applied revision is left alone: + Git's change is pending an apply, and the repair must not revert it; +- every other field that differs takes the desired value, as the API-wins rule says today. + +**This is a gate on part 1:** the two views and this policy have to be accepted, and covered by the +reference model, before field ownership is built. + +### The baseline lifecycle + +The field set is only as right as the baseline it was compared with. Each kind of event has to say +what it does to the baseline: + +| Event | Field set | Baseline afterwards | +| --- | --- | --- | +| Accepted write, baseline known | the fields that differ from the baseline | the event's object | +| Accepted write, no baseline (first sight, after a restart) | none: the write owns the whole object | the event's object | +| No-op write | empty, no window entry | unchanged | +| Write after a watch restart or relist of its type | none: whole object (two people's changes may have arrived as one) | the event's object | +| Recreate (same name, new UID) | none: whole object | the new object | +| Delete | whole object | cleared | +| Applier write withheld by the apply window | none, it is not mirrored | observed: the event's object, so the next author's write is compared with what the cluster runs. Desired: unchanged for the fields the write reverted | + +`resourceVersion` cannot detect a missed event: it is opaque, so the worker cannot tell whether one +value directly follows another. Restarts and relists are what the worker can see, so they reset. + +### What stays whole-object + +Field ownership is an optimization over the base proposal's footprint rule, and every case it cannot +decide falls back to that rule: + +- creates, deletes, recreates, and writes with no baseline (the table above); +- an encrypted document (field patches refuse them), so SOPS Secrets gain nothing; +- a list without a schema that says how it merges: until the worker reads schemas, a list is one + field, so two edits to different containers conflict; +- a field whose Git destination is not the object's own document (a `configMapGenerator` literal in + `kustomization.yaml`, next to the literals of other ConfigMaps). + +### Side effect: replays stop reverting Git + +When both sides changed one object, the API value wins today for the **whole** object +(`internal/git/apply_window.go`). A pending whole-object write replayed onto a newer tip sets every +field back to the live value, including the ones the newer commit changed. A field-owned write +replays onto that tip without touching the fields it does not own. A Git edit to `spec.replicas` +and an API edit to a label of the same Deployment both survive; today one of them loses. This holds +only because no field-owned commit writes an unowned field. + +## Part 2: an incoming commit holds the apply + +### Why windows close for an incoming commit today + +Two close triggers fire when commits arrive in Git: + +- **`foreign_commit`** (`closeWindowForPushHint`, `internal/git/applier_claim.go`) is how a busy + branch **learns** about a push. A push notice is a hint, and with a window open the worker reads + nothing from the remote first (the refresher skips a busy branch, `noteBusyPushNotice`). Closing + the window and publishing is what discovers the move: the push is rejected, the worker fetches, + replays its writes onto the foreign commit, and an apply becomes due. Without the close, the push + would be found only when the window closed on its own timers. A window carrying a + `CommitRequest` is left open. +- **`apply_due`** (`branchWorkerEventLoop.flushed`) is the flush + ([S1 in applier-echoes.md](applier-echoes.md#s1-flush-then-apply)). It closes the window, takes + the flush mark, and publishes every write up to the mark, so the apply includes them. An edit + decided after the mark does not hold the apply back: the applier's apply window records it, and an + apply that overwrites it is withheld and repaired. The flush does not begin while a request is + registered (`writeBusy`). + +No edit is lost by either today. What the incoming commit costs is a **forced split**: every open +window is closed by a commit the windows had nothing to do with, so a person's change lands in two +commits. That split is the only thing part 2 can remove. + +### Part 2 brings back a bounded wait + +The flush replaced the old "quiet branch" rule, under which an apply went out only when no window +was open and no write was pending. That rule starved: a person who kept editing held a Git-side +change back for as long as they typed. Holding the apply until windows close is the same wait. +Part 2 is acceptable only with [its own deadline](#the-hold-has-its-own-deadline), and the trade +has to be argued with measurements: forced splits removed against apply latency added. + +### The worker has to read the incoming commit first + +The overlap check compares the incoming commit with the open windows **while they are still open**. +Today a busy branch does not have that commit until it publishes (see `foreign_commit` above), and +that is deliberate: [push-notification-and-reconcile-trigger.md, section 3.3](push-notification-and-reconcile-trigger.md#33-learning-that-somebody-else-pushed) +makes a delivery a hint that is never trusted, and reading nothing first is what makes late +deliveries and rollbacks come out right without bookkeeping. + +So part 2 needs a **fetch on a busy branch**: on a push notice with windows open, the worker reads +the advertisement and fetches the new tip into its object store, without touching the worktree, the +pending writes or the windows. The tip it fetched is the one the check uses; the publication that +follows still finds the move by compare-and-swap, as today, so the hint stays a hint. It costs one +fetch per notice on a busy branch, counted in `git_fetches_total` under a reason of its own. + +### The rule + +| Incoming commit vs. open windows | Action | +| --- | --- | +| Disjoint: no object in common (or, with part 1, no field in common) | **Hold the apply.** The windows keep collecting; the apply runs when they have closed, or when the hold's deadline passes, through today's flush. | +| Overlapping, not provably disjoint, or not fetched | Today's behavior: the windows close now. | + +The hold is possible because the operator decides when an applier applies: the `FluxClaim` keeps +its Kustomization suspended and the `ArgoCDClaim` keeps automated sync off. + +### The hold has its own deadline + +Window timers cannot bound the hold. A window's `max_duration` is not fixed: attaching a +`CommitRequest` replaces the window's timers with the request's, running from the attach +(`attachToOpenWindow`, `internal/git/commit_request_attach_loop.go`). A hold that waited for the +windows' own deadlines could be extended by every attach. + +So the hold has an **absolute deadline, measured from when it began**. It ends at the first of: + +- every window open when the hold began has closed; +- the deadline passes. + +Either way it ends by entering today's flush (`flushed`), which then sends the apply, so the +flush mark and the late-apply bookkeeping (withheld writes, repairs) see the same sequence as today. +A window opened during the hold does not extend it; the flush closes it as today. + +The hold's deadline bounds **only the delay the hold adds**. Two existing delays stay on top of it, +unchanged, and must not be described as part of the hold's bound: + +- **save protection:** the flush still does not begin while a request is registered (`writeBusy`), + so a save in progress delays the apply exactly as it does today; +- **publication retries:** a flush that cannot publish retries as it does today. + +A request that attaches during the hold does not move the hold's deadline. Whether the apply waits +for that request is decided by `writeBusy`, as it is today. + +### Computing the overlap + +**In process (first):** compare the **rendered** tree at the fetched tip with the rendered tree at +the applier's **last applied revision**, map the changed documents to objects (and, with part 1, to +fields), and intersect with the open windows' footprints or fields. Rendered, because a change to a +`kustomization.yaml`, a base, a component, a generator or a values file changes objects whose own +files the commit did not touch. Against the last applied revision, because after a withheld or +failed apply, or several pushes, the applier applies everything since then. A tip that does not +descend from the last applied revision (a force push) counts as overlapping. No API calls and no +extra permissions. + +**As a server-side apply dry-run (later, only if the in-process check is too coarse on lists):** +apply the changed objects with `dryRun=All`, as the applier's field manager, without force. The API +server's conflict errors name the fields and their current managers, using the type's own schema. +The costs are problems H8 to H10. + +## How to test it + +Both parts extend the reference model of +[concurrent-author-windows.md](concurrent-author-windows.md#how-to-test-it-a-reference-model), which +must already exist. + +- **Part 1** adds field sets, the baseline table, the observed and desired views, and the repair + operation to the model. On every commit: a field-owned commit changes only fields its window owns, + every changed field is credited to the window that owns it, an older accepted write to a field + never lands after a newer one to the same field, and a field a window set back to its start value + is released: no commit of that window writes it, and Git holds the value from before the window. + A field-owned commit writes the window's own values, never a value from its latest event that its + writes did not set. The repair is checked + as its own commit, with its own attribution: it never writes a value only a withheld write + produced, never runs while a covering `Repair` is pending, and never touches a field an unapplied + incoming commit changed. +- **Part 2** adds busy-branch fetches, incoming commits with rendered diffs, and the hold's deadline + to the model. On every apply the worker requests, the checks are the flush's own guarantee: + **every edit decided up to the flush mark is published** in the commit the apply names, and + **every edit decided after it, to an object the applier manages, is recorded** by the apply + window, so an apply that overwrites it is withheld and repaired. A check that demanded no + unpublished edit at all would forbid what keeps continuous editing from starving the apply. On top + of that: no apply was delayed past the hold's deadline for a reason other than `writeBusy` or a + publication retry. + +## Potential problems + +Most failures here are **silent**: Git gets a plausible commit with the wrong content or the wrong +author, and nothing reports an error. + +### Part 1: field ownership + +- **F1. A wrong baseline gives the wrong field set.** *Mitigation:* the baseline table; every case + without a trusted baseline owns the whole object. +- **F2. Lists without a schema.** For Deployments, where most edits are in lists, part 1 gains little + until the worker reads schemas (`x-kubernetes-list-type` and merge keys from `/openapi/v3`, with + structured-merge-diff). That adds discovery traffic and schema version skew. *Mitigation:* ship + with lists as one field; add schemas only if measurements show it pays. +- **F3. Git shows states the cluster never had.** With Bob's window closing first, Git's first commit + holds Bob's `data.b` next to Alice's **old** `data.a`, a combination that never ran. Fields that + only work together (an image and the environment it reads, two ConfigMap keys read as a pair) + make that commit a broken state for anyone who checks it out, bisects onto it, or reverts the + other commit. *Mitigation:* document it; an open question asks whether a `GitTarget` can opt out. +- **F4. The repair is noise, or reverts.** A repair after every close clutters history. A repair that + compares with the live object copies a withheld apply's reverted value into Git and undoes an + author's committed edit; one that ignores the last applied revision reverts an unapplied Git edit. + *Mitigation:* the repair compares with the desired view, waits while a covering `Repair` is + pending, leaves fields changed since the last applied revision alone, and runs only when + something differs. This is the gate on part 1. +- **F5. Removals and parent fields.** Field patches today only assign values; a removal needs a + "remove" assignment. Removing a map conflicts with a key another window added inside it. + *Mitigation:* a removal owns its path, and owning a path conflicts with owning any parent or child. +- **F6. Sanitized fields create phantom ownership.** Compared before sanitizing, `managedFields`, + `resourceVersion` or a stripped tool label would show up as changed fields. *Mitigation:* compare + sanitized objects only. + +### Part 2: the apply hold + +- **H1. The overlap is judged on files.** *Mitigation:* compare rendered trees; anything the renderer + cannot render counts as overlapping. +- **H2. The overlap is judged against the wrong revision.** *Mitigation:* the applier's last applied + revision, and a non-descendant tip counts as overlapping. +- **H3. A bound that is not a bound.** Described as "the windows' `max_duration`", the hold's delay can + grow with every attach. *Mitigation:* the hold's own absolute deadline, and status and docs that + name save protection and publication retries as separate delays. +- **H4. Visible apply latency.** Someone who pushed to Git waits for the cluster with no sign of why. + *Mitigation:* a held reason on the applier claim's status, and a histogram of held time. +- **H5. A window counted against the wrong applier.** Windows are per `GitTarget`; applies are per + applier and its folders. *Mitigation:* a window counts for every applier whose folders contain one + of its footprint's files. +- **H6. The busy-branch fetch changes a deliberate rule.** Reading the remote before publishing is + what section 3.3 of the push-notification design avoided. A fetched tip that is already stale, or + a notice for the worker's own push, must not change what the publication does. *Mitigation:* the + fetched tip feeds only the overlap check; publication still finds the move by compare-and-swap, + and a notice naming the tip the worker last observed fetches nothing (as `noteBusyPushNotice` + already decides). +- **H7. The starvation S1 removed comes back.** *Mitigation:* the absolute deadline, the rule that + a window opened during the hold does not extend it, and the held-time histogram to show the cost. +- **H8. Dry-run permissions.** A dry-run apply needs `patch` on every kind it applies; the operator's + identity on a source cluster may only read. +- **H9. Dry-run admission.** Dry-run requests go through admission, and a webhook that does not + declare `sideEffects: None` rejects them. +- **H10. Ownership is not an open window.** A dry-run conflict means another manager owns the field, + which can be an edit from months ago. Flux applies with force, and Argo CD applies client-side + unless the Application syncs with `ServerSideApply=true`, so a conflict-free dry-run does not model + what Argo CD does. Every conflict has to be checked against the open windows anyway. + +## Alternatives + +1. **A text merge of the files (`git merge-file`) instead of field ownership.** Line-based, so it + knows nothing about YAML structure, list keys or the formatting `manifestedit` preserves. A + conflict-free text merge can still produce a wrong object. +2. **A log of every write instead of the owned fields.** Adds nothing while `field_conflict` keeps + one window per field ([what a window stores](#what-a-window-stores)); it would only be needed to + merge two windows' edits to one field. +3. **A last whole-object write instead of a separate repair.** Undoes another author's work or an + unapplied Git edit, as [the rule](#the-rule) shows. +4. **Holding the apply until the windows' own timers end.** Unbounded under attaches (H3). + +## Open questions + +1. Can a `GitTarget` opt out of field-level splitting, for repositories where fields of one object + only make sense together (F3)? +2. What is the hold's deadline: a fixed default, or a `GitTarget` setting next to the commit-window + timers? +3. Is the repair commit's author the operator, or the last author whose window closed on the object, + with a trailer that marks it as a repair? +4. Is a fetch per push notice on a busy branch (H6) acceptable, or should the overlap check run only + for notices that arrive while a window has collected for longer than some threshold? diff --git a/docs/facts/crd-upgrade-strategies.md b/docs/facts/crd-upgrade-strategies.md index e33cfb9dc..32531719f 100644 --- a/docs/facts/crd-upgrade-strategies.md +++ b/docs/facts/crd-upgrade-strategies.md @@ -3,7 +3,7 @@ > **facts** — durable reference. Index: [`../INDEX.md`](../INDEX.md) > > Written while shipping the `GitTarget` API wave -> ([`../design/gittarget-api-wave.md`](../design/gittarget-api-wave.md)), which removed or renamed +> (`../design/gittarget-api-wave.md` (removed)), which removed or renamed > five fields across three kinds and tried both of the strategies below on the way. Every claim > marked **measured** was run, not read: the envtests are named, and two were run against a live > k3d cluster. Claims marked *inferred* are reasoning from documented behaviour and are labelled so @@ -225,9 +225,6 @@ That reasoning has an expiry date, and it is worth writing down so the reversal ## See also -- [`../design/gittarget-api-wave.md`](../design/gittarget-api-wave.md) — the wave these facts came - from, including the version-strategy argument for keeping `v1alpha4` as a *sweep* rather than a - carrier. - [`../UPGRADING.md`](../UPGRADING.md) — the migration guide this page's ordering constraint shaped. - [`../spec/where-validation-lives.md`](../spec/where-validation-lives.md) — the repo-wide rule for which rung a check belongs on, and why a webhook is not one of them. diff --git a/docs/finished/config-plane-split.md b/docs/finished/config-plane-split.md index f64b60858..95c1b01c8 100644 --- a/docs/finished/config-plane-split.md +++ b/docs/finished/config-plane-split.md @@ -663,7 +663,7 @@ No `observedDestination` / `retargetingTo` — those belong to #6. - `spec.kubeConfig` is a new optional field. Existing installs are unaffected: absent → local cluster → today's behavior, byte for byte. - This design intentionally closed the older `SourceCluster` CRD proposal. The later - [`ClusterProvider` attribution design record](multi-cluster-author-attribution.md) explains why + `ClusterProvider` attribution design record explains why the source connection was subsequently given a cluster-scoped home. ## Future shape (designed-in, not built) @@ -932,7 +932,7 @@ YAML applied to the cluster. The source-cluster corner is its own gated leg 4. **Least-privilege ClusterRole shape on the remote.** Ship a documented broad read ClusterRole, or require the operator to grant per-type read and degrade unobservable cells gracefully (ties into - [`watch-and-catalog-architecture.md`](../design/watch-and-catalog-architecture.md) §1.7)? + `watch-and-catalog-architecture.md` (removed) §1.7)? 5. **`GitProviderReady` projection trigger and `Ready` gating.** Wire a `Watches(&GitProvider{})` so a provider going un-ready promptly re-reconciles its GitTargets (best, but adds an edge to the set that diff --git a/docs/finished/e2e-aggregated-apiserver-test-design.md b/docs/finished/e2e-aggregated-apiserver-test-design.md deleted file mode 100644 index f0594e7ce..000000000 --- a/docs/finished/e2e-aggregated-apiserver-test-design.md +++ /dev/null @@ -1,756 +0,0 @@ -# E2E Test Design: Aggregated API Server (Extension API Server) - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -> Last updated: April 2026 - -## Motivation - -GitOps Reverser currently has e2e coverage for built-in resources (ConfigMaps, Secrets, Deployments, -etc.) and CRD-backed resources (IceCreamOrder). However, Kubernetes supports a third category of -resources: those served by **aggregated API servers** (extension API servers), registered via -`APIService` objects. - -These resources participate in the Kubernetes API surface differently from CRDs: - -- They are served by a separate process (not etcd-backed through the core API server) -- The core API server proxies requests to the extension server -- Audit events are generated by the core API server's proxy layer, not the extension server -- Watch streams are proxied through the aggregation layer -- Status subresources may behave differently depending on the extension server implementation - -We need to verify that the full gitops-reverser pipeline — audit ingestion, watch/informer -reconcile, sanitization, file path generation, and git commit — works correctly with aggregated API -resources. - -### What the existing docs already say - -[kubernetes-api-discovery.md](../facts/kubernetes-api-discovery.md) already states that aggregated API -resources are theoretically supported: "yes, you can watch resources served through `APIService`; -CRD is not required." However, this has **never been tested end-to-end**. This design closes that -gap. - -The concrete bootstrap and wiring plan for the proxy-backed environment lives in -[e2e-aggregated-apiserver-proxy-hookup-plan.md](e2e-aggregated-apiserver-test-design.md). -This document focuses on **what the tests must prove** once that environment exists. - -### What we need to prove about gitops-reverser vs. what is Kubernetes behavior - -This document is careful to separate concerns: - -- **Gitops-reverser behavior** (our code): audit ingestion, rule matching, sanitization, file path - generation, watch/informer reconcile, content dedup, git commit. These are what the tests must - assert on. -- **Kubernetes aggregation mechanics** (not our code): audit event generation for proxied requests, - watch stream proxying, APIService health checks. We depend on these but do not test them directly. - The tests will *exercise* them but should not assert on Kubernetes internals. - ---- - -## Scope - -This test suite validates: - -1. **Audit webhook behavior** — do audit events for aggregated API resources arrive with the same - structure (verb, objectRef, responseObject) as CRD/built-in resources? -2. **Watch/informer behavior** — can informers list+watch aggregated API resources for - snapshot/reconcile? -3. **Status subresource handling** — does a status update on an aggregated resource get correctly - deduplicated (no spurious commit)? -4. **Non-standard resource names** — do resource instance names with underscores and dots (permitted - by the sample-apiserver's path-segment validation) survive the full pipeline? -5. **API discovery integration** — does `FilterDiscoverableGVRs()` correctly discover and accept - aggregated API resources? -6. **Sanitizer field preservation** — does the sanitizer correctly handle aggregated API resources, - including the known limitation for non-standard top-level fields? -7. **Reconcile/snapshot** — does an initial snapshot correctly capture existing aggregated API - resources? - -Out of scope: -- Extension API server high availability or failure modes -- Performance comparison between CRD and aggregated API watches -- mTLS between the core API server and extension server (handled by Kubernetes, not our code) - ---- - -## Test Infrastructure: Which Extension API Server to Use - -### Decision: `kubernetes/sample-apiserver` - -The Kubernetes project maintains -[sample-apiserver](https://github.com/kubernetes/sample-apiserver) — a minimal extension API -server that serves resources in the `wardle.example.com` API group. This is the canonical example -used by Kubernetes itself for testing aggregated APIs. - -**Resources it serves:** - -| Version | Resource | Kind | Scope | Notes | -|---|---|---|---|---| -| `v1alpha1` | `flunders` | Flunder | Namespaced | Has `spec` and `status` fields | -| `v1alpha1` | `fischers` | Fischer | Cluster | Has `disallowedFlunders` as a **top-level** field (not under `spec`) | -| `v1beta1` | `flunders` | Flunder | Namespaced | `spec` uses `flunderReference`/`fischerReference` instead of v1alpha1's `reference` | - -Note: `fischers` is only served at `v1alpha1`, not `v1beta1`. - -**Why this is the right choice:** -- Official Kubernetes project, actively maintained alongside k8s releases -- Minimal footprint: single binary, simple schema -- Has both namespaced and cluster-scoped resources -- Has a `status` subresource on Flunder -- Supports all CRUD verbs plus `list` and `watch` -- Used in Kubernetes' own e2e tests -- Pre-built container images at `registry.k8s.io/e2e-test-images/sample-apiserver` -- **The upstream `artifacts/example/` directory provides battle-tested deployment manifests** - including all required RBAC - -**Alternatives considered and rejected:** -- **metrics-server**: read-only (no create/update/delete), unsuitable for mutating audit tests -- **Custom extension server**: duplicates sample-apiserver's purpose, adds maintenance burden -- **HTTP mock behind APIService**: too fragile, must implement full K8s API contract - -### Name validation behavior (verified from source) - -The sample-apiserver uses `genericregistry.Store` which defaults to `path.ValidatePathSegmentName` -for instance name validation. This rejects only `/`, `%`, and `..` — it **does allow** underscores, -dots, and other characters that would be invalid in DNS labels. This is confirmed by reading the -flunder strategy -and REST store. - -This means S7 (non-standard names with underscores and dots) is viable without modifying the -sample-apiserver. - ---- - -## Setup: Providing `sample-apiserver` in the e2e environment - -The concrete installation mechanism is intentionally separated from this test-design document. - -- This document defines the required environment capabilities and the assertions the tests should - make. -- The active bootstrap ownership and rollout sequence are defined in - [e2e-aggregated-apiserver-proxy-hookup-plan.md](e2e-aggregated-apiserver-test-design.md). - -At the time of writing, the preferred direction is to have Flux bootstrap the aggregated API stack -through the `apiservice-audit-proxy` chart rather than keep a permanently hand-maintained parallel -manifest bundle in this repo. - -### TLS approach: `insecureSkipTLSVerify` - -The upstream `artifacts/example/apiservice.yaml` uses `insecureSkipTLSVerify: true` on the -APIService. This skips TLS verification between the core API server and the extension server. - -**We use this approach for e2e.** It avoids: -- Creating a cert-manager Certificate (the repo has a namespaced - `gitops-reverser-selfsigned-issuer` Issuer, not a ClusterIssuer, and cert-manager's - `cainjector` annotation wiring for APIService caBundle injection is fiddly) -- Injecting CA bundles into the APIService -- Debugging cert-related startup failures - -The sample-apiserver generates its own self-signed serving cert at startup when no cert files are -provided. Combined with `insecureSkipTLSVerify`, this just works. - -This matches how the sample-apiserver is deployed in Kubernetes' own test infrastructure. - -### Deployment artifacts - -Add the manifests to the existing `test/e2e/setup/manifests/` directory, following the same -pattern used for Prometheus (plain manifests applied via kustomize). This keeps all non-Helm -e2e infrastructure in one place: - -``` -test/e2e/setup/manifests/ -├── kustomization.yaml # add sample-apiserver/ resources here -├── prometheus/ # existing -└── sample-apiserver/ - ├── namespace.yaml # wardle namespace - ├── sa.yaml # ServiceAccount: apiserver - ├── deployment.yaml # wardle-server + etcd sidecar - ├── service.yaml # ClusterIP service on port 443 - ├── apiservice.yaml # v1alpha1.wardle.example.com with insecureSkipTLSVerify - ├── auth-delegator.yaml # ClusterRoleBinding → system:auth-delegator - ├── auth-reader.yaml # RoleBinding → extension-apiserver-authentication-reader - ├── rbac.yaml # ClusterRole for namespace/webhook/flowcontrol access - └── rbac-bind.yaml # ClusterRoleBinding for above -``` - -The existing `test/e2e/setup/manifests/kustomization.yaml` gets the new resources added: - -```yaml -resources: - - prometheus/instance.yaml - - prometheus/prometheus-rbac.yaml - - prometheus/servicemonitor.yaml - - sample-apiserver/namespace.yaml - - sample-apiserver/sa.yaml - - sample-apiserver/deployment.yaml - - sample-apiserver/service.yaml - - sample-apiserver/apiservice.yaml - - sample-apiserver/auth-delegator.yaml - - sample-apiserver/auth-reader.yaml - - sample-apiserver/rbac.yaml - - sample-apiserver/rbac-bind.yaml -``` - -This means the sample-apiserver manifests are installed automatically as part of `_services-ready` -(which already runs `kubectl apply -k test/e2e/setup/manifests`). There is no separate -"apply sample-apiserver" step in the Taskfile; we only add a readiness-wait task afterwards. - -### RBAC: full set from upstream (all four resources required) - -The upstream example defines four RBAC resources, and **all four are required** for the -sample-apiserver to function: - -1. **`auth-delegator.yaml`** — ClusterRoleBinding granting `system:auth-delegator` to the - ServiceAccount. Required for the extension server to delegate authentication decisions to - the core API server. - -2. **`auth-reader.yaml`** — RoleBinding in `kube-system` granting - `extension-apiserver-authentication-reader` to the ServiceAccount. **This is the one most often - missed.** Without it, the extension server cannot read the `extension-apiserver-authentication` - ConfigMap in `kube-system`, and delegated auth fails even when the pod is healthy and the - APIService exists. - -3. **`rbac.yaml`** — ClusterRole granting access to namespaces, admission webhook configs, and - flow control resources. Required for the extension server's internal machinery. - -4. **`rbac-bind.yaml`** — ClusterRoleBinding for the above ClusterRole. - -### Key resources - -**APIService** (the simple approach): -```yaml -apiVersion: apiregistration.k8s.io/v1 -kind: APIService -metadata: - name: v1alpha1.wardle.example.com -spec: - insecureSkipTLSVerify: true - group: wardle.example.com - groupPriorityMinimum: 1000 - versionPriority: 15 - service: - name: api - namespace: wardle - version: v1alpha1 -``` - -**Deployment** (with etcd sidecar for storage): -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: wardle-server - namespace: wardle -spec: - replicas: 1 - selector: - matchLabels: - apiserver: "true" - template: - metadata: - labels: - apiserver: "true" - spec: - serviceAccountName: apiserver - containers: - - name: wardle-server - image: registry.k8s.io/e2e-test-images/sample-apiserver:TAG - args: ["--etcd-servers=http://localhost:2379"] - - name: etcd - image: registry.k8s.io/etcd:v3.6.8 -``` - -The etcd sidecar is required because the sample-apiserver's `--etcd-servers` flag does not accept -an empty value — it needs a real etcd. The sidecar pattern from the upstream example is the -simplest approach. Data does not need to survive pod restarts for e2e. - -### Image tag: must match k3s version - -The e2e cluster uses `rancher/k3s:v1.36.1-k3s1`. The sample-apiserver image must match the -Kubernetes minor version. Use a kustomize variable for easy updates: - -```yaml -# kustomization.yaml -images: - - name: registry.k8s.io/e2e-test-images/sample-apiserver - newTag: "1.36.1" # match k3s v1.36.x -``` - -**Action required before implementation:** verify that `registry.k8s.io/e2e-test-images/sample-apiserver:1.36.1` -exists. If not, check the available tags and use the closest match. The sample-apiserver images -are published per Kubernetes release; newer k3s versions may require building from source. - -### gitops-reverser RBAC: no changes needed - -The default kustomize install still binds a wildcard read, but it no longer sits in -`config/rbac/role.yaml`: it moved to a ClusterRole of its own, -[`config/rbac/watch-any-role.yaml`](../../config/rbac/watch-any-role.yaml) (the Helm equivalent is -`rbac.watchTypes.mode: any`). - -```yaml -- apiGroups: ['*'] - resources: ['*'] - verbs: [get, list, watch] -``` - -This covers `wardle.example.com` flunders and fischers, so no additional RBAC overlay is needed. An -install that narrowed `watchTypes` to `selected` would have to name them explicitly; the e2e setup -does not. - -### Integration into `prepare-e2e` - -The sample-apiserver is deployed as part of the standard `prepare-e2e` flow, not as an opt-in. -This keeps the provisioning story simple and avoids runtime skip-or-fail ambiguity in the test -suite. - -Since the manifests are now part of `test/e2e/setup/manifests/`, they are applied by the existing -`_services-ready` task. We only need a readiness-wait task that depends on `_services-ready`: - -```yaml -_sample-apiserver-ready: - method: timestamp - sources: - - '{{.CS}}/services.ready' - - test/e2e/setup/manifests/sample-apiserver/**/* - generates: - - '{{.CS}}/sample-apiserver.ready' - status: - - test -f "{{.CS}}/sample-apiserver.ready" - cmds: - - task: _services-ready - - | - # Wait for the deployment to be available - kubectl --context "{{.CTX}}" -n wardle wait deployment/wardle-server \ - --for=condition=Available --timeout=120s - # Wait for APIService to become available (core API server can reach extension server) - kubectl --context "{{.CTX}}" wait apiservice/v1alpha1.wardle.example.com \ - --for=condition=Available --timeout=120s - # Verify the resource is discoverable - kubectl --context "{{.CTX}}" api-resources --api-group=wardle.example.com | grep -q flunders - touch "{{.CS}}/sample-apiserver.ready" -``` - -Add `_sample-apiserver-ready` to the `prepare-e2e` flow before `_prepare-e2e-ready`, and make -`_prepare-e2e-ready` consume the resulting stamp: -```yaml -prepare-e2e: - cmds: - # ... existing steps ... - - task: _webhook-tls-ready - - task: _sample-apiserver-ready # NEW - - task: _prepare-e2e-ready - - task: portforward-ensure - -_prepare-e2e-ready: - method: timestamp - sources: - # ... existing sources ... - - '{{.CS}}/sample-apiserver.ready' # NEW - # ... -``` - -And add a standalone task for running just the aggregated-api tests: -```yaml -test-e2e-aggregated-api: - desc: Run aggregated API server e2e scenarios - cmds: - - | - export CTX="{{.CTX}}" - export INSTALL_MODE="{{.INSTALL_MODE}}" - export NAMESPACE="{{.NAMESPACE}}" - export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v \ - -ginkgo.label-filter=aggregated-api -``` - -### Audit policy - -The existing audit policy at `test/e2e/cluster/audit/policy.yaml` has a catch-all rule: - -```yaml -- level: RequestResponse - omitManagedFields: true - verbs: [create, update, patch, delete, deletecollection] - # No resources listed = Matches everything else (including CRDs) -``` - -This matches `wardle.example.com` resources. **No audit policy change needed.** - -However, note that the existing noise filter drops `*/status` for `""`, `apps`, and -`networking.k8s.io` groups — but does **not** drop `*/status` for `wardle.example.com`. This -means status-subresource updates on Flunders **will** produce audit events. This is relevant for -S6. - ---- - -## Test File Structure - -``` -test/e2e/ -├── aggregated_api_e2e_test.go # Main test file, Label("aggregated-api") -└── templates/ - └── aggregated-api/ - ├── watchrule-flunder.tmpl # WatchRule for namespaced flunders - ├── clusterwatchrule-fischer.tmpl # ClusterWatchRule for cluster-scoped fischers - ├── flunder-instance.tmpl # Flunder resource template (v1alpha1) - └── fischer-instance.tmpl # Fischer resource template (v1alpha1) -``` - ---- - -## Test Scenarios - -All tests live under `Label("aggregated-api")` and are `Ordered`. -There is no separate smoke label split; `task test-e2e` runs the whole package, -and `task test-e2e-aggregated-api` remains the focused entry point for this -category. - -### Prerequisites (BeforeAll) - -1. Verify `flunders.wardle.example.com` appears in `kubectl api-resources` — if not, fail fast - with a clear error ("sample-apiserver not deployed; run task prepare-e2e") -2. Create test namespace -3. Set up Gitea repo and credentials (same pattern as manager tests) -4. Create GitProvider and GitTarget - -### S1 — Discovery + WatchRule: aggregated resource is accepted - -**What it validates:** the combination of `FilterDiscoverableGVRs()` in -internal/watch/discovery.go and the WatchRule controller -correctly accepting an aggregated API resource. - -**Action:** -1. Apply a WatchRule targeting `wardle.example.com/v1alpha1/flunders` -2. Wait for WatchRule to reach `Ready=True` - -**Assert:** -- `verifyResourceStatus("watchrule", name, ns, "True", "Ready", "")` — the WatchRule is Ready - -This is the only status signal the WatchRule currently exposes. We do **not** assert on informer -startup or discovery internals — `Ready=True` is the contract. If the discovery filter silently -dropped the GVR, the WatchRule would not reach Ready. - -**Why it matters:** if `ServerPreferredResources()` returns a different response shape for -aggregated APIs, or the discovery filter has an edge case, this catches it at the entry point. - ---- - -### S2 — Audit: CREATE a Flunder produces a git commit - -**What it validates:** the full audit pipeline for aggregated API mutations: -Redis → AuditConsumer → RuleStore match → sanitize → BranchWorker → git commit. - -**Labels:** `aggregated-api`, `smoke` - -**Action:** -1. Create a Flunder resource in the test namespace via `kubectl create --as=e2e-user@test` -2. Wait for git commit - -**Assert:** -- A file appears at `{target-path}/wardle.example.com/v1alpha1/flunders/{namespace}/{name}.yaml` -- The YAML content contains `apiVersion: wardle.example.com/v1alpha1`, `kind: Flunder` -- The file contains the Flunder's `spec` fields -- The commit message attributes the author (from `--as`) - -**Why it matters:** this is the core smoke test. If audit events for proxied API requests have a -different structure than native requests, the pipeline would silently drop them. - -This is the best candidate for smoke coverage because it proves the end-to-end aggregated API happy -path with a single resource create, without pulling in the more brittle edge-case behaviors from the -rest of the suite. - ---- - -### S3 — Audit: UPDATE a Flunder produces an updated git commit - -**What it validates:** update mutations on aggregated resources flow through correctly. - -**Action:** -1. Patch the Flunder's spec (e.g., change `referenceType`) -2. Wait for git commit - -**Assert:** -- The file is updated with the new spec -- The git log shows a new commit - ---- - -### S4 — Audit: DELETE a Flunder removes the file from git - -**What it validates:** delete operations on aggregated resources. - -**Action:** -1. Delete the Flunder -2. Wait for file removal in git - -**Assert:** -- The file no longer exists in the repo -- A commit records the deletion - ---- - -### S5 — Watch/Informer: snapshot reconcile captures existing Flunders - -**What it validates:** informers can list+watch aggregated API resources for snapshot/reconcile, -exercising internal/watch/informers.go. - -**Action:** -1. Create two Flunder resources **before** the WatchRule exists -2. Apply the WatchRule -3. Wait for the initial snapshot sync (GitTarget condition `SnapshotSynced`) - -**Assert:** -- Both pre-existing Flunders appear in git as part of the snapshot commit -- The commit is atomic (both resources in one commit, as per snapshot behavior) - -**Why it matters:** the informer must successfully `list` aggregated API resources during initial -cache sync. If the aggregation proxy introduces different pagination or response shapes, this -catches it. - ---- - -### S6 — Status subresource: no spurious commit - -**What it validates:** a status-only update on a Flunder does not produce a spurious git commit. - -**Expected behavior (concrete):** the sanitizer strips `status` because it only preserves `spec`, -`data`, and `binaryData` (plus the hardcoded top-level fields list in -[sanitize.go:74](../../internal/sanitize/sanitize.go#L74)). After sanitization, the content hash -is identical to the previous version. The content-hash deduplication in -[GitTargetEventStream](../../internal/reconcile/git_target_event_stream.go) drops it. **No new -commit should appear.** - -**Action:** -1. Create a Flunder, wait for the CREATE commit, record the git HEAD SHA -2. Update the Flunder's status subresource: - ``` - kubectl --context $CTX patch flunder $NAME -n $NS --subresource=status \ - --type=merge -p '{"status":{}}' - ``` - (Note: FlunderStatus is an empty struct, but the patch still triggers an audit event) -3. Wait a reasonable interval (e.g., 15 seconds — longer than the push interval) -4. Check git HEAD SHA - -**Assert:** -- Git HEAD SHA is unchanged — no new commit was created -- No error in controller logs related to flunder status processing - -**Why it matters:** the audit policy does not filter `*/status` for `wardle.example.com`. Without -sanitization + dedup working correctly, every status update would create a spurious commit. - ---- - -### S7 — Non-standard resource instance names - -**What it validates:** the full pipeline handles resource instance names that contain underscores -and dots — characters that are valid per the sample-apiserver's path-segment validation but would -be invalid as DNS labels. - -**Confirmed:** the sample-apiserver uses `path.ValidatePathSegmentName` (via -`genericregistry.Store` defaults), which allows underscores and dots. Only `/`, `%`, and `..` are -rejected. - -**Action:** -1. Create a Flunder named `test_flunder_underscore` -2. Create a Flunder named `test.flunder.dotted` -3. Wait for git commits - -**Assert:** -- Files are created at: - - `{path}/wardle.example.com/v1alpha1/flunders/{ns}/test_flunder_underscore.yaml` - - `{path}/wardle.example.com/v1alpha1/flunders/{ns}/test.flunder.dotted.yaml` -- Content is correct YAML with the expected apiVersion/kind -- Subsequent update and delete work for both names - -**Why it matters:** [internal/types/identifier.go](../../internal/types/identifier.go) uses -`fmt.Sprintf` to build paths with no validation or escaping. Underscores and dots are safe on all -common filesystems and in git. This test confirms the pipeline handles them end-to-end. - ---- - -### S8 — Cluster-scoped aggregated resource (Fischer) + sanitizer edge case - -**What it validates:** `ClusterWatchRule` works with cluster-scoped aggregated API resources. -Also surfaces a **known sanitizer limitation**. - -**Action:** -1. Apply a ClusterWatchRule targeting `wardle.example.com/v1alpha1/fischers` with scope `Cluster` -2. Create a Fischer resource with `disallowedFlunders: ["bad-flunder"]` -3. Wait for git commit - -**Assert:** -- File appears at `{path}/wardle.example.com/v1alpha1/fischers/{name}.yaml` (no namespace segment) -- ClusterWatchRule reaches `Ready=True` - -**Known issue to document:** Fischer has `disallowedFlunders` as a top-level field, not under -`spec`. The sanitizer -([sanitize.go:37](../../internal/sanitize/sanitize.go#L37)) only preserves `spec`, `data`, -`binaryData`, and a hardcoded list of other top-level fields. `disallowedFlunders` is **not** in -that list, so it will be silently dropped from the git output. - -This is the same behavior for any CRD or aggregated API resource that uses non-standard top-level -fields. The test should: -1. Assert that the file is created (the resource is tracked) -2. Assert that `disallowedFlunders` is **absent** from the committed YAML (documenting current - behavior) -3. Add a code comment noting this is a known limitation of the sanitizer, not a bug in aggregated - API support - ---- - -### S9 — Extension server restart: watch recovery - -**What it validates:** the informer reconnects after the aggregated API server restarts. - -**Action:** -1. Have a WatchRule active for flunders with a Flunder already committed -2. Restart the sample-apiserver: `kubectl -n wardle rollout restart deployment/wardle-server` -3. Wait for the deployment to become Available again -4. Wait for the APIService to become Available again -5. Create a new Flunder -6. Wait for git commit - -**Assert:** -- The new Flunder appears in git -- No persistent errors in the gitops-reverser controller logs about the GVR becoming unavailable - -**Why it matters:** informer reconnection through the aggregation proxy is handled by client-go's -reflector. The Watch Manager's exponential backoff for unavailable GVRs -([internal/watch/manager.go](../../internal/watch/manager.go)) should handle the transient -unavailability gracefully. - ---- - -## Risks and Edge Cases - -### Audit event structure for proxied requests - -The core API server generates audit events when proxying to an aggregated API server. The event -structure should be identical (same `objectRef`, same `responseObject` embedding). S2 will confirm -or disprove this assumption. - -Specific things to watch for: -- `objectRef.apiGroup` — should be `wardle.example.com`, not empty -- `objectRef.apiVersion` — should be `v1alpha1` -- `objectRef.resource` — should be `flunders` -- `responseObject` — should contain the full Flunder object - -### Sanitizer field-loss for non-standard schemas - -As detailed in S8, any aggregated API resource (or CRD) that places data in top-level fields -outside the hardcoded preservation list will lose that data. Fischer's `disallowedFlunders` is the -concrete example. This is a pre-existing product limitation — not introduced by aggregated API -support — but the test suite should document it clearly. - -### etcd sidecar adds resource overhead - -The sample-apiserver deployment includes an etcd sidecar. This adds ~100MB memory to the test -cluster. Acceptable for e2e, but worth noting for resource-constrained CI runners. - -### Version pinning - -The sample-apiserver image must match the k3s Kubernetes version. Currently k3s `v1.36.1-k3s1`. -The kustomize image tag makes this a one-line change when k3s is upgraded. - ---- - -## Template Examples - -### `templates/aggregated-api/watchrule-flunder.tmpl` - -```yaml -apiVersion: configbutler.ai/v1alpha3 -kind: WatchRule -metadata: - name: {{ .Name }} - namespace: {{ .Namespace }} -spec: - gitTargetRef: - name: {{ .GitTargetName }} - rules: - - operations: ["CREATE", "UPDATE", "DELETE"] - apiGroups: ["wardle.example.com"] - apiVersions: ["v1alpha1"] - resources: ["flunders"] -``` - -### `templates/aggregated-api/flunder-instance.tmpl` - -```yaml -apiVersion: wardle.example.com/v1alpha1 -kind: Flunder -metadata: - name: {{ .Name }} - namespace: {{ .Namespace }} -spec: - referenceType: "Flunder" - reference: "some-flunder" -``` - -### `templates/aggregated-api/clusterwatchrule-fischer.tmpl` - -```yaml -apiVersion: configbutler.ai/v1alpha3 -kind: ClusterWatchRule -metadata: - name: {{ .Name }} -spec: - gitTargetRef: - name: {{ .GitTargetName }} - namespace: {{ .GitTargetNamespace }} - rules: - - operations: ["CREATE", "UPDATE", "DELETE"] - apiGroups: ["wardle.example.com"] - apiVersions: ["v1alpha1"] - resources: ["fischers"] - scope: Cluster -``` - -### `templates/aggregated-api/fischer-instance.tmpl` - -```yaml -apiVersion: wardle.example.com/v1alpha1 -kind: Fischer -metadata: - name: {{ .Name }} -disallowedFlunders: - - "bad-flunder-one" -``` - ---- - -## Implementation Plan - -### Phase 1: Infrastructure (1 PR) - -1. Create `test/e2e/setup/manifests/sample-apiserver/` with manifests adapted from the upstream - `external-sources/sample-apiserver/artifacts/example/` directory -2. Add the new resources to `test/e2e/setup/manifests/kustomization.yaml` -3. Pin the image tag to match k3s `v1.35.x` (via kustomize `images` in the kustomization) -4. Add `_sample-apiserver-ready` readiness-wait task to Taskfile -5. Wire it into `prepare-e2e` before `_prepare-e2e-ready`, and add the stamp to - `_prepare-e2e-ready` sources -6. Verify manually: `kubectl get flunders.wardle.example.com` works, audit events arrive in Redis - -### Phase 2: Core test scenarios (1 PR) - -1. Create `test/e2e/aggregated_api_e2e_test.go` with Label `aggregated-api` -2. Create template files in `test/e2e/templates/aggregated-api/` -3. Implement S1 (discovery + WatchRule Ready) -4. Implement S2–S4 (CRUD lifecycle) -5. Implement S5 (snapshot reconcile) -6. Add `test-e2e-aggregated-api` task - -### Phase 3: Edge cases (1 PR) - -1. Implement S6 (status — no spurious commit) -2. Implement S7 (non-standard instance names) -3. Implement S8 (cluster-scoped Fischer + sanitizer documentation) -4. Implement S9 (extension server restart recovery) - -### Phase 4: CI integration - -1. Keep `aggregated-api` included in `task test-e2e`, which runs all labels. -2. Keep `task test-e2e-aggregated-api` as the focused entry point for aggregated - API work. -3. Document the test category in `E2E_DEBUGGING.md`. diff --git a/docs/finished/e2e-declare-investigation.md b/docs/finished/e2e-declare-investigation.md deleted file mode 100644 index 140bb3510..000000000 --- a/docs/finished/e2e-declare-investigation.md +++ /dev/null @@ -1,318 +0,0 @@ -# Investigation handoff: a GitTarget registers its event stream and never declares - -Working document for PR #315 (`feat/target-watch-cell-identity`). Written so a fresh -context can continue without re-deriving anything. **Facts are separated from -hypotheses.** Nothing below is a conclusion unless it says so. - -Status at the time of writing: **the branch is NOT mergeable.** A reproducible e2e -failure is open and its cause is not established. That status is preserved as written; -the investigation is closed, and the note below records how. - -> **Closed.** Two structural causes were found and removed by -> [`docs/design/watch-manager-ownership.md`](../design/watch-manager-ownership.md), and the -> temporary `declare-trace` phase logging this document asked for has been deleted with them: -> -> - `refreshRunningTargetWatches` re-planned only targets already present in `m.targetWatches`, so -> **a target whose first declare never completed was never picked up again** — the exact shape of -> "registers its event stream and then never declares". The owner's dirty set is marked on intent -> rather than on success and does not have that property. -> - the local-cluster discovery call carried no request timeout at all (`cluster_context.go` applied -> one only for a remote), so the legacy non-context `ServerGroupsAndResources()` could hang the -> controller worker running it. It is bounded unconditionally now, and no watch-manager work runs -> on a controller worker at all. -> -> A pass that fails no longer disappears either: it leaves the target dirty on a stated backoff and -> publishes `WatchPlanFailing` on the GitTarget, so the silent version of this failure cannot recur. -> The rest of this document is kept as the record of how it was tracked down. - ---- - -## 1. What the branch does - -Commits on top of `main` (`ff966f5d` is the merge-base): - -| Commit | Content | -| --- | --- | -| `164adc28` | cell, not GVR, is the sweep boundary's identity | -| `e5a86a74` | one target watch per cell, not per served version | -| `4a168948` | stamp queued work with the producing cell | -| `099233e3` | key stream readiness, retention and fidelity on the cell | -| `e8bd4808` | delete the stream lease; collapse `git.Provenance` to the cell | -| `94c43263` | **change 1**: compute the per-cell plan diff and log it (no behavior) | -| `b501954d` | **prerequisite**: render-fidelity revision becomes per scope | -| `e591ebd1` | **change 2**: apply the plan per cell (per-stream cancel) | -| `060bb739`, `e2685f51` | docs | -| `eeb231ab` | write-divergence guard for an empty plan + doc fixes (**unpushed**) | - -Design: [`docs/design/target-watch-plan.md`](../design/target-watch-plan.md). - -### The behavior change that matters here (`e591ebd1`) - -Before: `replaceGitTargetWatches` compared the whole rendered spec map. Equal meant an -early return; different meant one `prior.cancel()` for the entire target, then every -stream restarted. - -After: `targetWatchSet` is `map[types.CellKey]*runningTargetWatch`, each with its own -`cancel`. The plan is diffed into `keep` / `start` / `restart` / `stop` and applied per -cell. `internal/watch/target_watch.go:172-215` is the whole function. - -Two orderings are NEW in that function, and both happen while `targetWatchesMu` is held: - -- `set.stop(cell)` calls a stream's `context.CancelFunc` **under the mutex** - (`target_watch.go:194-199`). -- `reconcileTargetRenderFidelityLocked` takes the render-fidelity gate's own mutex - **under `targetWatchesMu`** (`target_watch.go:204`), establishing the lock order - `targetWatchesMu → RenderFidelityGate.mu`. - ---- - -## 2. The failure - -### 2.1 Symptom, identical in two independent runs - -A WatchRule's `Ready` condition stays `False` past the 90s e2e timeout. The e2e helper -that fails is `verifyResourceStatus` → `test/e2e/e2e_test.go:234`, comparing the -condition status `False` against the expected `True`. - -`Ready` on a WatchRule is a roll-up that includes `ConditionTypeStreamsRunning` -(`internal/controller/watchrule_controller.go:209`), which is projected from -`WatchManager.StreamSummaryForWatchRule`. With no declared streams, the rule's expected -cells have no entry in `targetStreamStates`, and `streamStatusesByType` -(`internal/watch/stream_readiness.go:228-245`) treats a missing entry as `Replaying`. -So the rule can never reach Ready. **That part is understood and is a consequence, not -the cause.** - -### 2.2 The cause is upstream: the GitTarget never declares - -In both failing runs, for the affected GitTarget only: - -- `Registered GitTargetEventStream` is logged (emitted from - `internal/controller/gittarget_controller.go:687`, inside the worker-wiring gate); -- **no `target watch plan reconciled` line is ever logged for that GitTarget.** - -That log line is emitted unconditionally at `target_watch.go:212`, on every path that -reaches it, including the all-`keep` no-op case. Its absence means -`replaceGitTargetWatches` never got that far for that target. - -> **Correction (review, 2026-08-27).** An earlier version of this document read the pair -> "registered, then nothing" as *one* reconcile that got past wiring and then blocked. -> That inference is **not supported**. `RegisterGitTargetEventStream` is only reached when -> the stream does not already exist (`gittarget_controller.go:682-688` returns the existing -> stream silently), so the log line marks the FIRST reconcile that passed the wiring gate -> and says nothing about any later one. What the evidence supports is only: the target -> passed wiring at least once, and **no** reconcile ever completed a declare. -> -> The review also placed a candidate EARLIER than declare: -> `resolveSourceClusterProvider` (`gittarget_controller.go:193`) is a synchronous API read -> that runs after wiring and before `DeclareForGitTarget`. An error there is ruled out (no -> `Reconciler error` in the log); a block there is not. -> -> And it found a concrete unbounded-wait mechanism in code this branch did not touch: the -> discovery call in `refreshClusterForDeclare` gets a request timeout **only for a remote -> cluster**. `cluster_context.go:695` applies `sourceClusterDialTimeout` to a config copy -> under `if !cc.isLocalLocked()`, so for the LOCAL cluster the legacy, non-context -> `ServerGroupsAndResources()` runs with no deadline at all. - -Meanwhile the WatchRule reconciles successfully every 10s -(`WatchRule reconciliation via GitTarget successful`), and other GitTargets declare -normally throughout the same window. - -### 2.3 Run-by-run evidence - -| Run | Code | Result | Failing spec | -| --- | --- | --- | --- | -| Local full `task test-e2e` | `e2685f51` | **FAIL** 69 passed / 1 failed | Commit Signing / custom reconcile message template `[signing]` | -| CI `E2E (full-manager)` run `33057523968` | `e2685f51` | **FAIL** 54 passed / 1 failed | Manager WatchRule / delete Git file when ConfigMap is deleted `[manager]` | -| Local `[signing]` only (5 specs) | `eeb231ab` | **PASS** 5/5 | — | -| Local `[manager]` subset (55 specs) | `eeb231ab` | **PASS** 55/55 in 8m23s | — | -| Local full, trace armed (71 specs) | `bd2d186b` | **PASS** 71/71 in 11m26s | — | - -Both failures are the same shape, in **different specs**. The failing GitTargets were -`signing-reconcile-dest` and `watchrule-delete-test-dest` respectively. - -**The two green rows do not exonerate anything, because they change two variables at -once.** They ran a smaller spec set AND a different commit (`eeb231ab` rather than -`e2685f51`). The local `[manager]` run is the same 55-spec set as CI's failing -full-manager leg and passed, so the failure is not a deterministic property of that -spec set. The only code difference between the two commits is the empty-plan -write-divergence guard in `RenderFidelityGate.Reconcile`, which has no path to the -declare code and is very unlikely to be the reason; but it is a difference, and the -comparison is confounded until a full run is repeated on one commit. - -**A repeat of the failing configuration did NOT reproduce it.** The full suite was run -again on `bd2d186b`, which carries the phase trace described below and an armed watcher -that would have captured any GitTarget sitting at `StreamsRunning=False` for more than -45s. 71 specs passed, and the watcher never fired. So the fault is **intermittent**: it -is not a deterministic consequence of change 2, and it did not reproduce on the third -large run. - -Tally so far: two large runs failed (both on `e2685f51`), three runs passed (two subsets -and one large, on `eeb231ab` and `bd2d186b`). No large run has yet been made on `main` -for comparison, so the branch is still the only place it has been seen. - -Verbatim, from the local run, all 19 log lines mentioning `signing-reconcile-dest` -reduce to: - -```text -1 "msg":"Registered GitTargetEventStream" -11 "msg":"Starting WatchRule validation" -1 "msg":"Unregistered GitTargetEventStream" -1 "msg":"GitTarget not found, was likely deleted" -``` - -Timeline of the CI failure (`watchrule-delete-test-dest`): - -```text -09:27:39 STEP creating GitTarget 'watchrule-delete-test-dest' -09:27:39 Registered GitTargetEventStream <- last trace of the target -09:27:39 WatchRule reconciliation via GitTarget successful -09:27:49 .. 09:29:09 the same WatchRule reconcile, every 10s, always "successful" -09:29:09 [FAILED] Timed out after 90.000s: : False to equal : True -09:29:17 Unregistered GitTargetEventStream (namespace teardown) -``` - -### 2.4 CI green baselines on this branch - -| CI run | Commit | Contents | Result | -| --- | --- | --- | --- | -| `33052920251` | `e8bd4808` | up to the lease deletion | success | -| `33055342913` | `94c43263` | + change 1 (log only) | success | -| `33057086134` | `060bb739` | + change 2 | **cancelled** (no signal) | -| `33057523968` | `e2685f51` | + docs | **failure** | - -So the first CI run that contained change 2 failed, and every run before change 2 passed. -That is **one** data point, not a bisect: `060bb739` was cancelled, so change 2 has never -had a clean CI run of its own. - ---- - -## 3. Ruled out, with the evidence - -All of these were checked against the CI job log, not reasoned about: - -- **No `Reconciler error`** anywhere in the job log (`grep -c` → 0). So no reconcile - returned an error to controller-runtime. -- **No `watch-first declare skipped; surface not observable`**. That is the Info log - `DeclareForGitTarget` writes whenever `EnsureGitTargetWatches` returns an error - (`internal/watch/materialization.go:38-41`), so none of the four error paths in - `EnsureGitTargetWatches` (`target_watch.go:141-170`) was taken. -- **No `aborting watch setup`** (the two error strings inside that function). -- **No global stall**: `target watch plan reconciled` lines appear steadily through the - 90s window for other targets (13 lines at 09:27:28, 17 at 09:27:39, 11 at 09:28:20, - 12 at 09:29:09, and so on). -- **`enqueueGitPathChange` cannot block**: it is a non-blocking `select` with a - `default` (`internal/watch/gitpath_events.go:33-47`). -- **`ResourceReference.Key()` is `"namespace/name"`** and does not include the UID - (`internal/types/reference.go:44`), so a UID-bearing reference and a rule-derived one - map to the same key. No key mismatch there. -- **The error census** for the whole CI job, by message: - `690 sendInitialEvents unsupported` (a known k3s fallback warning), - `185 Resync commit failed; dropping request`, `9 Failed to get referenced GitTarget`, - `4 Repository connectivity check failed`, `4 Remote connectivity check failed`, - `4 Failed to build pending write; dropping open window`, - `3 audit Redis not yet reachable`, `2 Failed to resolve GitProvider from GitTarget`, - `1 per-type reconcile failed`, `1 Failed to build resync pending write`. - -The elimination is uncomfortable: every path into `replaceGitTargetWatches` either logs -an error or reaches the unconditional log line, and neither happened. - ---- - -## 4. Hypotheses (none confirmed) - -**H1 — the reconcile blocks before reaching the log.** (Strongest.) If the GitTarget controller's -reconcile blocks inside `DeclareForGitTarget`, there is no log and no error, which is -exactly what we observe. It would also be self-sustaining: `refreshRunningTargetWatches` -(`target_watch.go`, called from `ReconcileForRuleChange`) only refreshes targets already -present in `m.targetWatches`, and a target whose first declare never completed is not in -that map, so no later WatchRule reconcile can rescue it. Candidate blocking points, in the order they are reached: -`resolveSourceClusterProvider` (an API read, BEFORE declare), `refreshClusterForDeclare` -(a discovery call with no request timeout on the local cluster, `cluster_context.go:695`), -`refreshWatchedTypeTables`, or `targetWatchesMu` itself. - -**H2 — lock-order inversion introduced by change 2.** (Weakened by review: the fidelity -recording path releases `gate.mu` before taking `targetWatchesMu`, so no cycle is -demonstrated.) `targetWatchesMu → gate.mu` is new -(`target_watch.go:204`). A reverse path would deadlock. `MarkTargetRenderFidelityScopeClean` -releases `gate.mu` inside `gate.RecordScopeClean` before `recordRenderFidelityStatus` -takes `targetWatchesMu`, so that specific path looks safe, but the whole set of -`gate.mu` holders has not been enumerated. Note `sync.RWMutex` is write-preferring: a -blocked `Lock()` blocks later `RLock()`s, so a slow reader plus a waiting writer can -serialise more than it appears. - -**H3 — cancelling under the mutex.** `set.stop()` runs `cancel()` while holding -`targetWatchesMu` (`target_watch.go:194-199`). Cancellation itself is non-blocking, but -it wakes stream goroutines that immediately contend for `targetWatchesMu` -(`markTargetStreamState`, `recordRenderFidelityStatus`). This should only be contention, -not deadlock, but it is new behavior under a lock and is worth ruling out. - -**H4 — not caused by this branch at all.** No run of change 2 has been compared against a -`main` run under the same conditions, and the failing spec moved between runs, which is a -race signature. `docs`-level note: my memory has several pre-existing e2e flake entries, -but none matches this signature. - -H1 is the only hypothesis that explains *all* the evidence including the silence. H2 and -H3 are candidate mechanisms *for* H1. - ---- - -## 5. The decisive next step - -Repeat the **full local suite** (the 73-spec default filter, which is what failed the -first time) on a single commit, and if it reproduces, take a **goroutine dump** from the -manager while the GitTarget is still stuck. That distinguishes H1/H2/H3 from H4 in one shot: either the -GitTarget reconcile goroutine is parked on a mutex or a network call, or it is not there -at all. - -A goroutine dump IS possible: the image is `gcr.io/distroless/static:debug`, which ships -busybox, and the manager is PID 1. Verified on a live pod: - -```bash -kubectl exec -n gitops-reverser -- /busybox/kill -QUIT 1 -kubectl logs -n gitops-reverser --previous | head -200 -``` - -SIGQUIT makes the Go runtime print every goroutine's stack and exit, so the pod restarts: -do it only once the spec has already failed. What to look for, per the review: - -| Frame at the top of the GitTarget reconcile goroutine | Reading | -| --- | --- | -| `resolveSourceClusterProvider` / client GET | blocked BEFORE declare | -| `refreshClusterCatalog` / `ServerGroupsAndResources` | inside declare, discovery blocked | -| `targetWatchesMu.Lock` | plan replacement contention (H2/H3) | -| no such goroutine at all | the registration log was from an earlier attempt (H4) | - -`Manager.DeclaredSourceCluster` (`cluster_context.go:317`) is a second, cheaper -discriminator: it is recorded at the START of `DeclareForGitTarget`, before any discovery. -Set, with no plan log, means the block is inside declaration; unset means the reconcile -never reached declaration. - -Practical notes: - -- `E2E_LABEL_FILTER` **replaces** the default filter, so re-AND the exclusions: - `E2E_LABEL_FILTER='manager && !image-refresh && !bi-directional && !source-cluster && !ado'`. - A zero-match filter skips both suite hooks and passes vacuously. -- The cluster survives a run; the manager pod is redeployed by the next run, so pull - logs or dumps **before** starting another. -- Read the Ginkgo JSON report rather than the truncated console log. - -A watcher can catch it without babysitting: poll every GitTarget for -`StreamsRunning=False` older than ~60s, and dump goroutines the moment one appears. - -If the block is confirmed, the smallest candidate fixes are: move `set.stop()` calls out -of the critical section (collect the cancels, run them after `Unlock`), and take the -fidelity gate outside `targetWatchesMu` (compute the revisions first, then apply). - ---- - -## 6. Also open, unrelated to the failure - -Ten CodeRabbit review findings were triaged; nine were valid and are fixed in the -unpushed `eeb231ab`, one was rejected (a "dangling godoc fragment" that is a complete -sentence ending in "…authorized for."). The one real code finding was mine: `Reconcile` -cleared a write divergence when the scope set was empty, because `fresh == len(scopes)` -holds vacuously at `0 == 0`. - -Two debts change 2 left, recorded in the design doc: `streamRevisions`' zero-revision -fallback for the no-gate wiring, and the retention roll-up's install-then-report -ordering requirement. diff --git a/docs/finished/image-refresh-test-design.md b/docs/finished/image-refresh-test-design.md deleted file mode 100644 index 7715ca7c6..000000000 --- a/docs/finished/image-refresh-test-design.md +++ /dev/null @@ -1,260 +0,0 @@ -# Image Refresh Test Design - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -## What This Tests - -The Task-driven image-refresh dependency chain is load-bearing infrastructure. When it breaks, -developers waste time chasing stale pods, and CI can silently run tests against the wrong -build. This document defines the scenarios that must hold, and how to validate them. - -The chain under test: - -``` -GO_SOURCES → controller.id → project-image.ready → image.loaded → controller.deployed -``` - -Each arrow is a Task-managed dependency edge. When any input is newer than its stamp, the chain -invalidates forward. The end result: `kubectl rollout restart` fires, the pod picks up -the newly loaded image from containerd. - -## Scenarios - -### S1 — No-op: second run without changes does nothing - -**Setup:** run `task prepare-e2e` once to reach a known good state. - -**Action:** run `task prepare-e2e` again immediately, with no source changes. - -**Assert:** -- `controller.deployed` stamp mtime is unchanged -- no `rollout restart` was issued -- the running pod is the same pod (same name, same start time) - -**Why it matters:** if every run restarts the pod, tests become slow and flaky. The -stamp mechanism exists precisely to avoid this. - ---- - -### S2 — Go source change triggers pod restart - -**Setup:** cluster in known good state from S1. - -**Action:** append a comment line to `cmd/main.go`, run `task prepare-e2e`. - -**Assert:** -- `controller.id` stamp is newer than before -- `image.loaded` stamp content changed (new digest) -- `controller.deployed` stamp content changed (new `IMAGE@digest`) -- a new pod is running (different name or newer start time) -- the old pod is gone - -**Why it matters:** this is the primary correctness invariant. A Go change must reach -the running pod. - ---- - -### S3 — Second Go change also triggers (not one-shot) - -**Setup:** cluster in state after S2 (pod already restarted once). - -**Action:** append another comment to `cmd/main.go`, run `task prepare-e2e`. - -**Assert:** same as S2 — new pod, new digest in stamps. - -**Why it matters:** rules out accidental one-shot behavior where the first change works -because a stamp was missing, but subsequent changes are silently ignored. - ---- - -### S4 — Non-Go file change does not trigger rebuild - -**Setup:** cluster in known good state. - -**Action:** append a comment to `test/e2e/helpers.go` (excluded from `GO_SOURCES`), -run `task prepare-e2e`. - -**Assert:** -- `controller.id` stamp mtime is unchanged -- `image.loaded` stamp mtime is unchanged -- `controller.deployed` stamp mtime is unchanged -- pod start time is unchanged - -**Why it matters:** confirms `GO_SOURCES` filtering is correct. Test-only changes must -not cause a rebuild. - ---- - -### S5 — Dockerfile change triggers rebuild - -**Setup:** cluster in known good state. - -**Action:** append a comment to `Dockerfile`, run `task prepare-e2e`. - -**Assert:** same as S2 — new pod, new digest. - -**Why it matters:** `Dockerfile` is listed in `GO_SOURCES` (as a build input). A -changed Dockerfile must produce a fresh image. - ---- - -### S6 — Stamp content matches what was actually deployed - -**Setup:** any state after a successful `task prepare-e2e`. - -**Assert:** -- `image.loaded` contains `gitops-reverser:e2e-local@sha256:` -- `controller.deployed` contains the same value -- the running pod's image matches (via `kubectl get pod -o jsonpath`) - -**Why it matters:** stamps are only useful if they reflect reality. This catches -drift between stamp content and cluster state. - ---- - -### S7 — Load-image is idempotent when digest is unchanged - -**Setup:** cluster in known good state. - -**Action:** touch `cmd/main.go` (update mtime without changing content) — this forces -`controller.id` to rerun `docker build`, but the resulting image should be identical -(same digest, since content is unchanged). - -**Assert:** -- `image.loaded` stamp mtime may update, but stamp content is unchanged (same digest) -- no `rollout restart` was issued (since `controller.deployed` is still up-to-date - relative to `image.loaded`) - -**Why it matters:** validates that the `IMAGE@digest` stamp in `load-image.sh` -correctly short-circuits the import when digest is unchanged. - -> Note: this scenario depends on Docker build cache producing a deterministic digest. -> If the build is not reproducible (e.g. timestamps embedded in the binary), this -> scenario may not hold and should be skipped or marked expected-to-fail. - ---- - -## Decision Tree: Where to Test - -``` -Is this testing Task/stamp dependency semantics (stamps, mtimes, rebuild triggers)? -├── Yes → needs to run `task prepare-e2e` and inspect stamp files -│ → shell script OR Go subprocess test -│ -└── No, this is testing the running cluster state (pod identity, image digest)? - → needs kubectl access and structured assertions - → Go test (Ginkgo) is a better fit -``` - -For this suite, both apply. The recommendation is: - -### Use Go (Ginkgo) in the existing e2e suite - -**Reasons:** - -- The e2e suite already owns cluster setup and teardown. These tests need the same - cluster. Putting them in Go avoids a second setup path. - -- Ginkgo `BeforeEach` / `AfterEach` can handle file modifications and `git checkout` - cleanup reliably, even on test failure. - -- Kubernetes assertions (pod name, start time, image digest) are better expressed - with the typed client than with `kubectl ... -o jsonpath` in shell. - -- Make can be invoked from Go via `exec.Command("make", ...)` and stdout/stderr can be - captured and asserted against (e.g. check for "Restarting deployment" or absence of it). - -- The existing `test/e2e/helpers.go` already has patterns for waiting on pod state. - -**Structure:** - -``` -test/e2e/ - image_refresh_test.go ← new file, Ginkgo suite -``` - -The suite should be tagged or placed in a separate `Describe` block so it can be run -independently: - -```bash -go test ./test/e2e/ -v -ginkgo.v --label-filter="image-refresh" -``` - -or as part of the full suite without filtering. - -**Each scenario maps to a Ginkgo `It` block:** - -```go -Describe("image refresh dependency chain", Label("image-refresh"), func() { - It("S1: no-op run does not restart the pod", func() { ... }) - It("S2: Go source change triggers pod restart", func() { ... }) - It("S3: second Go change also triggers restart", func() { ... }) - It("S4: test-only file change does not trigger rebuild", func() { ... }) - It("S5: Dockerfile change triggers rebuild", func() { ... }) - It("S6: stamp content matches deployed pod image", func() { ... }) -}) -``` - -**File modification helper:** - -```go -func appendComment(t GinkgoTInterface, path, marker string) { - f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0644) - Expect(err).NotTo(HaveOccurred()) - _, err = fmt.Fprintf(f, "\n// image-refresh-test: %s\n", marker) - Expect(err).NotTo(HaveOccurred()) - f.Close() - DeferCleanup(func() { - Expect(exec.Command("git", "checkout", "--", path).Run()).To(Succeed()) - }) -} -``` - -**Make invocation helper:** - -```go -func runPrepare(ctx context.Context) string { - cmd := exec.CommandContext(ctx, "make", - "CTX="+kubeContext, - "NAMESPACE="+namespace, - "INSTALL_MODE="+installMode, - "prepare-e2e", - ) - out, err := cmd.CombinedOutput() - Expect(err).NotTo(HaveOccurred(), string(out)) - return string(out) -} -``` - -**Pod identity helper:** - -```go -func currentPodIdentity(ctx context.Context) (name, startTime string) { - // list pods matching CONTROLLER_DEPLOY_SELECTOR - // return name + startTime of the newest pod -} -``` - -### What NOT to use Go for - -Stamp mtime comparison is awkward in Go (`os.Stat(...).ModTime()`). For scenarios -where the assertion is purely "stamp was not touched", capture make output and assert -absence of "Restarting deployment" — that is more reliable than mtime arithmetic. - -## CI Integration - -Add a Makefile target that runs only the image-refresh suite: - -```make -.PHONY: test-image-refresh -test-image-refresh: ## Validate the Make image-refresh dependency chain - export CTX=$(CTX) - export INSTALL_MODE=$(INSTALL_MODE) - export NAMESPACE=$(NAMESPACE) - export E2E_AGE_KEY_FILE=$(CS)/age-key.txt - go test ./test/e2e/ -v -ginkgo.v --label-filter="image-refresh" -``` - -In CI, run this after cluster setup and before the main `test-e2e` suite, so a broken -refresh chain fails fast with a clear message rather than silently passing with stale -pods. diff --git a/docs/finished/materialization-tail-and-live-readiness-review.md b/docs/finished/materialization-tail-and-live-readiness-review.md deleted file mode 100644 index dbd0e56fc..000000000 --- a/docs/finished/materialization-tail-and-live-readiness-review.md +++ /dev/null @@ -1,492 +0,0 @@ -# Materialization, the freshness tail, and GitTarget liveness — review + plan - -> **SUPERSEDED — moved to `finished/` 2026-06.** A dated `poc/redis-copy` branch review. Its metric -> inventory has been removed rather than corrected (see §6): the names predate two cleanups and -> almost none survive. For the live metric set see -> [`../interpreting-metrics.md`](../interpreting-metrics.md). Kept for the materialization and -> freshness-tail analysis, which still reads true. - -Status: review / proposal · Branch: `poc/redis-copy` · Date: 2026-06-12 - -This document reviews how a resource type goes from *claimed* to *checkpointed* to -*live-tailing*, what is already correct, and where the gaps are after commit -`8f2ad84` ("only backfill-reconcile a type on its first TypeSynced"). It then proposes -how to (a) restore periodic checkpoint healing without stealing commit windows, -(b) make the backfill→tail boundary explicit, and (c) give a GitTarget a clear -`Initializing / PartiallyLive / Live` signal that operators and e2e tests can gate on. - -It is the companion to: - -- [api-source-of-truth-reconcile.md](../architecture.md) — the R-stage - splice/tail architecture. -- `demand-driven-type-materialization-lifecycle.md` - — the demand axis (claims, phases) this builds on. -- `audit-log-ingestion-and-ordering.md` — the - per-type RV-ordered stream and the late lane. - ---- - -## 1. How it works today - -The live path is entirely **per-type stream → checkpoint + tail**. There is no direct -watch→git route anymore: the only caller of `RouteToGitTargetEventStream` is the audit -tail (`internal/watch/audit_tail.go`). - -For one resource type the lifecycle is: - -1. **Ingest** — the audit webhook mirrors every event into a strictly RV-ordered per-type - stream `…:audit:stream` (`RedisByTypeStreamQueue.Enqueue`). An out-of-order event (RV - below the stream high-water) is diverted to a diagnostic `…:audit:late` lane and is - **never replayed** by the ordered log. -2. **Demand** — the GitTarget controller `Declare`s its watched-type set as a renewing - lease; the registry feeds followability in as `LifecycleEvent`s. The `Materializer` - moves a claimed + followable type `Dormant → Requested` and emits `SyncRequested`. -3. **Checkpoint** — the driver LISTs the type into `…:objects:items` and pins the list's - collection revision `R` (`mirrorTypeObjects`), then `SyncSucceeded(gvr, R)` emits - `TypeSynced`. -4. **Backfill** — on `TypeSynced`, `reconcileTypeForSyncedTargets` fans a per-type - **splice resync** (mark-and-sweep; desired = checkpoint@R folded with the log `(R, head]`) - to every watching GitTarget. -5. **Freshness tail** — a per-type goroutine then loops `…:audit:stream` from the anchor - `R-` and applies each later entry as a sweep-free upsert/delete. -6. **Trim** — each successful checkpoint trims the log to `R`. - -Backfill resyncs and tail events land on the **same FIFO queue** per worker -(`BranchWorker.eventQueue`), which is what keeps them ordered. - -**One worker serves N GitTargets.** A `BranchWorker` is keyed by `(provider, branch)`, not by -GitTarget, so every GitTarget writing to the same branch (each under its own non-overlapping -`spec.path`) shares one worker: one FIFO queue, one goroutine, and **one open commit window**. -That window is a worker singleton bound to a single GitTarget at a time — an event for a -different GitTarget finalizes it (`windowFinalizeReasonIdentityChange`) and opens a fresh one. -This sharing is mostly benign (worker teardown is branch-keyed and N:1-safe via the -`WorkerManager` orphan sweep), but it has one sharp consequence for the heal below: because a -resync/heal force-finalizes *whatever* window is open, a heal for GitTarget A can finalize -GitTarget B's window — see Rec 1. - -### 1.1 Component overview - -```mermaid -flowchart LR - subgraph K8s["Kubernetes API server"] - AUD["Audit events"] - end - - subgraph Ingest["Ingestion (internal/queue)"] - WH["Audit webhook"] - Q["RedisByTypeStreamQueue.Enqueue"] - MAIN[("…:audit:stream
RV-ordered log")] - LATE[("…:audit:late
diagnostic lane")] - end - - subgraph Demand["Demand + phase (internal/typeset)"] - REG["Registry
followability"] - MAT["Materializer
Dormant→Requested→Syncing→Synced⇄Resyncing"] - end - - subgraph Driver["Driver (internal/watch.Manager)"] - DRV["driveMaterialization
handleMaterializationEvent"] - MIR["mirrorTypeObjects
LIST @ R"] - OBJ[("…:objects:items @ R
checkpoint")] - TAIL["per-type audit tail
ReadTypeAuditChanges"] - end - - subgraph Route["Reconcile + route (internal/watch + internal/reconcile)"] - SPL["SpliceType
checkpoint + log fold"] - ER["EventRouter
EmitTypeReconcileForGitDest"] - STR["GitTargetEventStream"] - end - - subgraph Git["Git (internal/git)"] - BW["BranchWorker.eventQueue
FIFO"] - REPO[("Git repo")] - end - - CTRL["GitTarget controller
Declare + status roll-up"] - - AUD --> WH --> Q - Q -->|in order| MAIN - Q -->|out of order| LATE - LATE -->|NudgeTypeResyncForLateEvent| MAT - - CTRL -->|Declare claim/renew| MAT - REG -->|LifecycleEvent| MAT - MAT -->|SyncRequested| DRV - DRV --> MIR --> OBJ - MIR -->|SyncSucceeded R| MAT - MAT -->|TypeSynced| DRV - - DRV -->|first sync: backfill| ER --> SPL - OBJ --> SPL - MAIN --> SPL - SPL -->|EnqueueResync desired@R| BW - - DRV -->|startTypeAuditTail anchor R| TAIL - MAIN --> TAIL - TAIL -->|per-event upsert/delete| STR --> BW - BW --> REPO - - MAT -->|Inventory roll-up| CTRL -``` - -### 1.2 Startup → fully synced → tailing - -The same flow as a sequence, to show how many components hand off before a type is -serviceable and the tail is live. (`alt` blocks call out the two behaviours that matter -for the gaps below.) - -```mermaid -sequenceDiagram - autonumber - participant CT as GitTarget controller - participant MAT as Materializer (typeset) - participant DRV as Driver (watch) - participant K8s as API server - participant OBJ as Redis :objects - participant LOG as Redis :audit:stream - participant ER as EventRouter / Splicer - participant BW as BranchWorker (git) - participant TL as Audit tail - - Note over CT: reconcile - validate, encryption, ensure worker, register event stream - CT->>MAT: Declare(gitDest, claimed GVRs) - MAT->>MAT: claim recorded, followable and claimed so Dormant becomes Requested - MAT-->>DRV: SyncRequested(gvr) - - DRV->>MAT: BeginSync(gvr) -- Requested to Syncing - DRV->>K8s: LIST type (mirrorTypeObjects) - K8s-->>DRV: items and collection rv R - DRV->>OBJ: ReplaceTypeObjects(items, R) - DRV->>MAT: SyncSucceeded(gvr, R) -- Syncing to Synced - MAT-->>DRV: TypeSynced(gvr, rv=R) - DRV->>LOG: TrimTypeAuditLog(minID=R) - - Note over DRV: TypeSynced handling, tail not yet running - DRV->>ER: reconcileTypeForSyncedTargets then EmitTypeReconcileForGitDest - ER->>OBJ: read checkpoint at R - ER->>LOG: fold log after R - ER->>BW: EnqueueResync(desired, revision=R) - Note over BW: backfill committed (mark-and-sweep), system-authored - - DRV->>TL: startTypeAuditTail(anchor = R-maxuint64) - loop freshness - TL->>LOG: ReadTypeAuditChanges(after lastID, block) - LOG-->>TL: new entries (rv after R) - TL->>BW: per-event upsert/delete (real author) - end - Note over BW,TL: backfill enqueued BEFORE tail starts, one FIFO queue, no sweep race - - rect rgb(255, 244, 224) - Note over MAT,DRV: Later - periodic sweep re-anchor or late-event nudge - MAT-->>DRV: SyncRequested then re-LIST then SyncSucceeded(R2) then TypeSynced(R2) - alt tail already running (current behaviour, post-8f2ad84) - DRV-->>DRV: re-splice SKIPPED, refreshed checkpoint never reaches git - else proposed heal mode (Rec 1) - DRV->>ER: heal resync (non-finalizing, drift only) - ER->>BW: EnqueueResync(heal) corrects orphans and late events, keeps open window - end - end -``` - ---- - -## 2. What is already correct (do not "fix" these) - -Two instincts about live events are *already satisfied at first sync*, which is worth -knowing before changing anything: - -- **"Store the RV we start at"** — already done. The tail anchors at the checkpoint - revision `R` (`auditTailAnchor`), and `SpliceType` returns that same `R`. The backfill - owns `≤R`; the tail owns `>R`. -- **"Filter live events until the reconcile is done"** — at first sync this holds *by - construction*: `reconcileTypeForSyncedTargets` enqueues the backfill resync - **synchronously before** `startTypeAuditTail` runs, and both share one FIFO worker - queue. So the resync is processed before any tail event, and the mark-and-sweep cannot - delete a not-yet-created object (its file does not exist yet). **There is no sweep race - at first sync.** - -So the first-sync path is sound. The real problems are elsewhere. - ---- - -## 3. The gaps - -### Gap 1 — periodic checkpoint healing no longer reaches git (HIGH; the `8f2ad84` regression) - -After `8f2ad84`, **any `TypeSynced` while the tail is running skips the re-splice** -(`handleMaterializationEvent`, the `!isAuditTailRunning` guard). The periodic sweep -re-anchor and the late-event nudge both refresh the Redis checkpoint and **trim the -log**, but the corrected state is never reconciled into git. Lost until a process restart -or a fresh first-sync (which will not happen while the tail runs): - -- **Late-lane / out-of-order events** — `NudgeTypeResyncForLateEvent → RequestResync →` - checkpoint refresh `→ TypeSynced →` *skipped*. The nudge is now effectively a no-op for - git. -- **`deletecollection`** — name-less, skipped by both the tail and the splice's log-fold; - only a full LIST reflects it. Disabled. -- **Orphans / missed deletes** — the checkpoint sweep's entire stated purpose - ("CORRECTNESS … is the checkpoint sweep's job") no longer applies to git. -- **The trim now runs without its backstop.** `TrimTypeAuditLog`'s safety argument is "a - reconcile that finds the log trimmed past its cursor re-reads the checkpoint" — but that - re-read *is* the re-splice `8f2ad84` disabled. A tail that stalls (Redis backoff) across - a re-anchor can have unread entries trimmed away with nothing to recover them. - -The tail keeps git fresh for in-order live edits, but the authoritative full-LIST -reconcile no longer corrects drift. **This is the thing to get right.** - -### Gap 2 — a failed first backfill leaves the tail running ahead of an un-backfilled target (MEDIUM-HIGH) - -`startTypeAuditTail` runs unconditionally even if a per-target backfill errored or was -skipped. The tail is keyed per **type**, so once any first-sync starts it, it is "running" -for *all* GitTargets. A GitTarget that claims an already-Synced type gets its backfill from -the `Declare` path — but if that `EmitTypeReconcileForGitDest` fails, it is only logged, -and the next reconcile will not retry it because `newlyDeclaredSyncedGVRs` already recorded -the type as declared. With Gap 1, no re-anchor heals it. Result: a transient backfill -failure = permanent per-target hole for objects that existed at checkpoint and never -change again. - -### Gap 3 — GitTarget reports `Ready=True` before any type is Synced (MEDIUM; the status ask) - -`evaluateEventStreamGate` flips `Ready=True` the moment the worker/stream is wired. The -materialization roll-up (`Claimed/Synced/Pending/Failing/NotFollowable`) is computed and -stored but **does not gate readiness or surface a phase**. There is no -init / partial-live / live distinction, so neither an operator nor an e2e test can tell -"all claimed types are actually mirrored." The data exists — it just is not projected into -a clear signal. - -### Gap 4 — the backfill/tail boundary is implicit and slightly noisy (LOW) - -It works by call-ordering + FIFO, not by an explicit gate, so any future refactor that -makes the splice async or starts the tail earlier silently reintroduces a sweep race. -Also, because the splice folds to `head` but the tail anchors at `R`, the `(R, head]` -window is applied **twice** at first sync — once by the system-authored backfill commit, -once by the tail with real authorship. Idempotent in content, but it produces redundant -commits. - -### Gap 5 — the checkpoint always plain-LISTs; the streaming WATCH is design intent only (MEDIUM) - -The demand-driven checkpoint fill (`mirrorTypeObjects`) does an unconditional -`dynamicClient.Resource(gvr).List(ctx, metav1.ListOptions{})`. There is **no** WATCH with -`sendInitialEvents=true` anywhere in the codebase — that string appears only in a stale -doc-comment on `typeset/materializer.go`, and the functions it names -(`StreamSnapshotForType`, `StreamClusterSnapshotForGitDest`) no longer exist. So the -intended "stream the initial events via a watch-list, fall back to a consistent LIST only -when the server can't" behaviour is **not** implemented; every checkpoint and every -re-anchor is a full LIST. For a large type on a busy cluster that is a heavier, less -incremental call, and it forgoes the bookmark-based `initial-events-end` revision the design -assumes for the anchor `R`. - -### Gap 6 — the materialization roll-up mis-buckets serviceability, and liveness is duplicated in test code (MEDIUM) - -`MaterializationSummaryForGitTarget` counts `Resyncing` as `Pending` and counts only -`PhaseSynced` as `Synced`. But a `Resyncing` type **still serves its prior checkpoint**, and -a `Failing` type with a prior checkpoint does too — they are live. So the roll-up's notion of -"synced" undercounts what is actually serviceable, and "pending" overcounts it. Consequences: - -- Any liveness signal built on `pending == 0 && synced == claimed` **flaps on every periodic - re-anchor** (Synced→Resyncing→Synced), even though nothing was ever unavailable. -- The fast-requeue `materializationSettling = sum.Pending > 0` - (`gittarget_controller.go`) fires on routine re-anchors, churning reconciles. -- The signal is **already duplicated in test code** — `waitForGitTargetMaterializationSettled` - re-derives the same buggy predicate from raw counts instead of reading one first-class - status field. - -The correct predicate is **serviceability** = `checkpointRV != ""` (true for `Synced`, -`Resyncing`, and `Failing`-with-prior-checkpoint), not `phase == Synced`. Separately, -`status.snapshot` (`GitTargetSnapshotStatus`) is **vestigial** — nothing has written it since -the pre-R2 bootstrap gather was removed — and should be deleted. - ---- - -## 4. Recommendations - -### Rec 1 — Restore periodic healing, deferred until the commit window is idle (fixes Gap 1; the headline) - -The reason re-anchor re-splices were disabled is that **every resync finalizes the open -commit window first** (`handleResyncRequest → finalizeOpenWindowWithReason`, -`windowFinalizeReasonResyncBeforeApply`) — so a re-anchor stole a user's held CommitRequest -window. Because **one worker serves N GitTargets** (§1.1) and the window is a worker -singleton, this is worse than same-GitTarget: a heal for GitTarget A finalizes whatever window -is open, so it can steal GitTarget **B**'s CommitRequest window. The gate therefore has to be -**worker-level** ("is *any* window open?"), which is exactly what a worker-singleton window -makes natural. - -The simplest correct fix is **not** to build elaborate "don't influence the window" -machinery, but to **defer the periodic heal until the worker is idle** — apply it only when -no commit window is open (`openWindow == nil`). The application is allowed to do its own -housekeeping; it just waits for a quiet moment to do it. This is safe from starvation -because a window always self-finalizes: the rolling commit-window silence timer -(`DefaultCommitWindow` = 5s, `windowFinalizeReasonTimer`), the buffer-size backstop -(`windowFinalizeReasonBufferLimit`) under sustained load, and a CommitRequest's own -`finalizeAt = receipt + delaySeconds`. So a deferred heal always gets a turn within a -bounded time, and when it runs there is, by definition, no window to steal and no live edit -to re-attribute. In steady state the heal finds git already matching desired and commits -nothing; it writes a (system-authored) commit only on genuine drift — exactly an orphan, a -`deletecollection`, or a late-lane event the in-order tail could not express. - -Concretely: - -- give the re-anchor resync a `Heal` kind that the worker **defers** while a window is open - (stash one pending heal; apply it at the next window-finalize boundary) instead of - force-finalizing; -- re-enable the re-anchor re-splice (drop the `!isAuditTailRunning` skip) routed through - that heal kind. - -The deferral must be **internal to the worker**, not an external `HasOpenWindow()` the driver -polls before enqueuing: an external check is a TOCTOU race against the FIFO (the window state -changes between the check and the heal's dequeue, so it would not actually prevent the steal). -The worker, dequeuing a `Heal` item, checks `openWindow != nil` itself and applies the heal at -the next boundary where no window is open. That boundary comes up frequently — every silence -timeout *and* every identity switch between the N GitTargets — so the heal never starves, even -under sustained multi-GitTarget load, and it never force-finalizes. (An external idle query is -fine as an *observability* signal — a "worker busy" gauge — but not as the heal's gate.) - -This restores the checkpoint's correctness role with minimal new code and **subsumes most of -Gap 2** — a heal re-anchor recovers a failed first-backfill within one sweep interval. -Ordering stays correct: a heal that runs at idle has no window to reorder against, and a heal -deferred past an open window lands strictly after that window's commit — the same arrival -order the old force-finalize preserved. (A fully non-finalizing, drift-only resync that runs -*concurrently* with an open window is the more complex alternative; it is not worth the extra -machinery given that windows self-finalize quickly.) - -### Rec 2 — Make the "tail waits for backfill" boundary explicit (fixes Gap 2; the live-event filter ask) - -Today it is emergent. Make it a precondition: only start/deliver the tail for a type once -its first backfill has been **enqueued without error for every current watcher**, and -retry a failed per-target backfill instead of recording it as "declared." This closes the -failed-backfill hole and makes the invariant robust to refactors. Once Rec 1 lands this is -belt-and-suspenders, but it is cheap and removes the fragility. - -**Implementation note (landed):** the **retry half** is implemented — a failed Declare-time -backfill is un-recorded (`forgetDeclaredGVR`) so the next reconcile re-classifies the type as -newly-declared and re-attempts it, and the Declare path no longer starts the per-type freshness -tail for a target whose backfill just errored. The broader "tail does not *deliver* to an -un-backfilled target" per-target gate was deliberately **not** added: with Rec 1 landed, the -periodic re-anchor heal re-fans the reconcile to *every* watcher (~1h backstop) and the next -reconcile retries within minutes, while the tail's per-event apply is benign for a momentarily -un-backfilled target — an upsert simply creates the file, and a delete for an object the target -never held is a no-op. The only genuine hole is unchanging baseline objects, which the retry and -the heal both fill; tracking per-(GitTarget, type) tail-delivery state to gate that narrow, -self-healing window is not worth the new state and the risk of dropping legitimate events. - -### Rec 3 — (Optional) capture one explicit fold tip and anchor the tail there (the start-RV ask, cleaner) - -Have the backfill return the concrete stream tip `H` it folded to and anchor the tail at -`H` instead of `R`. That makes the partition exact (`backfill = [0, H]`, -`tail = (H, ∞)`), eliminating the double-commit of `(R, head]`. It is a moderate refactor -(the materializer's `TypeSynced` carries `R`, not `H`, so the splice would have to hand -`H` back to the tail-start call). Treat as follow-up polish, not a correctness fix — -`R`-anchoring is the *safe* choice (overlap is idempotent; a gap would not be), so do not -rush it. - -### Rec 4 — Add a serviceability-based liveness phase to GitTarget status, orthogonal to Ready (fixes Gaps 3 + 6; the status ask) - -**`Ready` should NOT depend on liveness.** The reason is not backward-compat (there are no -end users) — it is that one boolean cannot honestly represent the four states a GitTarget can -be in: (1) config/plumbing broken, (2) plumbing fine but mirror still building, (3) mirror -fully current, (4) mirror as-current-as-possible **but a claimed type is not followable** (a -CRD that is not installed, or a typo in a WatchRule). State 4 is decisive: folding liveness -into `Ready` makes such a GitTarget `NotReady` *forever* despite being correctly configured -and mirroring all it can — and excluding not-followable types from the gate instead makes -`Ready=True` silently ignore a type the user asked to watch. Only two orthogonal axes tell the -truth (`Ready=True` + `Synced=Degraded, 1 type not followable`). This is the Deployment -`Available` vs `Progressing` split. - -Keep `Ready` = control-plane correctness (validated + encryption + stream wired), unchanged -and non-flapping. **Add** a data-plane signal computed in the controller from a -**serviceability-based** roll-up (Gap 6 fix): a type is *serviceable* when -`checkpointRV != ""` (i.e. `Synced`, `Resyncing`, or `Failing`-with-prior), evaluated over the -**followable, claimed** types only: - -| Phase | Meaning | -| --- | --- | -| `Initializing` | ≥1 followable claimed type not yet serviceable (first sync in flight) | -| `Synced` | all followable claimed types serviceable (fully live) | -| `Degraded` | serviceable ones are serviceable, but ≥1 claimed type is not followable, or failing without a checkpoint | - -Surface it as **both** a condition and a `status.phase` string (good printcolumn). **Name the -condition `Synced` (or `Materialized`), not `Live`** — `EventStreamLive` already uses "Live" -for "stream wired," so `Live` would collide. Tests then `kubectl wait --for=condition=Synced` -and the bespoke `waitForGitTargetMaterializationSettled` helper is deleted (its logic, and its -latent re-anchor flap, move into the controller and are fixed there). Because `Resyncing` -counts as serviceable, a periodic re-anchor does **not** flap the phase. Also **delete the -vestigial `status.snapshot`** while here. The full two-axis status contract (conditions, -phase derivation, reasons, tests) is written up in -[../status-design-git-target.md](../spec/status-conditions-guide.md). - -### Rec 5 — Tests to add - -- **watch (regression for `8f2ad84`):** a re-anchor while the tail is running heals git - (orphan deleted / late-event folded) **and** leaves an open CommitRequest window intact — - the test that would have caught the tradeoff. -- **git:** a deferred heal resync waits while a commit window is open and applies once it - finalizes; it never finalizes the open window or resolves a pending CommitRequest — - including the **shared-worker** case, where a heal scoped to GitTarget A must not steal - GitTarget B's open CommitRequest window on the same branch worker. -- **watch:** a failed first backfill is retried; the tail does not deliver to an - un-backfilled target. -- **queue/watch:** a late-lane (out-of-order) event reaches git within one sweep interval - via heal. -- **controller:** status phase `Initializing → Synced` as checkpoints land; a periodic - re-anchor (Synced→Resyncing→Synced) does **not** flap the phase; a claimed-but-not-followable - type yields `Synced=False, phase=Degraded` while `Ready` stays `True`. -- **e2e:** a `deletecollection` is reconciled into git within one sweep; a GitTarget - reports `Live` only after all claimed types synced. - -### Rec 6 — Implement the WATCH-first checkpoint with LIST fallback (fixes Gap 5) - -Make `mirrorTypeObjects` open a streaming watch-list (`sendInitialEvents=true`, -`resourceVersionMatch=NotOlderThan`, `allowWatchBookmarks=true`), fold the initial `ADDED` -events into the checkpoint, and pin the `initial-events-end` bookmark revision as the -anchor `R`. Fall back to the current consistent LIST **only** when the server rejects -`sendInitialEvents` (e.g. an aggregated apiserver) — detected from the watch error, not -chosen blindly. Until that lands, fix the stale `materializer.go` comment so it stops -describing behaviour that is not there. This is independent of Recs 1–5 and can be done on -its own. - -**Status (landed / deferred):** the **stale comment is fixed** — `materializer.go` now describes the -actual consistent-LIST fill and frames the watch-list as the planned optimisation, instead of naming -`StreamSnapshotForType` / `StreamClusterSnapshotForGitDest` (functions that do not exist) and -behaviour that is not there. The **watch-first LIST replacement itself is deliberately deferred**: -it is the lowest-priority item here ("whenever LIST cost matters"), no current consumer needs the -bookmark-pinned `R` (the LIST collection rv works), and — decisively for "test both paths" — the -`client-go` dynamic **fake client cannot simulate a `sendInitialEvents` watch-list** (no initial -`ADDED` replay, no `initial-events-end` bookmark), so a watch-first `mirrorTypeObjects` would block -forever in the ~10 existing fake-client tests that drive it, and new tests could only exercise a -hand-rolled synthetic watcher (testing our fold/bookmark consumption, not real server behaviour). -Shipping an under-tested watch loop that can hang the controller, for an optimisation nothing needs -yet, is the wrong trade. **A future implementation needs:** (1) `streamTypeObjects(ctx, dc, gvr)` -that folds `ADDED` until the `initial-events-end` bookmark and returns `(items, bookmarkRV)`; -(2) a `sendInitialEventsUnsupported(err)` classifier driving the LIST fallback; (3) a shared test -helper that installs a `PrependWatchReactor` on the fake client to replay tracker objects as initial -`ADDED` + an `initial-events-end` bookmark, which the existing `mirrorTypeObjects` tests migrate to -so they exercise the watch path instead of hanging; (4) an e2e assertion against a real apiserver. - ---- - -## 5. Priority - -1. **Rec 1** restores correctness and is the right way to "get this part right" — it is - also what makes the late-event tradeoff disappear. -2. **Rec 4** is independent and small, and directly serves the testability described. -3. **Recs 2 / 3** are hardening; **Rec 6** (WATCH-first checkpoint) is independent and can - slot in whenever LIST cost becomes a concern. - -Suggested order: Rec 4 (self-contained, unblocks tests) → Rec 1 (deferred-until-idle heal + -re-enable re-anchor healing) → Recs 2/3 → Rec 6. - ------- - -## 6. Metrics exposed today (reference) - -**Removed.** This section carried a full inventory of the instrument set as it stood in June 2026. -Almost none of those names survive: the June cleanup removed some, and the September 2026 rebuild -renamed or merged most of the rest. A stale inventory in a superseded document is worse than no -inventory, because it is what a grep for a metric name finds first. - -The live set is [`../interpreting-metrics.md`](../interpreting-metrics.md); the reasoning behind its -shape is [`../design/metrics-observability-plan.md`](../design/metrics-observability-plan.md). - -The one observation here that outlived its table: instruments are registered with a **name only** — -no `WithDescription` or `WithUnit` — so `/metrics` carries no HELP or UNIT metadata. That is still -true, and still a cheap separate improvement. diff --git a/docs/finished/multi-cluster-author-attribution.md b/docs/finished/multi-cluster-author-attribution.md deleted file mode 100644 index 212496e50..000000000 --- a/docs/finished/multi-cluster-author-attribution.md +++ /dev/null @@ -1,194 +0,0 @@ -# `ClusterProvider` multi-cluster author attribution: decision record - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current -> behaviour see [`../architecture.md`](../architecture.md), [`../configuration.md`](../configuration.md), -> and [`../../SECURITY.md`](../../SECURITY.md). Index: [`../INDEX.md`](../INDEX.md) -> -> Produced the cluster-scoped `ClusterProvider`, immutable `GitTarget.spec.clusterProviderRef`, -> provider-name-partitioned attribution facts, reconcile-time namespace authorization, and named -> audit routes. Remaining ingress work: -> [`../design/multi-source-audit-ingress-hardening.md`](../design/multi-source-audit-ingress-hardening.md). -> Purge-on-delete was answered separately and rejected, and its record has since been removed. The -> reasoning that still binds lives on `LegacyClusterProviderFinalizer` in -> [`kubeprovider_controller.go`](../../internal/controller/kubeprovider_controller.go): no -> finalizer, because `helm uninstall` would strand the object in `Terminating`. - -## The problem - -Author attribution is a **join keyed by `(group/resource, object-uid, resourceVersion)`** with no -cluster dimension. The write side records a fact per accepted mutation from the local apiserver's -audit stream ([`audit_handler.go`](../../internal/webhook/audit_handler.go) → `RecordFact`); the -read side resolves it back on each live watch event -([`author_resolver.go`](../../internal/watch/author_resolver.go)). - -Config-plane split broke the symmetry: a remote `GitTarget` watches a **remote** cluster (remote -UIDs and RVs) while the audit webhook was **local-only**. A remote watch event found no fact and -shipped as the committer. - -Two things were missing, and both needed a **name for the cluster**: an ingress a remote apiserver -can reach, tagged with which cluster it is, and a cluster dimension in the keyspace so a fact from -cluster A never joins an event from cluster B. - -## What shipped - -`ClusterProvider` is the cluster-scoped read-side peer of `GitProvider`. A `GitTarget` references -one by immutable name; that name partitions the source client, discovery and watch state, and the -attribution facts. `spec.allowedNamespaces` is deny-by-default and is checked on every reconcile, -before any watch starts. - -```text -GitTarget ─ spec.clusterProviderRef ─▶ ClusterProvider (source: READ + attribution, authorized per namespace) - ─ spec.providerRef ────────▶ GitProvider (destination: WRITE) -``` - -Audit ingestion uses `/audit-webhook/` and stores facts in that provider's partition. The -server verifies the sender is signed by the audit CA and that the named provider exists; it does -**not** bind an individual certificate to a provider. That is an accepted shared-control-plane -trust boundary, not tenant isolation — see the hardening doc. - ---- - -## The decisions, and what each one replaced - -### The name is the identity; the UID only authenticates - -`GitTarget.SourceClusterID()` (`//`) was **deleted, not adapted**. It fused -two identities and leaked a Secret reference. One identity replaced it — the provider **name** — -used for the audit route, the fact-index key, the GVK→GVR registry key, and the `clusters` map key. -`git.Event.SourceClusterID` became `SourceCluster`, carrying the name. - -An earlier revision keyed facts by **UID**. Reversed, for two reasons: - -1. **UID does not deliver the incarnation safety it appears to.** The audit *route* is name-based, - so a delayed batch from a deleted cluster still hits `/audit-webhook/prod-eu-1` and resolves to - the *current* provider. UID-in-the-key does not stop that; only rejecting the stale sender does. -2. **The name needs no translation.** It is already on the audit path and already known to the - watch side. UID-keying would add a `name → uid` lookup on *both* paths for no gain, and make - Redis keys opaque. - -### `default` is a defaulting convention, **not** a reserved local name - -This one was reversed late and matters most, because the rejected version is the intuitive one. - -An earlier draft made `default` **reserved for the in-cluster cluster**, enforced by CEL -`(self.metadata.name == 'default') == !has(self.spec.kubeConfig)`, with the chart shipping the -object. **None of that shipped.** - -What shipped: local-vs-remote follows from **`spec.kubeConfig` alone** — omitted means the -operator's own cluster, for *any* name -(`clusterprovider_types.go`, since replaced by [`kubeprovider_types.go`](../../api/v1alpha3/kubeprovider_types.go)). A -provider named `default` may carry a `kubeConfig` and mirror a remote cluster; a provider named -anything else may omit it and be in-cluster. `default` is only the value that -`GitTarget.spec.clusterProviderRef` defaults to -([`gittarget_types.go:98`](../../api/v1alpha3/gittarget_types.go#L98)), and the operator never -creates that object — a user does. - -Two things drove the reversal. Pinning a *name* to a physical cluster is the same -silent-retarget hazard that `kubeConfig` immutability exists to prevent — it just moves it into the -schema. And the reserved name bought nothing the existence check did not already give: a -`GitTarget` naming a provider that does not exist is held unready through the ordinary -"provider not found" path, so turning local watching off needs no special case. - -The **ref is defaulted, never `nil`**, which did survive: every reference is populated and jumpable, -so a "follow reference" traversal never hits an implicit hop with nowhere to land, and -`kubectl get gittarget -o yaml` is self-describing. An `isDefault` **flag was rejected** — a movable -default would reintroduce the retarget hazard directly. - -> Downstream consequence: locality is derived from `kubeConfig`, never from the name, and -> `GitTarget.IsLocalSource()` — despite the name — **is a name test** that only seeds the -> pre-discovery `SourceClusterReachable` default. It is not a locality predicate. The source-namespace work later took -> the further step of not deriving *permissions* from locality at all: a -> ClusterWatchRule's attachment to a GitTarget is authorized explicitly, never inferred from the -> two being in the same cluster. - -### Namespace authorization enforced once, on reconcile - -The confused deputy to close is **data export, not credential read**. A cluster-scoped provider -holds a credential that can read a lot of a remote cluster; any `GitTarget` referencing it makes -the operator mirror that state into the target's destination. So a tenant who can create a -`GitTarget` *and their own `GitProvider`* could export whatever the operator can read into a repo -they control. Pinning the kubeconfig Secret to the operator namespace does **not** help — the -tenant never touches the Secret. (An earlier claim that Secret-pinning "closes the confused deputy -by construction" was wrong and is retracted.) - -The fix is `spec.allowedNamespaces` on the provider, **deny-by-default** (empty admits nobody), -names and/or a label selector, ORed. - -This draft specified enforcement **"in two places"**, one of them a validating admission webhook. -**One shipped**, on every reconcile, returning before `DeclareForGitTarget` -([`gittarget_controller.go:311`](../../internal/controller/gittarget_controller.go#L311)) — which -also covers a policy tightened *after* the `GitTarget` was created, something admission cannot see. -That reasoning was generalized into a repo-wide rule: -[`../spec/where-validation-lives.md`](../spec/where-validation-lives.md). - -### Keyspace and the recorder API - -A `cluster:` infix on `factKeyExact/Last/RV`, the in-cluster provider included, so there is no -`""` special case. The rv-only hatch became `(cluster, group/resource, rv)` — **a correctness fix**, -since RV is not globally unique and the no-UID hatch was cross-cluster-ambiguous without it. -`RecordFact` gained an explicit `providerName` argument; `ResolveAuthor` / `LookupAuthorResolution` -the same. - -### Admission attribution: cut entirely - -An earlier draft sold a remote `ValidatingWebhookConfiguration` as the *"easy, identical-identity"* -fallback for managed clusters. Wrong on both counts, and removed from the enum: - -- **Not identical.** A `ValidatingWebhookConfiguration` configures the webhook URL and CA, not a - client identity for the *apiserver*. For the apiserver to authenticate *to* the webhook needs an - `AdmissionConfiguration` via `--admission-control-config-file` — an apiserver flag. -- **Not a managed-cluster unlock.** That flag is the same class of thing managed control planes - hide. [`attribution-setup-guide.md`](../attribution-setup-guide.md) already scopes attribution to - self-managed control planes. - -So it bought an *unauthenticated* callback on exactly the clusters where audit also fails, plus a -fail-open webhook that risks blocking tenant writes, plus a weaker (`:last`-only, no post-write RV) -join. The honest managed-cluster answer is a **source-side agent** — a separate, larger feature. -Command authorship (`/validate-operator-types`) is untouched by this and is unrelated. - -### Timing is a prerequisite, not a bug - -Exact attribution needs the fact inside the resolver's grace window -([`DefaultAttributionGraceWindow`](../../internal/watch/author_resolver.go) = 3s), while the -apiserver's `--audit-webhook-batch-max-wait` **defaults to 30s**. Not new — the local path always -relied on the same relationship, and e2e sets `1s`. Made explicit and per-provider here. A missed -fact degrades to a committer commit — never wrong, just less rich — so this is a freshness SLO. - -### Status: a quiet cluster is not unready - -`Validated` / `Reachable` / `DiscoveryHealthy` on the provider, aggregated into `Ready`, projected -onto `GitTarget` as a one-line `ClusterProviderReady` (the `GitProviderReady` pattern) so -per-cluster detail lives once on the shared object. Attribution health is a **`lastAuditEventTime` -timestamp**, deliberately *not* an `AuditIngestionActive` condition — a normally-quiet cluster with -no recent mutations must not read as unready. - ---- - -## Deferred, and why - -| Deferred | Why | -|---|---| -| Per-provider client-certificate binding | Real gap; the current CA-only check is a shared-control-plane boundary. Moved whole to [`../design/multi-source-audit-ingress-hardening.md`](../design/multi-source-audit-ingress-hardening.md). | -| Admission attribution | Cut entirely (above) — needs a source-side agent, not a webhook. | -| Mutable `kubeConfig` / endpoint repointing | v1 promised it; a mutable *endpoint* silently retargets and misattributes. `kubeConfig` is immutable; Secret **contents** rotation stays transparent. | -| Workload identity (`configMapRef`), ServiceAccount impersonation | Present in the embedded Flux type; rejected by CEL for now. | -| Fact purge on delete | Answered separately and **rejected**. No finalizer: `helm uninstall` would strand the object in `Terminating`. See the `KubeProviderReconciler` doc comment in [`kubeprovider_controller.go`](../../internal/controller/kubeprovider_controller.go). | -| Managed control planes (EKS/GKE/AKS) | Needs the source-side agent. | -| `ReferenceGrant`-style per-namespace consent | `allowedNamespaces` is one object for the admin; revisit if self-service consent is wanted. | - -## Why this was config-plane-split's planned step - -Config-plane split argued against a dedicated CRD, but conditionally: its remaining benefits — -reuse across many `GitTarget`s, platform-admin RBAC ownership, a home for connectivity status — -were *"real but not needed for the first version"*, and its load-bearing rationale was gone because -`main` had removed the `/audit-webhook/` path. Re-adding attribution restored the audit -story and made all three benefits needed. It even pre-drew the shape: a sibling ref naming a CRD -*"that only platform admins may create … referenced by name from the `GitTarget`."* - -Moving the kubeconfig from inline into the provider embeds the **same** -`meta.KubeConfigReference`, so the Flux-shaped roadmap survives intact (`value`→`value.yaml` key -order, reject-not-strip `exec`/insecure-TLS, target-scoped GVK→GVR, split -`Validated`/`SourceClusterReachable`). The object also **unlocked** what config-plane split parked -for lack of a home — per-provider `qps`/`burst`, per-provider attribution mode, and a real status -surface. The one departure from verbatim-Flux reuse: the cluster-scoped `secretRef` needs a -namespace Flux's type lacks, pinned to the operator namespace. diff --git a/docs/finished/redis-key-schema-v3.md b/docs/finished/redis-key-schema-v3.md deleted file mode 100644 index ee04bed71..000000000 --- a/docs/finished/redis-key-schema-v3.md +++ /dev/null @@ -1,421 +0,0 @@ -# Redis key schema v3 — author and watch namespaces - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -> Status: **IMPLEMENTED** — 2026-06-29. The `v2 → v3` key-format bump landed for the live Redis families: -> the audit author index, the watch-resume cursor, and the new command-author **store** (storage -> mechanics only — the CommitRequest admission/controller rewrite in -> `commitrequest-admission-authorship.md`, since removed, is a separate round and -> is **not** wired up yet). Scope: (1) replace the opaque `attr` namespace with a top-level `author` -> domain; (2) re-lay audit-sourced resource author facts as a readable, **object-grouped** hierarchy -> (`:object::` + `:last`, plus a type-scoped `:rv:` escape hatch), carrying -> name/namespace in the value; (3) put command authorship from admission in the same `author` domain but -> a separate `command` subfamily; and (4) replace the old `watch-cursor` family with a smaller `watch` -> domain whose leaf is the per-shard `last-rv`. -> -> **Implementation note (resolver / rv: hatch).** §4.2 reads as if an exact-capable event consults -> *only* `object::`. As shipped, an exact-capable event tries the exact key **and then** the -> type-scoped `rv:` key (still never `:last`). This is what makes the `rv:` escape hatch reachable — -> §5's own parenthetical ("…the watch side always carries a UID, so it would resolve via -> `object::…` first and never reach it") confirms `rv:` sits in the fallback chain. Without that -> fallback the `rv:` key would be dead. `rv:` is type-scoped and per-write, so consulting it does not -> reintroduce the stale-LWW hazard `:last` would. -> Related: -> `internal/queue/attribution_index.go`, -> [`internal/queue/redis_store.go`](../../internal/queue/redis_store.go), -> [`internal/watch/author_resolver.go`](../../internal/watch/author_resolver.go), -> [attribution](../spec/attribution.md), -> [watch event ordering & attribution grace](../spec/watch-event-ordering-and-attribution-grace.md), -> [watch-first ingestion architecture](watch-first-ingestion-architecture.md). - -## 1. Today (`v2`) and the companion plan - -Two live key families share the `gitops-reverser` prefix today. The CommitRequest admission plan adds a -third family unless it is folded into this naming pass before implementation: - -| Family | Owner | Key (`:`-joined, fields escaped) | TTL | -|---|---|---|---| -| Audit-sourced resource author fact | `AttributionIndex` | `gitops-reverser:attr:v2::::::[:]` | 10 min (fact), 20 min (`:seen`), 20 min (`:miss`) | -| Watch cursor | `RedisStore` | `gitops-reverser:watch-cursor:v2:::::` | 1 h | -| Command author (planned) | `CommandAuthorStore` | `gitops-reverser:command-author:v1:` in the companion sketch | 1 h | - -Older `docs/finished/` material mentions per-type Redis streams, object snapshots, and demand gates from -the retired `poc/redis-copy` line. Those are not live Redis owners in this branch, so this proposal does -not assign them new keys. - -The audit author family writes **up to three keys per audit event**, strongest first -(`factKeyVariants`): - -- `e` — **exact**: `(group, resource, ns, name, uid, rv)` — only when both uid *and* rv are known. -- `u` — **uid-only**: `(group, resource, ns, name, uid)` — only when uid is known. -- `r` — **rv-only**: `(group, resource, ns, name, rv)` — only when rv is known. - -A watch event's resolver reads the same three variants in the same order and takes the first hit. In v2, -each fact key carries a `:seen` tombstone (2× TTL — lets a late lookup return `expired` vs `absent`) and a -`:miss` marker (lets a late `RecordFact` emit `op="late"`). The v3 proposal deletes both. Correctness -moves into the resolver (§4.2), and late-arrival metrics become a separate observability problem rather -than extra Redis keys in this schema. - -Two real `:seen` tombstones, verbatim: - -``` -gitops-reverser:attr:v2:u::configmaps:1782713884-test-commit-author:commit-author-oidc-cm-1782713884:bbdecba6-4d33-40ee-8007-8269e28e799c:seen -gitops-reverser:attr:v2:u:apps:deployments:1782713884-test-commit-request-bundle:bundle-deploy-a-1782713884:bb1c4d09-4dce-4cc2-be34-f9bff17406c4:seen -``` - -Note the `u::configmaps` double-colon: the core group is empty, so it occupies an empty key field. The -key is wide, redundant (ns+name are also on the value-to-be), and the variant prefix `e/u/r` is opaque. -`attr` has the same problem at the family level: it is short, but it hides the thing Redis is actually -storing. The clearer root is **`author`**. `audit` belongs one level below it as provenance, not as the -top-level domain. - -## 2. The proposal — `author:v1:audit`, grouped under the object - -The same three join keys as v2, but **grouped under the object** and made self-describing. Everything -that was a key field except the type, the UID, and the RV moves into the value. - -``` -# exact — one immutable fact per write (uid + rv) -gitops-reverser:author:v1:audit:apps/deployments:object:bb1c4d09-4dce-4cc2-be34-f9bff17406c4:101 -# uid latest — last-writer-wins pointer for the object (the RV-mismatch fallback) -gitops-reverser:author:v1:audit:apps/deployments:object:bb1c4d09-4dce-4cc2-be34-f9bff17406c4:last -# rv-only — type-scoped escape hatch for a fact that has an RV but no UID (see §5) -gitops-reverser:author:v1:audit:apps/deployments:rv:101 -``` - -- **GroupResource** = `resource` for core (`configmaps`, `secrets`), `group/resource` otherwise - (`apps/deployments`, `rbac.authorization.k8s.io/roles`). Readable, `SCAN`-friendly per type. -- **`object::…`** groups every fact for one object under one prefix — `SCAN - author:v1:audit:apps/deployments:object::*` shows the whole history-in-flight for that object. -- **`author:v1`** is the durable naming decision. `audit` is the source-specific subfamily for - persisted-resource authorship; command authorship uses `author:v1:command` (§9). -- **`:`** is the exact per-write fact; **`:last`** is the latest-writer pointer. -- **`rv:`** is the rv-only fallback, sitting under a `rv:` sibling of `object:` (no UID in it), - **type-scoped** because RV is not globally unique (§5). Shipped from the start, not deferred. -- **Both `:last` and `:` hold the full author fact** — `:last` is just a copy of the most recent - write, so the resolver never needs a second round-trip. (In the end that *is* what `:last` is.) -- **Value** carries what the key no longer encodes: `groupResource`, `namespace`, `name`, `uid`, plus - the existing author/evidence fields. - -```jsonc -// Audit AuthorFact in author:v1:audit (added fields marked +) -{ - "groupResource": "apps/deployments", // + - "namespace": "team-a", // + - "name": "web", // + - "uid": "bb1c4d09-…", // + - "author": "alice@example.com", - "displayName": "Alice", "email": "alice@example.com", - "verb": "update", "auditID": "…", "resourceVersion": "101", "stageTimestamp": "…", - "isServiceAccount": false - // note: no "conflict" field — see §4 -} -``` - -No sibling keys are retained: no `:seen`, no `:miss`. Without tombstones, an aged-out fact is -indistinguishable from one that never arrived, so the new contract treats both as `absent`. Late-arrival -metrics still matter, but they should be designed separately from this key schema. - -### 2.1 GroupResource format — recommendation - -Use the **`group/resource`** form (core drops the group: `configmaps`). It reads like an API path. The -Go stdlib `schema.GroupResource.String()` is **not** a drop-in: verified against this repo's pinned -[apimachinery v0.36.2](../../go.mod), it returns the *reversed, dot-separated* form — - -```go -// k8s.io/apimachinery/pkg/runtime/schema/group_version.go -func (gr GroupResource) String() string { - if len(gr.Group) == 0 { - return gr.Resource - } - return gr.Resource + "." + gr.Group // "deployments.apps", not "apps/deployments" -} -``` - -So `{apps, deployments}` → `deployments.apps`. Two differences from the proposal: **order** (resource-first) -and **separator** (`.`). The dot is also *ambiguous* — group names contain dots -(`roles.rbac.authorization.k8s.io`), so the dot form can't be split back into group/resource; the `/` -form can (and `/` never appears in a group or resource name). We keep `/`. A two-line shared helper keeps -write-side and read-side identical (they must never drift): - -```go -func groupResourceKey(group, resource string) string { - if group == "" { - return resource - } - return group + "/" + resource -} -``` - -Group/resource and a UUID never contain `:` or `/`, so the per-field escaping -(`joinKeyFields`) that v2 needed for RBAC names like -`system:node-proxier` in the *name* field is no longer load-bearing for attribution (the name is a value -now). RVs are numeric in practice; treat them as opaque and reject/escape a stray delimiter defensively. - -## 3. How the join works — the three lookups map cleanly to v2 - -| Lookup (resolver order) | v3 key | v2 equivalent | When it wins | -|---|---|---|---| -| exact | `…:object::` | `e` | normal create/update — the watch event's post-write RV matches the fact. | -| uid latest | `…:object::last` | `u` | known RV-mismatch events: deletes and deletecollection. | -| rv-only | `…:rv:` | `r` | a fact that has an RV but no UID (§5) — the escape hatch. | - -The resolver keeps the same join power as v2 but no longer uses the same unconditional fallback chain: -exact-capable events stop at the exact key, while known RV-mismatch events may use `:last` (§4.2). -CommitRequest authorship leaves this path entirely and uses `author:v1:command` from admission (§9). - -## 4. The win: `:` is immutable, so conflict-marking retires - -A given `(uid, rv)` had **exactly one writer** — that RV exists *because* of that single write. So -`object::` is **written once and never contended**. That dissolves the machinery v2 needed: - -- **v2:** the shared `u` (uid-only) key is written by every author of the object, so a second, different - author collapses it to a `{conflict:true}` marker (`storeFactKey`), - and only the `e` (exact) key rescues precise per-write credit. -- **v3:** each write owns its own immutable `:` key, so there is **nothing to conflict**. The - resolver rule becomes **"exact for exact-capable events; `:last` only for known RV-mismatch events."** - -### 4.1 `:last` is last-writer-wins, and that is correct here - -`:last` is overwritten with the newest author on every write. It is consulted **only** when the exact -`:` misses — i.e. RV-mismatch events: - -- A **delete** writes `:last` = the deleter; the `DELETED` watch event (RV ≠ any write RV) resolves the - deleter. Correct. -- A **deletecollection** expands one `:last` per member = the actor; each per-object removal joins it. - The whole reason that design pinned itself to the uid-only variant — body RV ≠ removal RV — is just - "use `:last`" here. **Superseded:** there is no expansion any more. A collection delete is published - as ONE fact describing the collection, which every removal in its scope joins by uid membership or - by scope; see [the attribution spec](../spec/attribution.md). The uid-not-RV argument is the half - that survived, and it is why the uid tier outranks the scope tier. -- A **burst** (author₁ rv₁, author₂ rv₂): `:`=author₁, `:`=author₂ (both precise), `:last`=author₂. - Watch events for rv₁ and rv₂ each hit their exact key → both precise. This is the precision an earlier - single-key sketch gave up; the grouped scheme keeps it. - -### 4.2 Safety rule — exact-capable events do not fall through to `:last` - -No tombstone. The fundamental correctness rule is simpler and stronger: - -- If a watch event has a UID **and** RV and is an exact-capable event (`ADDED` / `MODIFIED`), lookup only - `object::`; if it misses after the attribution grace, return `absent` and commit as the - committer. -- Use `:last` only for known RV-mismatch events (`DELETED`, deletecollection-expanded removals, and any - future path that deliberately records author intent without a matching post-write RV). -- Use `rv:` only when the audit fact had RV but no UID; a normal UID-bearing watch event tries exact - first and never needs the rv-only key. - -This removes the whole "was there once a fact here?" tombstone question. It also means `expired` goes away -as a resolver outcome: after the 15 minute audit fact TTL (§10), a miss is just `absent`. - -## 5. The `rv:` escape hatch — type-scoped, shipped from the start - -A `resourceVersion` is **opaque** per Kubernetes' API contract: you may not assume it is comparable or -unique across resource types, and aggregated/CRD API servers can mint their own. So an rv-only key -**must** include the type — `author:v1:audit:apps/deployments:rv:101`, never a bare -`author:v1:audit:rv:101`. The proposal gets this right. - -`rv:` is the join path for a fact that has an RV but **no** UID — a hollow / metadata-only body *and* -an `objectRef` without a UID. Under the `RequestResponse` audit level this project relies on, -`IdentityFromAuditEvent` ([`identity.go`](../../internal/auditutil/identity.go)) backfills the UID from -the body even for `generateName` creates, so a no-UID fact is uncommon — but when it happens, `rv:` -is the difference between a precise author and an explicit unresolved author. It is **cheap and shipped -now**: one -extra key, written *only* when the fact has no UID (a UID-bearing fact's `rv:` key would be dead — the -watch side always carries a UID, so it would resolve via `object::…` first and never reach it). This -is a deliberate improvement over v2, which wrote a dead rv-only key for *every* event. - -The `object:`/`rv:` discriminator is what keeps the two cohabiting unambiguously: without it, `…:` -and `…:` would collide in the same key position, distinguishable only because a UUID happens to look -unlike a number (fragile; don't rely on it). With the discriminator, the rv-only path is self-describing -and can never be mistaken for an object key. - -## 6. Reason codes simplify - -Because `:` is a real exact match again, the v2 reason tiers keep their meaning — unlike a single-key -collapse, which would have erased `exact` vs `weak`: - -| Reason | Source | -|---|---| -| `exact_user` / `exact_serviceaccount` | `object::` matched, `isServiceAccount` flag | -| `weak` | `object::last` or `rv:` matched (known RV-mismatch or rv-only fallback) | -| `exact_deletecollection_item` | matched fact with `verb == deletecollection` (read from the **value**, not a key variant) | -| `absent` | no usable author fact matched before the grace elapsed | - -`conflict` **disappears** as a resolver outcome (§4 — there is nothing to conflict). `expired` also -disappears because `:seen` tombstones are gone (§4.2). Net dashboard change: drop the `conflict` and -`expired` series; everything else on `AttributionResolutionsTotal{result=…}` is stable. - -## 7. deletecollection stops being a special case at the key layer - -`RecordDeleteCollectionFacts` exists to write "only the -uid-only variant" because the body RV is dead. In v3 that is simply "write `object::last`" — the -same key any RV-mismatch event uses. The only deletecollection-specific thing left is the **reason code**, -now driven by `fact.verb` in the value, not by which key variant matched. - -## 8. The GitTarget watch store - -Applying "the same line of thinking" honestly: the object-grouping core **does not transfer** — the -cursor is not keyed by an object UID. But `watch-cursor` is still too much ceremony. The durable thing is -a watch shard's **last processed RV**, so the family should say that directly: - -``` -v2: gitops-reverser:watch-cursor:v2::::: -v1: gitops-reverser:watch:v1:target::::last-rv -``` - -- **`watch:v1`** is the durable namespace. This is not an author record, so it should not live under - `author`, and `cursor` does not need to be in the family name. -- **`target:`** keeps the recreated-GitTarget safety property. The UID is the identity; the - namespace/name are only labels. -- **``** uses the same helper as §2.1. Drop the version: a `resourceVersion` is a - per-resource counter shared across served versions of that resource, so the version is redundant in a - resume-cursor key. If one GitTarget ever watched a resource at two versions, their cursors would merge; - that is harmless because the RV space is shared. In practice each shard watches one version. -- **`` stays, without a `scope:` label.** This is where I would push back on the shorter - `gitops-reverser:watch:v1:uid::last-rv` form: the live data plane opens one raw watch - per `(GitTarget, GVR, namespace scope)`, not just per `(GitTarget, type)`. A namespaced `WatchRule` and - a cluster-wide `ClusterWatchRule` are different watch collections, and two named namespaces can also be - separate watch shards. Sharing one RV key across those shards risks resuming the wrong collection. -- **`:last-rv`** names the leaf. The value is just the resourceVersion string. - -`scope` encoding: - -| Watch scope | Key segment | -|---|---| -| Cluster-wide watch, including cluster-scoped resources and ClusterWatchRule over a namespaced resource | `cluster` | -| One namespace-scoped watch | `namespace:` | - -Examples: - -``` -gitops-reverser:watch:v1:target:gtuid-3:apps/deployments:namespace:team-a:last-rv -gitops-reverser:watch:v1:target:gtuid-3:configmaps:cluster:last-rv -``` - -The GitTarget UID stays — it is what stops a recreated GitTarget inheriting a dead predecessor's cursor -([`types/reference.go`](../../internal/types/reference.go#L26)). - -## 9. The command-author store - -The CommitRequest admission plan should use the same top-level author namespace, but it should **not** -reuse the audit subfamily. It has different provenance, different lookup semantics, and no wait: - -``` -gitops-reverser:author:v1:command: -``` - -- **`author:v1`** says what the record is: a commit-author candidate. -- **`command`** says which invariant applies: admission captured the command submitter before the - object became visible, and the controller only uses it after the object persisted. -- **``** is enough. Command objects are read back by persisted object UID; namespace/name/kind are - value/debug fields at most. -- **No `:seen` / `:miss`.** There is no asynchronous audit arrival and no grace-window join. A miss is - present-or-never and immediately falls back to the committer. -- **No RV, auditID, or conflict marker.** Those belong to audit-sourced resource joins, not command - authorship. - -This replaces the companion sketch's `gitops-reverser:command-author:v1:` before it lands. Keeping -`command-author` as a separate top-level family is workable, but less clear once resource attribution has -already moved under `author`. - -## 10. Migration & rollout — no data migration - -All families here are **ephemeral**: audit author facts ≤ 15 min, cursors ≤ 1 h, command authors ≤ 1 h. -A version-prefix bump needs **no dual-read and no backfill** — old keys TTL out while new pods write/read -the new families. - -| Family | Rollout effect of the bump | Self-heals by | -|---|---|---| -| Audit author facts | Facts written by old `attr:v2` pods aren't read by new `author:v1:audit` readers during the overlap → those events ship as **committer** for the rollout window. | Next writes land in `author:v1:audit`; gap ≤ the 15 minute fact TTL. | -| Watch cursor | A new pod reads `watch:v1` (miss) and **rebuilds each shard from a fresh replay** — one cold resync per shard, the same cost as any pod cold-start. | First successful `watch:v1` cursor write; one replay. | -| Command author | No live key exists yet if the admission plan has not landed. If a prototype used `command-author:v1`, let it TTL out and read only `author:v1:command`. | Next command create. | - -The cursor consequence is heavier: **bumping the cursor version forces a one-time full resync** per shard -on the upgrading pod (≈ a normal restart's cost, since watch-first replays on restart anyway). So bump -author facts and cursor together only if a resync is already expected; otherwise bump them independently. - -## 11. Code changes (sketch) - -- **`attribution_index.go`** - - Rename `attributionKeySuffix` from `:attr:v2:` to `:author:v1:audit:`. - - Raise `DefaultAttributionFactTTL` from 10 minutes to 15 minutes. - - Replace `factKeyVariants`/`factKey` with three small builders: `factKeyExact(gr, uid, rv)`, - `factKeyLast(gr, uid)`, `factKeyRV(gr, rv)`, all over the `groupResourceKey` helper. - - `RecordFact`: write `…:` (immutable `SET NX`-ish — last write for an RV is identical anyway) **and** - overwrite `…:last`; **also** write `rv:` when `uid == ""` (the §5 escape hatch — shipped, just - skipped for UID-bearing facts whose `rv:` key would be dead). No conflict read/merge. - - `AuthorFact` gains `GroupResource`, `Namespace`, `Name`, `UID`; **drop `Conflict`**. - - `storeFactKey`'s conflict-detection block is **deleted** (§4). - - Delete `factTombstoneSuffix`, `factTombstoneTTLMultiplier`, `factTombstoneKey`, `factMissSuffix`, - `factMissKey`, and `RecordAuthorMiss`; no sibling keys. - - Resolver: exact-capable events try `:` only; known RV-mismatch events use `:last`; no tombstone - branch, no `expired` result, and no `op="late"` metric from Redis miss markers. - - `attributionResultForMatch` derives the reason from the matched key kind + `fact` (§6); remove - `AttributionConflict` and `AttributionExpired`; keep `AttributionExactDeleteCollectionItem` (now keyed - off `fact.Verb`). - - `RecordDeleteCollectionFacts`/expander: write one `…:last` per member (§7). - - `recordFactIndexSize` scans `author:v1:audit:*`. -- **`redis_store.go`** — `watchCursorKeySuffix` → `:watch:v1:`; `watchCursorKey` emits - `target::::last-rv` (drop `gvr.Version`). `CursorStore` interface - signatures unchanged. -- **`command_author_store.go`** — add the CommitRequest plan's store with - `commandAuthorKeySuffix = ":author:v1:command:"`, not `:command-author:v1:`. -- **`author_resolver.go` / `watch`** — drop the `conflict` result label; otherwise no logic change. -- **Tests** — key-shape assertions in `attribution_index_test.go`; burst → both-precise; delete → `:last`; - no-uid fact → `rv:` written and joined (no dead `rv:` for a UID-bearing fact); deletecollection - join-shape; `watch:v1` cursor shape; command-author key shape and miss-is-immediate behavior. - -## 12. Consequences at a glance - -| Change | Win | Cost / risk | -|---|---|---| -| `author` top-level namespace | Names what Redis stores; avoids `attr` ambiguity | Slightly longer keys | -| `audit` and `command` subfamilies | Keeps provenance and invariants legible | Callers must choose the right builder | -| Object-grouped keys (`object::…`) | Self-describing, `SCAN`-per-object, ns/name off the key | Same ~3 keys/write as v2 (no key-count reduction — readability over count) | -| `:` immutable + `:last` LWW | Retires conflict-marking; burst stays precise; `:last` only for known RV-mismatch events | Requires event-kind-aware resolver branching (§4.2) | -| `rv:` type-scoped, shipped | Closes the no-UID-fact gap precisely; one extra key only when needed | Needs the `object:`/`rv:` discriminator (already in the scheme) | -| Reason codes simplified | `conflict` and `expired` disappear | Dashboards drop both series | -| deletecollection un-specialed at key layer | Less special-casing | None (reason via `fact.verb`) | -| Watch: `watch:v1:target:...:last-rv` | Names the store and leaf directly; version redundancy gone | One cold resync per shard on the bump (§10) | -| Command author under `author:v1:command` | CommitRequest plan fits the same naming system | None if changed before code lands | -| Version bump | No migration code | Brief committer/replay window during rollout (§10) | - -## 13. Recommendation - -Adopt `author` as the top-level domain. Use `audit` below it for mirrored-resource facts: - -``` -gitops-reverser:author:v1:audit::object:: # exact, immutable -gitops-reverser:author:v1:audit::object::last # latest, LWW -gitops-reverser:author:v1:audit::rv: # rv-only, type-scoped -gitops-reverser:author:v1:command: # command submitter -gitops-reverser:watch:v1:target::::last-rv -``` - -For audit facts: value-carried name/namespace, **conflict-marking retired** in favor of immutable -`:` + LWW `:last`, no tombstones, a 15 minute audit fact TTL, and an event-kind-aware resolver: -exact-capable events do not fall through to `:last` (§4.2). Reason codes are preserved minus `conflict` -and `expired` (§6). All three audit keys ship together; the `rv:` escape hatch is written only for a -no-UID fact (§5). For command authorship: keep the separate `command` subfamily from the first -implementation, with no audit fields and no wait (§9). Move watch cursors to the simpler -`watch:v1:...:last-rv` shape, keeping the namespace scope because the live watch shard includes it (§8). -Bump author facts and watch cursors in the same release only if a resync is already expected. - -### Definition of done (when implemented) - -- Audit author keys are under `:author:v1:audit:` as `…:object::`, - `…:object::last`, and `…:rv:`, with no `:seen` or `:miss` siblings. `AuthorFact` carries - group-resource/namespace/name/uid and **no `Conflict`**. -- Command author keys are under `:author:v1:command:`; records carry only command author fields and - a cleanup TTL. -- Audit fact TTL is 15 minutes. -- Conflict-marking removed; resolver is event-kind-aware: exact-capable events try exact only; known - RV-mismatch events may use `:last`; `rv:` keys are type-scoped and written only for no-UID facts (§5). -- Reason set: `exact_user` / `exact_serviceaccount` / `weak` / `exact_deletecollection_item` / `absent`; - `conflict` and `expired` dropped; dashboards updated. -- Watch-cursor key is `:watch:v1:target::::last-rv`; version dropped, - namespace scope retained. -- Unit tests assert the v3 key shapes and the burst→both-precise, delete→`:last`, and no-UID→`rv:` - behaviours; the deletecollection join-shape test passes against `:last`. -- Full validation per AGENTS.md once code lands: `task fmt → generate → manifests → vet → lint → test → - test-e2e` (e2e sequential). diff --git a/docs/finished/sensitive-resource-classification-plan.md b/docs/finished/sensitive-resource-classification-plan.md deleted file mode 100644 index 9aea8c28b..000000000 --- a/docs/finished/sensitive-resource-classification-plan.md +++ /dev/null @@ -1,108 +0,0 @@ -# Implementation Plan: Startup Sensitive Resource Classification - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -Tracks [#147](https://github.com/ConfigButler/gitops-reverser/issues/147) — -support CozyStack's `tenantsecrets.core.cozystack.io` so Secret-shaped custom -resources are encrypted before they land in Git. - -## Goal - -GitOps Reverser currently encrypts only core Kubernetes Secrets. The decision is -hardcoded in the Git write path: - -```go -if !types.IsSecretResource(event.Identifier) { - return content, nil -} -``` - -CozyStack `tenantsecrets.core.cozystack.io` carry the same `.data` payload as a -core Secret. The first increment adds one explicit process-level way to send -that resource type through the existing Secret encryption path. - -## Design - -Add one comma-separated startup flag: - -```text ---additional-sensitive-resources=core.cozystack.io/tenantsecrets,example.io/credentials -``` - -Core `secrets` is always built in, so current Secret behavior never depends on a -new flag or Helm value. The flag only carries additional resource types: - -- core resources use a bare resource name, for example `secrets` -- grouped resources use `group/resource`, for example - `core.cozystack.io/tenantsecrets` - -The sensitive-resource policy matches **group + resource**, not API version. -Sensitivity is a resource-type property; a CRD version bump must not silently -send the same kind back to the plaintext path. - -The parser splits each entry on one optional `/`, rejects empty segments and -entries with extra `/`, normalizes and deduplicates entries, and logs the final -resource set at startup. - -## Scope - -The sensitive-resource policy affects the Git layer only in this increment: - -1. `content_writer.buildContentForWrite` chooses encryption from the policy. -2. `generateFilePath` chooses `.sops.yaml` from the policy so writes and deletes - use the same path contract as core Secrets. -3. Git write-path encryption-failure logging uses the policy instead of a - core-Secret-only check. - -`IsSecretResource` stays for code that specifically means core Secrets. New Git -write code that means "must be encrypted" should use the policy. - -Do not change the bootstrapped `.sops.yaml` here. It encrypts `data` and -`stringData`, which fits core Secrets and CozyStack `tenantsecrets`. Sensitive -custom resources with other field shapes are tracked in [TODO.md](../TODO.md). - -## Helm - -Expose the additions through Helm values: - -```yaml -controllerManager: - additionalSensitiveResources: - - core.cozystack.io/tenantsecrets -``` - -The chart renders the list into the manager flag. The default list is empty -because core Secrets stay built in. - -The operator trade-off is explicit: a custom secret-bearing resource omitted -from the startup flag still follows the ordinary plaintext path. This increment -avoids a GitTarget API, name heuristics, and denylist carve-outs. - -## Implementation - -1. Add a small group/resource `SensitiveResourcePolicy` with parser and tests. -2. Parse `--additional-sensitive-resources` in `cmd/main.go`, keep built-in - core `secrets`, and log the resolved set. -3. Thread the policy from startup into branch workers and content writers. -4. Switch content encryption, `.sops.yaml` path generation, and encryption - failure logging from `IsSecretResource` to the policy. -5. Add Helm values and user docs for the CozyStack-style opt-in. -6. Add focused tests for parser validation, group/resource matching, non-core - sensitive encryption, nil-encryptor fail-closed behavior, and non-core - `.sops.yaml` paths. - -## Testing - -- `task fmt` -- `task vet` -- `task lint` -- `task test` -- `task test-e2e` after confirming Docker is running - -Run the required validation sequence sequentially. - -## Follow-up - -Catalog validation, rule-status diagnostics, runtime skipping for unencrypted -targets, and the open questions around those behaviors are split into -[sensitive-resource-diagnostics-follow-up.md](../design/sensitive-resource-diagnostics-follow-up.md). diff --git a/docs/finished/signing-snapshot-tail-replay-failure-investigation.md b/docs/finished/signing-snapshot-tail-replay-failure-investigation.md index 8726f72d1..68c8ef8d4 100644 --- a/docs/finished/signing-snapshot-tail-replay-failure-investigation.md +++ b/docs/finished/signing-snapshot-tail-replay-failure-investigation.md @@ -11,7 +11,7 @@ > attempt 1, commit `113b4bc0224cfc0e5f900e38f877b8676828dfe9`. > > Related: -> [materialization-tail-and-live-readiness-review.md](materialization-tail-and-live-readiness-review.md) +> materialization-tail-and-live-readiness-review.md (removed) > (Gap 2 / Rec 2), > [github-e2e-per-type-tail-failure-investigation.md](signing-snapshot-tail-replay-failure-investigation.md). > @@ -288,7 +288,7 @@ flowchart LR ## 6. Reconciliation With the Readiness Review The readiness review's -[Rec 2 "Implementation note (landed)"](materialization-tail-and-live-readiness-review.md) +Rec 2 "Implementation note (landed)" explicitly declined to add a per-`(GitTarget, type)` tail-delivery gate: > The broader "tail does not *deliver* to an un-backfilled target" per-target gate diff --git a/docs/finished/sops-repo-bootstrap-and-key-management-architecture.md b/docs/finished/sops-repo-bootstrap-and-key-management-architecture.md deleted file mode 100644 index c1fe912d7..000000000 --- a/docs/finished/sops-repo-bootstrap-and-key-management-architecture.md +++ /dev/null @@ -1,78 +0,0 @@ -# SOPS Repo Bootstrap and Key Management Architecture - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -## Scope for This Increment - -This document is intentionally narrow. It defines only what we will implement now. - -## Decisions - -- Do not auto-generate key material. -- Do not support a generic controller-level `.sops.yaml` override. -- Ensure root `.sops.yaml` only from the write path, and only for branches we write to. -- For now, only ensure presence of `.sops.yaml`; do not validate content equality. -- "Matches Secret semantics" is defined by `isSecretResource` in `internal/git/content_writer.go`: move this Secret-semantics function to `internal/types/identifier.go` so it can be reused consistently. -- Align naming with Flux-style shape, but mirrored for reverse-gitops encryption: - - use `spec.encryption.provider` - - use `spec.encryption.secretRef.name` - - do not overload existing Git auth `spec.secretRef` -- Do not include `spec.encryption.serviceAccountName` in this increment. - -## Behavior Contract - -- On any write flow to a branch: - - If root `.sops.yaml` is missing on that branch, write it as part of the same commit flow. - - If root `.sops.yaml` exists, leave it unchanged. -- Secret detection for encryption-related behavior uses the same Secret-semantics function everywhere. -- `GitProvider.spec.secretRef` continues to mean Git authentication only. -- Encryption configuration is read from `GitProvider.spec.encryption`. -- Supported provider value in this increment: `sops`. -- Encryption key/credential material is sourced from `spec.encryption.secretRef.name`. -- If encryption reference is absent or invalid, Secret writes fail safely (no plaintext fallback). - -## GitProvider API Direction - -- Keep existing: - - `spec.secretRef`: Git credentials for clone/fetch/push. -- Add Flux-aligned mirrored encryption section: - - `spec.encryption.provider`: required when `spec.encryption` is set; value `sops`. - - `spec.encryption.secretRef.name`: Secret containing SOPS key material or static cloud credentials. -- Resolution and scope: - - Referenced Secret is namespace-local to the `GitProvider`. - - Key material is runtime-only and must never be committed to git. -- Deferred: - - `spec.encryption.serviceAccountName` is intentionally not included yet. -- Validation/status intent: - - Surface clear status/condition errors when encryption reference is missing/invalid and Secret semantics are matched. - -## `.sops.yaml` Definition Strategy - -- For this increment, keep `.sops.yaml` source simple and static: - - ensure presence in repo root during write flow. - - do not attempt to store/transport full `.sops.yaml` content in `GitProvider.spec`. -- Rationale: - - keeps API small and avoids embedding large policy blobs in CRDs. - - lets users customize `.sops.yaml` directly in git after bootstrap. - - avoids introducing policy merge/validation semantics in this phase. -- Revisit later if needed: - - if teams need declarative policy management, add an explicit reference-based source (for example Secret/ConfigMap ref), not an inline multi-line spec blob. - -## Implementation Plan - -1. Add/keep a single ensure helper in the Git write pipeline that checks/creates root `.sops.yaml` for the branch being written. -2. Keep ensure logic "existence only" (no content reconciliation). -3. Add `GitProvider.spec.encryption` wiring with fields aligned to Flux naming (`provider`, `secretRef.name`) and pass resolved encryption config into write workers. -4. Move `isSecretResource` from `internal/git/content_writer.go` to `internal/types/identifier.go` and use it from call sites that need Secret semantics. -5. Add/update tests for: - - `.sops.yaml` created when missing during write. - - `.sops.yaml` untouched when present. - - `GitProvider` Git auth `spec.secretRef` and encryption `spec.encryption.secretRef.name` are independently resolved and validated. - - only `provider: sops` is accepted. - - Secret-semantics helper behavior remains correct after move. - -## References - -- `docs/design/sops-repo-bootstrap-out-of-scope.md` -- `internal/git/content_writer.go` -- `internal/types/identifier.go` diff --git a/docs/finished/sops-repo-bootstrap-out-of-scope.md b/docs/finished/sops-repo-bootstrap-out-of-scope.md deleted file mode 100644 index ac56fef1f..000000000 --- a/docs/finished/sops-repo-bootstrap-out-of-scope.md +++ /dev/null @@ -1,19 +0,0 @@ -# SOPS Repo Bootstrap: Out of Scope (Current Increment) - -> **finished** — shipped or closed. Kept for context only; **nothing here binds**. For current behaviour see [`../spec/`](../spec/). Index: [`../INDEX.md`](../INDEX.md) - -This file tracks explicit non-goals and deferred items, so the main architecture plan stays small and execution-focused. - -## Explicitly Not Doing Now - -- Auto-generating AGE or other key material. -- Generic controller-level `.sops.yaml` override support. -- `.sops.yaml` content reconciliation (enforcing exact template match). -- Workload identity integration. -- Full key lifecycle orchestration (rotation, backup, recovery automation). -- Replacing external `sops` invocation with in-process encryption. - -## Notes - -- `.sops.yaml` remains user-adjustable once present in the repo. -- Future increments can re-introduce any item above with explicit API and operational design. diff --git a/docs/finished/watch-first-ingestion-architecture.md b/docs/finished/watch-first-ingestion-architecture.md deleted file mode 100644 index 5d377006c..000000000 --- a/docs/finished/watch-first-ingestion-architecture.md +++ /dev/null @@ -1,36 +0,0 @@ -# Watch-first ingestion architecture — retired design record - -> **Historical context only.** The current implementation contract is -> [`../architecture.md`](../architecture.md). This short record replaces the superseded rollout plan that -> previously occupied this path, so older links remain useful without presenting abandoned design work as live -> behaviour. - -The shipped model is watch-first: one raw watch per `(GitTarget, GVR, scope)` supplies object state. A -watch’s initial replay and any recovery replay establish the desired state; Git is the persisted mirror. -Audit events never create or repair object state. They are optional evidence used only to name a live -mutation’s Git author. - -## Current invariants - -- A branch worker serializes writes. A commit window belongs to one GitTarget and either one named actor or - an unnamed author state. -- With attribution disabled, and for replay/resync writes, Git uses the configured committer as author. -- With attribution enabled, a strong audit match names the authenticated actor. A weak, conflicting, late, - or missing fact produces `unknown (attribution unresolved) `. - A late fact never rewrites an already-created commit. -- Redis is optional in configured-author mode. It is required when audit-backed attribution is enabled and - can also retain watch-resume cursors. -- A `CommitRequest` captures its command submitter at validating admission when that optional Redis-backed - path is enabled. A request without a captured actor claims no actor; the matching window, not the request, - determines the final Git author. - -For the live details, use the [watch ingestion and reconcile](../architecture.md#watch-ingestion-and-reconcile), -[attribution](../architecture.md#attribution-audit-names-the-author-never-the-state), and -[CommitRequest finalize](../architecture.md#commitrequest-finalize) sections. - -## Metrics - -Metric names and operational interpretation are maintained in -[`../interpreting-metrics.md`](../interpreting-metrics.md). The telemetry design backlog is -[`../design/metrics-observability-plan.md`](../design/metrics-observability-plan.md); a planned metric is not -evidence that the runtime currently emits it. diff --git a/docs/future/config-surface-for-a-structured-repository.md b/docs/future/config-surface-for-a-structured-repository.md index f04c15e8c..2c6a026ae 100644 --- a/docs/future/config-surface-for-a-structured-repository.md +++ b/docs/future/config-surface-for-a-structured-repository.md @@ -5,7 +5,7 @@ > Absorbs and extends the API-surface block left unbuilt by the status and configuration-model > review — `spec.suspend`, `spec.interval` and the reconcile-request annotation, the > `CommitRequest` lifecycle, and the `ClusterWatchRule` scope question — which is now sequenced in -> [`gittarget-api-wave.md`](../design/gittarget-api-wave.md). +> `gittarget-api-wave.md` (removed). ## The one-sentence finding diff --git a/docs/future/direction-and-configuration-surface.md b/docs/future/direction-and-configuration-surface.md index 878f99c76..ec6fe7989 100644 --- a/docs/future/direction-and-configuration-surface.md +++ b/docs/future/direction-and-configuration-surface.md @@ -6,7 +6,7 @@ > Companions: [`config-surface-for-a-structured-repository.md`](config-surface-for-a-structured-repository.md) > (the field-level review this extends), > [`../layout/model.md`](../layout/model.md) and -> [`../design/gittarget-api-wave.md`](../design/gittarget-api-wave.md) (the API work this +> `../design/gittarget-api-wave.md` (removed) (the API work this > sequences), and > [`../design/support-boundary/helm-light-support-boundary.md`](../design/support-boundary/helm-light-support-boundary.md) > (the Helm option this deliberately parks). diff --git a/docs/future/idea-unify-pending-write-kinds.md b/docs/future/idea-unify-pending-write-kinds.md deleted file mode 100644 index 1d20374a2..000000000 --- a/docs/future/idea-unify-pending-write-kinds.md +++ /dev/null @@ -1,119 +0,0 @@ -# Idea: Unify `PendingWriteAtomic` and `PendingWriteCommit` into one shape - -> Status: deferred — captured for future re-assessment. -> Date: 2026-05-05 - -## The proposal in one sentence - -Collapse `PendingWriteKind` (today's `PendingWriteCommit` vs `PendingWriteAtomic`) so -the branch worker has a single open-window pipeline; the snapshot/reconcile case becomes -"a `WriteRequest` with `CommitMessage` set and a request-boundary finalize trigger", -not a parallel `Kind`-tagged path. - -## Why it keeps coming up - -`PendingWriteCommit` and `PendingWriteAtomic` carry the same fields and end with the -same three lines (`commitPendingWrites`, `append(pendingWrites, ...)`, `pendingWritesBytes += ...`). -The structural symmetry is real. The user-visible commit shape is `CommitMessage` set vs -unset. The temptation is to make that the dispatch and delete the `Kind` enum. - -## Use case walkthrough - -The four use cases in [commit-window-refactor.md](../spec/commit-window-refactor.md#use-cases): - -1. **Burst collapse.** Per-event-only behavior (atomic is one commit by definition). - Unification neither helps nor breaks this. -2. **Honest authorship.** Reconcile snapshots are configured-author; live commits carry - either a resolved audit actor or the explicit unresolved identity. Today this is enforced by - *type*: the atomic path is `AttributionNotAttempted` and never reads `event.UserInfo`. Under - unification it becomes a *convention*: producers must retain the correct - `AttributionOutcome` for snapshot batches. Same observable result, weaker compile-time guarantee. -3. **Target isolation.** Single-target on both paths today. Unaffected. -4. **Safe replay.** Both paths carry resolved metadata. Unaffected. - -So one use case is per-event-only, two are unaffected, and one (Honest authorship) is -where the two paths intentionally differ. Whether unification is "fine" depends on whether -that one is comfortable being a convention rather than a type-system invariant. - -## Where the two shapes actually diverge - -| Concern | Per-event / grouped | Atomic | Survives unification? | -|---|---|---|---| -| Authorship source | resolved actor or explicit outcome | Configured author (`AttributionNotAttempted`) | Yes, by preserving `AttributionOutcome` rather than relying on an empty username | -| Commit message | Template render | Caller-provided string | Yes, via `CommitMessage` non-empty as the dispatch signal | -| Finalize timing | Author/target change, silence, byte cap, shutdown | Request boundary | Yes, by adding "request has `CommitMessage` set" as a finalize trigger | -| Byte-cap behavior | Mid-stream finalize is fine | Must not split one snapshot across commits | Yes, via the rule "finalize the *next* organic window early; don't actively split a request" | -| Cross-request coalescing | Same (author, target) merges | Snapshot must not merge with audit-author events | Yes, the new finalize trigger forces a boundary on `CommitMessage`-bearing requests | -| Target identity source | `event.GitTargetName` | `WriteRequest.GitTargetName` (back-fills events) | Unchanged | - -Each row is mechanically resolvable. The "survives" column is only "yes" if the producer -side cooperates — most importantly, if the reconcile producer stops fabricating -`UserInfo: {Username: "gitops-reverser"}` and lets the git-layer fall-through populate -`Author`. - -## Reasoning — both directions - -**Argument for unification.** One open-window pipeline, one `PendingWrite` shape, one -`commitOptionsFor` builder (instead of three: `forEvent`, `forGroup`, `forBatch`). The -architectural principle becomes clean: producers describe *what* changed and declare -intent (`CommitMessage` set vs unset); the git layer decides *how* it's attributed. The -reconcile producer stops asserting an identity it has no business asserting. - -**Argument against unification.** The Honest-authorship guarantee is currently enforced -at the type level. Under unification it becomes a population convention: a producer that -forgets to clear `UserInfo` on snapshot events would silently leak audit-style attribution -onto reconcile commits — no compile-time check would catch it. The two `PendingWriteKind` -values are also the cheapest legible encoding of "request-level intent vs per-event -data": deleting them moves intent from request scope onto the substrate (events). - -After [commit-window-cleanup-q1](https://github.com/) (the planner deletion, now landed), -the savings from unification have shrunk: it removes one enum, one dispatch arm in -`handleQueueItem`, and the three `commitOptionsFor*` builders. It adds: a new -finalize trigger, a do-not-split byte-cap rule, an event/window invariant -("`UserInfo` empty == snapshot intent"), and producer-side cleanup. - -## Complexity assessment - -A rough sketch of the change footprint: - -- **Producers** (small): three `UserInfo` fabrication sites in - folder_reconciler.go deleted; the - reconcile `WriteRequest` site sets `CommitMessage` instead of `CommitMode`. The audit - producer in [git_target_event_stream.go](../../internal/reconcile/git_target_event_stream.go) - is unchanged. -- **Branch worker** (medium): merge `buildAtomicPendingWrite` into the open-window flow; - add request-boundary finalize trigger; rework byte-cap to "don't split a request". -- **Git layer** (small): collapse three signature builders into one driven by - `PendingWrite.author()`. -- **Types** (small): delete `PendingWriteKind`, `PendingWriteCommit`, `PendingWriteAtomic`, - `CommitMode`, `CommitModePerEvent`, `CommitModeAtomic`, `WriteRequest.CommitMode`. -- **Tests** (medium): rewrite anything asserting on `PendingWriteKind == PendingWriteAtomic` - or `CommitMode == CommitModeAtomic`. Add tests for the new boundary rules (force-finalize - on `CommitMessage`, byte-cap don't-split, empty-`UserInfo` fall-through). Reconcile tests - that assert on the fabricated `UserInfo="gitops-reverser"` get rewritten to assert on - the *commit's* Author/Committer instead — a healthier surface anyway. -- **Behavior change** (one): a multi-event per-event request that exceeds the byte cap - mid-stream is no longer split. Pathological case (thousand-event request); worth a - changelog line. -- **Migration**: stageable in roughly 8 PR-shaped steps that each leave the tree green. - Not a single all-or-nothing change. - -Roughly: a couple of weeks of focused work, mostly tests and migration order. Not a -weekend project; not a quarter-long rewrite. - -## Recommendation - -**Deferred.** The savings are modest after the Q1 planner cleanup, and the strongest -objection (losing the type-system check on snapshot authorship) doesn't dissolve — it -becomes a producer convention enforced by review and tests. Worth revisiting if any of -the following changes: - -- A second snapshot-style producer appears and the duplication of "build atomic pending - write + custom signature builder + custom dispatch" becomes painful. -- A new feature requires the open-window pipeline to also handle the snapshot case (e.g. - cross-request coalescing rules that would naturally extend to snapshots too). -- The reconciler's fabricated `UserInfo` causes a bug (wrong attribution leaking into a - commit). At that point removing the fabrication becomes a fix, and unification is the - cleanest way to do it. - -If none of those happen, the two-`Kind` model remains the cheapest honest encoding. diff --git a/docs/future/source-side-save-plan.md b/docs/future/source-side-save-plan.md deleted file mode 100644 index fb8ebbe32..000000000 --- a/docs/future/source-side-save-plan.md +++ /dev/null @@ -1,322 +0,0 @@ -# Plan: saving from the source cluster - -A `CommitRequest` saves the writes in its author's open commit window. It lives in the control -cluster, where the operator runs, while the edits it saves happen on the source cluster. This plan -adds a second way to save: a small request created **on the source cluster**, next to the edits, -attributed from that cluster's audit stream like the edits are. - -It also plans the second step: letting a save take over a window whose writes name no author, as an -opt-in, for a source cluster that cannot send audit events at all. There it is the only way a save can -put its author's name on the commit. - -Status: superseded. It was built on `feat/gitops-api-asks-converge-root-ca` as a second kind, -`RemoteSaveRequest` in `source.configbutler.ai/v1alpha1`, with a takeover for clusters that send no -audit events. The [review that replaced it](../review/20261027-save-request-simplification.md) folded -both kinds into one `CommitRequest`, served on the control cluster and on a source cluster through one -core, and removed the takeover with the admission webhook. The -[configuration reference](../configuration.md#serving-the-commitrequests-of-a-source-cluster) describes -what shipped; this page keeps the reasoning that led there. - -This revision takes in gitops-api's review of the first draft, and a later API review of the -implementation: see [After the API review](#after-the-api-review). - -## Decisions - -1. **Build the source-side save first.** It gives the right author wherever the source cluster has - audit, and an honestly unresolved one where it does not. -2. **It is a new kind, defined by GitOps Reverser.** A `CommitRequest` with another schema, or a - kind gitops-api defines and translates, were both considered. See - [Why a new kind](#why-a-new-kind). -3. **The takeover is the second step.** Some people cannot configure their API server's audit, so - for them it is the only way to get their name on a commit. It is opt-in on the GitTarget, which - chooses how much a save may claim; a request that asks without permission fails loudly. See - [The takeover](#the-takeover). -4. **`cf3b97be` does not merge as it is.** Its automatic takeover becomes the opt-in, or comes out. - -## Why: two identity domains - -Today the save and the edits are attributed in two places: the `CommitRequest`'s author comes from -the control cluster's admission webhook, the edits' authors from the source cluster's audit events. -They match only when one person has the same username in both. - -gitops-api makes them match. Its server impersonates the human when it creates the `CommitRequest`, -so the admission webhook records the same Dex email that kcp's audit records for the edits, and the -ordinary author rule attaches. gitops-api does not need a takeover. What it pays for that is -elsewhere: - -- **A grant into the control cluster.** Every tenant's identity may `create commitrequests` in its - `tenant-` namespace there. It is the one place a tenant reaches outside its kcp - workspace. -- **A server on the save path.** The browser cannot reach that namespace, so every save goes through - gitops-api's `/editor/commit`, which impersonates and creates the request. - -A save created in the workspace removes both. "A customer's only API surface is their kcp workspace" -then holds with no exceptions, and the browser creates the save itself. That is the production -benefit this plan is for. It also helps any setup where people can reach only the source cluster. - -```mermaid -flowchart LR - subgraph Source["Source cluster (kcp workspace)"] - Edit["Alice edits cm/a"] - Save["Alice creates a save"] - end - subgraph Control["Control cluster"] - Op["Operator"] - end - Edit -->|"watched, attributed
from audit"| Op - Save -->|"watched, attributed
from the same audit"| Op - Op -->|"status written back
(a new write on the source cluster)"| Save -``` - -## The source-side save - -### What it does - -The operator watches the new kind on each source cluster through its KubeProvider. It attributes -each request exactly as it attributes the edits, from the source cluster's audit facts. It then -attaches the request to the matching commit window, and writes the request's status back where its -author can read it. - -- **On a source with audit**, Alice's save and Alice's edits resolve to the same user, so the save - attaches under the ordinary author rule. No takeover and no trust are involved, and Git names the - person who made the edits. -- **On a source with no audit**, the save names nobody, like the edits. A request that names no actor - may attach to a window that names none, which is today's rule - (`TestPendingCommitRequest_UnresolvedRequestAttachesToUnresolvedWindow`). The save gets its message - and its timers, and the author in Git stays unresolved. The window still holds everyone's - unattributed writes, but nobody is named as their author. - -### Why a new kind - -gitops-api could define its own kind and translate it into a `CommitRequest`, but it cannot learn who -created the object: watching the workspace shows the object and nothing about its creator. Only the audit stream -says that, and GitOps Reverser is what reads it. A translation would also spend part of the attach -wait on an extra hop, and add a second status to keep in step. - -Reusing `CommitRequest` with a different schema would be worse than a separate kind: - -- **Another shape.** It names a GitTarget on another cluster than the one it is created on. -- **Another trust model.** Its author comes from audit; a `CommitRequest`'s comes from admission. -- **One name for two things** invites the confusion that `tenant-` already costs gitops-api. -- **Installed alone.** A workspace gets one small CRD instead of the whole `configbutler.ai` set, so - nobody is offered GitTargets that nothing reads. gitops-api installs CRDs into workspaces through its - `workspaceCRDs` list, so no `APIExport` is needed there. -- **Easy to never mirror.** In its own API group, or at least as its own kind, keeping it out of Git - whatever a `WatchRule` selects is one check, like `kube-root-ca.crt`. - -### Which GitTarget it saves - -It names one, in a required `spec.gitTargetRef` of `namespace` and `name` on the control cluster. -The people creating it need no access to GitTargets: their tooling supplies the address. This draft -had it name none and save every GitTarget mirroring its namespace; review showed that an unchanged -request then does more whenever another GitTarget is added, and that its caller cannot say "this one -only". See [After the API review](#after-the-api-review). - -### When its wait starts - -On a source with audit, a request is attributed only once its audit fact has joined its watch event, -and the edits it saves wait in the same audit pipeline (kcp batches audit with a `batchMaxWait` of -1s). The request's wait for a window starts when the branch worker registers it, which is after the -request itself was attributed. That is how a `CommitRequest`'s `attachTimeout` is anchored today -(`handleAttachCommitRequest` in `internal/git/commit_request_attach_loop.go`). The timeout then covers -only the spread between the request's stream and the edits' streams. Its -default can still be set for a source with audit, rather than copied from `CommitRequest`'s `2s`. - -### An unresolved save on a source that has audit - -On a provider whose `AuditFactsReceived` latch is `True`, a save that resolves no author means its -audit fact went missing. Attaching it to a window that names no author would claim exactly the -window that should stay unattributed. So the latch applies here too: - -```mermaid -flowchart TB - S["A source-side save is attributed"] --> N{"Did it resolve
an author?"} - N -->|yes| Own["Attach to that author's window"] - N -->|no| L{"Has its KubeProvider ever
received an audit fact?"} - L -->|no| None["Attach to a window that names no author"] - L -->|yes| Refuse["Refused: its fact went missing.
Ready=True with reason AuthorUnresolved"] -``` - -Refusing rather than waiting keeps the outcome visible, and the person saves again. - -### What it writes on the source cluster - -This is GitOps Reverser's first write on a source cluster. Today a source credential only lists and -watches the mirrored types ([rbac.md](../rbac.md)). The new access is scoped to the new kind: list, -watch, status update, and delete (for the finished-request TTL). It is off unless the KubeProvider -turns it on, and the KubeProvider's docs say what it adds. - -### Notes for clients - -- **Create it as JSON.** gitops-api found that kcp drops audit events whose request body is YAML. A - save sent as YAML would lose its author without any error. A browser sends JSON anyway; the docs - must say it for everyone else. -- **One save at a time per GitTarget and author.** gitops-api queues saves in its server - (`commitGate`). If the browser creates them directly, that queue is gone. A window carries one - message, so a second save of the same author on the same GitTarget, made while the first still - waits for a window or collects one, is refused at once with `SaveInProgress`, rather than losing - its message to the first in silence. The client saves again once the first resolved. A second - person's edit that lands while a window is open still closes that window early: windows are one at - a time per branch worker, whoever writes. That is - [a design question of its own](../design/concurrent-author-windows.md). - -## The takeover - -### Who needs it - -A source-side save names its author only when the source cluster sends audit events. Many people -cannot make it: a managed Kubernetes service usually does not let the customer point the API -server's audit at a webhook. On such a cluster no write and no save can name who made it, and Git -records every save as attribution-unresolved. A takeover is the only way to put a name on those -commits, and that name is a claim on trust by definition: the cluster cannot confirm it. - -### The risk - -On a cluster with no audit, every write names no author, so all of them share one window on the -GitTarget, whoever made them. A save that takes the window over names its submitter as the author of -everyone's writes. - -```mermaid -flowchart LR - subgraph Source["Source cluster with no audit stream"] - A["Alice edits cm/a"] - B["Bob edits cm/b"] - end - A --> Win["One window on the GitTarget
author: none"] - B --> Win - CR["Alice's save"] -.->|"a takeover would name Alice
as the author of Bob's edit"| Win -``` - -That is harmless when one person edits the GitTarget's namespaces at a time, and wrong as soon as two -do. Only the GitTarget's owner knows which it is, so the GitTarget decides. - -### What the GitTarget allows - -An enum, because there are two levels of trust worth offering: `unattributedWindows`, on each grant -in the GitTarget's `spec.allowedCommitRequests`. A request claims only what the grant admitting its -namespace allows; admitted by several, it may claim the most any of them allows. (The first version -was one field, `spec.commit.unattributedWindows`, for requests in the GitTarget's own namespace; -it moved into the grants when a `CommitRequest` could name a GitTarget in another namespace.) - -- **`"Keep"`** (the default): a save attaches only to its author's window, as today. On a cluster - with no audit, a save from the control cluster finds no window of its own. -- **`"SavedBy"`**: a save may attach its message to a window whose writes name no author, without - claiming them. The author stays unresolved, and the commit records who saved it: - - ```text - Saved-By: alice@example.com - ``` - - The message may still describe Bob's edit, but nobody is named as the author of something they - did not write. A misleading message is a much smaller harm than a false author. -- **`"TakeOver"`**: a save may take such a window over. Its submitter authors the commit, and the - message ends with a trailer that says the name was taken on trust: - - ```text - Author-Source: CommitRequest team-a/save-1; no audit record names who made these changes - ``` - -A grant on the KubeProvider was considered and set aside: the provider's owner usually does not know -who edits which namespace. The GitTarget's owner does, and grants it per consuming namespace. - -### Three conditions, all required - -```mermaid -flowchart TB - Start["A save finds an open window on its GitTarget
whose writes name no author"] --> Q1{"1. Did the save ask
for more than its own window?"} - Q1 -->|no| Own["Its own windows only"] - Q1 -->|yes| Q2{"2. Does the grant admitting it
allow what it asked (unattributedWindows)?"} - Q2 -->|no| Refuse["Refused: Ready=False, Stalled=True
reason TakeOverNotAllowed"] - Q2 -->|yes| Q3{"3. Has the KubeProvider ever
received an audit fact?"} - Q3 -->|yes| Own - Q3 -->|no| Take["SavedBy: attach the message, author unresolved
TakeOver: the submitter authors the commit"] -``` - -1. **The save asks for it.** An enum under `spec.window` beside `attach`, matching the GitTarget's: - `spec.window.author`, `"Own"` (the default), `"SavedBy"` or `"TakeOver"`. A save asks for what it - wants each time, and the grant bounds it: `"TakeOver"` allows a save that asks for `"SavedBy"`, - and not the other way round. -2. **The GitTarget's grant allows it.** A save that asks for more than the grants admitting it allow - fails at once, - when it is first reconciled: `Ready=False`, `Stalled=True`, reason `TakeOverNotAllowed`. The spec - is immutable, so it cannot recover by itself. -3. **The KubeProvider has never received an audit fact** (`AuditFactsReceived` is not `True`). On a - cluster that has sent audit, a write that names no author means its fact went missing, so no - save claims it, whatever 1 and 2 say. The save then attaches to its own windows only. - -### How it relates to the source-side save - -The two cover different clusters: - -| Source cluster | Source-side save | Control-cluster save with `"TakeOver"` | -| --- | --- | --- | -| Sends audit | The saver's name, from audit | Not needed; the latch blocks it | -| Cannot send audit | Message only, author unresolved | The saver's name, on trust | - -A source-side save cannot offer `"TakeOver"`: on a cluster with no audit, nothing records who created -it either, so it has no name to give. The name has to come from the control cluster's admission -webhook, which is why a takeover is always made by a control-cluster `CommitRequest`. - -## What happens to `cf3b97be` - -It takes over automatically. Most of it is the worker side of `"TakeOver"`: matching a window whose -writes name no author, authoring the commit as the submitter, and the `Author-Source` trailer. Two -ways forward before the branch merges: - -- **Turn it into the opt-in here.** Add `spec.commit.unattributedWindows` on the GitTarget and - `spec.window.author` on the `CommitRequest`, the controller's check and refusal, `"SavedBy"`, and - tests for each row above. The source-cluster e2e's save spec then sets both. -- **Take it out here**, and build the takeover in its own PR after the source-side save. - -## Open questions - -1. The source-side kind's name and API group. Proposed: `RemoteSaveRequest` in - `source.configbutler.ai/v1alpha1`. "Save" is the word its user means and the one the project - already uses for what a request asks; `SimpleCommitRequest` was considered and set aside, since - it is a different kind with another trust model, not a simpler `CommitRequest`. Its spec carries - no Git terms (a message, a wait, a catch-up), and its status still reports each GitTarget's - commit, so a save stays traceable. Its own group makes "never mirror it" and "install it alone" - one check each. -2. The default of its attach wait on a source with audit. -3. The reason name for an unresolved save on a source that has audit (`AuthorUnresolved` above). -4. Whether the operator should hold a second save on a GitTarget until the first has resolved. -5. Names for the takeover: `spec.commit.unattributedWindows` on the GitTarget and - `spec.window.author` on the `CommitRequest`, each `"Keep"`/`"Own"`, `"SavedBy"`, `"TakeOver"`. -6. `cf3b97be`: turn it into the opt-in on this branch, or take it out and build the takeover later? - -## After the API review - -A review of the implementation found three things: - -1. **A remote save is not confined to its namespace.** A window groups by GitTarget and author, not by - namespace, so a save from `shop` commits the same author's edits in `billing` too, or, with no - audit, anyone's. The review offered two answers: confine the save, or let the GitTarget's grant - authorize the whole window. The grant is the answer: an admitted `RemoteSaveRequest` saves its - creator's whole window on that GitTarget, and the docs say so. Confining it was built and taken out - again: a namespace-shaped exception to windows that are not namespace-shaped, with an outcome of its - own (`WindowOutsideNamespace`) that the person saving could do nothing about. -2. **A cluster-scoped watch matched every namespace.** A collection with no namespace is either - cluster-scoped or every namespace, and the match did not tell them apart. It now reads the - GitTarget's watched types and counts only namespaced ones. -3. **The GitTarget's owner could not consent.** Enabling the requests on a KubeProvider made every - GitTarget mirroring the namespace eligible, and a request steers a window's message, its timers - and an apply. The provider's switch still makes them reachable; `GitTarget.spec.allowedRemoteSaveRequests` - admits them, by source namespace (names or `"*"`; no `Same`, since the namespaces are another - cluster's) and an optional `requestSelector`. Omitted, it admits none: `NotAllowed`. Labels pick - requests and prove nothing about who made them. - -A second review then found that implicit target selection still made a save broader than its caller -may intend: a request reached every admitting GitTarget mirroring its namespace, so adding a -GitTarget changed what an unchanged request did. A `RemoteSaveRequest` now names its GitTarget in a -required `spec.gitTargetRef`, saves that one, and reports a flat status (`status.commit`, -`status.appliedRevision`, `Ready`, `Applied`). The request's namespace decides only admission. That -also removed a cold-start failure: with no named target, discovery that had not caught up reported -`NoGitTarget`. The same review fixed two recovery paths: a request served again after a restart -keeps the author found when it was first served (its status writes moved its resourceVersion, so its -audit fact no longer matches), and keeps an outcome already decided, following only its apply. - -A `CommitRequest` gained a cross-namespace `spec.gitTargetRef`, and with it -`GitTarget.spec.allowedCommitRequests`: a grant list like the others, defaulting to one `from: Same` -grant, whose grants also carry the `unattributedWindows` bound. The GitTarget that admitted a request -is recorded by UID (`status.gitTargetUID`), so a GitTarget created after the request under the same -name must admit it before anything is attached. diff --git a/docs/layout/README.md b/docs/layout/README.md index 34e4ba0aa..ad696328e 100644 --- a/docs/layout/README.md +++ b/docs/layout/README.md @@ -37,13 +37,6 @@ that produced them. What moved is the evidence. ## What is deliberately not here -- [`../design/gittarget-api-wave.md`](../design/gittarget-api-wave.md) sequenced the breaking work on - `GitTarget`, which shipped in #330. The placement work was additive and was never part of it. -- [`../design/placement-visibility-and-declared-defaults.md`](../design/placement-visibility-and-declared-defaults.md) - holds the three questions the sibling-inference deletion left. Decided; `status.placement` has - since shipped and the rest is unbuilt. Its - Question 2 (a CRD default for `placement.default`) is **reopened** by [`model.md`](model.md)'s - reversal, not superseded by it. - [`../design/support-boundary/`](../design/support-boundary/README.md) owns what the operator may edit and what it refuses. Layout decides **where** a document goes; the support boundary decides **whether** it may be written at all. diff --git a/docs/layout/model.md b/docs/layout/model.md index ba233fa71..8e58720b7 100644 --- a/docs/layout/model.md +++ b/docs/layout/model.md @@ -68,7 +68,7 @@ new file is registered with the nearest kustomization that governs it, whatever |---|---| | `byType` into a subdirectory produces a file no kustomization lists | **No.** That was [#295](https://github.com/ConfigButler/gitops-reverser/issues/295), and it is fixed | | `placement.default` does the same to every type at once | **No.** Same bug, same fix | -| A CRD default cannot be added: a non-empty template shadows the kustomize-root rung | **Weakened**, and [`placement-visibility-and-declared-defaults.md`](../design/placement-visibility-and-declared-defaults.md) already concedes the fix turns this from a correctness wall into a legibility trade | +| A CRD default cannot be added: a non-empty template shadows the kustomize-root rung | **Weakened**, and `placement-visibility-and-declared-defaults.md` (removed) already concedes the fix turns this from a correctness wall into a legibility trade | | "Where do my files go" needs a metric and a status field to be legible | **True** — but that is an argument for status, not against templates | | Nothing can **create** structure, so an empty repository cannot be bootstrapped | **True**, and untouched by either design. Creating a `kustomization.yaml` is not a path question | @@ -473,7 +473,7 @@ reads only the scan. Neither PR is breaking, so neither waits for a coordinated consumer bump. What is breaking on `GitTarget` is unrelated to placement and is sequenced in -[`gittarget-api-wave.md`](../design/gittarget-api-wave.md). +`gittarget-api-wave.md` (removed). **PR 1 is the corpus, and it is the reason the rest is reviewable.** [`shapes/README.md`](../../test/fixtures/layout-corpus/shapes/README.md) already has the shape of a golden-file suite — diff --git a/docs/misconceptions.md b/docs/misconceptions.md new file mode 100644 index 000000000..518d3e271 --- /dev/null +++ b/docs/misconceptions.md @@ -0,0 +1,361 @@ +# Misconceptions we corrected + +Between July and October 2026 the project held a number of beliefs that turned out to be false: +about its own code, about Kubernetes, and about Flux, Argo CD, kustomize and go-git. Most of them +sounded right, several shipped, and a few were documented as intended behavior. Applying Git back +to the cluster (the bi-directional mode) only works reliably since #419, #425 and #443, and most of +the hard lessons of the last months are about that mode. + +This page collects them, so a claim is checked here before it is raised again. Each entry gives the +belief, what is true, what it broke or nearly broke, and where it was corrected. Many of the design +pages that recorded these steps have since been deleted; `git log` keeps them, and the references +below point at what still exists: pull requests, commits, specs and current design pages. + +Two habits caused more of these than any single gap in knowledge. They are listed first. + +## How we investigated + +- **A green run supports a fix.** Two hypotheses for PR #315's status failure were implemented and + withdrawn. One fix passed its spec once while contradicting the logs, and would have swept under + `always` after a user tightened the prune mode. *True:* a pass on a flaky spec proves nothing; the + diagnostic has to fire, and the logs have to agree. Recorded in + [watch-plane-status-convergence-failures.md](design/watch-plane-status-convergence-failures.md). +- **Several failing specs are several flakes.** Five failing e2e specs were one bug: a deleting + `GitTarget` flooded the shared 100-slot queue with 595 resyncs in 16 seconds. Fixed in #312. +- **A theory built from code reading is good enough.** The Argo CD e2e failure was read as a push + race, then as a stale cache, and a workaround and a doc rewrite were built on that. The controller + log showed the API server rejecting an enum value. *True:* read the applier's and the controller's + logs before theorizing. +- **"Flux does it this way", from memory.** Twice it was the opposite. Flux is also internally + inconsistent (its enums are not one convention). *True:* check `external-sources/flux`, see + [definitions.md](definitions.md). +- **`task test-e2e 2>&1 | tail -N` shows whether the suite passed.** It reports `tail`'s exit code, + so a failing suite read as green. +- **A label filter narrows the default e2e set, and a zero-match run proves something.** The filter + replaces the default, and a run that matches nothing skips `BeforeSuite` and reports green. + +## Applying Git back (bi-directional) + +- **Forwarding only foreign push webhooks controls when the applier applies.** Argo CD's timed + refresh syncs every new tip two to three minutes after a push, even with `selfHeal` off, and Flux + applies on every interval. Every timer is a trigger. *Cost:* the applier reverted edits GitOps + Reverser had not published yet. *Fixed:* GitOps Reverser makes every apply request itself, and the + applier's own triggers are held off (#419, [apply-trigger.md](design/apply-trigger.md)). +- **A 24h Flux interval amounts to exclusive control.** Flux still applies on a kustomize-controller + restart, an apply retry, a spec change, and every new `GitRepository` artifact, including one + another consumer fetched. *Fixed:* the `FluxClaim` keeps the Kustomization suspended and resumes + it once per apply (#419). +- **A PreSync hook, or the self-heal timeout and backoff, can delay Argo CD's revert.** A sync's + manifests are frozen when the operation starts, a hook cannot tell a heal from a deploy, and the + timeouts throttle only repeats, so the first revert is immediate. *Fixed:* automated sync off, + and GitOps Reverser starts every sync + ([argocd-bi-directional.md](design/support-boundary/argocd-bi-directional.md)). +- **Holding Argo CD `selfHeal` off is needed to stop reverts.** `selfHeal` is a step inside + automated sync and never runs while automated sync is off. Forcing it would lose the user's value + on hand-back. Shipped as "leave it alone" in #437. +- **Argo CD prunes an undeclared `automated.enabled` on versions before 3.1.** The API server + refuses the server-side apply with a 500, "field not declared in schema". *Fixed:* a dry-run gate + and a refusal reason (#437). +- **An apply must wait for a quiet branch** (no open window, no pending write). The wait starved: + someone who kept editing held a Git-side change back for as long as they typed. *Fixed:* the flush. + The worker closes the window, takes the flush mark, publishes up to the mark and sends the apply. + An edit after the mark is recorded, and an apply that overwrites it is withheld and repaired (#425, + [S1 in applier-echoes.md](design/applier-echoes.md#s1-flush-then-apply)). +- **Closing the commit window on a push notice is about latency.** It is discovery. With a window + open the worker reads nothing from the remote first, so publishing is how a busy branch learns + that the branch moved at all ([concurrent-author-windows.md](design/concurrent-author-windows.md)). +- **kustomize-controller's `lastAttemptedRevision` means the apply has happened.** It is written + before the build, about 0.8 seconds early per 200 objects. An artifact holding the commit, or a + five-minute timer, proves nothing either. *Cost:* the apply window closed early. *Fixed:* applies + are observed operations, and one past its budget is `Unresolved` but still followed (#419). +- **Closing the apply window ten seconds after the applier reports done catches its writes.** On a + cluster with no audit stream every event waits the full attribution grace, one after another, so + the applier's writes arrived after the window, were mirrored as edits, and reverted the newer API + edit. *Fixed:* the window closes on watch progress, with the mark taken as the newest + `resourceVersion` among the collection's objects, never the list's + ([review](review/20261026-apply-window-late-events.md), `internal/git/apply_barrier.go`). +- **The newest `managedFields` entries being the applier's alone proves the applier wrote.** An + update that removes every field a manager owns deletes that manager's entry, so an older + kustomize-controller entry ends up newest. *Cost:* a `kubectl label x tmp-` undo was withheld, and + `Repair` put `tmp=1` back in the cluster and in Git. Fixed in #420 (519b483e). +- **An applier write in the same second as a recorded edit is ambiguous, so mirror it.** Events + arrive in `resourceVersion` order, so it is the apply over the edit. Mirroring it reverted the edit + (caught by e2e E6, #425). +- **Flux is immune to the applier-echo push race.** It passed only because its echo arrived after + the five-second push cooldown; Argo CD failed every time. A stale pending decision was ordered + after a Git-side push, and the replay re-planned it, reverting the push. Fixed in #425. +- **Taking over `spec.suspend` from `flux suspend` with server-side apply makes us its sole owner.** + A forced apply of the same value co-owns the field, so the release's empty apply left the + Kustomization suspended while the Event said it resumed. The Flux simulator had modeled one owner + per field. Fixed in #420, measured in envtest. +- **A history entry newer than our request time proves a recovered Argo CD sync ran.** History times + use the controller's clock, so skew lets an older sync pass. *Fixed:* a recovered sync is repeated + when no entry proves it (#443). +- **An unanswered Argo CD update can be timed out after two minutes.** That assumed the remote API + server's default request timeout, which nothing establishes. Replaced by a `resourceVersion`-guarded + refresh write that makes the lost update impossible to land (#420). +- **Argo CD only writes as `kubectl-client-side-apply`.** `argocd-controller` is the field manager + for client-side and server-side apply, `Replace=true` writes with an empty manager, and a retrying + failed operation is `Running` with `finishedAt` set (#419). +- **A newly bound Argo CD `Application` must not converge, and Argo CD needs no `CatchUp`.** Reversed + on 2026-10-20: Argo CD catches up and converges on first naming, the same as Flux (S2 in + [applier-echoes.md](design/applier-echoes.md)). +- **Each tool works out which of its objects syncs a changed folder.** Neither does. Argo CD + refreshes every `Application` with a matching `repoURL` and narrows only on a hand-written + `manifest-generate-paths` annotation; a Flux `Receiver`'s resource list is hand-written. This ruled + out "interpret ownership, then patch the orchestrator" (#383). +- **The sanitizer keeps GitOps-tool bookkeeping out of Git.** It stripped Flux keys and no Argo CD + key, so `argocd.argoproj.io/tracking-id` was committed. Argo CD stamps it on every non-CRD manifest + and never validates it against the object, and a foreign tracking-id fails another `Application`'s + sync ("Shared resource found"). *Fixed:* an exact-key deny-list, because `sync-wave` and friends are + user intent (`internal/sanitize/types.go`). Flux's `config.kubernetes.io/origin` joined it in #455. +- **Argo CD reconciles every 180 seconds and tracks by label.** The defaults are 120 seconds plus up + to 60 seconds of jitter, and annotation tracking + ([e2e-bi-directional-corner.md](spec/e2e-bi-directional-corner.md)). +- **Filtering writes by identity or field manager is needed to stop the echo of our own commit.** + The echo ends by itself. gitops-api's filter caused a two-and-a-half-minute flip-flop with every + commit unattributed. Withdrawn on 2026-07-14 + ([gitops-api-open-asks.md](future/gitops-api-open-asks.md)). +- **A live edit that meets a pending render-fidelity gate is safely dropped, because the replay + covers it.** The scope was marked clean on a different goroutine from the worker's answer, so + edits in that gap were lost for good, and + [api-first-publication.md](api-first-publication.md) documented the drop as intended. *Cost:* the + audit-route-override flake, nine CI failures. Fixed in #420 and Fix 2a + ([live-edit-dropped-while-render-fidelity-pending.md](design/live-edit-dropped-while-render-fidelity-pending.md)). + +## Attribution + +- **A removal may be credited to the object's last writer.** That named whoever last edited the + object as the person who deleted it. Removals wait for delete evidence, with no last-writer + fallback, since 2026-10-22 (#431, [spec/attribution.md](spec/attribution.md)). +- **A `DELETED` frame resolves on the write at its exact (uid, `resourceVersion`).** A deletion + never carries that `resourceVersion`: an immediate delete answers with a Status holding only the + uid, and a finalizer-held one is stamped one step earlier. Every deletion waited out the grace, so a + Flux prune of N objects stalled the stream for N times the grace (the E12 `NoWindow` failure). Two + earlier theories were wrong. Fixed in #431 (a52caca2), which credits a removal by its cause. +- **The deleter's fact outranks the finalizer-clearing controller's.** It is replaced: both writes + return the same `resourceVersion`, land on one (uid, rv) key, and the last one wins, so the commit + named the controller. Fixed in #289 (fcdc3bcd). +- **Collection-delete tiers rank below `latest`.** `latest` answers "who last wrote"; a removal asks + "who deleted". Ranking above the collection's uid set credited every collection removal to the last + editor. Corrected during #283 to #287 + ([attribution-fact-stream.md](finished/attribution-fact-stream.md)). +- **Attribution tiers are tried in sequence, each with its own wait.** There is one wait, and every + tier is evaluated again on every wake + ([attribution-removal-wait-options.md](design/attribution-removal-wait-options.md)). +- **A `CommitRequest`'s author can be looked up by its current `resourceVersion`.** A label patch + before the claim named whoever patched it. *Fixed:* a creation query (#459). A no-op patch then + returned the create's own (uid, rv) and replaced the creator, so creation facts got their own slot, + `factKindCreation` (#460). +- **`Metadata`-level audit is enough for `CommitRequest`s.** The uid and `resourceVersion` come only + from `responseObject`, and server-side apply creates with a patch, so both create and patch need + `RequestResponse` (#460). +- **The admission `userInfo` on the control cluster is the audit user on the source cluster.** It + matched only because gitops-api impersonates the Dex email; for anyone else it was a coincidence. + Admission and takeover were removed in #456 + ([review](review/20261027-save-request-simplification.md)). +- **A matched fact that names nobody is a miss for the audit-route signal.** It proves the route is + alive; the observation was wrong in both directions. Fixed in #341 (0e38cf48). +- **`unknown_route` is a possible audit rejection.** Any named route is accepted as a partition name; + the only route rejection is the bare endpoint, `bare_endpoint_disabled`, and configuration.md had + said the opposite ([metrics-observability-plan.md](design/metrics-observability-plan.md)). +- **Audit ingress is authenticated per provider.** It is not, yet: one shared client certificate + routes any `/audit-webhook/`. Still open, in + [multi-source-audit-ingress-hardening.md](design/multi-source-audit-ingress-hardening.md). + +## Commit windows, saves and status + +- **`closeDelay: 0` is a tight but valid default.** It was shorter than the time a window takes to + open, because admission was the fast path and the edit's audit join the slow one, so every save + resolved `NoWindowInGrace`. The default became two seconds, and the race went away with admission + in #456. +- **A save can wait on the attribution fact index.** A fact arrives before the watch event, so such a + wait fires before the window opens, and it cannot see deduplication, render fidelity or the queue. +- **`windowPathKey` is the Git path after placement.** It is the `GitTarget` path joined with the + generated file name; the write plan resolves the destination only at commit + ([concurrent-author-windows.md](design/concurrent-author-windows.md)). +- **The push cooldown is inert now that the commit window batches.** Under alternating authors an + identity change closes the window, the silence timer never fires, and the cooldown is the only + batching left. The earlier "remove it" was retracted ([push-cooldown.md](design/push-cooldown.md)). +- **An idle apply follow ends when the worker's record ages out.** That happened only when another + save resolved, so long applies were reported `Lost` and quiet workers followed forever. *Fixed:* an + explicit `SaveApplyTimeout` (#460). +- **Commit shaping belongs on `GitProvider`.** A branch worker serves every `GitTarget` on one + provider and branch, so those targets could not disagree; the setting describes a folder. Moved to + `GitTarget.spec.commit` in #330. +- **The status write that beat us re-enqueued us.** The winning write is status-only, and every + `For()` uses `GenerationChangedPredicate`, which filters it out. A lost write left a stale `Ready` + for five minutes, 74 times per run. Fixed in #315. +- **Our own push is not news for `status.remote`.** A new revision always writes status, and + observation freshness is not checkout readiness: keying on trust alone sent never-published + targets to Git on every reconcile (#388). +- **A delayed requeue (`AddAfter`) always fires.** controller-runtime v0.25's priority queue folds it + into an item already queued, so a foreign push reached `status.remote` a whole five-minute tick + late. Fixed in #424 (3f00e098). +- **Duration fields accept what they emit.** `0.5ms` round-trips as `500µs`, which the Flux-style + pattern rejected, so the object could never be updated again; CEL `self == oldSelf` also compared + spellings (`1m` against `1m0s`). Fixed in #388. +- **The `StreamsReady` condition should be absent until it knows.** `kubectl wait` errors on an + absent condition, so it exists from the first reconcile as `Unknown` + ([streaming-readiness-status-machine-design.md](finished/streaming-readiness-status-machine-design.md)). +- **`ReasonRefusedStructural` is a permanent boundary.** It carries `Solvable`, and a consumer + shipped "can never be synced" for a typo + ([open-asks-priority.md](design/open-asks-priority.md)). + +## Watches and the watch plane + +- **A watch ending is a failure.** The API server ends every watch on a randomized timeout. Grading + it as `WatchError` flipped `Ready` and emitted a Warning about every 40 minutes per type. Fixed in + #363 (d0601a59). +- **A watch stream may be parented to the plan pass's context.** The pass deadline canceled every + stream once the plan was applied, while the logs looked healthy and readiness sat at `Replaying`. + A stream's parent is the manager's lifetime + ([watch-manager-ownership.md](design/watch-manager-ownership.md)). +- **A per-pass deadline gives failure isolation.** Isolation came from taking network I/O off the + owner loop; a first cut let one unreachable cluster stall every healthy target. A uid-less delete + also matched every incarnation and could tear down a same-name successor (same page). +- **Never debounce the first `GitTarget` declaration.** Declaring before the rules arrive builds an + empty plan, and that empty plan cleared a write divergence in the fidelity gate. Replaced by a + two-second rolling settle window (same page). +- **A per-`GitTarget` latch shows the initial snapshot is done.** A target declared before its rule's + collection was planned counted as done, its initial snapshot was thrown away, and it deadlocked + against the converge. Tracked per collection since #425. +- **A resync payload can be replaced by a bare dirty set.** The snapshot and its position in the + queue define the snapshot-then-deltas boundary; without it, mark-and-sweep reverts edits + ([target-watch-plan.md](design/target-watch-plan.md) keeps the fence). +- **A cluster-wide collection can absorb a named-namespace one of the same type.** Collapsing them + widened the named rule's stream to every namespace the credential could read. They are peers, and + the newer rule is refused ([source-scope-simplification.md](design/source-scope-simplification.md)). +- **Legacy discovery calls are bounded.** client-go's `ServerGroupsAndResources()` runs with no + deadline. +- **A controller `ownerReference` marks a derived object.** Measured against live controllers, it + catches one producer in five: objects a Flux `HelmRelease` renders or a flux-operator `ResourceSet` + expands carry none. The `*.toolkit.fluxcd.io/` labels also give opposite verdicts + ([expansion-provenance-markers.md](facts/expansion-provenance-markers.md)). + +## Git, go-git and Git hosts + +- **A `file://` remote is a faithful test server.** go-git v6's in-process receive-pack never + compares the old value, so every racing push won and the compare-and-swap tests were vacuous. Tests + use `startRealGitServer` (#297). +- **go-git v6 ignores ambient configuration, as v5 did.** It honors a global `commit.gpgSign` and + refuses to commit without a signer, and it loads `known_hosts` whenever `HostKeyAlgorithms` is + empty, even with a callback set. Every SSH remote would have failed in the distroless image; only + e2e caught it. Fixed by `PinExplicitSigningPolicy` and `ssh.KeyAuth` (#297, + [azure-devops-multi-ack.md](design/azure-devops-multi-ack.md)). +- **Azure DevOps accepts go-git, because a bare probe returned 200.** It returns 400 on any capability + list without `multi_ack`, and v5 cannot parse the multi-ACK response. Flux escapes only because it + never fetches. *Fixed:* go-git v6, instead of #292's bundled `git` + ([azure-devops-multi-ack-requirement.md](facts/azure-devops-multi-ack-requirement.md)). +- **Context cancellation bounds go-git calls.** HTTP honors it only through `List` and + `FetchContext`, and SSH ignores it past the dial, so handshakes hang + (`internal/git/network_bound.go`, #413). +- **A head-of-cycle fetch with `Force: true` cleans a worktree a failed write left dirty.** Only for + modified tracked files; new files and directories survive a hard reset. Fixed in #382 (9535d6a8). +- **go-git cannot learn an empty repository's default branch.** GitHub advertises it over protocol + v2 (`unborn`), and go-git asks and then discards the reply. Status reports "unknown"; guessing + `main` was rejected + ([gittarget-parent-empty-repository.md](design/gittarget-parent-empty-repository.md)). +- **A receive-pack probe proves push access.** On Gitea any valid user with read access gets 200, so + it proves only a valid credential (#429). +- **Gitea's `gpg.error.no_gpg_keys_found` means a GPG problem.** For SSH signatures it means no + verified SSH key matched, and verification needs the web-only `verify_ssh` flow + ([e2e-signing-followups.md](finished/e2e-signing-followups.md)). +- **Migrating to Forgejo is required to unpin Gitea.** Both moved to Go's `CrossOriginProtection`, + which a plain Go client passes; deleting the `_csrf` scrape unpinned Gitea in place + ([e2e-git-server-choice.md](design/e2e-git-server-choice.md)). + +## Kustomize, rendering and placement + +- **Our re-implemented transformers render what kustomize renders.** Three shipped bugs: + `images[].name` is a regex over the whole image string (`ngin.` matches nginx), the replicas field + spec includes `ReplicationController`, and image traversal rewrites OCI `volumes[].image.reference` + but skips `ephemeralContainers`. A malformed `images[].name` also nil-dereferences inside kustomize. + *Fixed:* ask kustomize instead of re-implementing it (#232, #233, + [render-attribution.md](design/support-boundary/render-attribution.md)). +- **`alpha.config.kubernetes.io/transformations` records which transformers touched an object.** It + records every transformer that ran, on every object of the build, with duplicates per `images:` + entry (same page). +- **Rendered objects are valid unstructured with `int64` numbers.** Numbers come back as Go `int`, + so `NestedInt64` reports not found and `DeepCopyJSON` panics (same page). +- **A structural "contains `${...}`" check fences off postBuild substitution.** Flux's own CRD + carries `${var:=default}` literally in a description, and one such CRD refused the whole folder. + The difference is visible only against the live object. Reverted on the #234 branch + ([render-fidelity.md](design/support-boundary/render-fidelity.md)). Kustomize itself never emits + `${...}` ([kustomize-never-emits-dollar-brace.md](facts/kustomize-never-emits-dollar-brace.md)). +- **Mirroring the live object into its source file is safe.** Under `labels:` or + `commonAnnotations:` it wrote the build's output into the build's input, one commit per reconcile + of an in-sync folder. Where live and render agree, the source keeps its bytes (#234, 8444cd00). +- **A hand-kept deny-list of kustomize fields is enough.** `vars` was tolerated, and `$(VAR)` sources + got the substituted value mirrored back. The unsupported set is derived from kustomize's + `Kustomization` struct (#229). +- **kustomize folds `patchesStrategicMerge` and `patchesJson6902` into `patches:`.** It does not; they + refuse under their own names ([patch-authoring.md](design/support-boundary/patch-authoring.md)). +- **Kustomize means flat files beside one root, with no nesting.** Measured on v5.8.1: a root can + list nested paths, nested roots work, nothing is picked up by glob, and an unlisted file renders + nothing. The `spec.layout` redesign rested on the opposite and was reversed into two booleans + (#319, [layout/model.md](layout/model.md)). +- **A created `kustomization.yaml` must carry `namespace:`.** A namespace-less root is an ordinary + base; Flux `targetNamespace` and Argo CD `destination.namespace` supply the namespace, and what + refused it was our own render-fidelity gate (#328, + [created-root-namespace.md](design/created-root-namespace.md)). +- **Placement inference is safe when every candidate agrees.** `allSameDir` was true for a single + directory, so a whole type collapsed into one multi-namespace file. Inference was deleted (#291). +- **Top-level `clusters/`, `apps/` and `infra/` identify a Flux fleet root.** That missed the + canonical `infrastructure/` and matched unrelated repositories; the evidence is + `flux-system/gotk-*.yaml` + ([orchestrator-knowledge-boundary.md](design/support-boundary/orchestrator-knowledge-boundary.md)). +- **SOPS lets one document of a multi-document file be edited.** One data key and one file-wide MAC + cover the file, and metadata is read from the first document only + ([sops-single-file-no-multidoc.md](spec/sops-single-file-no-multidoc.md)). + +## Kubernetes API behavior + +- **Deleting a CRD field is pruned quietly and harmlessly.** Strict `kubectl apply` rejects the field + loudly while controllers prune it silently, and removing `allowedSourceNamespaces` failed open, + widening `"*"` to every readable namespace + ([crd-upgrade-strategies.md](facts/crd-upgrade-strategies.md)). +- **`managedFields` separates declared values from schema defaults, and a map default is + additive.** `managedFields` is bookkeeping, and a map default disappears once any key is set + (#319). +- **Pinning the kubeconfig Secret to the operator namespace closes the confused deputy.** The deputy + is data export through any `GitTarget` that references the provider, and the tenant never touches + the Secret. Fixed with deny-by-default grants on the provider (#441, + [provider-access-rules.md](finished/provider-access-rules.md)). + +## Release, CI and e2e + +- **`draft: true` in release-please is vestigial.** It is required, because immutable releases + reject asset uploads after publish. The spurious whole-history PR came from drafts being + invisible to the release lookup; fixed by splitting the invocations (#203). +- **Unparseable commits come from nested `* feat!:` squash lines.** The cause is a line-leading + `word(a(b))`, or `BEGIN_COMMIT_OVERRIDE` mentioned in prose. #388's changelog was lost, so 0.50.0 + shipped three undocumented breaks (#403). +- **Several `BREAKING CHANGE:` footers in one squash all reach the changelog.** Only the first is + kept; on #450 it described an undone intermediate state. +- **A `container:` job runs bash.** It runs dash, so `set -o pipefail` killed the 0.40.0 publish and + stranded a draft release (#279). +- **Coverage dips mean missing tests.** go1.27's coverprofile charges each comment-separated block + the function's whole statement count; a 17-fold inflation was observed. +- **A metric e2e check that skips an absent series, or asserts `increase() == 0`, proves the + producer works.** One reason had no producer at all, and deleting every recording call left the + suite green; `increase()` cannot see an event before the first sample + ([proving-the-metric-surface.md](design/proving-the-metric-surface.md)). +- **Gauges published from inside a loop show that loop stalling.** Queue depth read 0 through the + first stall, and oldest-dirty-age froze when the loop hung + ([metrics-observability-plan.md](design/metrics-observability-plan.md)). +- **The local e2e suite covers the Helm chart.** `INSTALL_MODE=config-dir` never renders it, and a + `{{ . | quote }}` in an argument leaked literal quotes and crash-looped the manager (#221). +- **`k3d image import` succeeding means the image is imported.** Exec polling misreads completion + and node failures are only logged; fixed with `-m direct` and a retry (#185). +- **k3s DiskPressure means a full disk.** The percentage thresholds fire with gigabytes free on a + shared runner; absolute thresholds since #187. +- **A zero-agent CI cluster behaves like the local three-agent one.** The node restart bounced + CoreDNS, Valkey and the manager together, and DNS died on k3s 1.37. Mitigated in #415; the root + cause is unproven ([k3s-1.37-dns-after-node-restart.md](ci/k3s-1.37-dns-after-node-restart.md)). +- **Name isolation made `crd_lifecycle` safe to run in parallel.** Cluster-wide CRD discovery churn + made it flaky again, and it was serialized again + ([e2e-serial-registry.md](spec/e2e-serial-registry.md)). +- **Gating the docs lint on the Git diff is safe.** A shallow CI checkout made the merge base + unreachable, the file list came back empty, and the gate passed; it reads a committed list, + `.docs-lint-scope` ([docs-linting.md](design/docs-linting.md)). diff --git a/docs/review/20261025-crd-text-proposal.md b/docs/review/20261025-crd-text-proposal.md deleted file mode 100644 index e18546871..000000000 --- a/docs/review/20261025-crd-text-proposal.md +++ /dev/null @@ -1,365 +0,0 @@ -# Proposal: clearer CRD descriptions - -The branch makes the API reference more consistent, but several descriptions still ask the reader -to unpack too many rules at once. Keep the useful detail. Give each sentence one job, and use -bullets where readers need to compare values, check requirements, or follow a precedence order. - -This proposal changes wording and line wrapping, and adds one lint check for line width. It proposes -no schema or controller behavior changes. - -## Scope - -Reviewed `chore/crd-text-sweep` at `0b8334eb`, against its merge base with `origin/main` -(`1bc97904`). The focus is newcomer-facing text in the ten `configbutler.ai/v1alpha3` CRDs and -the branch's changes to user documentation. This is an editorial review with targeted behavior -checks. A full correctness review of the controller changes is outside its scope. - -The examples below are replacement descriptions, without Go comment prefixes. They use the -plain-text conventions required by [AGENTS.md](../../AGENTS.md). Markdown commentary follows the -[documentation style guide](../style-guide.md). - -## What's good - -- Field descriptions usually start by explaining the field's purpose. -- Defaults, namespace resolution, and immutable references are documented near the setting. -- The short descriptions already work. Keep fields such as `GitTarget.spec.path`, - `GitProvider.spec.url`, and `GitTarget.spec.encryption` compact. -- `onRefusal` and `serializeNamespace` already give each value its own paragraph. Their structure - works; converting those paragraphs to bullets is optional polish. -- The generated YAML excludes the long type-level design rationale I checked. For example, - `GitTarget.spec.placement` uses the field comment. The internal discussion on - `GitTargetPlacementSpec` stays in the source. Review the generated description before deciding - a comment needs editorial work. - -## Top risks (ranked) - -1. **Some wording changes the reader's understanding of behavior.** - `CommitRequest.spec.message` says the message is used as written, without explaining that - `GitTarget.spec.commit.message.requestTemplate` can wrap it. `GitProvider.status.lastVerifiedAt` - claims to show how long a connection has been broken; it records the last successful check. - Failure may begin later, and `Ready=False` can also reflect signing or configuration failures. - These need corrections before cosmetic edits. -2. **Most descriptions render badly in `kubectl explain`.** It keeps each source line break and - wraps every line again at 76 characters of text, so a longer line leaves its last words on a - line of their own. 616 of the 1,067 description lines are longer than that. See the - [rendering measurements](../facts/crd-descriptions-in-kubectl-explain.md). Restructuring a - description does not help until its lines fit. -3. **Grant defaults and selector pruning need clearer separation.** Grant descriptions compress - omission, explicit lists, and combined permissions into a paragraph. Selector descriptions mix - matching, deletion behavior, and conflicting rules. Separate these cases so readers can find - the one that applies to them. -4. **Some references leave readers guessing which object or condition is meant.** Both watch-rule - `objectSelector` descriptions refer to `spec.prune.mode` without naming the `GitTarget`. - `ArgoCDClaim.status.conditions` mixes `True` and `False` with `NotConfigured`, which is a - reason, not a condition status. It also omits `False` with reason `ArgoCDVersionNotSupported`. -5. **Long troubleshooting prose is squeezed into a table.** The new row about repeated - `ClaimedResourceReconciled` events in the Flux guide contains the symptom, implementation - cause, ownership explanation, and remedy. Give that case its own subsection and leave a short - link in the table. - -Evidence for the behavior corrections: [request message rendering](../../internal/git/commit_executor.go), -[provider reconciliation](../../internal/controller/gitprovider_controller.go), and -[Argo CD control conditions](../../internal/controller/argocdclaim_hold.go). - -## What Flux does - -Flux has a [documentation style guide](https://fluxcd.io/contributing/docs/style-guide/), maintained -in its [website source](https://github.com/fluxcd/website/blob/main/content/en/contributing/docs/style-guide.md). -It recommends active voice, present tense, direct wording, and avoiding idioms that make reading -harder for people who speak English as a second language. It puts related items that must appear -in a specific order into a list, and keeps paragraphs under six sentences. This is a website -writing guide; it does not prescribe a format for CRD comments. - -The source offers useful examples too: - -- In [GitRepository's API types](https://github.com/fluxcd/source-controller/blob/main/api/v1/gitrepository_types.go), - `SecretRef` separates HTTPS and SSH requirements into sentences. `Provider` states its default - separately. `GitRepositoryRef` describes precedence beside each choice. -- In [Kustomization's API types](https://github.com/fluxcd/kustomize-controller/blob/main/api/v1/kustomization_types.go), - `Wait` states the action, its interaction with `HealthChecks`, and its default in separate - sentences. Some other descriptions are still dense, so these are examples to learn from, rather - than a uniform standard to copy. -- The [GitRepository reference](https://github.com/fluxcd/source-controller/blob/main/docs/spec/v1/gitrepositories.md) - starts with a manifest and explains it with bullets. It gives authentication methods and branch - selection their own sections. - -Adopt those structural habits, and keep this repository's conventions for field names and literal -values. Flux's guide supports numbered lists for ordered items, such as a precedence order. Bullets -for unordered choices are this proposal's own recommendation. Sources checked on October 25, 2026; -the links track their default branches. - -## Recommended changes (actionable) - -### Use a small set of writing rules - -- Lead with what the field controls. Follow with the default or omitted behavior. -- Use bullets for alternatives and independent requirements. Use numbered lists when order matters. -- Keep a short explanation in prose when there is no choice to compare. -- Keep existing per-value paragraphs when they already make the choices clear. -- Wrap description text at 76 characters after the Go comment prefix (risk 2). Fix that before - optional paragraph-to-bullet changes. -- Name the subject: the `GitTarget`, the source cluster, or the referenced `GitBranch`. -- Prefer familiar verbs: `allows`, `uses`, `rejects`, and `restores`. Replace phrases such as - `a written list`, `admits nobody`, and `hands the switch back` with concrete behavior. -- Keep deletion behavior, authorization, failure reasons, and immutability visible. Shorter text - must preserve these details. -- Aim for two to four sentences in ordinary descriptions. Treat the existing 80-word target as - a prompt to restructure longer text. Preserve necessary warnings. - -### 1. Explain the message override accurately - -Location: [CommitRequest.spec.message](../../api/v1alpha3/commitrequest_types.go). - -```text -Message supplies the requested commit message. Omitted, the GitTarget's -spec.commit.message.liveTemplate generates the message. - -The text is never evaluated as a template. If the GitTarget configures -spec.commit.message.requestTemplate, that template wraps this text. -Otherwise, the text is used as written. - -The message must contain 1–1024 characters and at least one non-whitespace -character. Newlines are allowed for a subject and body; other control -characters are rejected. -``` - -Short paragraphs work here: the message policy and validation are separate topics, but neither -needs an artificial list. - -### 2. Describe the timestamp without claiming an outage duration - -Location: [GitProvider.status.lastVerifiedAt](../../api/v1alpha3/gitprovider_types.go). - -```text -LastVerifiedAt is the time of the last successful repository connectivity -check using this GitProvider's credentials, or anonymous access when -spec.secretRef is omitted. Failed checks leave this timestamp unchanged. -Check status.conditions for the reason the GitProvider is not ready. -``` - -The controller updates the timestamp only after a successful remote check. The timestamp cannot -identify when a later failure began. - -### 3. Distinguish condition status from reason - -Location: [ArgoCDClaim.status.conditions](../../api/v1alpha3/argocdclaim_types.go). - -Replace the paragraph beginning `InControl is True` with: - -```text -InControl reports whether GitOps Reverser controls when the Application -syncs: - -- True, reason AutomatedSyncOff: GitOps Reverser controls sync timing. -- False: control is being released or revoked, or no GitTarget references - the claim. -- False, reason NotConfigured: another field manager has enabled automated - sync. -- False, reason ArgoCDVersionNotSupported: the Application has no - spec.syncPolicy.automated.enabled field (Argo CD before 3.1), so GitOps - Reverser cannot turn automated sync off. -- Unknown, reason NotConfigured: the claim is not configured to control the - Application. - -A claim with no referencing GitTarget reads no applier and reports -Ready=True, reason NotReferenced. -``` - -These cases come from `holdArgoCD`, `ownedSwitchCondition`, and `switchOff`. The version check in -`switchOff` also catches an Argo CD downgrade after the configuration check. Apply the same -status/reason distinction when editing the longer `Configured` paragraph and Flux conditions. - -### 4. Turn namespace choices into bullets - -Location: [ConsumerNamespaces.From](../../api/v1alpha3/access_grants.go). - -The existing sentence puts four enum values and four explanations in one line of thought. -Each value is a separate choice: - -```text -From selects which namespaces this grant allows consumers from: - -- "Same": the namespace of the object that defines the grant. -- "Names": the namespaces listed in names. -- "Selector": namespaces whose labels match selector. -- "All": every namespace. -``` - -### 5. Make the omitted and empty grant lists easy to distinguish - -Location: [GitBranch.spec.allowedGitTargets](../../api/v1alpha3/gitbranch_types.go). - -The distinction is already present, but `a written list is the whole policy` makes the reader -translate it. State each case: - -```text -AllowedGitTargets controls which GitTargets may write through this -GitBranch. - -- Omitted: allows GitTargets in this GitBranch's namespace. -- Empty list ([]): allows no GitTargets. -- Nonempty list: allows only GitTargets matched by at least one grant. - -An explicit list replaces the default. Include a grant for this GitBranch's -namespace if GitTargets there still need access. -``` - -Apply the same structure to `KubeProvider.spec.allowedGitTargets` and `allowedApplierClaims`, -preserving their source-namespace and applier-namespace defaults. Keep -`GitProvider.spec.allowedGitBranches` distinct: both omission and an empty list deny access. - -### 6. Give each receiver type its own line - -Location: [GitReceiver.spec.type](../../api/v1alpha3/gitreceiver_types.go). - -```text -Type selects the push payload and authentication format: - -- "github": GitHub payload with X-Hub-Signature-256. -- "gitlab": GitLab payload with X-Gitlab-Token. -- "gitea": Gitea payload with X-Gitea-Signature. -- "generic-hmac": a JSON payload with repository, branch, and after fields, - authenticated by X-Signature: sha256=. -``` - -This also replaces the JSON-like `{"repository", "branch", "after"}` fragment, which names fields -but is not valid JSON. - -### 7. Keep template fields available, but group them - -Location: [GitTarget.spec.commit.message.liveTemplate](../../api/v1alpha3/gitprovider_types.go). - -The current description combines the template context, per-resource fields, empty-value handling, -and request overrides. It also omits Kind, Labels, and the Label accessor, which the -[commit message guide](../commit-messages.md) documents. Group the context and add the missing -fields: - -```text -LiveTemplate is the Go template for commit messages generated from live -edits. An empty value uses the built-in template. - -The template receives: - -- Author: the username, which may be empty. -- GitTarget: the target name. -- Count: the edit count after coalescing and before comparison with Git. -- Operations: counts of retained entries by operation kind. -- Resources: the resources in the commit window. - -Each resource provides Operation, Group, Version, Resource, Kind, Namespace, -Name, APIVersion, ResourceVersion, Generation, and Labels. Read one label -with {{.Label "key"}}. ResourceVersion identifies the state written by the -commit and may be empty. Kind and Labels are empty for a deletion. -Generation is 0 for a kind without a spec. Guard Kind, ResourceVersion, and -Generation with {{with}}. - -A CommitRequest message takes precedence without changing the author. The -requestTemplate field controls whether that message is wrapped. -``` - -Keep examples and longer context definitions in the existing [commit message guide](../commit-messages.md). - -### 8. Move the long Flux troubleshooting row into a subsection - -Location: [Flux troubleshooting](../bi-directional-flux.md#troubleshooting). - -Proposed subsection text: - -> Repeated `ClaimedResourceReconciled` events mean something is undoing the claim's suspension. -> A rising `claimed_resource_reconciled_total` records these corrections. If the claim also reports -> `OperationUnresolved`, inspect the controller that writes the `Kustomization`. -> -> A controller that replaces the whole spec can remove `spec.suspend` by leaving it out. Flux can -> then apply before GitOps Reverser suspends the `Kustomization` again. This removal does not take -> ownership of `spec.suspend`, so `SuspendOwnedElsewhere` does not identify it. -> -> Update that controller to use one of these approaches: -> -> - Preserve the `spec.suspend` value from the object it read when replacing the spec. -> - Use server-side apply and omit `spec.suspend` from the fields it manages. - -Keep the symptom in the table and link to this explanation. The adjacent `Overriding by hand` -bullet also benefits from two sentences: first the unchecked apply, then the claim's response. - -### 9. Optional: use bullets for refusal choices - -Location: [GitTarget.spec.onRefusal](../../api/v1alpha3/gittarget_types.go). - -The existing description already separates `Ignore` and `Reapply` into paragraphs. Keep it unless -a rendered comparison shows a benefit. This alternative mainly separates the restrictions on -`Reapply` from the value's immediate effect: - -```text -OnRefusal controls what happens when a live edit cannot be written to the -folder. Defaults to "Ignore". A refused edit makes no commit and remains on -the cluster until an applier overwrites it from Git. - -- "Ignore": reports the refusal in conditions and events. -- "Reapply": uses spec.applierClaimRef to reapply Git after pending edits - are published, reverting the refused edit. Requires spec.applierClaimRef. - -Reapply runs only when the document already exists in the folder and the -write removes no documents. It does not run for folder-wide failures, such -as invalid YAML. A refusal that persists triggers one reapply, rather than -one per check. -``` - -This is a justified exception to the word-count target: the restrictions prevent a reader from -expecting a reapply to repair every refusal. - -### 10. Optional: use bullets for namespace serialization - -Location: [GitTarget.spec.serializeNamespace](../../api/v1alpha3/gittarget_types.go). - -Omission, `true`, and `false` already have separate paragraphs. Bullets may make the alternatives -easier to scan, but this is a lower-priority presentation choice. Correct line wrapping first: - -```text -SerializeNamespace controls whether written documents include -metadata.namespace. It applies to every write of a namespaced resource. -Cluster-scoped resources ignore this setting. - -- Omitted: leaves out metadata.namespace only when the kustomization - governing the document sets the same namespace. -- true: always includes metadata.namespace. Use this for a flat folder - applied directly. -- false: always omits metadata.namespace. The namespace must come from a - kustomization, a Flux Kustomization's targetNamespace, or an Argo CD - Application's destination.namespace. - -With false, the folder accepts only one source namespace. A WatchRule that -adds a second is rejected with GitPathAccepted=False, reason -MultipleSourceNamespaces. -``` - -### Remaining pass, in priority order - -| Location | Proposed treatment | -| --- | --- | -| Both watch-rule `objectSelector` fields | Name `GitTarget.spec.prune.mode`; separate pruning and selector conflicts. | -| `GitTarget.spec.placement` | Separate existing-document behavior, path variables, and fallback restrictions. | -| `KubeProvider.spec.attribution.auditRoute` | Separate routing, default, namespace ownership, and conflict behavior. | -| `GitBranch.spec.parentBranch` | Separate default-parent selection from missing-parent behavior. | -| `GitProvider.spec.knownHostsRef` | Use a numbered list for the three sources in precedence order. | -| `GitTarget.spec.prune.mode` | Use one bullet for each mode, with the default first. | -| Both claim kinds' root descriptions | Lead with control of apply timing; separate the trigger cases. | -| `GitTarget.spec.suspend` | Separate stopping new commits, the final push, and resuming from cluster state. | -| Upgrade entry for removed message templates | Separate removal, client behavior, and the migration action. | - -## Implementation order - -1. Correct the message, timestamp, condition, template-field, and cross-object-reference wording. -2. Rewrap every description at 76 characters, in a commit of its own so the diff stays mechanical. - In the same change, add a `task lint` check that fails on any `description` line longer than 76 - characters under `config/crd/bases`, and add the limit to the CRD house style in - [AGENTS.md](../../AGENTS.md). Without the check, the next edit brings long lines back. -3. Apply the enum and grant-list examples. -4. Restructure the remaining long descriptions and the Flux troubleshooting case. -5. Preview `kubectl explain` output, using the envtest recipe in the - [rendering measurements](../facts/crd-descriptions-in-kubectl-explain.md#how-to-reproduce). - Consider the optional changes to `onRefusal` and `serializeNamespace` only if they improve - that output. -6. Regenerate CRDs into a scratch directory and compare with the pre-edit branch snapshot after - stripping every `description` key. Preserve scopes, defaults, validation, and list semantics. -7. Run the repository's required validation for an API-comment implementation. For this proposal - alone, run `task lint-docs` and Vale on this file. diff --git a/docs/review/20261026-metrics-and-labels.md b/docs/review/20261026-metrics-and-labels.md deleted file mode 100644 index 8b17780be..000000000 --- a/docs/review/20261026-metrics-and-labels.md +++ /dev/null @@ -1,95 +0,0 @@ -# Review: the metrics and their labels, before the breaking release - -Every instrument GitOps Reverser exports was read at its recording site. For each one, this review -asks whether it answers a question an operator has, and whether its labels let that operator find -the object to fix. This release breaks the API anyway, so a label change costs the least now. Every -finding below was acted on in the same branch. - -## Scope - -Reviewed on `docs-metrics-architecture-inventory` (PR #450) from `918fa804`: the 65 instruments -registered in [`internal/telemetry/exporter.go`](../../internal/telemetry/exporter.go) (38 -counters, 8 histograms, 5 synchronous gauges, 14 observable gauges), their recording sites, and -[`interpreting-metrics.md`](../interpreting-metrics.md). The cardinality model is -[§7.1 of the plan](../design/metrics-observability-plan.md#71-cardinality-budget). - -## Verdict - -The surface is in good shape. Each stage of the pipeline counts once with a bounded `outcome`, -the loss paths are separate counters, the saturation gauges are read at scrape time, and every -instrument in the guide has an interpretation and a query. None of them is noise: each answers a -question that no condition, Event or log line answers as well. - -The labels had one structural flaw. The branch worker's 21 families were labeled by -`{provider_namespace, provider_name, branch}`, a label set from before `GitBranch` existed. It did -not name the object an operator acts on, so a failing push could not be matched to the -`GitBranch` that reports it. The smaller findings follow. - -## The branch worker's identity - -Every branch-worker series now carries `{gitbranch_namespace, gitbranch_name, provider_namespace, -provider_name}`, set in one place (`BranchWorker.gitBranchAttrs`). The `GitBranch` is the object to -act on; the `GitProvider` groups the branches one credential writes, which is how an expired -credential shows. Neither adds a series: a worker serves one `GitBranch`, and a `GitBranch`'s -provider is immutable. `branch` is dropped because the `GitBranch` already names it. - -A first version of this change dropped the provider too and published it on a separate -`git_branch_info` series for a join. That was undone: the join cost every "by provider" query a -`group_left`, and a `group_left` errors when two pods export the info series at once, as during a -rollout. Labels that add no series belong on the series. - -`git_branch_targets` maps a `GitTarget` to its `spec.gitBranchRef`, so it names the target even -when the `GitBranch` or its provider does not resolve. Before, it published empty provider labels -in that case. - -## Findings - -| # | Finding | What was done | -| --- | --- | --- | -| R1 | `audit_events_total` carried no cluster. With several `KubeProvider`s posting audit, "whose audit stopped" could not be asked. | It gains `audit_route`: the route an event was posted under, when a live `KubeProvider` declares it (the reconcile publishes each provider's route into `attribution.RouteHealth`); `unrecognized` for any other route; `none` for a shared-endpoint event naming none. The raw route is producer-chosen, so it is never a label unchecked. The request histogram stays route-free: a histogram pays its bucket count per label. | -| R2 | `git_fetches_total{reason="bootstrap"}` could never emit. Its caller, `BranchWorker.EnsurePathBootstrapped`, had no production caller, and neither did `ensureRepositoryInitialized`: a worker's first contact is the publication cycle's fetch. | The dead entry point, its five helpers and the reason are deleted. `ensureRepositoryInitialized` moved to a test fixture. The template staging it wrapped is live, and its tests now call `ensureBootstrapTemplateInPath` directly. Ledger rows 1 and 2 keep their numbers, which docs cite, and measure the same two requests. | -| R3 | Suspected: `git_resync_failures_total` overlaps `git_commit_failures_total`. Traced: it does not, because a resync's commit failure never reaches `recordCommitFailure`. It did overlap two others: a resync refused at intake is also `git_queue_refusals_total{kind="resync"}`, and a resync deferred while the remote is unreachable stays pending, yet was counted as failed. | `git_resync_failures_total` gains `reason`: `error` and `timeout` lose a snapshot; `refused` and `deferred` lose nothing. A deferred reply wraps the new `git.ErrResyncDeferred`. The alert selects `reason=~"error\|timeout"`. | -| R4 | `attribution_resolution_wait_seconds` carried `tier`, `event_kind` and the type triple: as a histogram, the largest family in §7.1 (7,680 series in the model install). No query used the type on it. | The type is gone from the histogram; `attribution_resolutions_total` keeps it. | -| R5 | `secret_encryptions_total{outcome="failed"}` named no `GitTarget`. | It gains `gittarget_namespace` and `gittarget_name`, from the write batch's own target, which every path sets. | -| R6 | `InitOTLPExporter` set up a Prometheus exporter and printed to stdout. Nothing is pushed over OTLP. | Renamed `InitPrometheusExporter`; the prints are gone, and the no-op shutdown says why it is one. | -| R7 | `appliers_held`, `stale_snapshots_total` and `claimed_resource_reconciled_total` had no `apply_` prefix. | Renamed `apply_appliers_held`, `apply_stale_snapshots_total` and `apply_claimed_resource_reconciled_total`. | -| R8 | New, found tracing R3: nothing in production sent an atomic write request (`CommitModeAtomic`), so `kind="atomic"` on `git_commit_failures_total` and `git_writes_dropped_total`, and `close_reason="atomic_before_apply"`, never emitted. The commit-failure doc also called `atomic` "a snapshot/resync request", which it was not. | The atomic write path is deleted: `handleAtomicRequest`, `buildAtomicPendingWrite`, `PendingWriteAtomic`, `CommitMode`, and every `WriteRequest` field but `Events`. With one value left, `kind` comes off both counters. Its enqueue logs now name the event's GitTarget; they printed an always-empty request field. Tests of shared behavior were moved onto live writes. | - -Other changes in the same pass: - -- `deadcode -test -tags e2e ./...` found four functions nothing calls, in tests or any binary: - `git.AuthFromSecretData`, `git.GetCommitSigner`, `giteaclient.Client.RegisterUserKeyAsUser` and - `test/utils.UncommentCode`. They are deleted, and the sweep now reports nothing. - -- `apply_requests_total` and `apply_revision_mismatches_total` had no identity at all. They carry - the `GitBranch` and provider that asked and the claim that was asked. -- `watch_sessions_ended_total`, `watch_replay_duration_seconds` and `watch_event_handling_seconds` - gain `source_cluster`: an `expired` storm is one API server's watch cache under pressure. -- The rule "object names are labels in one family only" was already false. It now says what §7.1 - says: configuration objects may be labels, mirrored resources never are. - -## Family by family - -**Configuration state** (`resource_condition`). The best-designed family: status as a label, -`Unknown` synthesized, deleted objects stop publishing. It keeps its generic `resource_namespace` -and `resource_name` keys, the Flux shape; matching a `GitBranch`'s conditions to its worker series -takes a `label_replace`, which the guide shows. - -**Watch ingest.** Labeled by the `GitTarget` and the type, which is what "which tenant stopped -receiving events" needs; the per-type stream instruments now carry the cluster. - -**Write boundary and windows.** Correctly per `GitTarget`: a document, a placement, a window and a -save each belong to one target. - -**Branch worker and apply.** Useful and well split: pushes, retries, fetches, refusals, losses and -the retained backlog each answer a different question. - -**Audit attribution.** The fact-loss trio stays three counters on purpose: their units differ. - -**Catalog, watch plane, Secret encryption, receiver.** Right as they are, with R5 applied. - -## Consequence to plan for - -A dashboard or alert that selects a branch-worker series by `branch`, or names one of the three -renamed apply series, matches nothing after this release. -[`UPGRADING.md`](../UPGRADING.md) lists every series that changed. diff --git a/docs/review/20261027-save-request-simplification.md b/docs/review/20261027-save-request-simplification.md index 2be9ad39c..2736b1779 100644 --- a/docs/review/20261027-save-request-simplification.md +++ b/docs/review/20261027-save-request-simplification.md @@ -594,7 +594,7 @@ The design history, in the steps that matter here: 4. **The race admission created** (#367, September). The request took the fast path (admission) and its edit the slow one (audit join). Saves were "finalized before the write it exists to publish reached the worker", so `attachTimeout` got a 2s default to cover the gap - ([wait-options](../design/commitrequest-save-wait-options.md), l.16-43, 97-114). + (wait-options, l.16-43, 97-114). 5. **"Works without audit" was never true on its own.** A named request on an audit-less cluster meets windows that name nobody: `WindowMismatch`, "every time, permanently" (wait-options l.277-281). It became true only on 2026-10-26, with `SavedBy`/`TakeOver` @@ -602,7 +602,7 @@ The design history, in the steps that matter here: 6. **`RemoteSaveRequest`** (`beddcdef` to `70d07d91`). It is authored from source audit. It was built so that "a customer's only API surface is their kcp workspace" holds with no exceptions: no tenant grant into the control cluster, and no `/editor/commit` on the save path - ([source-side-save-plan](../future/source-side-save-plan.md), l.46-59). + (source-side-save-plan, l.46-59). In short: admission was the right answer to a pipeline that no longer exists. Every argument for it since then has been about either takeover or gitops-api's current save path, and the source-side diff --git a/docs/sops-age-guide.md b/docs/sops-age-guide.md index 1c7fe0e33..18d489bfd 100644 --- a/docs/sops-age-guide.md +++ b/docs/sops-age-guide.md @@ -194,5 +194,3 @@ The bootstrap template currently contains a static recipient in - For install and starter resource setup: [`../README.md`](../README.md) - For chart quickstart values: [`configuration.md`](configuration.md) -- For the deeper bootstrap/encryption architecture: [`design/sops-repo-bootstrap-and-key-management-architecture.md`](finished/sops-repo-bootstrap-and-key-management-architecture.md) -- For explicit non-goals and deferred work: [`design/sops-repo-bootstrap-out-of-scope.md`](finished/sops-repo-bootstrap-out-of-scope.md) diff --git a/docs/spec/README.md b/docs/spec/README.md index 7ca326264..fc276c127 100644 --- a/docs/spec/README.md +++ b/docs/spec/README.md @@ -41,7 +41,6 @@ three lifecycle classes and was unreadable spread across them. The rows below st | [`manifestedit-field-ownership-spike.md`](manifestedit-field-ownership-spike.md) | "the API wins" — full-object ownership, and the do-not-build list | | [`type-followability.md`](type-followability.md) | is a type followable, and if not, the single reason | | [`type-lifecycle-events-and-wobble-settling.md`](type-lifecycle-events-and-wobble-settling.md) | removal grace and flap coalescing | -| [`gvk-gvr-mapping-layer.md`](gvk-gvr-mapping-layer.md) | the GVK↔GVR bijection contract | | [`where-validation-lives.md`](where-validation-lives.md) | schema → CEL → **the reconciler**; a webhook only for what exists solely at admission | | [`sops-single-file-no-multidoc.md`](sops-single-file-no-multidoc.md) | one encrypted file is one document | | [`scale-subresource-audit-rehydration.md`](scale-subresource-audit-rehydration.md) | history: the `/scale` field-patch design; a scale is now mirrored from the watch, and the field-patch writer is what remains | diff --git a/docs/spec/current-manifest-support-review.md b/docs/spec/current-manifest-support-review.md index f5d8b4d71..fddd7f9d7 100644 --- a/docs/spec/current-manifest-support-review.md +++ b/docs/spec/current-manifest-support-review.md @@ -17,7 +17,7 @@ > Related: > `implementation-plan.md`, > [reconcile-via-watchlist-mark-and-sweep.md](reconcile-via-watchlist-mark-and-sweep.md), -> [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md), +> gvk-gvr-mapping-layer.md (removed), > [current-manifest-support-review-feedback.md](current-manifest-support-review.md), > [manifest-inventory-file-agnostic-placement.md](manifest-system.md), > [manifestedit-abstraction-plan.md](manifest-system.md), @@ -357,7 +357,7 @@ Two constraints to respect from the start, not retrofit: - **The GVK↔GVR resolver must be built**, as an injectable source (live informers, kubeconfig, static snapshot, or nil for structure-only). Everything downstream depends on it; see - [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md). + gvk-gvr-mapping-layer.md (removed). - **Materialization must be bounded — spatially and temporally.** *Spatially:* identity indexing needs only a cheap header parse; build the full `manifestedit` node tree only for documents a plan action touches (`DocumentModel.Snapshot` as a @@ -926,7 +926,7 @@ model rather than a second write path. usually carry GVR/name, while manifests are keyed by GVK/name; the writer should receive a resolved `ResourceIdentifier`/manifest identity pair from the planning layer and delete by `RecordRef`. See - [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md). + gvk-gvr-mapping-layer.md (removed). - Add a resource-identity index beside the existing manifest-identity index. The writer should locate watched resources by `ResourceIdentifier`, while retaining GVK for YAML fidelity and diagnostics. @@ -1001,7 +1001,7 @@ The **starter requirement** (structure-only, no cluster needed) is to define *recognized = parses as KRM* and to fail the GitTarget when any YAML file falls in bucket 2 (non-KRM YAML), alongside the duplicate-identity gate. Once the watched API surface is available through the -[GVK/GVR mapping layer](gvk-gvr-mapping-layer.md), bucket 4 (unwatched API-backed +GVK/GVR mapping layer, bucket 4 (unwatched API-backed KRM) joins the same refusal. Bucket 5 is the explicit exception: allowlisted non-API KRM is accepted but not materialized. Non-YAML files (bucket 1) are always ignored and never cause failure. @@ -1141,7 +1141,7 @@ core logic. > > Deliberately deferred: comparing those GVKs against a live API to decide what > is *watched / unwatched / orphaned*. The next model needs the -> [GVK/GVR mapping layer](gvk-gvr-mapping-layer.md), the +> GVK/GVR mapping layer, the > watched/unwatched/orphan comparison, the plan computation, and wiring the same > library into the live writer. > @@ -1198,7 +1198,7 @@ the live writer. diagnostic message text with structured reasons from `manifestedit`. 2. **Add the API/source-of-truth input.** Define an injectable source for the watched API surface and current desired resources by implementing - [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md). It should work with live + gvk-gvr-mapping-layer.md (removed). It should work with live controller state, a CLI kubeconfig-backed source, and test snapshots. Resolve every API-backed KRM document's GVK to a `ResourceIdentifier` where possible, record unresolved GVKs as diagnostics, and add indexes by both manifest diff --git a/docs/spec/e2e-test-design.md b/docs/spec/e2e-test-design.md index 2e9b4aabb..bb7191917 100644 --- a/docs/spec/e2e-test-design.md +++ b/docs/spec/e2e-test-design.md @@ -45,8 +45,6 @@ What it does not promise: - removal of all remaining shell from the harness - total independence from shared cluster-side services -Those broader open items live in [e2e-finish-plan.md](../design/e2e-finish-plan.md). - ## Cross-Cutting Requirements These requirements should shape any further e2e reorganization: diff --git a/docs/spec/gittarget-isolation-on-rule-change.md b/docs/spec/gittarget-isolation-on-rule-change.md index 24bb3bd25..26cfd49df 100644 --- a/docs/spec/gittarget-isolation-on-rule-change.md +++ b/docs/spec/gittarget-isolation-on-rule-change.md @@ -13,7 +13,7 @@ > exist. The isolation guarantee is still pinned end to end by > `test/e2e/gittarget_isolation_e2e_test.go` ("Manager GitTarget Isolation", > label `manager`). Rule `operations` were later removed from both rule kinds -> ([collection rename](../design/collection-terminology-rename.md)). +> (collection rename). ## Problem diff --git a/docs/spec/gvk-gvr-mapping-layer.md b/docs/spec/gvk-gvr-mapping-layer.md deleted file mode 100644 index 45307b4d5..000000000 --- a/docs/spec/gvk-gvr-mapping-layer.md +++ /dev/null @@ -1,473 +0,0 @@ -# Manifest GVK/GVR Mapping Layer - -> **spec**, kept as design history. Index: [`../INDEX.md`](../INDEX.md) -> -> Status: built 2026-06-04, then superseded by [type-followability.md](type-followability.md). The -> layer this page designs (`internal/mapping` with `ResourceMapper`, `StructureOnlyMapper`, -> `StaticSnapshotMapper` and `mapping.Status`; `watch.CatalogMapper`; the catalog's `byGVK`, -> `LookupGVK`, `LookupGVR` and `APIResourceEntry`; `RuleGVRResolver`) no longer exists. The mapper -> is now `typeset.Lookup` ([`internal/typeset/lookup.go`](../../internal/typeset/lookup.go)): -> `Ready()` plus an exact-match `ByGVK(gvk) (TypeRecord, bool)`, and a caller gates on -> `TypeRecord.Followable()` instead of a status vocabulary. The controller hands the git writer its -> `typeset.Registry`; tests use `typeset.NewSnapshotRegistry`; a Lookup that is not ready is the -> structure-only mode, which is still the `manifest-analyzer` default. The kubeconfig-backed mapper -> and the `APIMappingReady` condition were never built. What carries over is the reasoning: -> discovery is the only source of GVK/GVR truth, a mapping must match exactly, and degraded -> discovery is never evidence of absence. -> -> Captured 2026-06-04. -> Related: -> [current-manifest-support-review.md](current-manifest-support-review.md), -> [reconcile-via-watchlist-mark-and-sweep.md](reconcile-via-watchlist-mark-and-sweep.md), -> [kubernetes-api-resource-catalog.md](../facts/kubernetes-api-resource-catalog.md), -> [`internal/watch/api_resource_catalog.go`](../../internal/watch/api_resource_catalog.go), -> [`internal/manifestanalyzer/analyzer.go`](../../internal/manifestanalyzer/analyzer.go) - -## Summary - -The manifest materialization plan depends on a layer that does not exist yet: -mapping manifest identity (`apiVersion`, `kind`, namespace, name) to resource -identity (`group`, `version`, `resource`, namespace, name), and back again. - -`APIResourceCatalog` and `RuleGVRResolver` already solve the watch side of this -problem: rules become concrete watched GVRs through trusted Kubernetes discovery. -They do not yet provide a manifest-facing `GVK <-> GVR` abstraction, and the -manifest analyzer deliberately has a no-cluster mode. The missing layer must -therefore be an injected interface, not a package-global call to discovery. - -This is not decorative plumbing. It is the boundary that decides whether an -existing manifest document is watched, unwatched, stale, ambiguous, disallowed, -or unknowable. Every later phase of the materialized model depends on that -decision being stable and explicit. - -## What Kubernetes Actually Provides - -Kubernetes gives us an API-surface catalog through discovery. From `/api`, -`/apis`, and the per-group/version discovery documents, a client can learn: - -- API groups and versions the server currently supports. -- The preferred version for each API group. -- Resource entries for each group/version. -- The REST resource name, normally plural, such as `configmaps` or - `deployments`. -- The `kind` served by that resource entry. -- Whether the resource is namespaced. -- The verbs the endpoint advertises, such as `get`, `list`, `watch`, `create`, - `update`, `patch`, and `delete`. -- Short names, categories, and singular names when the server reports them. - -In client-go terms, this is the data returned by discovery calls such as -`ServerGroupsAndResources()`, `ServerPreferredResources()`, and -`ServerResourcesForGroupVersion()`. The project already stores the important -part of that shape in `APIResourceEntry`: - -```text -GVR -GVK -Namespaced -Verbs -Preferred -Subresource -Allowed -PolicyReason -``` - -That is enough for the mapping layer's first job: exact mapping between a served -GVK and its served GVR, plus scope and verb metadata. - -Kubernetes does **not** give us everything: - -- Discovery is not a live object snapshot. -- Discovery is not OpenAPI schema validation. -- Discovery does not prove the operator's RBAC can list or watch the resource. -- Discovery can partially fail; an incomplete discovery response must not be - treated as authoritative absence. -- Discovery can expose subresources like `deployments/status` and - `deployments/scale`, which are not normal mirrored objects. -- Discovery does not mean a later `List()` call will succeed against every - aggregated API server. - -So the right mental model is: discovery tells us what resource types the API -server claims to serve, with enough metadata to map GVK and GVR. Object state, -permissions, and destructive reconciliation still require their own trust gates. - -## How Much Theory Matters - -In ordinary Kubernetes usage, the types usually line up: - -- `apiVersion: v1`, `kind: ConfigMap` maps to `v1/configmaps`. -- `apiVersion: apps/v1`, `kind: Deployment` maps to - `apps/v1/deployments`. -- A CRD with `spec.names.kind: Widget` and `spec.names.plural: widgets` appears - in discovery as `example.com/v1`, kind `Widget`, resource `widgets`. - -That practical reality is useful. It means the mapper should be boring in the -happy path and should not block the whole project on theoretical edge cases. - -But pluralization cannot be guessed safely, and the edge cases matter exactly -where GitOps Reverser is most dangerous: - -- Some resource names are irregular or non-obvious. -- Multiple resources can share a kind shape across groups or versions. -- A manifest can use an API version the current cluster no longer serves. -- A CRD can serve multiple versions for the same kind. -- A resource can be served but excluded by GitOps Reverser policy. -- Subresources can share familiar-looking kinds while not being objects we mirror. -- Aggregated APIs can appear in discovery but fail later when listed. - -The conclusion is deliberately modest: do not build a giant schema system, but -also do not infer GVRs from kind strings or path layout. Use discovery/catalog -data when available; otherwise mark the mapping as unresolved and keep the -analyzer structure-only. - -## Responsibilities - -The mapping layer owns these decisions: - -- Map a manifest GVK to one concrete served GVR. -- Map an event or object GVR to the served GVK used for manifest identity. -- Attach namespaced/cluster-scoped information to the mapping. -- Report whether the mapping is trusted, stale, ambiguous, disallowed, or - unavailable. -- Tell the materialized manifest store whether a document belongs to a watched - set for this GitTarget. -- Preserve the analyzer's no-cluster mode by allowing a nil implementation that - records only structure. - -The mapping layer does not: - -- validate object schemas, -- convert one API version to another, -- prove RBAC, -- list objects, -- choose Git file placement, -- decide whether unwatched content is refused, reported, or pruned. - -## Source Of Truth Versus Mapper Source - -Kubernetes discovery is the source of truth for served GVK/GVR mappings. The API -server is the only authority that can say "this cluster serves kind `Deployment` -at `apps/v1` through resource `deployments`" or "this CRD's plural resource name -is `icecreamorders`." GitOps Reverser should not invent that relationship from -English pluralization, generated paths, or hard-coded assumptions. - -`MapperSource` does **not** mean there are four competing truths. It describes how -this process obtained, cached, replayed, or intentionally declined to obtain -Kubernetes discovery data. - -- `live-catalog`: the controller reads the in-process `APIResourceCatalog`, which - is refreshed from Kubernetes discovery and shared with watch planning. The API - truth is current cluster discovery, cached locally. -- `kubeconfig`: a CLI command contacts a cluster through kubeconfig, builds a - temporary catalog, then maps through the same catalog semantics. The API truth - is current cluster discovery, fetched on demand. -- `static-snapshot`: tests or offline review load a serialized catalog-shaped - fixture. The API truth is a captured or declared snapshot, not live truth. -- `structure-only`: no API discovery data is available or desired; the analyzer - only parses manifest structure. There is no API truth, so mapping is - deliberately unknown. - -That distinction is why `MappingStructureOnly` is not an error. It is the honest -answer for "we know this YAML looks like KRM, but we did not ask any API surface -what REST resource serves it." Conversely, when a mapper is backed by discovery -data, absence can only be trusted if the relevant group/version discovery is not -degraded. - -## Interface - -> **Status (2026-06-04): implemented.** The interface and the runtime-independent -> implementations live in `internal/mapping` -> (`ResourceMapper`, `StructureOnlyMapper`, `StaticSnapshotMapper`, and the shared -> `ResolveGVK` reduction); the catalog-backed implementation is -> `watch.CatalogMapper`, built on the -> catalog `byGVK`/`LookupGVK` additions -> ([`api_resource_catalog.go`](../../internal/watch/api_resource_catalog.go)). -> The doc warned the concrete Go names could move, and two did: to satisfy the -> repository's no-stutter lint, `MappingResult` is `mapping.Result` and -> `MappingStatus` is `mapping.Status`. The `Mapping*` status constants below kept -> their names. `internal/mapping` has no dependency on `internal/watch`, so the -> analyzer can resolve mappings without importing the watch manager. All of it was -> later replaced by `internal/typeset`, which keeps that leaf property (see the -> status at the top). - -The concrete Go names can move, but the interface should make the dependency -explicit: - -```go -type ResourceMapper interface { - Source() MapperSource - Ready() MapperReadiness - Generation() uint64 - - GVRForGVK(ctx context.Context, gvk schema.GroupVersionKind) (Result, error) -} - -type MapperSource string - -const ( - MapperSourceLiveCatalog MapperSource = "live-catalog" - MapperSourceKubeconfig MapperSource = "kubeconfig" - MapperSourceStaticSnapshot MapperSource = "static-snapshot" - MapperSourceStructureOnly MapperSource = "structure-only" -) - -type MapperReadiness struct { - Ready bool - Degraded bool - Generation uint64 - Reason string -} - -type Result struct { - GVK schema.GroupVersionKind - GVR schema.GroupVersionResource - - Namespaced bool - Verbs []string - Preferred bool - Allowed bool - - Status Status - Reason string -} - -type Status string - -const ( - MappingResolved Status = "Resolved" - MappingUnserved Status = "Unserved" - MappingAmbiguous Status = "Ambiguous" - MappingDisallowed Status = "Disallowed" - MappingSubresource Status = "Subresource" - MappingCatalogUnavailable Status = "CatalogUnavailable" - MappingDiscoveryDegraded Status = "DiscoveryDegraded" - MappingStructureOnly Status = "StructureOnly" -) -``` - -`GVRForGVK` must require an exact group/version/kind match. It should not -silently map `extensions/v1beta1 Deployment` to `apps/v1 Deployment`, even though -a human understands the relationship. Version conversion is a later feature and -requires a real conversion source, not a REST mapping guess. - -Resource-to-manifest identity is not part of this mapper contract. Delete planning -starts from the GitTarget folder's `ByResourceIdentity` inventory; if that inventory -does not already contain the resource, the planner does not re-derive a manifest -identity through a reverse lookup. - -Errors should be reserved for implementation failures: discovery call failed, -snapshot could not be loaded, malformed static data, context cancellation. -Expected lookup outcomes should be returned as `Status` so callers can -make policy decisions without parsing error strings. - -## Implementations - -These implementations share the same mapping contract. They differ only in how -the catalog data is obtained and how much trust callers can place in freshness. - -### Live Catalog Mapper - -Used by the controller and watch manager. - -This wraps the existing `APIResourceCatalog`. The catalog already carries GVR, -GVK, scope, verbs, preferred-version, subresource, allowed-policy, readiness, and -generation data. The main missing pieces are lookup methods/indexes for GVK and -status-rich mapping results. - -Required catalog additions: - -- `byGVK[schema.GroupVersionKind] -> []APIResourceEntry` -- exported lookup for exact GVK -- mapping status helpers that preserve degraded lookup state -- generation-aware result reporting - -The live mapper must not call Kubernetes discovery directly. The catalog owns -discovery refresh and trust state; the mapper reads that trusted local view. - -### Kubeconfig Discovery Mapper - -> **Status (2026-06-04): deferred.** `MapperSourceKubeconfig` exists as a constant, -> but the kubeconfig-backed mapper itself is not built yet. The controller uses -> `live-catalog` and tests use `static-snapshot`, so nothing on the B3/M3/M6 path -> needs this; it lands with the optional CLI cluster-check mode (Implementation -> Order step 5). - -Used by a CLI or offline command that is allowed to contact a cluster. - -This implementation builds an `APIResourceCatalog` from a kubeconfig-backed -discovery client, then exposes the same mapping behavior as the live catalog -mapper. It gives humans a way to run the manifest analyzer in "check this folder -against this cluster's API surface" mode without starting the controller. - -This mode can report mapping and watched/unwatched classification, but it should -still not list live objects unless a separate command explicitly asks for a full -plan against cluster state. - -This is not a second interpretation of Kubernetes semantics. It is the same -discovery truth as `live-catalog`, just fetched by a short-lived command instead -of by the controller's continuously refreshed catalog. - -### Static Snapshot Mapper - -Used by tests, CI, and offline review. - -This implementation loads a serialized API-resource catalog snapshot. The shape -should be close to `APIResourceEntry`, not raw Kubernetes discovery JSON, because -tests and design fixtures should express the project contract directly. - -Static snapshots are also the right way to make manifest materialization tests -deterministic. A fixture can say "this cluster serves apps/v1/deployments and -v1/configmaps" without needing a kube-apiserver. - -Because a static snapshot is not live discovery, it should be treated as an -explicit test/review input. It can model old clusters, partial catalogs, policy -exclusions, ambiguity, and degraded discovery on purpose, but it must not be -mistaken for proof about the cluster currently running the controller. - -### Structure-Only Mapper - -Used by the current `manifest-analyzer` default. - -This implementation always returns `MappingStructureOnly`. It is not a failure. -It means the analyzer can inventory valid KRM, duplicate identities, -multi-document files, encryption boundaries, and GVK counts, but cannot decide -whether a document is watched or orphaned. - -This is the mode that preserves the analyzer's current no-cluster promise. -It should never produce creates, deletes, watched/unwatched conclusions, or -destructive adoption decisions. - -## Manifest Store Integration - -The materialized store should carry both identities when a mapper can provide -them: - -```text -ManifestIdentity = apiVersion + kind + namespace + name -GVK = group + version + kind -ResourceIdentity = group + version + resource + namespace + name -mapping.Status = resolved / unserved / ambiguous / ... -``` - -Store construction should do the cheap parse first: - -1. Read YAML headers and manifest identity. -2. Parse `apiVersion` and `kind` into GVK. -3. Ask the injected mapper for GVR. -4. If resolved, populate the resource-identity index. -5. If unresolved, keep the document in the manifest-identity and GVK indexes and - attach a diagnostic. - -This keeps bounded materialization intact. Full YAML node trees are still built -only for documents a plan action touches. - -## Watched Classification - -Mapping a document to a GVR is not enough to say it is managed by a GitTarget. -The document must also match the GitTarget's effective watch selection. - -Classification should be a second step: - -```text -document GVK -> mapper -> document GVR -GitTarget rules -> RuleGVRResolver -> watched GVR set -document GVR in watched set? -> tracked/untracked/orphan decision -``` - -This intentionally reuses the existing WatchRule resolution semantics instead -of making the manifest layer interpret rules on its own. - -Status categories: - -| Category | Meaning | -|---|---| -| `tracked` | Mapping resolved and GVR is selected by this GitTarget. | -| `unwatched` | Mapping resolved but GVR is not selected by this GitTarget. | -| `unserved` | GVK is not served by trusted catalog data. | -| `ambiguous` | More than one served resource could match. | -| `disallowed` | Served, but excluded by GitOps Reverser resource policy. | -| `unknown` | Mapper is unavailable, degraded, or structure-only. | - -Only `tracked` documents can be swept as managed resources. `unwatched`, -`unserved`, `ambiguous`, `disallowed`, and `unknown` documents are acceptance or -status facts; they are not delete targets by default. - -## GitTarget Start Conditions - -The GitTarget lifecycle should gain an explicit API-surface/mapping gate before -the initial snapshot can make destructive decisions. - -Proposed condition: - -```text -Type: APIMappingReady -Reason: Resolved | CatalogUnavailable | DiscoveryDegraded | MappingFailed -Message: API resource mapping is ready for all watched resource types -``` - -`Ready=True` should require this gate in addition to the existing validation, -encryption, snapshot, and event-stream gates. - -Startup rule: - -1. The catalog must have trusted initial data. -2. The GitTarget's WatchRules and ClusterWatchRules must resolve through - `RuleGVRResolver`. -3. Every watched GVR needed for the initial snapshot must have a resolved watched-type - table entry carrying its GVK. -4. Any manifest-store acceptance mode that needs watched/unwatched decisions must - have a mapper source stronger than `structure-only`. -5. If discovery is degraded for a lookup scope that could affect the target, hold - the target at `APIMappingReady=False` instead of starting a destructive - snapshot from partial knowledge. - -This does not mean every KRM document in the repository must be served before -GitTarget startup. It means the system must know enough to classify the watched -set it is about to manage. Unwatched or unserved documents can still be refused -by adoption policy without being deleted. - -## Failure Policy - -Mapping failures should be boring and visible: - -- `CatalogUnavailable`: wait or fail closed; do not start initial snapshot. -- `DiscoveryDegraded`: preserve last-known-good mappings; do not turn absence - into deletes. -- `Unserved`: refuse or report the document, depending on adoption policy. -- `Ambiguous`: ask for a more specific rule or static mapping; do not guess. -- `Disallowed`: surface as policy, not as "not served." -- `StructureOnly`: continue inventory-only analysis; skip watched/orphan - classification. - -The destructive invariant is the same as the watch/catalog architecture: an -inability to observe the API surface is not evidence that a resource should be -deleted from Git. - -## Implementation Order - -1. ✅ Add `byGVK` and exact GVK lookup to `APIResourceCatalog` (also `LookupGVR` - and the `CatalogLookup` trust state). -2. ✅ Introduce the `ResourceMapper` interface and a catalog-backed implementation - (`watch.CatalogMapper`). -3. ✅ Add a static-snapshot implementation for unit tests - (`mapping.StaticSnapshotMapper`), alongside the structure-only mapper. -4. Change manifest store construction to accept a mapper and record the resolved - `mapping.Status` per document. -5. Keep `manifest-analyzer` structure-only by default; add an optional - kubeconfig/static-catalog mode later. -6. Add the GitTarget mapping readiness gate before initial snapshot planning. -7. Teach status/reporting to summarize mapped, unmapped, watched, unwatched, and - degraded counts. - -## References - -- Kubernetes API discovery: - -- Kubernetes API concepts: - -- Kubernetes API definitions: - -- client-go discovery: - -- metav1 `APIResource`: - diff --git a/docs/spec/manifest-system.md b/docs/spec/manifest-system.md index 8a48e529e..3c20f2c72 100644 --- a/docs/spec/manifest-system.md +++ b/docs/spec/manifest-system.md @@ -145,7 +145,7 @@ dropped, and flapping types are coalesced by a settle window ([`type-lifecycle-events-and-wobble-settling.md`](type-lifecycle-events-and-wobble-settling.md)). Full model: [`type-followability.md`](type-followability.md). The GVK/GVR contract -underneath it: [`gvk-gvr-mapping-layer.md`](gvk-gvr-mapping-layer.md). +underneath it: [`type-followability.md`](type-followability.md). ## Known gap: external Git drift diff --git a/docs/spec/mutation-capture-lab-design.md b/docs/spec/mutation-capture-lab-design.md index f28b399e7..f4b367f16 100644 --- a/docs/spec/mutation-capture-lab-design.md +++ b/docs/spec/mutation-capture-lab-design.md @@ -515,6 +515,5 @@ lab-only behavior to the Helm chart. - [Issue #168: Watch mode](https://github.com/ConfigButler/gitops-reverser/issues/168) - [Audit Ingestion Decision Record](../architecture.md) -- [Watch & Catalog Architecture](../design/watch-and-catalog-architecture.md) - [Architecture](../architecture.md) - `test/mutationlab/README.md` — the scenario index (Row → test → corpus dir → sources) diff --git a/docs/spec/reconcile-via-watchlist-mark-and-sweep.md b/docs/spec/reconcile-via-watchlist-mark-and-sweep.md index e7d3dc4bb..7c6707ed2 100644 --- a/docs/spec/reconcile-via-watchlist-mark-and-sweep.md +++ b/docs/spec/reconcile-via-watchlist-mark-and-sweep.md @@ -10,7 +10,7 @@ > once, replayed as decided" to "Stale snapshots are discarded" describe current behavior. > Related: > [current-manifest-support-review.md](current-manifest-support-review.md), -> [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md), +> [type-followability.md](type-followability.md), > [current-manifest-support-review-feedback.md](current-manifest-support-review.md), > [`internal/git/manifestedit/DECISION.md`](../../internal/git/manifestedit/DECISION.md) diff --git a/docs/spec/sops-single-file-no-multidoc.md b/docs/spec/sops-single-file-no-multidoc.md index c725fd61c..82b7ac416 100644 --- a/docs/spec/sops-single-file-no-multidoc.md +++ b/docs/spec/sops-single-file-no-multidoc.md @@ -7,8 +7,7 @@ > Related: > [../layout/new-file-placement-rules.md](../layout/new-file-placement-rules.md), > [../layout/contextual-namespace.md](../layout/contextual-namespace.md), -> [../finished/sops-repo-bootstrap-and-key-management-architecture.md](../finished/sops-repo-bootstrap-and-key-management-architecture.md), -> [../finished/sops-repo-bootstrap-out-of-scope.md](../finished/sops-repo-bootstrap-out-of-scope.md) +> [../sops-age-guide.md](../sops-age-guide.md) ## Decision diff --git a/docs/spec/type-followability.md b/docs/spec/type-followability.md index 9e24a9fa7..7d3541483 100644 --- a/docs/spec/type-followability.md +++ b/docs/spec/type-followability.md @@ -385,6 +385,5 @@ writer takes sensitivity from the operator's sensitive-resource policy. ## References - [scale-subresource-audit-rehydration.md](scale-subresource-audit-rehydration.md) -- [gvk-gvr-mapping-layer.md](gvk-gvr-mapping-layer.md) - [resource-types.yaml](../facts/resource-types.yaml) - [subresources.md](../facts/subresources.md) diff --git a/docs/spec/watch-event-ordering-and-attribution-grace.md b/docs/spec/watch-event-ordering-and-attribution-grace.md index 93d2eedf1..a64f30e9c 100644 --- a/docs/spec/watch-event-ordering-and-attribution-grace.md +++ b/docs/spec/watch-event-ordering-and-attribution-grace.md @@ -5,7 +5,7 @@ > Status: **implementation explainer** > Checked: 2026-10-20 > Related: [Kubernetes watch options](../facts/kubernetes-watch-options.md), -> [Watch-first ingestion architecture](../finished/watch-first-ingestion-architecture.md), +> [Architecture](../architecture.md), > [Attribution contract](attribution.md), > [Label-selection design](../design/watches-labels-simplification.md) diff --git a/test/mutationlab/README.md b/test/mutationlab/README.md index e42feddcc..1c21e7aed 100644 --- a/test/mutationlab/README.md +++ b/test/mutationlab/README.md @@ -69,7 +69,7 @@ because it carries a response body to recover the name from, which is what makes BODY rather than the name the thing the aggregated rows are actually missing. Rows 16 and 17 test the watch transport itself; the driver uses the lab's targeted `/watch-probe` endpoint so transport-only events can be scenario-attributed — see the -[watch-first ingestion architecture](../../docs/finished/watch-first-ingestion-architecture.md) +[watch-first ingestion architecture](../../docs/architecture.md) design notes. > ⚠️ **The "no audit" rows record this cluster's audit POLICY, not the API server.** The lab cluster