Skip to content
10 changes: 7 additions & 3 deletions docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,12 @@ should land before the next tag is cut (the last release is v0.55.0). Checked ag
`attachWaitingCommitRequests` survives its removal); a late stale applier write (the model
never withholds);
- the GitTarget-generation and checkout-reset parts of a footprint revision are untested;
- a counter for window fallbacks (`footprint_fallback` counts the windows it closes, not the
fallback).
- measure the footprint index on realistic repositories and commit rates before widening the
layouts it proves. It is rebuilt whenever its revision changes, every local commit included;
a review measured a rebuild at about 3.5 ms and 3.9 MB allocated for 100 small manifests, and
33 ms and 37 MB for 1,000 (allocation, not retained heap), against 19 µs for a reuse. The
retained-byte budget does not charge the indexes, the accepted versions or the withheld-field
records, so a full budget does not bound the worker's memory.

## Applier claims and bi-directional editing

Expand All @@ -46,7 +50,7 @@ should land before the next tag is cut (the last release is v0.55.0). Checked ag
[branch-worker-event-model.md](design/branch-worker-event-model.md#saves-lose-execution-identity-across-a-restart).
- [ ] **Gaps in the withheld-field record** (`internal/git/withheld_fields.go`): a field patch in the
apply window forgets the object's accepted version, so a withhold over it records nothing;
`objectKey` leaves out the GitTarget and the cluster (it predates #466); accepted versions held
accepted versions held
for render fidelity (`recordUnpublishedEdits`) and resyncs with `Initial: false` are untested;
no counter shows how many fields are withheld.
- [ ] **Reopen the render-fidelity gate after a Git repair, automatically.** Today a target on
Expand Down
13 changes: 8 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,10 @@ The solution, in the vocabulary used throughout this document:
`unknown (attribution unresolved)`, so an installation that expects attribution can investigate its audit
policy, ingress, Redis connection, or source identity instead of mistaking the result for configured-author mode.
- **One `BranchWorker` per Git branch serializes all writes.** Every write to a branch funnels through a
single worker and a single commit window, which keeps concurrent GitTargets and authors from racing
each other into a corrupt tree.
single worker and its one event loop, which owns the checkout and orders every Git operation, so
concurrent GitTargets and authors cannot race each other into a corrupt tree. The loop may hold
several commit windows, one per author, only while it proves their files do not overlap
([concurrent-author-windows.md](design/concurrent-author-windows.md)).

***

Expand Down Expand Up @@ -756,9 +758,10 @@ flowchart TD
PUSH --> DONE[Remote branch updated]
```

The important boundary is the open window: it accepts only the same author and `GitTarget`. A
`CommitRequest` from another author never closes someone else's window; it resolves as a benign no-commit
instead.
The important boundary is the window: each accepts only the same author and `GitTarget`. Two
authors keep a window each, side by side, while their files are proven not to overlap; otherwise the
newcomer's write closes the other window first. A `CommitRequest` attaches only to its own author's
window and never closes someone else's; with none to attach to, it resolves as a benign no-commit.

### Remote moved while we were writing

Expand Down
11 changes: 7 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -3311,8 +3311,9 @@ request produced a pushed commit:
request is refused at once instead. Save again with a new request once the earlier one is `Ready`.
A request for the `Next` window is not refused by one that is collecting: it closes that window
under the earlier request's message first. Saves by different authors do not race for a
message: a write by another author closes the open window, and its message, before it opens one
of its own.
message: each author's writes collect in a window of their own, side by side while their files
are proven not to overlap, and otherwise a write by another author closes the open window, and
its message, before it opens one of its own.

`SaveInProgress` is decided when the branch worker registers the request, which is after its
author was found, not when it is created. A client that relies on it to keep two of its saves
Expand All @@ -3339,8 +3340,10 @@ request produced a pushed commit:
A request is served again after GitOps Reverser restarts: the path that owns it finds its own
claim, and resumes with the author it recorded. A graceful stop, such as an upgrade or a rollout,
first finishes the saves it holds: a request collecting its window is committed with its message,
pushed, and its outcome written before the pod exits. A request still waiting for a window is
picked up again by the next pod. A crash, or a stop that runs out of its grace period, loses the
pushed, and its outcome written before the pod exits. Edits the branch worker had already accepted
are committed too, each in its author's window; an edit still on its way from the cluster is not,
and the next pod's startup replay writes it under a generated message. A request still waiting for
a window is picked up again by the next pod. A crash, or a stop that runs out of its grace period, loses the
open window: the next pod's startup replay writes its edits under a generated message, and the
request attaches again as if new, so it can end `NoWindow`.

Expand Down
29 changes: 21 additions & 8 deletions docs/design/concurrent-author-windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Neither is needed for the benefit this page delivers.

## The problem

A branch worker holds **one** open commit window at a time (`branchWorkerEventLoop.openWindow`). A
Before stage 2, a branch worker held **one** open commit window at a time (`branchWorkerEventLoop.openWindow`). A
window belongs to one author on one `GitTarget`, and a write that is anyone else's cannot join it
(`openWindow.canAppend`). It closes the window first, with close reason `identity_change`, and opens
its own.
Expand Down Expand Up @@ -51,6 +51,14 @@ gitops-api avoids it today by serializing every save and its edits behind one lo
(`commitGate`). A client that creates `CommitRequest`s on a source cluster from the browser has no
such lock, so the split comes back.

What this page delivers is **commit grouping**: when different authors edit files it can prove
independent, each gets a commit of their own. It is not a transaction. A save still collects the
writes that reach its author's window before its timers close it; events and the save arrive
independently, so a deadline can still split a save, and nothing proves a save holds every edit its
author meant. A footprint proves that two windows change different files, not that two objects in
those files are consistent with each other. "This save is exactly these edits" needs explicit
membership and a completion signal, which longer timers cannot give.

Two saves by **one** author on one `GitTarget` are a different problem, and it is already handled.
A window carries one message, so the second save is refused with `SaveInProgress` while the first
still waits for a window or collects one. That stays true under this proposal, which changes only
Expand Down Expand Up @@ -128,7 +136,8 @@ A write is **proven independent** only when all of these hold:
is off, so the write registers nothing in a `kustomization.yaml` and creates none (a new file is
always added to the kustomization governing it, and `useKustomize` creates one on the first new
document);
- the object's destination is the built-in path, with no `placement.byType` or `placement.default`
- an object with a document already is written where that document lives, at any path; a new
object's destination is the built-in path, with no `placement.byType` or `placement.default`
template for its type: a template can bundle several objects into one file (`all.yaml`);
- the destination file holds only this object's document, or does not exist yet. The built-in
path has no API version, so two versions of one object share a file: they count as one object.
Expand Down Expand Up @@ -286,8 +295,8 @@ window, so they are proven against today's behavior before they are used to judg

## Stages

Stages 1 and 2 are built; stages 3, 4 and 5 are proposed. Stage 3 gates the next release, because
stage 2 shipped without stage 4's flag.
Stages 1 and 2 are built; stage 3 is in progress, stage 4 was decided against, and stage 5 is
proposed. Stage 3 gates the next release, because stage 2 shipped without a flag.

1. **Built. Extract window management without changing behavior.** Move the window bookkeeping
(`openWindow`, its timers, attach and finalize) behind one type that holds a set of windows, with
Expand All @@ -300,10 +309,14 @@ stage 2 shipped without stage 4's flag.
conflicting writes, delete and recreate, `attach_next`, zero-duration windows, resync, shutdown,
failed publication and replay, and buffer pressure, each against the reference model, which
first has to keep work pending across failed pushes and retries.
4. **Proposed. Ship behind a startup flag**, off by default at first. The flag is read once, at start; changing
it takes a restart, and the restart drains every open window through the normal shutdown path
(opening order, close reason `shutdown`), so there is no live transition to get right. Measure concurrent-window
counts, forced splits by reason, save outcomes, and memory before turning it on by default.
4. **Decided against (2026-10-28): no startup flag.** Stage 2 is always on, so stage 3's tests gate
the release instead of a default-off rollout. What the flag's measurement period was for is read
from the metrics: `git_commit_windows_open` shows windows open side by side,
`git_commit_windows_single` a worker that fell back to one at a time, and
`git_commit_windows_total` by `close_reason` the forced splits (`identity_change`,
`path_conflict`, `footprint_fallback`). Memory is not measured yet: the retained-byte budget
charges windows and pending writes, but not the footprint indexes, the accepted versions or the
withheld-field records.
5. **Proposed. Reconsider the follow-ups** in
[window-field-ownership-and-apply-hold.md](window-field-ownership-and-apply-hold.md), using those
measurements. Field ownership and the apply hold are separate changes, each with its own
Expand Down
6 changes: 4 additions & 2 deletions docs/design/event-pipeline-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,9 +275,11 @@ admission check reads.
flowchart LR
E["live event"] --> G{"render fidelity<br/>established for<br/>this target?"}
G -- "no" --> DROP["dropped<br/>(the pending replay repairs it)"]
G -- "open" --> CAN{"same author<br/>and target as<br/>the open window?"}
CAN -- "no" --> FIN1["close window<br/>(identity change)"] --> NEW
G -- "open" --> CAN{"an open window of<br/>the same author<br/>and target?"}
CAN -- "yes" --> ADD["add to window"]
CAN -- "no" --> PROVE{"footprints proven<br/>and disjoint?"}
PROVE -- "yes" --> NEW
PROVE -- "no" --> FIN1["close the windows<br/>it cannot sit beside<br/>(identity change,<br/>path conflict)"] --> NEW
NEW["open window,<br/>snapshot target's timers"] --> ATT["attach a waiting save"] --> ADD
ADD --> CAP{"retained bytes<br/>over budget?"}
CAP -- "yes" --> FIN2["close window<br/>(buffer limit)"]
Expand Down
3 changes: 3 additions & 0 deletions docs/interpreting-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,8 @@ boundary, the commit, the push. Background:
| `git_writes_dropped_total` | counter | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name`, `reason` | A live event the branch worker dropped because its `GitTarget`'s render fidelity was not established. `reason` is `render_fidelity_diverged`: a rendered value does not match the live object, and every write is dropped until a fresh replay measures the folder clean. Or it is `render_fidelity_pending`: a collection of the `GitTarget` has not finished its replay, which closes every collection of that target until it does. A live event that meets a pending gate is held, not dropped, and committed with its author once the gate opens; it is counted here only when the hold gives up, 10 minutes after the first one was held or once the branch holds more than its retained-byte budget, and the worker then asks for a fresh snapshot, which commits the current state as the operator (a deletion comes back only under prune mode `Always`). Nothing replays a dropped write, so each one is an edit missing from Git until the object changes again or its collection replays. The worker logs the first drop under each gate status at Info, naming the collections the gate waits for, and each hold, release and give-up at Info. |
| `git_commit_windows_total` | counter | `gittarget_namespace`, `gittarget_name`, `close_reason`, `timer_source` | One per closed commit window, whether or not it then produced a commit. `close_reason` is `idle_timeout` / `max_duration` / `attach_next` / `identity_change` / `path_conflict` / `footprint_fallback` (a commit changed a file outside its window's footprint, so the worker holds one window at a time from then on, and closes all but the newest of the windows open) / `buffer_limit` / `resync_before_apply` / `foreign_commit` (a Git host said the branch may have moved, so the window is published now) / `apply_due` (an apply came due for an applier on the branch, so the window's edits are published before the apply is sent) / `shutdown`; `timer_source` is `target` or `commit_request`. See [tuning commit windows](#tuning-commit-windows). |
| `git_commit_window_duration_seconds` | histogram | `gittarget_namespace`, `gittarget_name` | How long each window collected: from the write that opened it to the start of its finalize. Its count equals `git_commit_windows_total` summed over `close_reason` and `timer_source`. |
| `git_commit_windows_open` | gauge | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name` | Commit windows the branch worker holds open. Above 1 means authors are collecting side by side, one window each, because their files are proven not to overlap. A branch that only ever reads 0 or 1 while several people edit it gets no concurrency: their writes are in shared files, kustomize folders, or another layout the worker cannot prove independent. Read at scrape time. |
| `git_commit_windows_single` | gauge | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name` | 1 once the branch worker has fallen back to one commit window at a time, after a commit would have changed a file outside its window's footprint (a bug: the worker refused the commit and asked for a resync). It stays 1 until the worker restarts. The worker logs the refused paths at Error (`SingleWindowFallback`). Read at scrape time. |
| `git_queue_depth` | gauge | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name` | Pending + in-flight + committed-but-unpushed. Read at scrape time. |
| `git_retained_bytes` | gauge | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name` | What the worker holds until a push lands, in the units of the retained-byte budget: the open window, the retained writes and waiting saves, each with a 1 KiB per-item charge. Serialized bytes, not heap. |
| `git_retained_writes` | gauge | `gitbranch_namespace`, `gitbranch_name`, `provider_namespace`, `provider_name` | Decided writes waiting for a push. |
Expand Down Expand Up @@ -1421,6 +1423,7 @@ sum by (outcome) (rate(gitopsreverser_secret_encryptions_total[5m]))
| `sum by (gitbranch_namespace, gitbranch_name, reason) (increase(gitopsreverser_git_writes_dropped_total[15m])) > 0` | Loss: writes met a closed render-vs-live gate and were dropped, and nothing replays them. `render_fidelity_diverged` goes with `RenderMatchesLive=False` on a `GitTarget`; `render_fidelity_pending` is a hold that gave up. |
| `sum by (gittarget_namespace, gittarget_name, reason) (increase(gitopsreverser_placement_refusals_total[1h])) > 0` | Loss: new resources the writer declined to place are absent from the mirror until the placement policy is fixed. |
| `sum by (gittarget_namespace, gittarget_name) (increase(gitopsreverser_placement_kustomization_entries_total{outcome="failed"}[1h])) > 0` | A new file is in Git without its `resources:` entry, so kustomize never builds it. |
| `max by (gitbranch_namespace, gitbranch_name) (gitopsreverser_git_commit_windows_single) == 1` | A branch worker found a commit that would have changed a file outside its window's footprint, refused it, and holds one commit window at a time until it restarts. It is a bug to report, with the `SingleWindowFallback` log line. |
| `max by (gitbranch_namespace, gitbranch_name) (gitopsreverser_apply_operations_unresolved) > 0`, `for: 15m` | An applier is stuck mid-apply, and nothing more is asked of it until the operation completes. |
| `max by (gitbranch_namespace, gitbranch_name) (gitopsreverser_apply_appliers_held) > 0`, `for: 15m` | An applier is held behind the initial-snapshot barrier: a `GitTarget` naming its claim watches nothing, or its folder is missing. |
| `sum by (applier_kind, claim_namespace, claim_name, reason) (rate(gitopsreverser_apply_requests_total{outcome="failed"}[15m])) > 0`, `for: 30m` | Apply requests keep failing for the named applier claim; the worker retries at its backoff, and the cluster runs older content than the branch meanwhile. |
Expand Down
Loading
Loading