Skip to content

attribution-setup-guide.md

Latest commit

 

History

History
180 lines (138 loc) · 8.08 KB

attribution-setup-guide.md

File metadata and controls

180 lines (138 loc) · 8.08 KB
Raw

Attribute commits to Kubernetes users

Attribution uses kube-apiserver audit events to name the Kubernetes actor as the Git author. The configured committer stays unchanged. Without attribution, both identities use the configured committer.

Start with a working Git mirror. You need control over kube-apiserver flags and files; managed control planes that hide these settings cannot use this webhook.

1. Enable the receiver

Add these chart values to your release and reconcile it:

replicaCount: 1
attribution:
  enabled: true
  transport: redis
queue:
  redis:
    addr: valkey.example.internal:6379
    # For authenticated Valkey, supply a Secret in the release namespace:
    # auth:
    #   existingSecret: valkey-auth

Redis/Valkey holds attribution facts and watch cursors. If its data is lost, recent facts are lost and watches cold-replay. For a disposable single-pod setup, attribution.transport: memory needs no Redis but loses facts whenever the reverser restarts, and leaves the watch resume cursors inactive unless you configure Redis separately.

The chart's KubeProvider, local in the release namespace, admits only the release namespace by default. Grant the namespaces whose GitTargets may mirror through it in kubeProvider.allowedGitTargets (see the grants), or install with the chart's values-playground.yaml to let every namespace mirror its own. See RBAC for the controller's read permissions.

The chart creates the audit Service and cert-manager certificates. Wait for the release to be ready, then read its Secret names and kubeconfig generator:

helm get notes gitops-reverser -n gitops-reverser

Use the Helm storage namespace here. For Flux this may be flux-system, even when the pods and certificate Secrets live in gitops-reverser.

2. Configure delivery on every API server

Generate the webhook kubeconfig from the audit CA and client certificate Secrets. Keep CA verification and tls-server-name; the audit listener requires mTLS. The kubeconfig embeds a private key: encrypt it before committing it to Git.

Set its server to an address reachable from each control-plane host, with a named route:

server: https://REACHABLE_ADDRESS:9444/audit-webhook/gitops-reverser.local
tls-server-name: gitops-reverser-audit.gitops-reverser.svc

Use /audit-webhook/<release namespace>.local for a normal installation: the chart creates the local KubeProvider in its release namespace, and a provider's route defaults to <namespace>.<name>. For another provider, use its spec.attribution.auditRoute, or <namespace>.<name> when it sets none. kubectl get kubeprovider -A -o wide prints each provider's resolved route in its Route column.

Only installations explicitly using attribution.auditRouteAnnotationKey should keep the bare /audit-webhook endpoint. That mode selects a route from each event's annotation; appending a route would instead select that one route for the entire request.

A ClusterIP works only when the host can route to the service CIDR. Otherwise use a reachable NodePort or load balancer. Do not assume cluster DNS or a loopback NodePort works from the host. Keep tls-server-name when connecting by IP so the service certificate's DNS SAN is verified.

Place the kubeconfig and the policy below on every control-plane node, mount them into kube-apiserver, and set:

--audit-policy-file=/path/to/audit-policy.yaml
--audit-webhook-config-file=/path/to/audit-webhook.kubeconfig
--audit-webhook-mode=batch
--audit-webhook-version=audit.k8s.io/v1
--audit-webhook-batch-max-wait=1s
--audit-webhook-batch-max-size=100

The files must exist and be readable by the API server's process user before it starts. Follow your distribution's file lifecycle: Talos 1.14 writes machine.files during boot, so stage the configuration and reboot each node. A successful live apply does not prove the files exist.

Restart one API server at a time and check its direct /readyz endpoint before continuing. Use batch: blocking modes couple API writes to receiver availability. Batching buffers and retries during short outages, but its finite buffer does not guarantee delivery through an outage. See k3s file placement for an example.

Choose the audit policy

Start with the tuned policy. It captures configuration writes at RequestResponse level, including CRDs and custom resources, and drops reads, heartbeats, core runtime noise, selected status updates, and HPA-driven scale changes. Keep the write catch-all so newly installed APIs are covered.

For broader noise filtering, add these resources to the first level: None rule:

resources:
  - group: ""
    resources: ["pods/*", "nodes/*"]
  - group: events.k8s.io
    resources: [events]
  - group: discovery.k8s.io
    resources: [endpointslices]
  - group: "*"
    resources: ["*/status"]

Merge these with the existing exclusions. A pods entry alone does not exclude pods/exec or other subresources. Keep manual scale writes if you want their authors recorded.

The example includes Secret writes at RequestResponse level. To keep Secret bodies out of the webhook, exclude core secrets too. Those writes then have no audit evidence for attribution.

Keep create and patch of commitrequests in the configbutler.ai group covered at RequestResponse level. A CommitRequest's author comes from the audit event that created it, matched by the uid and resourceVersion in the event's responseObject. Metadata level leaves the response out, so the request cannot be matched. patch is there because server-side apply creates an object with a patch. Without the creating event, a request names nobody and cannot save the window of an author the audit names. The write catch-all covers both verbs. The audit policy controls evidence collection; WatchRule and ClusterWatchRule control what gets written to Git.

3. Verify a live change

Check receiver readiness and the route:

kubectl -n gitops-reverser rollout status deployment/gitops-reverser
kubectl -n gitops-reverser get kubeprovider local

FACTSRECEIVED=True means the provider has received an attribution fact at least once. It is historical evidence, not a continuous health check. Receiver readiness alone does not prove API server delivery.

Using your normal user credentials, mutate a ConfigMap covered by an existing ready GitTarget and WatchRule:

kubectl -n YOUR_WATCHED_NAMESPACE create configmap attribution-check \
  --from-literal=check=first
kubectl -n YOUR_WATCHED_NAMESPACE patch configmap attribution-check \
  --type=merge -p '{"data":{"check":"second"}}'

After the target's commit window closes, fetch its branch and inspect the commits affecting the ConfigMap. The author should identify your Kubernetes user; the committer should be unchanged:

git show --format=fuller COMMIT

Repeat through each direct API server endpoint, using a unique ConfigMap name per test. Delete the test objects afterward. A baseline snapshot is not a substitute for this live-write check.

An unmatched live change is authored as unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid>. Check the route, policy coverage, API server webhook errors, and transport connectivity first. attribution.grace defaults to 3s; increase it only if facts arrive after that window. attribution.ttl defaults to 10m.

Rotate or disable

When the audit client certificate changes, regenerate the embedded kubeconfig and roll it out to all API servers. A server certificate renewal under the same CA needs no node update. Plan CA rotation with trust overlap; replacing the CA while nodes still trust the old one breaks delivery.

To disable attribution, remove webhook delivery from every API server first. Then set attribution.enabled: false in the release. This avoids sending events to a removed Service.

For remote clusters and shared routes, see configuration and the shared audit trust boundary.