docs: add etcd revision profiling guide - #10826
waveywaves wants to merge 2 commits into
Conversation
tekton-robot
added
release-note-none
tekton-robot
added
the
size/XXL
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10826 +/- ##
=======================================
Coverage 90.81% 90.81%
=======================================
Files 292 292
Lines 22196 22196
=======================================
Hits 20158 20158
Misses 2036 2036
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
/retest |
Extracted from the documentation originally contributed in tektoncd#10329. Implements deliverable 1 of tektoncd#10322. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: waveywaves <11972233+waveywaves@users.noreply.github.com>
b3e5a4f to
3182449
Compare
tekton-robot
added
size/L
waveywaves
changed the title
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vdemeester The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
tekton-robot
added
the
approved
|
cc @aThorp96 |
aThorp96
left a comment
aThorp96
left a comment
There was a problem hiding this comment.
I haven't finished the doc yet but there's some great stuff here. I've added a handful of comments I have so far. This is going to be really useful for devs and operators once it lands :)
| @@ -0,0 +1,283 @@ | |||
| <!-- | |||
| --- | |||
| linkTitle: "Profiling etcd Usage" | |||
There was a problem hiding this comment.
Nit-pick. I think "Profiling" may not communicate the full utility of this doc.
| linkTitle: "Profiling etcd Usage" | |
| linkTitle: "Measuring and Estimating etcd Usage" |
| This guide explains how a Tekton workload consumes etcd storage and how to | ||
| measure that consumption. It is intended for operators and platform builders | ||
| doing capacity planning or investigating etcd pressure. |
There was a problem hiding this comment.
If Operators doing capacity planning, I think docs/development may not be the right place for it. Maybe it could be moved to docs/ alongside docs/tekton-controller-performance-configuration.md. The performance configuration doc should also link to this doc as well, I believe.
| etcd is an MVCC store: each write creates a new version of a key. Until etcd | ||
| compacts old revisions, those versions consume storage. A TaskRun can be | ||
| written many times as the Pipelines controller, Chains, Results, and platform | ||
| controllers update it, so its write volume can be many times the size of its | ||
| current value. | ||
|
|
||
| Both factors matter: | ||
|
|
||
| - **Object size:** bytes written by each update. | ||
| - **Write count:** how often the object is created or updated. | ||
|
|
||
| A small object rewritten thousands of times can cost more than a larger object | ||
| written once. | ||
|
|
There was a problem hiding this comment.
Optional suggestion: a link or footnote to etcd's documentation on this would be valuable
| etcd is an MVCC store: each write creates a new version of a key. Until etcd | |
| compacts old revisions, those versions consume storage. A TaskRun can be | |
| written many times as the Pipelines controller, Chains, Results, and platform | |
| controllers update it, so its write volume can be many times the size of its | |
| current value. | |
| Both factors matter: | |
| - **Object size:** bytes written by each update. | |
| - **Write count:** how often the object is created or updated. | |
| A small object rewritten thousands of times can cost more than a larger object | |
| written once. | |
| etcd is an MVCC store: each write creates a new version of a key.[^etcd-mvcc] Until etcd | |
| compacts old revisions, those versions consume storage. A TaskRun can be | |
| written many times as the Pipelines controller, Chains, Results, and platform | |
| controllers update it, so its write volume can be many times the size of its | |
| current value. | |
| Both factors matter: | |
| - **Object size:** bytes written by each update. | |
| - **Write count:** how often the object is created or updated. | |
| A small object rewritten thousands of times can cost more than a larger object | |
| written once. | |
| [^etcd-mvcc]: More info on etcd's revision management can be found here: https://etcd.io/docs/v3.7/learning/api/#revisions |
| `CreateRevision` and `ModRevision` are positions in the store-wide revision | ||
| sequence. `Version` is the per-key write count used in this guide. |
There was a problem hiding this comment.
| `CreateRevision` and `ModRevision` are positions in the store-wide revision | |
| sequence. `Version` is the per-key write count used in this guide. | |
| `CreateRevision` and `ModRevision` are positions in the store-wide revision | |
| sequence and can be safely ignored for our purposes. `Version` is the per-key write count used in this guide. |
| Tekton resources are CustomResourceDefinitions and retain their API group in | ||
| the key. Many resources compiled into the API server use an ungrouped prefix, | ||
| and some have legacy names. For example, Nodes use `/registry/minions/`, and | ||
| Services are split between `/registry/services/specs/` and | ||
| `/registry/services/endpoints/`. | ||
|
|
||
| Do not infer arbitrary storage keys from API resource names. Confirm the layout | ||
| by listing a prefix: |
There was a problem hiding this comment.
[...] Many resources compiled into the API server use an ungrouped prefix,
and some have legacy names. For example, Nodes use/registry/minions/, and
Services are split between/registry/services/specs/and
/registry/services/endpoints/Do not infer arbitrary storage keys from API resource names. Confirm the layout
by listing a prefix:
Is this relevant here? Maybe I'm missing something, but it's unclear to me why this is included for the reader. If there's specific resources that fall into this which the reader needs to be aware of, we should include them, but I don't believe Nodes and Services are relevant for this debugging.
For general, non-tekton etcd debugging, if guidance applies generally to both tekton and non-tekton objects it's good to include here, but if the guidance only applies to non-tekton objects which aren't very relevent for Tekton debugging it might not be worth spending time on. Maybe linking to a general etcd debugging guide could cover those types of notes. E.g. etcd's Interacting with etcd guide is IMO required reading for this doc, and at the very least should be linked as an additional resource
| by listing a prefix: | ||
|
|
||
| ```bash | ||
| ETCDCTL_API=3 etcdctl ... get /registry/tekton.dev/ --prefix --keys-only |
There was a problem hiding this comment.
It would be good to mention that etcdctl is a required CLI for this doc. Installation instructions are here: https://etcd.io/docs/v3.5/install/
| of writes since the key's current lifetime began. Deleting and recreating a key | ||
| starts the count again. | ||
|
|
||
| From a kubeadm control-plane node, for example: |
There was a problem hiding this comment.
This should be more explicitly highlighted. All etcdctl commands in this doc assume the command is being run from a control-plane node or somewhere with etcd access. Even if there's a brief section on connecting to etcd or linking to external documentation on that, there should be a callout about the assumptions we're making in the bash blocks here.
When using a kind cluster for example, the easiest way I've found is using a debug etcdclient pod as described here: kubernetes-sigs/kind#3058.
| `user.username`. Metadata-level logging records the actor, verb, resource, and | ||
| timestamp without logging object bodies. | ||
|
|
||
| Count creates as well as updates: the first create gives the key version 1. |
There was a problem hiding this comment.
Count creates as well as updates: the first create gives the key version 1.
It's unclear to me what this is trying to communicate. Is this instructing on how to "count creations and updates"? If the first create gives the key version 1, what about subsequent creations? Do they increment the version if they succeed? Do they never suceed?
|
|
||
| ## Profiling a PipelineRun | ||
|
|
||
| To estimate one execution's write volume, profile the PipelineRun and the |
There was a problem hiding this comment.
I think it would be good to define "write volume" early in the doc. When I read "write volume", I think "the volume of writes" as in "the number of write requests". I think here we're measuring "the volume of storage required by etcd to store all writes"
5 participants
Changes
Carries forward the documentation from #10329 by @thc1006. The extracted documentation commit preserves @thc1006 as its author; the follow-up commit makes the guide self-contained on current
main.Add an operator guide for understanding and profiling the etcd write volume of Tekton workloads. It covers per-key versions and serialized size, storage-key layout, controller attribution through managed fields and audit logs, UID-based discovery of PipelineRun-related objects, revision-pinned reads, aggregation, and the limits of estimating backend cost.
The optional profiling helper is intentionally split into #10834. This PR implements deliverable 1 of #10322.
Validation:
git diff --check origin/main...HEAD/kind documentation
Submitter Checklist
As the author of this PR, please check off the items in this checklist:
/kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tepRelease Notes