docs: replace the guessed blast radius of the prefix bug with measured behaviour - #118
Merged
Conversation
…d behaviour Verified both runbook findings against live AWS in ap-southeast-1 (2026-08-17), via a dry-run changeset only — nothing was deployed. The EventBridge finding is worse than pitfall #62 originally guessed. That write-up hedged "best case ResourceNotFoundException, worst case they inspect the unrelated service". There is no best case: describe-rule on the default `cdr-s3-object-created` returns ENABLED and list-targets-by-rule shows it wired to the live askkaifbot cdr-lambda. An operator deployed at cdr-staging and following the old runbook gets a green light from somebody else's production. A command that errors is self-correcting; one that confidently answers about the wrong resource is not. Recorded, with the rule: run the wrong command, do not reason about what it would probably do. MaxMemoryUsed confirmed with controls in both directions: list-metrics returns 0, the old command returns null with exit status 0, and the identical shape against Duration returns 2 — so the query form was never the problem. The replacement Logs Insights query returns max_mem = 130 MB against the real log group. Surfaced the collision evidence in the runbook's own warning, where an operator reads it, rather than only in the pitfalls file. Dry-run result (not committed, recorded here): all 19 resources "+ Add", no Modify/Delete/Replacement; all 14 cdr-staging-* names confirmed free; changeset and its REVIEW_IN_PROGRESS stack shell deleted; live cdr-lambda LastModified unchanged at 2026-07-05T06:19:32. 454 tests pass; all 8 CI guards pass — including check_runbook_prefix.py, which flagged the first draft of this very commit and made me reword it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #116/#117. I verified both runbook findings against live AWS in ap-southeast-1 — dry-run changeset only, nothing deployed, nothing left behind.
The EventBridge finding is worse than I wrote
Pitfall #62 hedged: "best case
ResourceNotFoundException, worst case they inspect the unrelated production service." That was reasoning, not measurement. There is no best case:The rule exists, is
ENABLED, and targets the live askkaifbotcdr-lambda. So an operator deployed atcdr-staging, following the old runbook, does not get an error telling them they are looking at the wrong thing — they get a green light from somebody else's production and conclude their own pipeline is correctly wired.A command that errors is self-correcting. One that confidently answers about the wrong resource is not. Rule recorded: when estimating a doc bug's blast radius, run the wrong command — do not reason about what it would probably do.
MaxMemoryUsed, with controls in both directions
list-metrics --metric-name MaxMemoryUsednull, exit status 0Duration(negative control)max_mem = 130MB, statusCompleteDry-run changeset
All 19 resources
+ Add— no Modify, Delete or Replacement. The changeset reportsPhysicalResourceId: Nonefor Adds, so it cannot prove non-collision by itself; I resolved the names from the changeset template and checked all 14 against the account — every one free. Changeset and itsREVIEW_IN_PROGRESSstack shell deleted; livecdr-lambdaLastModifiedunchanged at2026-07-05T06:19:32.The collision evidence is now surfaced in the runbook's own warning, where an operator actually reads it, not only in the pitfalls file.
Note
check_runbook_prefix.py— added in #117 — flagged the first draft of this very commit, because I wrotedescribe-rule --name cdr-s3-object-createdin the new warning. Correct call by its own rule. I reworded rather than adding an exemption, so the guard stays strict.454 tests pass; all 8 guards pass.
Scope: documentation only. No code, no deployment. The pipeline is still verified only as far as changeset validation — no Lambda ran.
🤖 Generated with Claude Code