Originally published on dev.to. This repo is the archive copy.
Discussion: <!-- paste the dev.to URL here after publishing -->
Everything written about AI coding agents is about making them faster. Better prompts, cheaper model tiers, aggressive delegation, parallel subagents. Almost nothing is written about the part that actually bites you: the code the agent wrote is now on its way to production, and nobody read it.
I run tenant-to-tenant migrations — the kind where a bad script doesn't throw an exception, it silently copies thousands of files to the wrong place and you find out on Monday. My toolkit is ~50 PowerShell scripts, and an increasing share of the diffs are agent-authored. Two remote VMs pull from that repo and execute against live customer tenants.
So I had to answer the question in the title for real. This is what I ended up with: four layers, none of them clever, all of them mechanical. Plus the four things that leaked through anyway.
Agents changed how much code gets produced. They did not change how much code you can carefully read. The gap between those two numbers is the entire problem, and it only grows.
The instinctive fix is "review more carefully." That fails, because it's a discipline solution to a volume problem. Discipline is exactly the resource that's already exhausted at 6pm on a cutover day.
So: no gate that depends on me being alert. Every layer below either runs automatically or refuses to proceed.
The obvious objection to testing ops scripts is that they talk to live systems, so you can't test them. That's true of the scripts, and false of the guards.
The most expensive class of failure in my work isn't a crashing script — it's a script that runs happily against slightly wrong input. A stale user-mapping file doesn't error. It maps half your users to nobody and reports success.
So the guards are the thing under test, and they run against synthetic fixtures committed to the repo — a fake tenant, fake users, deliberately broken variants:
tests/
Run-GuardTests.ps1
fixtures/
contoso/
config/ # a complete, fake, self-consistent project
=== Guard regression tests (synthetic fixtures) ===
[PASS] MappingDrift: clean = in sync (exit 0)
[PASS] MappingDrift: count mismatch = drift (exit 1)
[PASS] MappingDrift: empty AccountName = drift (exit 1)
ALL PASS (3/3)
Three tests. That's it. It is not impressive and it does not need to be — it needs to be runnable by a machine with no credentials, which is the property that lets everything in Layer 2 work.
One detail worth stealing: the drift check compares file timestamps, and the first version threw a false positive when two files were written in the same operation. A guard that cries wolf gets disabled within a week, which is worse than no guard. It now allows 120 seconds of tolerance. Tune your guards for silence. A guard you ignore is a guard you don't have.
Here was the real hole, and it existed long before agents: the execution VMs pulled main. Every
commit — mine, the agent's, a 2am "quick fix" — was one git pull away from running against a
customer tenant.
Two branches now:
main— where work happens. Agents commit here freely.stable— what the VMs pull. Nothing writes to it except one script.
That script is the whole gate. It refuses to promote unless every condition holds:
[1/5] inside a git repository
[2/5] on the source branch, working tree clean
[3/5] source branch is pushed (no local-only commits)
[4/5] guard harness passes
[5/5] target can fast-forward (no divergence)
Fail any one and it exits non-zero having changed nothing. On success it fast-forwards, tags, and pushes.
Two design choices that matter more than they look:
Fast-forward only. If stable has diverged from main, the answer is never "merge it." It
means someone hotfixed production directly, and that's a human conversation, not a git operation.
The script refuses and says so.
The harness gate is inside the promote script, not in CI. CI tells you something broke after
it's already on main. This tells you before anything reaches the machines that touch customer
data. Same test, different position in the pipeline, completely different consequence.
The escape hatch is -SkipTests, and it prints a loud warning. Escape hatches that are hard to use
get worked around; escape hatches that are easy but noisy get used honestly.
Every project config carries an environment flag:
{
"tenantName": "contoso",
"environment": "dev"
}The interesting part is not the field. It's what happens when the field is absent:
# Unknown/missing values return 'prod' (fail safe, with a warning so it gets fixed).
if (-not (Test-Path $ConfigPath)) {
Write-Warning "environment: config not found -- assuming 'prod'."
return 'prod'
}An old config written before this field existed has no environment. The tempting default is
dev, because "it's probably a sandbox and I don't want warnings everywhere."
That's backwards. An unlabelled config in this repo is far more likely to be a real customer than
a forgotten sandbox — the sandboxes are new, the customer projects are old. So the unknown case
resolves to the strict answer, and destructive operations against a prod config refuse without an
explicit -Force.
This is the cheapest layer to build and the one I'd add first. The rule generalises: when your code can't tell where it is, it should assume the most dangerous place.
Every project config in my repo is prod. All of them. The sandbox is the exception.
That single fact decides the whole policy, and I got it wrong on the first pass. The instinct is to
guard everything that writes to a tenant — but nearly every script does, so every ordinary run would
refuse, and within a week I'd be typing -Force without reading the line above it. That isn't a
guard. It's a habit of ignoring guards, which is strictly worse than not having one, because now
there's a warning I've trained myself past.
So the test isn't "does this write to production." It's both of:
- irreversible — no delta run or re-copy undoes it, and
- not routine — it isn't part of the normal copy/verify/delta cycle.
Three scripts qualified out of ~40: delete migrated subsites, freeze the source tenant, strip a site's permissions to one user. The daily migration scripts are deliberately unguarded. They write to customer tenants constantly — that's the job.
Two exemptions apply wherever the guard is used. -WhatIf is never gated, because previewing has to
stay frictionless or nobody previews. And the undo direction is never gated — un-freezing a
source is ungated while freezing it is not. Gate the do, not the undo. A bad freeze staying in
place while someone hunts for the right flag is its own outage.
The three layers above are gates: they block bad things. This one is different — it's about state that would otherwise live only in my head.
Four hooks, at four lifecycle points:
| Event | What runs | Why |
|---|---|---|
| SessionStart | pull repos, diff commits + handoff notes since last session | The agent starts knowing what changed while it was gone |
| UserPromptSubmit | scan my message for correction-shaped language | See below — this is the one that took longest to get right |
| PostToolUse (Write/Edit) | lint every script written or edited | Catches environment-specific breakage at write time, not run time |
| Stop | push already-committed work | Two machines stay in sync without me remembering |
The PostToolUse linter is the highest-value-per-line thing in the entire setup, and it checks
almost nothing generic. It enforces this environment's rules: this codebase must be pure ASCII
(the shell it runs under mangles em-dashes and box-drawing characters), and a specific string-
interpolation footgun that had already broken two scripts. Both are things a general-purpose linter
has no opinion about, and both are things an agent reproduces confidently and incorrectly, forever,
because its training data is full of the general case.
Encode your environment's weirdness into a hook, not into a prompt. A prompt instruction is advice the model may drift from over a long session. A hook is a wall.
The remaining gap was subtler, and I only saw it after reading a lot of other people's setups: the agent drafts, you correct it, you move on — and the correction dies in the transcript. Next month, same mistake. I had a perfectly good directory of typed "lessons" files. Almost nothing was in it, because writing one required remembering to, at the exact moment I was busy being annoyed.
The trigger was the missing piece, not the storage. So now a hook watches my messages for correction-shaped language and injects a short reminder to decide whether the lesson is durable and, if so, write it down.
The engineering is entirely in the false-positive rate. My first pattern list fired on this:
"this script never works properly on the VM"
That's a bug report, not a standing rule — but it contains "never," and "never" was on the list.
The fix was to require an imperative aimed at the agent: never run, never suggest, always use. Bare never no longer counts.
Final: 8 correction phrasings fire, 10 lookalikes stay silent. Same principle as Layer 1 — I tuned it for silence, because the failure mode of a chatty hook is that you learn to ignore it, and then it's just noise you're paying tokens for.
Any writeup without this section is selling you something.
1. A warning is not a failure. A setup script needed to export a certificate. It shelled out to
openssl, which wasn't on PATH. It printed a warning and continued — exit code 0, script
"succeeded." The missing file surfaced days later as an unrelated-looking connection error.
Rewritten to export natively and then read the artifact back and verify it round-trips. If a
step produces a file, check the file. Exit codes lie by omission.
2. My secret-scrubber was itself the leak. I wrote a sanitizer to strip credentials out of code
before mirroring it to a public repo. It worked by find-and-replace over a list of known secrets —
which meant the sanitizer contained the secrets, in plaintext, in the repo it was protecting. I
caught it on the way to git push. It now matches on shape (a regex for the credential format)
rather than on a list, which is both safer and catches secrets nobody remembered to enumerate.
Enumeration-based redaction is a trap; it inverts into exactly what it's preventing.
3. Adding a .gitignore rule does not untrack anything. I found 42 config files with customer
data already tracked. I fixed .gitignore, committed, and felt good. They were still in the repo,
and still in history. .gitignore only governs untracked files — you need git rm --cached as a
separate step. Also: git check-ignore will cheerfully report a tracked file as "not ignored,"
which reads like your rule is broken when it isn't. Use --no-index to check the rule itself.
4. The bug none of this caught was found by a test I nearly didn't write. In an API wrapper:
return @(@{ type = 'text'; text = $System }) # returns a hashtable, not an array
return ,@(@{ type = 'text'; text = $System }) # correctPowerShell unrolls single-element arrays on return. The payload silently serialised as an object instead of an array, and the API would have rejected every call. It reads correctly. It reviews correctly. It's wrong. Language-level footguns are where agent-written code is most dangerous, because the code looks exactly like what a competent human writes.
5. The fail-safe in Layer 3 was never actually called. I wrote this article, then audited the
code it describes. Assert-SafeEnvironment — the function that refuses destructive work against a
production config — was defined, loaded into every script's scope, and invoked by exactly nothing.
grep found one hit: a comment. For weeks the guard existed as a thing I could point at, which is
the most dangerous state a safety mechanism can be in, because it stops anyone from looking for the
missing one.
It's wired now, and the section above is what the wiring taught me. But the useful part isn't the
fix — it's that four layers of mechanical gates did not catch a guard that was never connected, and
the thing that did catch it was sitting down and checking each claim against grep. Automated
checks verify the code you wrote. Nothing automatically verifies the code you only think you wrote.
Being honest about the boundary:
- It gates code, not judgment. Nothing here stops a correct script from being run against the wrong target. That's still a human decision, and it's still the biggest risk in the system.
- The guards only cover failure modes I already know about. Every one was written after something went wrong. This is a ratchet, not a proof.
- It scales with incidents, not with time. If you have no scars yet, you have nothing to encode. Start by writing down the last thing that broke.
Not the branch layout, not the hooks. This:
Every gate must be mechanical, and every default must fail toward the safe answer.
The version of this that depends on you being careful is not a system. It's a plan to be careful, and you already had one of those before the agent showed up.
Companion piece: Two Brains, Not One — the memory architecture that sits alongside this. That one is about what the agent knows; this one is about what it's allowed to ship.