fix(infra): GKE nodes ran as the default Compute Engine service account - #94
Merged
Conversation
The MEDIUM triage ADR-046 deferred. Three of the five findings were real. GCP-0050, twice: neither node pool set node_config.service_account, so GKE fell back to the default Compute Engine service account — typically roles/editor across the whole project. That is D-31 violated inside the module that implements D-31's other five identities. Bounded rather than harmless: Workload Identity means pods impersonate runtime/drift/retrain, so this is what an attacker inherits from a compromised node, not from a compromised pod. Real all the same. A sixth identity, `nodes`, now carries Google's documented minimum for a custom node service account, plus artifactregistry.reader scoped to the repository rather than granted project-wide. node_oauth_scopes gained devstorage.read_only in the same change: a scope is an upper bound on what a service account may do, so the Artifact Registry grant would have been unusable without it and private pulls would fail. cloud-platform stays avoided, per the variable's own description. GCP-0011: google_project_iam_member.ci_sa_user granted roles/iam.serviceAccountUser PROJECT-WIDE, under a comment reading "Scoped via condition (only acting on SAs in this project)". There was no condition block, so CI could impersonate any service account in the project — including runtime, drift and retrain, exactly the blast radius ADR-017 exists to bound. Replaced with three per-account bindings: deploy, runtime and nodes. drift and retrain are reached through Workload Identity, never by CI, and are deliberately absent. That makes three instances in this session of a comment asserting a control that was never built — the tfsec suppression's validation rule, the baseline gate's exclude-block claim, and now this condition. baseline-review.md records it as the open structural gap: a comment asserting a control exists is a checkable claim in exactly the way a path is, and nothing here verifies that class of claim. The gate is raised to CRITICAL,HIGH,MEDIUM. At HIGH these sat in the log enforced by nothing. The two remaining MEDIUM are object versioning on append-only access-log buckets, accepted with a dated justification rather than fixed. LOW stays out: 8 findings whose triage has not been done, and an unenforced threshold is honest about that in a way a suppression is not. The scaffolded service's gate stays at CRITICAL,HIGH deliberately. The higher bar is affordable here because this repo has .security-baselines/ and an expiry gate; a generated service has neither, and a MEDIUM gate with nowhere to record an acceptance trains adopters to delete the step. Adopter-visible and disruptive: service_account is replace-forcing, so Terraform will propose recreating both node pools. MIGRATION.md says to read the plan and check PodDisruptionBudgets first. ADR-017 and D-31 are amended to six identities; ADR-046's MEDIUM count is corrected from 4 to 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The MEDIUM triage ADR-046 deferred as "separate work". Three of the five findings were real, and two of them were security defects in the module.
GCP-0050×2 — nodes ran as the default Compute Engine service accountNeither node pool set
node_config.service_account, so GKE fell back to the default Compute Engine service account — typicallyroles/editoracross the whole project.That is D-31 violated inside the module that implements D-31's other five identities (
ci,deploy,runtime,drift,retrain). The identity split covered everything except the machines.Bounded rather than harmless, and worth being precise: Workload Identity means pods impersonate
runtime/drift/retrain, so this is what an attacker inherits from a compromised node, not from a compromised pod.A sixth identity,
nodes, now carries Google's documented minimum —logging.logWriter,monitoring.metricWriter,monitoring.viewer,stackdriver.resourceMetadata.writer— plusartifactregistry.readerscoped to the repository rather than granted project-wide.A second-order catch:
node_oauth_scopesdefaulted to logging + monitoring only. A scope is an upper bound on what a service account may do, so the new Artifact Registry grant would have been unusable and private image pulls would fail. Addeddevstorage.read_only— the narrow scope, notcloud-platform, which the variable's own description tells you to avoid. Granting a role the scopes cannot reach would have looked like hardening and done nothing.GCP-0011— CI could impersonate every service account in the projectThere is no condition block. The grant was project-wide and unconditional, so the CI identity could act as
runtime,driftandretrain— exactly the blast radius ADR-017 exists to bound.Replaced with three per-account bindings:
deploy,runtime, andnodes(attaching a service account to a node pool requiresserviceAccountUseron it).driftandretrainare reached by workloads through Workload Identity, never by CI, and are deliberately absent rather than carried over.The two that were accepted, not fixed
GCP-0078andAWS-0090are object versioning on the access-log buckets, GCP and AWS. Append-only sinks, already lifecycle-bounded. Versioning guards against overwrite and modification of existing objects — neither is a failure mode for a log sink, and it multiplies cost on the highest-volume bucket in the project. The buckets holding state, models and MLflow artifacts do have versioning, which is where the control matters.Baselined with a dated justification (2027-03-05) and a review question written into the entry: are those buckets still access-log-only? The acceptance rests entirely on that, so it is a property to verify in
storage.tf, not a rationale to re-read.This is the distinction the previous review learned to make. The first two are defects; these are an accepted risk where the check is correct in general and does not apply to this object lifecycle. Only the third kind belongs in a baseline.
The gate now runs at MEDIUM
CRITICAL,HIGHCRITICAL,HIGH,MEDIUMCRITICAL,HIGHCRITICAL,HIGH— deliberatelyAt HIGH those five findings were visible in the log and enforced by nothing, which is the same as not looking.
The service stays at HIGH on purpose: the higher bar is affordable here because this repo has
.security-baselines/plus an expiry gate, so an accepted finding can be recorded, dated and reviewed. A generated service has neither, and a MEDIUM gate with nowhere to write down an acceptance trains adopters to delete the step — which costs more than the two findings it would catch.LOW stays out. Eight findings whose triage has not been done, and an unenforced threshold is honest about that in a way a suppressed finding is not.
Third instance of the same pattern
GCP-0011makes three this session — a comment asserting a control that was never built, surviving because reviewing meant reading it:variables.tfvalidation rule — none existed (fix(infra): GKE control plane could be public with no allowlist #93);check_baselines_expiry.pyclaimed to readexclude:blocks — it filtered on uppercase ids and saw none (fix(security,docs): the baseline gate saw no entries, and Art. 11 cited a missing index #88);ci_sa_userclaimed an IAM condition — the block was absent.A comment asserting that a control exists is a checkable claim, in exactly the way a path is. Nothing here verifies that class of claim.
docs/audit/baseline-review.mdrecords it as the open structural gap, and it is a bigger finding than any of the three.Adopter impact — read this one
service_accountis replace-forcing. Terraform will propose recreating both node pools; workloads reschedule.MIGRATION.mdsays to read the plan, do it in a window, and confirm PodDisruptionBudgets (D-27) are in place first. Two further rows cover the scope change and the IAM move.Verification
ADR-017 and D-31 are amended to six identities; ADR-046's MEDIUM count is corrected from 4 to 5 — it was mine to get right.
🤖 Generated with Claude Code