[BUG] Coder gate is blind to non-Go changes (CI/shell/Ruby/docs), hollow GATE-PASS

[Defilan]

Bug Description

The coder gate only verifies Go (fmt/vet/build/lint/test), so a coder GO whose diff is entirely non-Go (GitHub Actions workflow YAML, shell scripts, Homebrew/Ruby formulae, docs) reaches GATE-PASS without any meaningful verification of the actual change. The gate is structurally blind to the very content that changed.

Evidence (M2/M3 batch run-20260711-182721)

• Add container image scanning and supply chain security #233 (supply-chain CI) GO'd + would GATE-PASS, but the diff has two runtime-fatal errors the Go gate can't see:
  • a SLSA step uses: slsa-framework/.../generator_container_slsa3.yml@v1.9.0 under steps: — a reusable workflow can only be called at job level, so the workflow is invalid.
  • cosign sign --yes (keyless OIDC) added without granting the job id-token: write, so it fails at runtime.
  • (Also the summary overclaimed adding Trivy + govulncheck; neither is in the diff.)
• [FEATURE] Ship foreman-agent (and metal-agent) via the Homebrew tap for one-command edge upgrades #822 (Homebrew tap) GO'd, but the LlmkubeForemanAgent formula defines def install twice (dead code / brew audit + RuboCop Lint/DuplicateMethods smell) — invisible to the Go gate.

Proposed Fix

Extend the gate (or add a reviewer stage) to lint/validate non-Go artifacts based on the changed file types:

• actionlint for .github/workflows/*.yml
• shellcheck for shell scripts
• brew style / rubocop for Homebrew formulae
• for docs- or config-only GOs, route to a reviewer instead of relying on a vacuous gate (this generalizes [FEATURE] Coder gate: surface a GO that changes no functional production code (docs/comment-only fixes) #1022's "GO with no functional production code" concern).

Environment

• LLMKube version: 0.9.4
• Component: Foreman coder gate

Additional Context

Surfaced by scrutinizing every GO in an M2/M3 batch. Companion to the anti-confab-on-prose finding filed alongside this.

[Defilan]

Fresh live reproductions (backlog batch run-20260712-100300, honest-verdict fleet)

Two GOs from a five-issue backlog batch reproduced this exact gap on issues the harness had never seen. Both are honest in outcome (I hand-verified each is actually correct), but the gate had no ground truth to verify them, so correctness was luck of good coder grounding, not something the harness enforced.

#1021 (make validate-samples, a Python validator + CI wiring). Diff: scripts/validate-samples.py (158 lines), Makefile (+6), .github/workflows/test.yml (+11). The gate ran make test (Go) and passed; it never ran make validate-samples. I ran the validator by hand: it correctly loads all 9 CRD schemas, passes the 22 real samples, and catches an injected Model.spec.url + bogus parameters block (the literal #699 hallucination) with 'source' is a required property / Additional properties are not allowed. Good code, zero gate verification. A subtly-broken validator (wrong path, missing additionalProperties: false) would have GATE-PASSed identically.

#1001 (Foreman CRD status metrics via kube-state-metrics CRS). Diff: charts/foreman/templates/crs-configmap.yaml (196), grafana-dashboard.yaml (364), prometheusrule.yaml (73), values.yaml. All YAML/JSON; the Go gate verified none of it. The load-bearing risk is hallucinated CRD status field paths (wrong path = silently empty metric). I grounded every [status, …] path against the CRD schema and live objects: all real, including status.verdict (a top-level mirror of status.result.verdict). Correct, again with no gate verification.

Sub-finding: the work-class classifier does not close the gap by itself

The honest-verdict work-class policy (PR #1078) correctly gates ci-policy/release-policy, but non-Go files that dominate a diff by changed-line count fall to a self-GO-able class and self-certify:

• [FEATURE] Validate config/samples against the CRD schemas (catch hallucinated manifests) #1021: 158 Python lines dominate; .py is not in the classifier glob table, so actualWorkClass = code-fix and it self-GO'd (coder honestly declared ci-policy; the footprint overrode).
• [FEATURE] Expose Foreman CRD status (Workload/AgenticTask/FleetNode) as Prometheus metrics via kube-state-metrics Custom Resource State #1001: chart YAML classifies as packaging, which is self-GO-able.

So classification alone is not enough. The fix is the slice-2 path-scoped checker registry (.foreman/verify.yaml): the gate must run the matching non-Go checks (actionlint for workflows, make validate-samples for samples, promtool/kubeconform for the chart), not merely classify the file type. Note the recursion: #1021's own deliverable, make validate-samples, is exactly the kind of check that registry should invoke.

[Defilan]

Correction / update on the #1001 example above. My initial hand-check only confirmed the [status, …] paths exist in the CRD schema. A full pre-merge review went further and found #1001 is not "unverified but correct" as I first characterized it: it is unverified and actually broken, in five independently-confirmed ways:

• Its own helm unittest suite fails 5 of 29 on the branch (18/18 pass on main). The gate GATE-PASSed a chart change whose own tests fail.
• foreman_workload_tasks_total (the metric the issue names explicitly) is wired to status.tasks, which is a list of ObjectReferences, not a count, so kube-state-metrics' toFloat64 can never compute it. The path resolves; the type is wrong. That is exactly the trap my path-existence check missed.
• The ForemanFleetNodeHeartbeatStale alert renders as > 5 (5 seconds) instead of 300, because the template strips the m/s unit suffix without converting to seconds. A shipped alert that flaps continuously, opposite its own documented "must be > 90s" comment.
• The PrometheusRule carries no operator selector label, so a default kube-prometheus-stack install would never adopt the alerts.
• spec.agent should be spec.agentRef.name, so the agent label is always empty.

This makes the point sharper than the original comment did: the Go-only gate did not merely fail to add confidence to a correct change, it passed a genuinely broken one, including one whose own unit tests fail. helm unittest / helm lint / promtool on chart-and-rule changes are exactly the path-scoped checks slice 2's registry needs to run. (The #1021 Python example still stands as verified-correct-but-unverified-by-gate; only the #1001 characterization needed this correction.)