CP3: Add custom RBAC role definitions to app-baseline chart

[nguforw-moj]

Summary

The container-platform-environments repo handles RBAC bindings (RoleBindings via app-baseline) and authentication (EKS access entries via identity/terraform/), but has no mechanism for BU teams to define custom Kubernetes Roles or ClusterRoles. Teams can only reference built-in ClusterRoles (view, edit, admin).

Problem

BU teams will need scoped-down or purpose-built roles that don't match the broad built-in ones:

• CI/CD deployer — can update Deployments and Services but not read Secrets
• Readonly debugger — can view pods, logs, and events but not exec into containers
• Namespace admin — full access within namespaces but no cluster-scoped permissions
• Restricted editor — edit minus certain resources (e.g. NetworkPolicies, ResourceQuotas)

Without this, teams get overly broad access or must raise manual requests — defeating the self-service GitOps model.

Proposed Solution

Extend the app-baseline chart with an optional roles: field in product.yaml:

access:
  - group: ci-deployer
    role: my-product-deployer
    clusters:
      - container-platform-octo-nonlive

roles:
  - name: my-product-deployer
    rules:
      - apiGroups: ["apps"]
        resources: ["deployments", "replicasets"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
      - apiGroups: [""]
        resources: ["services", "configmaps"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]

Implementation

1. New template charts/app-baseline/templates/clusterrole.yaml — creates ClusterRoles from .Values.roles
2. Naming convention — prefix with product name (<product>-<role-name>) to avoid collisions
3. Gatekeeper constraint — block roles that grant wildcard * on all apiGroups/resources/verbs
4. Documentation — update README with common role pattern examples

Why not a separate Terraform module?

• Roles are product-driven and BU-specific — they belong in product.yaml
• The platform AppProjects already permit ClusterRole creation
• No new pipeline or infrastructure needed
• Same self-service GitOps model as namespaces and RoleBindings

Acceptance Criteria

• app-baseline chart supports optional roles: list in values
• Custom ClusterRoles created with product-name prefix
• RoleBindings can reference custom roles (not just built-in ones)
• Gatekeeper constraint blocks overly permissive role definitions
• README documents roles: field with common examples
• Backwards compatible — products without roles: work unchanged
• Integration test validates custom role creation and binding

Related

• container-platform-environments — charts/app-baseline/
• container-platform-terraform-gatekeeper — guardrail constraint
• ADR-015: GitOps Repository Structure

[nguforw-moj]

Progress update + two design changes

Initial implementation is done on the chart side. Two decisions changed from the original proposal, and one new gap was found.

Change 1: role names are declared in full, not auto-prefixed

The original proposal had the chart auto-prefix custom role names (deployer in product.yaml → ClusterRole my-product-deployer). That required the RoleBinding template to maintain a lookup of custom role names and conditionally prefix, while leaving built-in roles alone.

Dropped in favour of declaring the full name in both places:

access:
  - group: my-team-ci
    role: my-product-deployer     # exact name
roles:
  - name: my-product-deployer     # exact name

The RoleBinding template goes back to a plain pass-through (name: {{ .role }}), with no resolution logic. The product-name prefix is now a documented convention rather than chart behaviour — what you write is the ClusterRole that gets created.

Trade-off accepted: the product name is repeated, and a product rename means updating both roles: and access:. Product renames already require changing namespaces and SSO entries, so this adds little.

Change 2: validation moved from Gatekeeper to Helm render time

The original AC put the guardrail in Gatekeeper. Gatekeeper only rejects at admission, which means a bad role reaches the cluster before anyone finds out.

Validation now runs at Helm render time in templates/_validation.tpl, so it fails during rendering. Gatekeeper is still worth adding as defence in depth for anything applied outside this repo, but it should not be the primary control.

New gap found: silent no-op on cluster-scoped rules

Custom roles are ClusterRoles bound via namespaced RoleBindings. Kubernetes silently ignores cluster-scoped rules in that arrangement — no error at apply time, the permission simply never applies. A team could write a rule for nodes, see a green sync, and be left debugging why access does not work.

Worth noting this fails closed, not open: the rule grants nothing rather than granting too much. So it is a developer-experience problem, not a security hole.

The validation now rejects:

Rejected	Detection
Well-known cluster-scoped resources (nodes, persistentvolumes, namespaces, clusterroles, storageclasses, ...)	Denylist, incl. subresources such as nodes/proxy
nonResourceURLs	Exact — only ever grantable via ClusterRoleBinding
* on all apiGroups + resources + verbs	Exact
Missing name or empty rules	Exact

Residual gap: cluster-scoped CRDs cannot be detected statically, as that needs API discovery. Documented in the README; also fails closed.

Verified locally

• Happy path renders helloworld-deployer ClusterRole + matching RoleBinding per namespace
• All four validation guards fire with actionable messages
• Product with no roles: field renders zero ClusterRoles (backwards compatible)
• helm lint clean

Remaining work

• CI to render the chart on PR — this repo has no workflow that runs helm template. identity-plan.yml is path-filtered to identity/terraform/**, so a product.yaml-only change triggers nothing. Without this, the render-time validation surfaces only as an ArgoCD ComparisonError after merge instead of blocking the PR. This is the single highest-value remaining item.
• Gatekeeper constraint as defence in depth (cluster-scoped rules + wildcards)
• Gatekeeper constraint enforcing the product-name prefix, so the collision guard is enforced rather than conventional
• Integration test in container-platform-integration-tests
• Update the product.yaml reference docs that currently state role must be one of view/edit/admin

Files changed (container-platform-environments)

• charts/app-baseline/templates/clusterrole.yaml (new)
• charts/app-baseline/templates/_validation.tpl (new)
• charts/app-baseline/templates/rolebinding.yaml
• charts/app-baseline/values.yaml
• charts/app-baseline/Chart.yaml — 1.0.0 → 1.1.0
• namespaces/octo/helloworld/product.yaml — worked example
• README.md

[nguforw-moj]

Alternative to consider: platform-curated role catalogue

Before we commit to per-product arbitrary role definitions, there is a third option that has not been costed. Raising it now because it would change where this code lives and who owns the security-sensitive judgement.

The option

Instead of letting each product.yaml define arbitrary RBAC rules, the platform team defines a fixed catalogue of roles — for example cp-deployer, cp-debugger, cp-secret-reader. These are installed once per cluster via cluster-core, alongside Gatekeeper and cert-manager. product.yaml can then only reference a name from that catalogue:

access:
  - group: my-team
    role: edit                # built-in
    clusters: [...]
  - group: my-team-ci
    role: cp-deployer         # platform-curated, must exist in catalogue
    clusters: [...]

Why it is worth considering

The driver for custom roles is the gap between the built-in roles. Checking what they actually grant (Kubernetes RBAC docs):

• view — excludes Secrets deliberately, because reading Secrets exposes ServiceAccount credentials
• edit — permits Secret access and running Pods as any ServiceAccount in the namespace. Per the docs, it can be used to obtain the API access level of any ServiceAccount in that namespace
• admin — edit plus creating Roles and RoleBindings

So view → edit is a cliff, not a step. Granting a CI pipeline edit so it can patch a Deployment also gives it every Secret in the namespace and the ability to assume any ServiceAccount identity there. For a long-lived automation credential that is the weakest link in the model.

That gap is real and worth closing. The question is who defines the roles that close it, not whether to close it.

(Content rephrased from the Kubernetes documentation for licensing compliance.)

Cost comparison

Measured against the current branch:

Option	Executing template code	Who reviews RBAC rules	Name collisions
Built-ins only	0 lines	N/A	N/A
Curated catalogue	0 lines in this chart	Platform team, once	Impossible — platform owns the names
Per-product custom (current branch)	89 lines	Every product PR	Convention only

The 89 lines break down as 62 lines of validation and 27 lines of actual feature. Roughly 70% of the code exists only to guard the footgun described in the comment above, not to deliver the capability.

The curated catalogue needs zero lines in app-baseline. Because the roles already exist on the cluster, rolebinding.yaml's existing pass-through binds them unchanged — the same reason that file needs no modification on the current branch. The only addition would be an allowlist check on the role field, and that is arguably better placed in Gatekeeper than in Helm.

This changes the earlier "no separate module" conclusion

The reasoning for keeping RBAC inside app-baseline was that role definitions are product-driven and BU-specific, so they belong next to the access entries that reference them.

A curated catalogue is the opposite: platform-wide, per-cluster, versioned, identical across all nine clusters. That profile matches Gatekeeper and cert-manager, so it would belong in cluster-core — either inline or as a container-platform-terraform-* module. Worth flagging explicitly, since it reverses the earlier decision.

Trade-offs

Gains:

• Removes 89 lines, ~70% of which is footgun mitigation
• Removes RBAC rule review from every product PR and puts it with the platform team
• Eliminates the ClusterRole name collision problem outright
• Makes the undetectable cluster-scoped CRD gap irrelevant
• Consistent roles across all BUs instead of per-product drift

Costs:

• No per-product flexibility — a team needing something outside the catalogue has to request a platform change
• Adding or amending a role becomes platform-team work with a release cycle, not self-service
• Requires an initial catalogue design pass to decide what the roles are

What we need to decide this

Concrete requirements from the BUs. The three patterns currently documented in the README — deployer, reader, secret-reader — are all generic; none is product-specific. If the real-world requests all look like those, the catalogue is the better fit and we should not ship per-product arbitrary rules.

Suggest we ask OCTO, LAA and HMPPS for actual cases before merging the current branch. The work done so far is not wasted either way: the validation logic and the documented failure modes carry over, and if we go with the catalogue the two new template files are a clean deletion (rolebinding.yaml is untouched, so there is nothing to unwind).

Recommendation

Hold the current branch pending BU input. If the requests are all generic access tiers, build the catalogue instead. If there are genuine product-specific needs, ship the current approach and add the outstanding CI and Gatekeeper guardrails.

[nguforw-moj]

Team decision: simplify to built-in roles only

Following team discussion, the scope of this issue has changed. Custom per-product role definitions are deferred. The agreed model is:

Decided

1. Three built-in roles only: view, edit, admin — no custom ClusterRole definitions via product.yaml
2. Live environments are view-only by default — edit/admin on a production cluster requires an explicit live_access_justification field (enforced at render time)
3. Non-live environments allow flexible permissions — edit or admin freely granted to support development workflows
4. Group naming convention: github-<org>-<team> (e.g. github-laa-developers, github-hmpps-dev-team)
5. Argo CD for live deployments — direct cluster mutation in live is discouraged; teams deploy via GitOps

Rationale for deferring custom roles

• The three patterns initially proposed (deployer, reader, secret-reader) are generic access tiers, not product-specific
• A platform-curated role catalogue is a better fit if the gap between view and edit needs closing later
• Removes 89 lines of template code (70% of which was footgun mitigation) and the associated PR review burden
• Eliminates ClusterRole name collision risk entirely

What's implemented (on branch, not yet merged)

File	Change
charts/app-baseline/templates/_validation.tpl	New — validates role is view/edit/admin, rejects edit/admin on prod without justification
charts/app-baseline/templates/rolebinding.yaml	Wired validation via include
charts/app-baseline/values.yaml	Documented role options and group naming convention
charts/app-baseline/Chart.yaml	Bumped to 1.1.0
namespaces/octo/helloworld/product.yaml	Worked example: github-cloud-platform with edit on nonlive, view on live
README.md	Full RBAC Permissions Model section (levels, live policy, naming, validation)

Custom role files (clusterrole.yaml, previous _validation.tpl) have been deleted.

Verified locally

• helm template + helm lint pass for helloworld on both cluster tiers
• edit on live without justification → hard failure with actionable message
• edit on live with live_access_justification → passes
• Invalid role name → hard failure
• Product with no access entries → renders cleanly (backwards compatible)

Updated acceptance criteria

• Chart supports view, edit, admin roles only
• Chart rejects invalid role names at render time
• Live environments enforce view-only default (edit/admin requires justification)
• Group naming follows github-<org>-<team> convention
• README documents the permissions model
• Backwards compatible — existing products work unchanged
• CI workflow to run helm template on PR (outstanding — not in scope of this branch)
• Update identity/terraform to align group names if needed
• Integration test validates RoleBinding creation

What this supersedes

The original proposal for per-product custom roles is deferred, not cancelled. If BUs later demonstrate genuine product-specific permission needs that the built-in tiers don't cover, we can revisit either:

• A platform-curated role catalogue (installed via cluster-core)
• Per-product custom roles (the implementation explored earlier in this issue)

Both options are documented in the earlier comments on this issue.