ADR-007: Tailored NetworkPolicies for RHDH deployments [RHDHPLAN-1032]#8
Conversation
Proposes adding default-deny NetworkPolicies scoped to RHDH-labeled pods for both Operator operands and Helm chart deployments, leveraging the additive nature of NetworkPolicies for user extensibility. Assisted-by: Claude
Add tailored NetworkPolicies to RHDH install methods to enforce least-privilege pod communication by default, addressing risks identified in the OCP Threat Model. Ensure policies are flexible for disconnected environments and do not break functionality on clusters without NetworkPolicy support.
Base policies cover only core traffic flows common to all deployments. All deployment-specific egress is the user's responsibility via additive NetworkPolicy resources. Assisted-by: Claude
…AT-819 reference Assisted-by: Claude
Updated section headers and improved documentation clarity regarding NetworkPolicies for RHDH.
Assisted-by: Claude
rm3l
changed the title
|
@rm3l: GitHub didn't allow me to request PR reviews from the following users: redhat-developer/rhdh-security, redhat-developer/rhdh-install. Note that only redhat-developer members and repo collaborators can review this PR, and authors cannot review their own PRs. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
requested review from
Fortune-Ndlovu,
OpinionatedHeron,
gazarenkov,
subhashkhileri and
zdrapela
|
I have few points as a reminder that is my concern that I can think of. What do you think about it ? @rm3l @Fortune-Ndlovu |
Co-authored-by: Subhash Khileri <subhashkhileri2@gmail.com> Assisted-by: Claude
Co-authored-by: Subhash Khileri <subhashkhileri2@gmail.com> Assisted-by: Claude
@polasudo For the "OCI registry egress in base policies", yes, it makes sense to add a note about that, especially when the default deployment would need to fetch plugins from registry.access.redhat.com or quay.io or even ghcr.io for community plugins. I'll add a note about that in the ADR. Anyway, we have a spike task to analyze and map RHDH traffic flows, which would help concretely define the base policies rules. |
Yes I was thinking abut some sort of policy template that would guide users to to pull plugins from ghcr for example or from some registry |
…to docs Add OCI (registry.access.redhat.com, quay.io) and NPM (npm.registry.redhat.com) plugin registries to base egress policies. Mention ready-to-use NetworkPolicy templates for common scenarios in the documentation guidance. Assisted-by: Claude Co-authored-by: Martin Polaško <50864147+polasudo@users.noreply.github.com>
makes sense. Updated in |
Fortune-Ndlovu
left a comment
Fortune-Ndlovu
left a comment
There was a problem hiding this comment.
this is a great ADR, thanks @rm3l , I've only added a few comments to help with planning!
…olicies fit RHDH is a layered product deployable in any user namespace without cluster-admin. ANP/BANP are cluster-scoped and require elevated privileges, making them unsuitable as the primary mechanism. Cluster admins can still layer ANP/BANP on top independently. Assisted-by: Claude Co-authored-by: Fortune Ndlovu <fndlovu@redhat.com>
Each component (e.g., RHDH backend, PostgreSQL) gets its own policies appropriate to its role, ensuring defense-in-depth across all components. Assisted-by: Claude Co-authored-by: Fortune Ndlovu <fndlovu@redhat.com>
…ularity The RHDH backend pod needs broad HTTPS egress since plugin registries, LLM endpoints, and other external destinations cannot be pinned by CIDR. Defense-in-depth comes from per-component granularity (PostgreSQL stays locked down) and blocking non-standard ports. Assisted-by: Claude Co-authored-by: Fortune Ndlovu <fndlovu@redhat.com>
Base policies cannot be disabled. On clusters without NetworkPolicy enforcement, they simply have no effect. Assisted-by: Claude
The existing Orchestrator NetworkPolicies in both rhdh-operator and
rhdh-chart use namespace-wide podSelector: {}, which contradicts the
label-scoped principle. Rewriting them is a breaking change for
existing Orchestrator deployments.
Assisted-by: Claude
Include ready-to-use NetworkPolicy templates and connectivity verification steps in migration guides, rather than just "communicate in release notes." Co-authored-by: Fortune Ndlovu <fndlovu@redhat.com> Assisted-by: Claude
…er-component granularity Assisted-by: Claude
gazarenkov
left a comment
gazarenkov
left a comment
There was a problem hiding this comment.
The analysis is thorough and the considerations are valuable, but the decision itself needs sharper definition — specifically, what policies will be created, how they'll be delivered, and who is responsible for what.
In the comments I preliminary propose 3 types of policies and we can consider 3 mechanisms accordingly:
A: base internal (known) - manifests are created either as CNP (Admin) once on operator deployed (always) or dynamically on instance deployed (dynamically, depending on config)
B: base external (known but conditional: external db) - manually per instance (unless we want to make external DB's CIDR to be a part of instance config) or automatically (if we want external DB's CIDR to be declared in controller container)
C: plugins/flavours dependent - to be documented per-plugin/flavour, as a rule requires manual intervention.
Let's discuss
| Additionally, RHDH is supported across multiple Kubernetes platforms (OCP, EKS, AKS, GKE), each with different levels of default NetworkPolicy enforcement. OCP enforces NetworkPolicies out of the box, but other platforms may require users to configure a compatible CNI plugin. | ||
|
|
||
| ### Disconnected/airgapped environments | ||
|
|
There was a problem hiding this comment.
How Airgap impacts on needs in defining NetworkPolicy?
To me it looks like another network (CIDR), local instead of external? Did I miss something?
There was a problem hiding this comment.
You're right that from a NetworkPolicy perspective, airgap mostly means different CIDRs (internal mirrors instead of public endpoints). I actually wanted to highlight why default egress policies can't hardcode specific destinations, since in disconnected environments every endpoint (registries, SCM, auth, ...) would point to a different internal address. The policies themselves don't change structurally; it's the user-added allow rules that might differ. Let me clarify this.
There was a problem hiding this comment.
I do not think we can hardcode specific destinations in non-airgap as well, CIDRs in cloud services tend to change.
There was a problem hiding this comment.
well, I mean a user can point to static public IP address ranges if they want (even in connected use cases), but would have to supply their own policies to allow that. Anyway, that's exactly the point about Alternative 2 (CIDR-based egress restrictions), applicable to both connected and disconnected use cases.
| ### Advanced deployment scenarios | ||
|
|
||
| These introduce additional traffic flows that policies must account for: | ||
|
|
There was a problem hiding this comment.
All of the plugins are "blackboxes" for installation, I think?
There was a problem hiding this comment.
WDYM by "blackboxes for installation"? During the install-dynamic-plugins init container execution?
This section is more about runtime traffic flows (pod-to-pod communication, egress to external endpoints), not plugin installation. Lightspeed, Orchestrator, and external database configurations each introduce distinct network traffic patterns that the base policies may need to take into account. Plugin installation (OCI/NPM fetching) is covered separately under the egress policies.
| - Deployment must not fail on clusters without NetworkPolicy-capable CNI plugins. But Kubernetes already handles this gracefully as policies will be created but not enforced | ||
| - The RHDH Operator itself is installed via OLM, and OLM NetworkPolicy support is still being backported (tracked separately in [RHDHPLAN-351](https://redhat.atlassian.net/browse/RHDHPLAN-351)). This ADR covers only the operands managed by the operator and the Helm chart resources | ||
| - Policies must be flexible enough for disconnected environments. Users must be able to allowlist their internal endpoints without modifying the default policies directly | ||
|
|
There was a problem hiding this comment.
What we consider by "default policies", cluster wide (CNP with Tier Admin or Baseline https://network-policy-api.sigs.k8s.io/api-overview/#the-adminnetworkpolicy-resource )?
There was a problem hiding this comment.
I mean standard namespace-scoped Kubernetes NetworkPolicy resources (not cluster-scoped CNP or ANP/BANP) tied to the current RHDH instance. So these default policies would be created by the Operator for its operands and by the Helm chart for its resources.
Does that make sense?
Also note that as per https://network-policy-api.sigs.k8s.io/#apis , CNP is still a proposal and does not seem available yet (checked on OCP and K8s as well):
K8s 1.35.5
❯ kubectl api-resources | grep -i polic | sort
Alias tip: k api-resources | grep -i polic | sort
accesscontrolpolicies hub.traefik.io/v1alpha1 false AccessControlPolicy
backendtlspolicies btlspolicy gateway.networking.k8s.io/v1 true BackendTLSPolicy
networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy
poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget
validatingadmissionpolicies admissionregistration.k8s.io/v1 false ValidatingAdmissionPolicy
validatingadmissionpolicybindings admissionregistration.k8s.io/v1 false ValidatingAdmissionPolicyBinding
OCP 4.22
❯ oc api-resources | grep -i polic | sort
adminnetworkpolicies anp policy.networking.k8s.io/v1alpha1 false AdminNetworkPolicy
adminpolicybasedexternalroutes apbexternalroute k8s.ovn.org/v1 false AdminPolicyBasedExternalRoute
baselineadminnetworkpolicies banp policy.networking.k8s.io/v1alpha1 false BaselineAdminNetworkPolicy
clusterimagepolicies config.openshift.io/v1 false ClusterImagePolicy
imagecontentpolicies config.openshift.io/v1 false ImageContentPolicy
imagecontentsourcepolicies operator.openshift.io/v1alpha1 false ImageContentSourcePolicy
imagepolicies config.openshift.io/v1 true ImagePolicy
mutatingadmissionpolicies admissionregistration.k8s.io/v1beta1 false MutatingAdmissionPolicy
mutatingadmissionpolicybindings admissionregistration.k8s.io/v1beta1 false MutatingAdmissionPolicyBinding
networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy
poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget
podsecuritypolicyreviews security.openshift.io/v1 true PodSecurityPolicyReview
podsecuritypolicyselfsubjectreviews security.openshift.io/v1 true PodSecurityPolicySelfSubjectReview
podsecuritypolicysubjectreviews security.openshift.io/v1 true PodSecurityPolicySubjectReview
validatingadmissionpolicies admissionregistration.k8s.io/v1 false ValidatingAdmissionPolicy
validatingadmissionpolicybindings admissionregistration.k8s.io/v1 false ValidatingAdmissionPolicyBinding
There was a problem hiding this comment.
Well, I think existence of CNP is the part of the story (it is referenced in OCPSTRAT issue), so it has to be taken into account and opted where feasible.
There was a problem hiding this comment.
where exactly did you see that reference to CNP? I don't see any mention of CNP in OCPSTRAT-819 or OCPSTRAT-939
There was a problem hiding this comment.
On https://redhat.atlassian.net/browse/HPSTRAT-104
"
Success Criteria
A Kube NP YAML file is delivered with each control plane operator that restricts ingress as well as egress communication to only the necessary communication. We may also choose to deliver a default Admin Network Policy that is applied when an OpenShift cluster is installed
...
Note: AdminNetworkingPolicy is a K8s mechanism to set strict secure rules for the cluster. The work under this Jira card should consider and prefer the use of AdminNetworkPolicy resources.
"
There was a problem hiding this comment.
Okay, I think there might be a slight confusion here.. My understanding from this is that HPSTRAT-104 also tracks all control plane operators and that the creation of cluster-wide ANP/BANP/CNP is recommended as part of a default OCP cluster installation. So not applicable to layered products like RHDH.
I still don't see the point for each individual layered Operator to ship its own cluster-wide policies tbh, (potentially resulting in inconsistent policies affecting all workloads).
Even in this case (creating cluster-wide policies), it would need to be done only once when the Operator is installed (not for each Operand), so part of https://redhat.atlassian.net/browse/RHDHPLAN-351 (currently blocked) I guess.
The goal of this work is specifically for NetworkPolicies to be added for Operands.
Anyways, I can get clarification from the OCP team..
There was a problem hiding this comment.
Okay, the OCP team provided clarification on this in this Slack thread. I've updated the ADR accordingly (and the alternative 5).
In a nutshell, we are expected to ship standard NPs and cluster admins can use ANP/BANP/CNP to override if needed. PTAL.
There was a problem hiding this comment.
Commented below (on Alternative 5)
| - **Clear boundary between base and user-managed policies**: The base policies shipped by the Operator and Helm chart cover DNS, API server access, PostgreSQL, broad HTTPS egress for the RHDH backend, and Orchestrator cross-namespace traffic. Egress on non-standard ports or to destinations not reachable over HTTPS (e.g., corporate proxies on custom ports) is the user's responsibility to allow via additive policies. | ||
| - **User-extensible via additive policies**: This split is possible because Kubernetes NetworkPolicies are **additive**: once a default-deny policy selects a pod, any additional NetworkPolicy matching that pod can only add more allow rules, never remove existing ones. Users who need to allow additional site-specific traffic simply create their own NetworkPolicy resources in the same namespace. These compose naturally with the base policies without any CRD or Helm values change. The base NetworkPolicies are always present and cannot be disabled. On clusters without NetworkPolicy enforcement, they simply have no effect | ||
| - **Documentation**: Clear documentation is essential to make this approach work in practice. Users need to understand which base policies are shipped, what traffic they allow and deny, how to create additional NetworkPolicies for their site-specific needs, and how to troubleshoot connectivity issues caused by policies. Documentation should also clarify the requirement for a NetworkPolicy-capable CNI plugin on non-OCP platforms, and provide ready-to-use NetworkPolicy templates for common plugin egress scenarios (e.g., GitHub/GitLab SCM access, Quay/Artifactory registries, OIDC providers) that users can copy and adapt for their environment | ||
| - **Label-scoped policies**: Since RHDH is a layered product deployed into potentially shared namespaces, policies use `podSelector` with RHDH-specific labels rather than namespace-wide selectors, following the OCP best practice for layered products |
There was a problem hiding this comment.
To make it implementation ready, need to mention what exactly labels we're going to use.
There was a problem hiding this comment.
That's an implementation detail, no? Each install method (Operator, Helm chart) already applies its own labels to the pods it manages. The ADR's design principle is that policies must use RHDH-specific label selectors (not namespace-wide podSelector: {}). The exact label keys and values will be determined during implementation and may differ between install methods.
There was a problem hiding this comment.
I believe the values are stable (even if not the same for Chart and Operator) and have to be specified and known outside of implementation so to me it is definitely a part of decision.
…to-the-install-methods-operands-and-helm-chart
Helm releases can be deployed by regular users without cluster-admin, so ANP/BANP cannot be created. For the Operator path, ANP/BANP would need to be shipped in the OLM bundle, which is blocked by RHDHPLAN-351. Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> Co-authored-by: Fortune Ndlovu <fndlovu@redhat.com> Assisted-by: Claude
RHDH is a platform whose egress is defined by its plugins, unlike typical workloads with known network flows. Even officially supported plugins can target customer-specific endpoints, limiting how restrictive base egress policies can be. Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> Assisted-by: Claude
Same policy structure applies; only the destinations differ (internal mirrors instead of public endpoints). Users configure their own additive policies for site-specific internal endpoints. Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> Assisted-by: Claude
Restructure the "clear boundary" section into three categories: always-on base policies, flavour-conditional (auto-managed), and user-managed additive policies. Also acknowledge that ANP/BANP are GA since OCP 4.16 while retaining the privilege and deployment model arguments for rejecting them. Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> Assisted-by: Claude
- Remove Orchestrator cross-namespace from base egress list (belongs in flavour-conditional) - Remove upstream Backstage chart reference from Alternative 3 - Clarify Lightspeed: HTTPS endpoints covered by base rule, only non-standard ports need additional policies - Fix negative consequence: base HTTPS rule covers most egress, users only need additive policies for non-standard ports/protocols - Fix ANP/BANP rejection: focus on privilege model and cross-platform availability, add OCPSTRAT-939 reference - Clarify airgapped section: base HTTPS rule covers most internal endpoints too Assisted-by: Claude
- Collapse advanced deployment scenarios into a key constraint - Keep context section focused on the problem, not the solution - Simplify Alternative 4 rejection (non-standard ports apply to any plugin, not just community ones) - Merge redundant positive consequences (OCPSTRAT-819 + OCP 5+) - Clarify Lightspeed as enabled by default, not optional Assisted-by: Claude
gazarenkov
left a comment
gazarenkov
left a comment
There was a problem hiding this comment.
Classification makes it more clear, thanks.
2 things:
- I guess I miss some decision details to understand how exactly we are going to implement it in each case (static manifest to provide, dynamicaly created etc)
- I would not totally reject CNP (as discussed)
| - **Clear boundary between base and user-managed policies**: The base policies shipped by the Operator and Helm chart cover DNS, API server access, PostgreSQL, broad HTTPS egress for the RHDH backend, and Orchestrator cross-namespace traffic. Egress on non-standard ports or to destinations not reachable over HTTPS (e.g., corporate proxies on custom ports) is the user's responsibility to allow via additive policies. | ||
| - **User-extensible via additive policies**: This split is possible because Kubernetes NetworkPolicies are **additive**: once a default-deny policy selects a pod, any additional NetworkPolicy matching that pod can only add more allow rules, never remove existing ones. Users who need to allow additional site-specific traffic simply create their own NetworkPolicy resources in the same namespace. These compose naturally with the base policies without any CRD or Helm values change. The base NetworkPolicies are always present and cannot be disabled. On clusters without NetworkPolicy enforcement, they simply have no effect | ||
| - **Documentation**: Clear documentation is essential to make this approach work in practice. Users need to understand which base policies are shipped, what traffic they allow and deny, how to create additional NetworkPolicies for their site-specific needs, and how to troubleshoot connectivity issues caused by policies. Documentation should also clarify the requirement for a NetworkPolicy-capable CNI plugin on non-OCP platforms, and provide ready-to-use NetworkPolicy templates for common plugin egress scenarios (e.g., GitHub/GitLab SCM access, Quay/Artifactory registries, OIDC providers) that users can copy and adapt for their environment | ||
| - **Label-scoped policies**: Since RHDH is a layered product deployed into potentially shared namespaces, policies use `podSelector` with RHDH-specific labels rather than namespace-wide selectors, following the OCP best practice for layered products |
There was a problem hiding this comment.
I believe the values are stable (even if not the same for Chart and Operator) and have to be specified and known outside of implementation so to me it is definitely a part of decision.
| ### Alternative 5: Use AdminNetworkPolicy (ANP) / BaselineAdminNetworkPolicy (BANP) instead of standard NetworkPolicies | ||
| - **Approach**: Use the AdminNetworkPolicy (ANP) or BaselineAdminNetworkPolicy (BANP) resources, as recommended for consideration by the parent outcome [HPSTRAT-104](https://redhat.atlassian.net/browse/HPSTRAT-104). | ||
| - **Rejected because**: Although ANP and BANP are GA since OCP 4.16 ([OCPSTRAT-939](https://redhat.atlassian.net/browse/OCPSTRAT-939)) and will be available in OCP 5, they are cluster-scoped resources that require cluster-admin privileges to create and manage. For the Helm chart path, releases can be deployed by regular cluster users who do not have cluster-admin privileges, so they simply cannot create ANP/BANP resources. ANP/BANP are designed for platform-level network governance enforced by cluster admins, not for individual workload self-protection. They are also not GA in upstream Kubernetes and not universally available across all RHDH-supported platforms (EKS, AKS, GKE). Standard namespace-scoped NetworkPolicies are the appropriate tool for a layered product. Note that cluster admins can still layer ANP/BANP on top of RHDH's standard NetworkPolicies for additional cluster-wide enforcement, but that is outside RHDH's scope. | ||
|
|
There was a problem hiding this comment.
I think Cluster admin privileges is a constraint for Helm chart, not a limitation.
I would not consider CNPs as a hard alternative, it is an option to consider where feasible, and as I can see it for the time, quite powerful option in a case of Operator at least (I would prefer to avoid excessive coding where we can use K8s features)
There was a problem hiding this comment.
As clarififed by the OCP team, we are expected to ship standard NPs, and cluster admins may use cluster-wide ANP/BANP/CNP to override if needed.
There was a problem hiding this comment.
Reading the conversation I can see "ANP may be provided but is not required", that's exactly what I mean.
I think there is some discrepancy with "ship" term. As correctly mentioned, normally Operator can not ship ANP (permissions) but I guess it can be shipped along with Operator (as part of manifest, by cluster admin)... Asked at the the same thread.
| - Deployment must not fail on clusters without NetworkPolicy-capable CNI plugins. But Kubernetes already handles this gracefully as policies will be created but not enforced | ||
| - The RHDH Operator itself is installed via OLM, and OLM NetworkPolicy support is still being backported (tracked separately in [RHDHPLAN-351](https://redhat.atlassian.net/browse/RHDHPLAN-351)). This ADR covers only the operands managed by the operator and the Helm chart resources | ||
| - Policies must be flexible enough for disconnected environments. Users must be able to allowlist their internal endpoints without modifying the default policies directly | ||
|
|
There was a problem hiding this comment.
Well, I think existence of CNP is the part of the story (it is referenced in OCPSTRAT issue), so it has to be taken into account and opted where feasible.
| Additionally, RHDH is supported across multiple Kubernetes platforms (OCP, EKS, AKS, GKE), each with different levels of default NetworkPolicy enforcement. OCP enforces NetworkPolicies out of the box, but other platforms may require users to configure a compatible CNI plugin. | ||
|
|
||
| ### Disconnected/airgapped environments | ||
|
|
There was a problem hiding this comment.
I do not think we can hardcode specific destinations in non-airgap as well, CIDRs in cloud services tend to change.
gazarenkov
changed the title
…tailored-networkpolicies-to-the-install-methods-operands-and-helm-chart
Document the exact labels used by each install method to scope NetworkPolicies to a specific component of a specific instance: Operator uses rhdh.redhat.com/app, Helm uses app.kubernetes.io/instance combined with app.kubernetes.io/component. Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> Assisted-by: Claude
OCP team confirmed operators are expected to ship standard NetworkPolicies, not ANP. ANP is for cluster admins to override or tighten operator-shipped policies as needed. Assisted-by: Claude
…tailored-networkpolicies-to-the-install-methods-operands-and-helm-chart
Move key paragraphs (multi-platform support, plugin-driven unpredictable egress, disconnected environments) from the problem statement into Key constraints, reordered by impact. Rename ADR title from "Install Methods" to "Application Workloads" to avoid confusion with the Operator itself. Summarize user-extensible policies section, add CNP to Alternative 5, clarify Lightspeed as default since 1.10, replace em dashes with semicolons, and use CR-specific/release-specific terminology for label-scoped policies. Assisted-by: Claude
Add three example policies to the Decision section illustrating the label-scoped approach for both Operator and Helm chart paths: default-deny (Helm), allow Router ingress (Operator, OCP-specific), and DNS-only egress for PostgreSQL (Operator, cross-platform). Assisted-by: Claude
Address threat model finding about insecure Prometheus metrics port by explicitly documenting port 9464 ingress restriction to the monitoring namespace and adding a corresponding NetworkPolicy example. Assisted-by: Claude
5 participants
Summary
Assisted-by: Claude
Related