KEP-2676: Support Istio ambient in Kubeflow by orfeas-k · Pull Request #891 · kubeflow/community

orfeas-k · 2025-07-31T19:38:46Z

Add proposal for supporting Istio ambient in Kubeflow.

Context

This has been discussed for more than a year in the Kubeflow community (see kubeflow/community-distribution#2676). Currently, there is an ongoing GSOC project working on implementing this (see
#809 (comment)), which will be greatly facilitated by agreeing on expected changes.

What is this proposal about?

This proposal:

defines the changes needed to make Kubeflow components ambient-compatible
outlines a setup that uses Istio in ambient and enables all the existing Authorization and Authentication use cases.

Ref kubeflow/community-distribution#2676

cc @kimwnasptd @juliusvonkohout @thesuperzapper

Add proposal for supporting Istio ambient in Kubeflow. This has been discussed for more than a year in the Kubeflow community (see kubeflow/community-distribution#2676). Given that right now, there is an ongoing GSOC project working on this (see kubeflow#809 (comment)), this proposal aims to facilitate the project and: * defines the changes needed to make Kubeflow component ambient-compatible * outlines a setup that uses Istio in ambient and enables all the existing Authorization and Authentication use cases. Ref kubeflow/community-distribution#2676 Signed-off-by: Orfeas Kourkakis <44405072+orfeas-k@users.noreply.github.com>

juliusvonkohout · 2025-08-01T17:06:16Z

+* A high-level architecture of Kubeflow that uses Istio ambient and showcases how to:
+  * Use HTTPRoutes instead of VirtualServices
+  * Enable the user login flow
+  * Use waypoints to enforce namespace isolation (L7 and L4 traffic)


Suggested change

* Use waypoints to enforce namespace isolation (L7 and L4 traffic)

* Use waypoints to enforce namespace isolation (L7 and L4 traffic)

* Provide Zero overhead namespaces, so without actual user workloads the Pods, CPU and memory used should be 0 (after the seaweedfs PR is merged). So no waypoint proxies are allowed in kubeflow profile namespaces.

This is also related to #891 (comment).

I think this is not needed since it's not an actual goal of this proposal.

juliusvonkohout · 2025-08-01T17:08:41Z

+To enable authorization for Kubeflow components, the whole `kubeflow` namespace is included in the mesh. 
+
+##### Waypoints
+To get the best performance out of ambient mode[^6][^7], services are enrolled to a waypoint only when the L7 features are required. For the Kubeflow control plane, only authorizationPolicies targeting `ml-pipeline` and `ml-pipeline-ui` make use of L7 attributes. Thus, a waypoint is deployed in the `kubeflow` namespace while only the services of those two components are enrolled to the waypoint.


Are you sure that notebooks do not use L7 ? I mean everyone with the kubeflow-userid header uses L7 so i guess almost all components in Kubeflow.

Yes. All those components trust the ingress gateway and the headers it provides. Thus, they accept all requests from the ingress gateway without any filtering done at the AuthorizationPolicy, like kfp components do (api server)

I was talking about the jupyterlab itself in the customer namespace. They should be created on demand based on the notebook custom resource.

Do you refer to the AuthorizationPolicy created by the profile-controller? This is created dynamically when a new user namespace is created and it also allows/denies traffic to jupyterlab server pods. This looks like this, which indeed has L7 rules.

juliusvonkohout · 2025-08-01T17:12:42Z

+##### One waypoint per namespace
+User namespaces are secured via an AuthorizationPolicy that is created dynamically by kubeflow's `profile-controller`. Most of the rules in this AuthorizationPolicy have L7 features[^8]. In sidecar mode, this AuthorizationPolicy is enforced to all traffic in the namespace by having no `selector`. This is due to the nature of those namespaces, where services are created dynamically. In Istio ambient mode, the only way to enforce a namespace-wide L7 AuthorizationPolicy is by proxying traffic to the whole namespace through the waypoint and then attaching the AuthorizationPolicy to the waypoint. This results in the authorization rules being enforced for all traffic proxied by the waypoint. This means that **a separate waypoint needs to be deployed in every user namespace**.


No, we want to have exactly the opposite. Never ever waypoint proxies in user namespaces. Also the istio documentation and my talk with @kimwnasptd and the istio/solo.io founder at Kubecon supports that you do not need them in user namespaces. See https://istio.io/latest/docs/ambient/usage/waypoint/#configure-resources-to-use-a-cross-namespace-waypoint-proxy. That is the most essential part that you do not have waypoint proxies there and do not destroy zero overhead user namespaces.

(replied in #891 (comment))

juliusvonkohout · 2025-08-01T17:15:00Z

+#### One waypoint per namespace
+Apparently, deploying one waypoint per namespace adds a resource overhead for every user namespace, while it also reduces network efficiency for requests to that namespace. As mentioned above, this architecture is required by Istio's ambient model to enforce namespace-wide L7 AuthorizationPolicies. Since the `profile-controller` creates a default policy with L7 rules  (e.g. path or headers based), one waypoint must correspond only to one namespace. The same Authorization model is not possible using a single cross-namespace waypoint. [Asking in Istio slack](https://istio.slack.com/archives/C041EQL1XMY/p1751905575407449?thread_ts=1751636618.350839&cid=C041EQL1XMY), they confirmed that:
+>Typically, the authorization policy would be applied to a service so the destination would be "the thing my policy is attached to".
+>...
+>I don't think there is a shorthand expression for "all services in X namespace".
+
+This leaves no other way to enforce an L7 AuthorizationPolicy to all services in a namespace, where services are being created dynamically.
+
+Note that the introduced resource overhead affects cases with a large number of **idle** namespaces. For namespaces that are being actively used, the resource footprint is still greatly reduced, given that there are no sidecar containers injected.


This sounds really problematic. So i would rather stay with Istio-cni until this is possible. Follow up of https://github.com/kubeflow/community/pull/891/files#r2248473959

I don't think we should block on this. Of course, providing zero overhead namespaces without any namespaced resources should be the north star we 're after. The fact though that Istio ambient does not enable us to do so should not be a reason for Kubeflow to not be ambient compatible. Both istio-cni and ambient mode have their pros and cons and Kubeflow should provide administrators the option to decide on their own which mode suits them. For example, clusters with few namespaces but hundreds of workloads per namespace would benefit greatly from using ambient, contrary to clusters with hundreds of idle namespaces and a few workloads per namespace.

As mentioned in the proposal, instead of blocking, I suggest:

Implement code that does not provide enforce any high-level architecture. That means that it will be configurable enough to support both waypoint per namespace and cross-namespace waypoint architectures. If Istio ambient starts supporting L7 namespace isolation using a single cross-namespace waypoint, then the code is already compatible or at minimum needs a few tweaks rather than the whole "sidecar -> istio ambient & gateway API" migration effort.

Provide an ambient overlay in manifests. The Istio-cni will still be the default one but providing an overlay enables users and distributions to try ambient and experience benefits and downsides. This can greatly help the next point.

Ask feature from Istio to support L7 namespace isolation using a single cross-namespace waypoint. I 've already asked them in Slack and plan to bring this up in their meetings too. However, I suspect that they have a good reason this is not supported right now. If there is adoption of Istio ambient by kubeflow users, then this puts us in a better position by providing valid use cases for it.

Agree with all of @orfeas-k's points.

While having waypoints per namespace will have a different resource footprint from CNI, still this is up to the vendors to decide if they are OK with or not rather than not supporting it in Kubeflow.

The example can keep the CNI, but the project should still give the option to end users to have Kubeflow with K8s Gateway API and ambient.

Yes, i agree, lets not make istio ambient the default until this is possible. So for the time being istio-cni is the default and istio-ambient opt-in overlay of istio-cni.

I think technically they 'll need to be two separate overlays but let's keep this implementation detail for implementation time. We all agree that Istio-cni will be the default one, provided in the example.

I just want to keep a single istio folder. In there we also have the legacy insecure overlay to disable istio-cni. https://github.com/kubeflow/manifests/tree/master/common/istio/istio-install/overlays/insecure. please lets not create multiple istio versions to maintain, but one base version with overlays.

This was brought up in the Istio community meeting by @ca-scribner and it looks like they don't see any blockers to this. It was not an intentional omission, just something that hasn't been implemented or decided on. There was nothing brought up that was a hard blocker, more just challenges to overcome. It would though be a big effort, so before they actually action it they'd want to see more than a single person requesting the feature.

I filed a feature request here https://github.com/istio/istio.io/issues/16781. What we need to do now is to:

get more consensus/demand, which can only be done by getting more people to use the ambient overlay (at least from Kubeflow's side)

bring it up again in a future community meeting once the demand is obvious

I 'm updating the proposal with this information to make sure it is documented.

Include information about the feature request in Istio. Signed-off-by: Orfeas Kourkakis <44405072+orfeas-k@users.noreply.github.com>

github-actions · 2025-11-13T00:19:46Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

juliusvonkohout · 2025-11-17T16:29:01Z

@kimwnasptd @orfeas-k do you want to continue here ?

monotek · 2025-11-18T11:12:07Z

We already have this running with ambient mesh.
We removed all istio inject labels from kubeflow resources, added "istio.io/dataplane-mode: ambient" to the profiles namespace default labels and patched / generated the resources in the workspace ns with Kyverno.

These are the rules we're using for this to use a shared waypoint:

Service:

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-kubeflow-notebook-service
spec:
  rules:
    - name: mutate-kubeflow-notebook-service
      match:
        any:
          - resources:
              kinds:
                - Service
              namespaces:
                - kubeflow-*
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              app: kubeflow-notebook
              istio.io/ingress-use-waypoint: "true"
              istio.io/use-waypoint: waypoint
              istio.io/use-waypoint-namespace: istio-system
              kyverno-mutate: mutate-kubeflow-notebook-service

AuthorizationPolicy needed for notebook frontend (this is an example without oauth flow):

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-kubeflow-notebook-authorization-policy
spec:
  rules:
    - name: generate-kubeflow-notebook-authorization-policy
      match:
        any:
          - resources:
              kinds:
                - Namespace
              namespaces:
                - kubeflow-*
      generate:
        synchronize: true
        apiVersion: security.istio.io/v1
        kind: AuthorizationPolicy
        name: kubeflow-notebook
        namespace: "{{request.object.metadata.name}}"
        data:
          metadata:
            labels:
              app: kubeflow-notebook
          spec:
            action: ALLOW
            rules:
              - from:
                  - source:
                      principals:
                        - "cluster.local/ns/istio-system/sa/waypoint"

juliusvonkohout · 2025-11-18T12:10:13Z

Thank you @monotek. Do you mind adding Staffabse to the Kubeflow Adopters ? https://github.com/kubeflow/community/blob/cb085e4311e48b906cec9c6c1653bb94928cbf9b/ADOPTERS.md? See also #833

juliusvonkohout · 2025-11-18T16:32:35Z

@monotek we have just merged ambient here kubeflow/community-distribution#3246. Maybe you want to evaluate it. CC @madmecodes

monotek · 2025-11-20T12:52:45Z

Thank you @monotek. Do you mind adding Staffabse to the Kubeflow Adopters ? https://github.com/kubeflow/community/blob/cb085e4311e48b906cec9c6c1653bb94928cbf9b/ADOPTERS.md? See also #833

We're still evaluating kubeflow. Not sure yet if we'll keep using it.

@monotek we have just merged ambient here kubeflow/manifests#3246. Maybe you want to evaluate it. CC @madmecodes

That's likely not enough because the notebook controller should be able to set waypoint labels to the notebook service too.

juliusvonkohout · 2025-11-20T16:39:05Z

If you are still evaluating then i strongly recommend to start of the master branch which will become the 1.11 release.

github-actions · 2026-02-19T00:28:31Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

juliusvonkohout · 2026-02-25T14:21:25Z

See kubeflow/dashboard#185

github-actions · 2026-05-27T00:48:03Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions · 2026-06-17T00:55:09Z

This pull request has been automatically closed because it has not had recent activity.You can reopen the PR if you want.

christian-heusel · 2026-07-02T16:23:15Z

@kubeflow/kubeflow-steering-committee this KEP got closed by the stale bot even though the people that raised this are still interested in driving this forward (we just discussed this in the Notebooks WG Call due to a proposed change in the dashboard kubeflow/dashboard#267). Could you clarify on the status of the proposal and make sure that it is being properly shepherded? 🤗

cc @deusebio

google-oss-prow · 2026-07-03T07:45:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chasecadet for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

deusebio · 2026-07-03T12:27:21Z

Yes, as I mentioned yesterday during the call, we have definitive interest seeing this feature consolidated upstream, since in the past year we've kept on working, re-opening pull-requests, rebasing and addressing comments in the various PRs that we have opened, naively hoping this could have happened for 26.03 already. The work we have done goes well beyond just the PRs opened there, since we have also already integrated those changes into our solution as an experimental feature to test and validate that the integration does indeed work as expected.

If anyone want to give it a go, you can already either:

deploy Charmed Kubeflow, also following the guidance here on how to migrate/enable ambient
Or you can also install Charmed Kubeflow with ambient using this terraform module and following the guidance here.

This should already provide an easy path to anyone that is also interest on this topic and they would like to give it a go to get started.

We strongly believe this would be a very sensible feature that we hope it can be of broader interest for the community. Beside providing support to ambient (that is more efficient and easy to manage than side-car), this also brings the integration with the Gateway API, that would also make a step forward to make Kubeflow more general and compliant with the new way to manage routing in K8s. But having said this (which are our considerations why we believe this is a useful feature), at this point I'd really like to have clearer indication whether there is indeed interests and commitment from upstream to get this in. I really wouldn't like to keep on working on this any further to then figure out that the feature was not of interest or will never make it to a release.

monotek · 2026-07-03T14:27:27Z

Please not.
Managing k8s resources with Terraform almost certainly lead to configuration drift.

Why can't we just have plain k8s resources or a Helm chart, which we can use in a real gitops tool like FluxCD, independent from a service mesh or anything else? 😢

deusebio · 2026-07-03T15:44:34Z

Let's not drift conversation. There is no proposal to use Terraform in the context of the changes discussed here in the KEP.

I just provided a way to consume Charmed Kubeflow with ambient if anyone wanted to give it a go and look around.

google-oss-prow Bot requested review from johnugeorge and juliusvonkohout July 31, 2025 19:38

google-oss-prow Bot added the size/L label Jul 31, 2025

orfeas-k force-pushed the proposal-support-istio-ambient branch from 2797df2 to 55c124c Compare July 31, 2025 19:39

juliusvonkohout reviewed Aug 1, 2025

View reviewed changes

update: Add information about waypoint per user namespace

aa9d8ba

Include information about the feature request in Istio. Signed-off-by: Orfeas Kourkakis <44405072+orfeas-k@users.noreply.github.com>

github-actions Bot added the lifecycle/stale label Nov 13, 2025

github-actions Bot removed the lifecycle/stale label Nov 18, 2025

juliusvonkohout mentioned this pull request Nov 18, 2025

Add more adopters #833

Open

52 tasks

github-actions Bot added the lifecycle/stale label Feb 19, 2026

github-actions Bot removed the lifecycle/stale label Feb 26, 2026

bkanuka mentioned this pull request Apr 26, 2026

Istio Ambient support kubeflow/community-distribution#2676

Open

github-actions Bot added the lifecycle/stale label May 27, 2026

github-actions Bot closed this Jun 17, 2026

juliusvonkohout reopened this Jul 3, 2026

github-actions Bot removed the lifecycle/stale label Jul 4, 2026

	* Use waypoints to enforce namespace isolation (L7 and L4 traffic)
	* Use waypoints to enforce namespace isolation (L7 and L4 traffic)
	* Provide Zero overhead namespaces, so without actual user workloads the Pods, CPU and memory used should be 0 (after the seaweedfs PR is merged). So no waypoint proxies are allowed in kubeflow profile namespaces.

		##### One waypoint per namespace
		User namespaces are secured via an AuthorizationPolicy that is created dynamically by kubeflow's `profile-controller`. Most of the rules in this AuthorizationPolicy have L7 features[^8]. In sidecar mode, this AuthorizationPolicy is enforced to all traffic in the namespace by having no `selector`. This is due to the nature of those namespaces, where services are created dynamically. In Istio ambient mode, the only way to enforce a namespace-wide L7 AuthorizationPolicy is by proxying traffic to the whole namespace through the waypoint and then attaching the AuthorizationPolicy to the waypoint. This results in the authorization rules being enforced for all traffic proxied by the waypoint. This means that a separate waypoint needs to be deployed in every user namespace.

Uh oh!

Conversation

orfeas-k commented Jul 31, 2025

Context

What is this proposal about?

Uh oh!

juliusvonkohout Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 13, 2025

Uh oh!

juliusvonkohout commented Nov 17, 2025

Uh oh!

monotek commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juliusvonkohout commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juliusvonkohout commented Nov 18, 2025

Uh oh!

monotek commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juliusvonkohout commented Nov 20, 2025

Uh oh!

github-actions Bot commented Feb 19, 2026

Uh oh!

juliusvonkohout commented Feb 25, 2026

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

christian-heusel commented Jul 2, 2026

Uh oh!

google-oss-prow Bot commented Jul 3, 2026

Uh oh!

deusebio commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotek commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deusebio commented Jul 3, 2026

juliusvonkohout Aug 1, 2025 •

edited

Loading

juliusvonkohout Aug 12, 2025 •

edited

Loading

juliusvonkohout Aug 13, 2025 •

edited

Loading

monotek commented Nov 18, 2025 •

edited

Loading

juliusvonkohout commented Nov 18, 2025 •

edited

Loading

monotek commented Nov 20, 2025 •

edited

Loading

deusebio commented Jul 3, 2026 •

edited

Loading

monotek commented Jul 3, 2026 •

edited

Loading