CFP-46532: Synthetic Probing Subsystem for Connectivity Testing - #99
CFP-46532: Synthetic Probing Subsystem for Connectivity Testing#99arighnac wants to merge 2 commits into
Conversation
|
cc @jrife |
|
|
||
| * `status.SourceNode` is a [selectable field](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/). cilium-operator also establishes filtered watches on the kube API server for any event that updates the field selector `status.SourceNode` to empty value. It also sets a client predicate to ensure that the older value and the new empty value are different. | ||
|
|
||
| ### flowgen Daemonset |
There was a problem hiding this comment.
We should just put this into the Cilium agent. Why a standalone DaemonSet?
There was a problem hiding this comment.
Yes, it makes sense to punt flowgen-agent into cilium-agent itself. We can keep this as a separate Hive, and just ensure that the flowgen-agent watchers can run as isolated background goroutines so that it doesn't impact cilium-agent.
The daemonset serves multiple purposes though, both an agent as well as a flowgen-client which generates the actual synthetic packets. It seems there are two options: have the client embedded within cilium-agent itself, or have the flowgen-client as a standalone process launched by cilium-agent and communication between the two over an UDS. I am erring towards having a separate process to perform synthetic traffic generation. Let me know what you think!!
There was a problem hiding this comment.
I have updated this in the CFP to remove the flowgend daemonset deployment, and instead have flowgen-agent run as a Hive within cilium-agent itself.
|
|
||
| ## Section 5: Synthetic Probing: Dataplane | ||
|
|
||
| ### Section 5.1: Dataplane Init Steps |
There was a problem hiding this comment.
Why not just deploy the flowgen-client as a non-host-network pod and use the normal loader machinery to do the BPF programming. We might do something like plugins to load cil_from_flowgen+cil_to_flowgen in place of the normal cil_from_container...
If not a pod then maybe something similar to the Cilium health check endpoint where we have the Cilium agent managing all of this then the flowgen client "endpoint" is programmed as any other would be. That way we could eliminate the client needing to be a standalone pod.
There was a problem hiding this comment.
Why not just deploy the flowgen-client as a non-host-network pod and use the normal loader machinery to do the BPF programming. We might do something like plugins to load cil_from_flowgen+cil_to_flowgen in place of the normal cil_from_container...
I am erring towards the side of this. The flowgen-client can be deployed within a netns, and cilium can just plug in a flowgen veth pair, on which we can load the special BPF programs using plugins. I hope the deployment of flowgen-client will still be managed by cilium, since we do not want users to deploy this separately???
If not a pod then maybe something similar to the Cilium health check endpoint where we have the Cilium agent managing all of this then the flowgen client "endpoint" is programmed as any other would be. That way we could eliminate the client needing to be a standalone pod.
I am erring towards Cilium managing deployment of flowgen-client process in an isolated namespace along with loading the flowgen BPF programs on the special flowgen veth pair. Wdyt?
| * Modify the packet source and destination IP’s and ports to the user 4-tuple, using `ctx_store_bytes`. | ||
|
|
||
| * BPF-Redirect the packet to the ingress hook of the user source pod lxc interface. | ||
| * `ctx_redirect(ctx, sep->ifindex, BPF_F_INGRESS)` |
There was a problem hiding this comment.
This ingress redirection won't work with netkit, since it would skip cil_from_container (I realize this is why you limited the scope initially to just veth), but considering this patch was merged to support BPF_F_EGRESS with bpf_redirect_peer, it's probably worth mentioning how we intend to make this work with netkit now. We'd basically just swap out one or the other depending on the datapath mode and would need a startup probe that disallows the use of flowgen with netkit if the system's kernel doesn't have BPF_F_EGRESS support.
There was a problem hiding this comment.
@jrife Thanks a lot for the patch!! I wil update the CFP with this.
Signed-off-by: Arighna Chakrabarty <arighnac@google.com>
Signed-off-by: Arighna Chakrabarty <arighnac@google.com>
@msune I saw your PRs and it definitely seems that the ideas are aligned (you are proposing the DSCP headers, I am proposing IPv4 options, which we already use for packet tracing in Cilium), although the CFP builds an API and a robust subsytem around generating the synthetic packets? Could you please review this CFP and see if you want any specific areas to be addressed? |
3 participants
Summary
An observability subsystem that allows an user to generate out-of-band synthetic unidirectional flows to perform on-demand connectivity check/test across running workloads, with zero intrusion on the user workloads. The synthetic flows needs to support a wide range of CUJs, be observable via the Hubble sub-system, and generate more fine-grained debugging data from the Cilium dataplane.
Motivation
On-Demand Connectivity Validation: Allow developers and operators to quickly generate network flows across multiple endpoints via an API. This enables users to perform "on-demand" connectivity status checks between any two workloads with less friction. This also enables users to perform network validations before productionalizing workloads on k8s clusters.
Troubleshooting in the Absence of Live Traffic: A challenge in resolving network connectivity failures for users is the indeterminate volume of traffic at the time of reproduction. By generating synthetic flows, engineers can proactively induce and diagnose issues without waiting for user traffic, expediting root cause analysis.
Extract Important Debugging State out of the Dataplane: Synthetic probes can act as a great interface to generate fine-grained debugging state out of Cilium, given that the generation of such packets would be controlled by the API.
(Discussion continued in CFP.....)