Enhancement: Allow Argo Label customization

[Capiru]

If we could add labels to argo workflows, we could more easily group workflows in the Argo UI.

Current state:
I see that here the labels are only being built using _base_labels:

metaflow/metaflow/plugins/argo/argo_workflows.py

Line 892 in 13a7cad

.labels(self._base_labels)

However, _base_labels is only being built from _base_kubernetes_labels: https://github.com/Netflix/metaflow/blob/13a7cad8081aff1f34db8d4e663d8940e65ebbdc/metaflow/plugins/argo/argo_workflows.py#L189C34-L189C57
Since _base_kubernetes_labels is effectively hardcoded, there's no customization possible in the labels, i see also that there's a TODO exactly what i would suggest here, that there is an environment variable to use here:

metaflow/metaflow/plugins/argo/argo_workflows.py

Lines 402 to 409 in 13a7cad

	def _base_kubernetes_labels(self):
	"""
	Get shared Kubernetes labels for Argo resources.
	"""
	# TODO: Add configuration through an environment variable or Metaflow config in the future if required.
	labels = {"app.kubernetes.io/part-of": "metaflow"}
	return labels

Proposed Solution:
Add METAFLOW_ARGO_WORKFLOWS_LABELS. Then use it to set the labels.

Example PR: #2781

[Sridhar1030]

Hi, I’m interested in working on this issue.

From my understanding, the current implementation only applies labels from _base_kubernetes_labels, which is effectively static. Adding support for a configurable environment variable like METAFLOW_ARGO_WORKFLOWS_LABELS would allow users to attach custom labels to Argo Workflows and make grouping/filtering in the Argo UI much easier.

My proposed approach:

• Introduce support for a new environment variable METAFLOW_ARGO_WORKFLOWS_LABELS.
• Accept labels in a key=value comma-separated format (e.g., team=data,env=prod).
• Parse and merge these labels with _base_kubernetes_labels, ensuring base labels are preserved.
• Add validation to ensure Kubernetes label format compliance.
• Include documentation and tests for the new configuration.

Before starting, I’d like to confirm:

1. Does the comma-separated key=value format work, or would JSON/YAML be preferred?
2. Should custom labels override base labels if the same key is provided?
3. Are there any existing configuration patterns in Metaflow that I should follow for environment-based settings?

If this sounds good, I’ll go ahead and start working on a PR.

[Sridhar1030]

/assign

[theOGognf]

We need this feature for organizing our generated WorkflowTemplate resources, but I'm not sure if adding a new and generic Argo Workflows configuration env var is the best way to do it. There's already a KUBERNETES_LABELS env var configuration option, and the ability to add labels via the @kubernetes decorator. The containers in a workflow (i.e., each step) already use the labels from the @kubernetes decorator; the parent WorkflowTemplate object just needs those labels to be propagated upwards, or for it to respect the KUBERNETES_LABELS env var itself

In cases where users want to specify labels only for the parent WorkflowTemplate resource, then it might make sense to have a new Argo Workflows configuration env var like ARGO_WORKFLOWS_TEMPLATE_LABELS that isn't applied to the child container templates

I'd be happy to give this an attempt if provided a bit of guidance on which approach is preferred

[Capiru]

I apologise for the late reply as i was out on vacation.

Does the comma-separated key=value format work, or would JSON/YAML be preferred?

IMO, it should follow the same principle as the kubernetes labels with comma-separated key=value as i suggested in my draft PR.

Should custom labels override base labels if the same key is provided?

Ideally no, because the purpose of the base labels is to distinguish metaflow to non metaflow tasks, but i don't see why not override if manually specified by the user.

Are there any existing configuration patterns in Metaflow that I should follow for environment-based settings?

There are some similarities in how it is assigning kubernetes labels to both argo workflows and kubernetes labels