Repository navigation
Loki output: set tenant header based on k8s label instead of static value #1145
Description
Activity
Update:
It seems logging-framework uses the loki output from fluentd.
fluentd does not yet support tenant_id_key which fluentbit does support:
https://docs.fluentbit.io/manual/v/1.8/pipeline/outputs/lokiI opened a ticket with fluend to also support this, then the logging-framework needs to pass through only?
fluent/fluentd#3992I believe this is already supported by the fluentd loki plugin, see https://grafana.com/docs/loki/latest/clients/fluentd/#tenant.
The tenant field also supports placeholders, so it can dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a k8s pod label:
<match **> @type loki url "https://logs-prod-us-central1.grafana.net" tenant ${$.kubernetes.labels.tenant} # ... <buffer $.kubernetes.labels.tenant> @type memory flush_interval 5s </buffer> </match>Also, the associated logging-operator output field is already passed through to the fluentd configuration: https://github.com/banzaicloud/logging-operator/blob/42402497f2116f014f14ba867fdce13aba0bf171/pkg/sdk/logging/model/output/loki.go#L74
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!
thanks @aslafy-z for clarifying this! @fekete-robert is there a good place in the docs to add this?
I think we can do two things:
- either add this example to the documentation of the Loki output (this is generated from the source code https://github.com/kube-logging/logging-operator/blob/master/pkg/sdk/logging/model/output/loki.go - everything has to be added as code comments, so it's not very convenient and prone to formatting errors),
- or we can create a separate page in the docs repo (under https://kube-logging.github.io/docs/examples/ ) that describes the scenario and how to do it.
I would opt for the separate page, but first someone should actually try this out whether it works correctly or not. I closed this prematurely, sorry.
Reacted by Robert FeketeThis issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!
- locked and limited conversation to collaborators
on Aug 30, 2023
Describe the solution you'd like
I want to define 1 default Loki ClusterOutput but not with a static tenant.
With this, tenants don't get to see the config (it's cluster level) and for 100 tenant's I don't need then to create 100 namespaced output's with the same loki endpoint and auth config with is unsecure and harder to maintain if something changes.
e.g.:
Additional context
We create multi tenant k8s clusters and use capsule to create Tenant's in the cluster.
It's not relevant to know how Capsule works internally, only that namespaces and pods have a certain label with their tenant id.
For logging, I need to extract this label to be forwarded as the tenant header (X-Scope-OrgId) to Loki.
I see in the code that something like this is available for kafka (headers_from_record in the crds):
More important: Fluentd also supports this:
https://docs.fluentd.org/output/http#headers_from_placeholders
For non-loki users it might be interesting to also add this feature (with a more generic name: headers_from_record) to the http output