Set Log Rotation Dinamically

[phgeraldeli]

Describe the question/issue

I need to set the log rotation dinamically by using a kubernetes label. It's supported? If It is, how can i do it?

I'm using below syntax to set log group name dinamically, but when I do it with logRotation key, It does not work.

[OUTPUT]
      log_group_name $(kubernetes['labels']['my_label'])
      ...

I've tried to do the same passing log_retention_days as $(kubernetes['labels']['log_retention_days']) but It did'nt work, I wish i could do a default value as well. Something like, if it has this label use It, instead use default.

Fluent Bit Log Output

Couldn't get the log

Fluent Bit Version Info

2.31.12

Cluster Details

EKS + cloudwatch

Application Details

Steps to reproduce issue

Deploy daemonset in EKS using aws-for-fluent-bit chart
Deploy a app with the specific labels
Set log_group_name dinamically
Do the same with logRotation
Let it creates a new LogGroup
Verify that is Never Expire

Related Issues

[trojaond]

Running into the same issue with log_retention_days. The reason for that might be that according the documentation both log_retention_days and log_group_name are not able to handle record_accessor syntax. Whereas log_stream_template and log_group_template are.

Desired functionality:

[OUTPUT]
    Name cloudwatch_logs
    Match kube.*
    log_group_name $log_group_name ❌ (this does not work)
    log_stream_prefix fallback-stream
    log_group_template $group_name.  ✅
    log_stream_template $stream_name   ✅
    auto_create_group true
    log_retention_days $log_retention ❌ (this does not work)

where record contains these items (for exxample):

{
  "log_group_name": "test_group_fallback",
  "log_retention": 7,
  "group_name": "/kubernetes/logs",
  "stream_name": "kubernetes"
}

So more specifically, we would like to access record fields the same log_group_template and log_stream_template . Can we request this feature to be added?