Support for running aws-for-fluent-bit on Bottlerocket nodes without runAsUser: 0

[amiyusau17]

Summary

The official aws-for-fluent-bit DaemonSet manifests and associated AWS documentation
(e.g. Logging for Amazon EKS,
Container Insights Fluent Bit setup)
require Fluent Bit containers to run as runAsUser: 0 in order to collect host-level and
dataplane logs. This is in direct conflict with Bottlerocket's security recommendations,
which explicitly discourage running workload containers as root.

Environment

• EKS node OS: Bottlerocket
• Logging setup: aws-for-fluent-bit:3.4.3 DaemonSet (Container Insights)
• Log targets: host logs via systemd journal (/var/log/journal), dataplane logs
  (kubelet, kube-proxy), and application logs (/var/log/containers)

Problem

On Bottlerocket nodes, host and dataplane logs are collected via the Fluent Bit systemd
input plugin, reading directly from the mounted journal path /var/log/journal:

[INPUT]
    Name                systemd
    Tag                 host.kernel
    Path                /var/log/journal
    DB                  /var/fluent-bit/state/flb_kernel.db
    Read_From_Tail      ${READ_FROM_TAIL}
    Systemd_Filter      _TRANSPORT=kernel

[INPUT]
    Name                systemd
    Tag                 host.messages
    Path                /var/log/journal
    DB                  /var/fluent-bit/state/flb_messages.db
    Read_From_Tail      ${READ_FROM_TAIL}

[INPUT]
    Name                systemd
    Tag                 host.secure
    Path                /var/log/journal
    DB                  /var/fluent-bit/state/flb_secure.db
    Read_From_Tail      ${READ_FROM_TAIL}
    Systemd_Filter      SYSLOG_FACILITY=4
    Systemd_Filter      SYSLOG_FACILITY=10

The journal path is mounted via a hostPath volume:

volumeMounts:
  - name: journal
    mountPath: /var/log/journal
    readOnly: true
volumes:
  - name: journal
    hostPath:
      path: /var/log/journal

The core issue is that /var/log/journal on Bottlerocket nodes is not readable by
non-root users without explicit group membership (e.g. systemd-journal), which
Bottlerocket does not expose in a documented or stable way for workload containers.
This means that runAsUser: 0 is currently the only working approach, which:

1. Conflicts with Bottlerocket's security model discouraging root workload containers.
2. Will trigger violations in environments enforcing Kubernetes PodSecurityStandards
   at restricted or baseline level.

A related community discussion exists in the Bottlerocket repo with no official resolution:
bottlerocket-os/bottlerocket#3527

The broader native logging support for Bottlerocket is tracked but unresolved:
bottlerocket-os/bottlerocket#2747

Request

One or more of the following would address this:

• Official documentation or a dedicated Bottlerocket variant of the DaemonSet manifest
  that provides guidance on the runAsUser: 0 requirement and its trade-offs.
• Investigation into whether Bottlerocket exposes a stable GID (e.g. systemd-journal)
  that could allow non-root journal access, enabling runAsNonRoot: true + supplementalGroups
  as a drop-in replacement for runAsUser: 0.
• Guidance on a minimal, hardened securityContext for Bottlerocket nodes that collects
  host/dataplane logs via the systemd input plugin while satisfying PodSecurityStandards
  at baseline level.

Workaround (Current)

Running with runAsUser: 0 but hardened with dropped capabilities:

securityContext:
  runAsUser: 0
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

This works functionally but is not ideal in a Bottlerocket security context.