systemd: build: fix service startup failure when using IMDSv2 by charltonstanley · Pull Request #10221 · fluent/fluent-bit

charltonstanley · 2025-04-16T18:26:09Z

PR #9845 added a service dependency sethostname.service for Amazon Linux 2, which uses Amazon's Instance Metadata Service (IMDS) to query and set the hostname. This works for instances using the legacy IMDSv1, however it fails when admins have enforced the usage of IMDSv2. This patch is a modification of @cpandya-we's comment/work on #10186. Closes #10186

Note: I used modified versions of the code samples @cosmo0920 submitted in #9845. Specifically, note the differences in the packer file, where IMDSv2 support was enabled.

Testing

Example configuration file for the change

[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    name dummy
    tag  dummy.local

    # Read interval (sec) Default: 1
    interval_sec 10

[FILTER]
    Name  record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record filter_type record_modifiler

[OUTPUT]
    name  stdout
    match *

[] Debug log output from testing the change

Log output confirming both fluent-bit.service and sethostname.service service start successfully.

[ec2-user@ip-172-31-10-223 ~]$ systemctl status fluent-bit -l
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2025-04-16 18:14:35 UTC; 52s ago
     Docs: https://docs.fluentbit.io/manual/
 Main PID: 3229 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3229 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [ctraces ] version=0.6.2
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] initializing
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [sp] stream processor started
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [output:stdout:stdout.0] worker #0 started
Apr 16 18:14:45 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827284.897177091, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:14:55 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827294.896876191, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:05 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827304.897044922, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:15 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827314.896878894, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:25 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827324.896980969, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
[ec2-user@ip-172-31-10-223 ~]$ systemctl status sethostname -l
● sethostname.service - Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: active (exited) since Wed 2025-04-16 18:14:35 UTC; 1min 21s ago
  Process: 3218 ExecStartPost=/bin/bash -c rm -f /run/imds_token /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3212 ExecStart=/bin/bash -c /usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)" (code=exited, status=0/SUCCESS)
  Process: 3204 ExecStartPre=/bin/bash -c curl -s "http://169.254.169.254/latest/meta-data/hostname"    -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3174 ExecStartPre=/bin/bash -c curl -sX PUT "http://169.254.169.254/latest/api/token"    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token (code=exited, status=0/SUCCESS)
 Main PID: 3212 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sethostname.service

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support...
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Started Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support.

Here is the packer template for creating AMI. Note the addition of the the imds_support property, as well as the additional metadata_options.

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.8"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "amazonlinux2" {
  ami_name      = "fbit-amazon-linux2-amd64"
  instance_type = "t3.micro"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      name                = "amzn2-ami-kernel-5.10-hvm-2.*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
      architecture        = "x86_64"
    }
    most_recent = true
    owners      = ["amazon"]
  }
  ssh_username = "ec2-user"

### Added IMDSv2 support ###
  imds_support = "v2.0"
  metadata_options {
    http_endpoint = "enabled"
    http_tokens = "required"
    http_put_response_hop_limit = 2
  }
}

build {
  name = "az2-fbit-packer"
  sources = [
    "source.amazon-ebs.amazonlinux2"
  ]

  provisioner "file" {
    source      = "/path/to/github/fluent-bit/packaging/packages/amazonlinux/2/2025-04-16-10_05_01/fluent-bit-4.0.1-1.x86_64.rpm"
    destination = "/home/ec2-user/fluent-bit.rpm"
  }

  provisioner "file" {
    source      = "path/to/fluent-bit.conf"
    destination = "/home/ec2-user/fluent-bit.conf"
  }

  provisioner "shell" {
    inline = [
      "sudo yum install -y /home/ec2-user/fluent-bit.rpm",
      "sudo mv /home/ec2-user/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf",
      "sudo systemctl enable fluent-bit"
    ]
  }
}

[N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[N/A] Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

charltonstanley · 2025-04-21T16:19:13Z

Hi there, any chance this could be merged in soon? We have paused updates on our servers because we don't want to have to write an automation script to disable the fluent-bit repository on each instance. (The related bug affects ~125 instance within my org.) Thanks!

cosmo0920

I'd confirmed that this patch works under IMDSv2.
Could you rebase off the current master?
This is because at the moment of sending this PR, our CI was broken.
So, we wanted to do CI at least once.

cosmo0920 · 2025-04-30T06:24:16Z

I'd confirmed that the same logs on my created EC2 instance:

[ec2-user@ip-172-31-22-3 ~]$ systemctl status fluent-bit -l
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: disabled)
   Active: active (running) since 水 2025-04-30 06:17:46 UTC; 1min 17s ago
     Docs: https://docs.fluentbit.io/manual/
 Main PID: 3215 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3215 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [sp] stream processor started
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [output:stdout:stdout.0] worker #0 started
 4月 30 06:17:58 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993877.020040385, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:08 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993887.020059688, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:18 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993897.020056403, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:28 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993907.020051693, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:38 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993917.020051760, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:48 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993927.020056001, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:58 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993937.020064700, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
[ec2-user@ip-172-31-22-3 ~]$ systemctl status sethostname -l
● sethostname.service - Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: active (exited) since 水 2025-04-30 06:17:46 UTC; 5min ago
  Process: 3201 ExecStartPost=/bin/sh -c rm -f /run/imds_token /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3192 ExecStart=/bin/sh -c /usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)" (code=exited, status=0/SUCCESS)
  Process: 3181 ExecStartPre=/bin/sh -c curl -s "http://169.254.169.254/latest/meta-data/hostname"    -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3153 ExecStartPre=/bin/sh -c curl -sX PUT "http://169.254.169.254/latest/api/token"    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token (code=exited, status=0/SUCCESS)
 Main PID: 3192 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sethostname.service

 4月 30 06:17:46 ip-172-31-22-3.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support...
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal systemd[1]: Started Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support.

@cpandya-we

PR fluent#9845 added a service dependency "sethostname.service" for Amazon Linux 2, which uses Amazon's Instance Metadata Service (IMDS) to query and set the hostname. This works for instances using the legacy IMDSv1, however it fails when admins have enforced the usage of IMDSv2. This patch is a modification of @cpandya-we comment/work on fluent#10186. Fixes fluent#10186 Signed-off-by: Charlton Stanley <charlton@charltonstanley.com>

charltonstanley · 2025-04-30T14:34:04Z

I'd confirmed that this patch works under IMDSv2.
Could you rebase off the current master?
This is because at the moment of sending this PR, our CI was broken.
So, we wanted to do CI at least once.

@cosmo0920 Thanks for verifying my patch. Rebase has been done!

charltonstanley requested review from edsiper, fujimotos, koleini and leonardo-albertovich as code owners April 16, 2025 18:26

github-actions Bot added the docs-required label Apr 16, 2025

charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch 2 times, most recently from cc450b4 to 08fdd4a Compare April 16, 2025 19:00

cosmo0920 reviewed Apr 30, 2025

View reviewed changes

charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch from d11d346 to 08fdd4a Compare April 30, 2025 14:17

charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch from 08fdd4a to 5d83094 Compare April 30, 2025 14:24

cosmo0920 added this to the Fluent Bit v4.0.2 milestone May 1, 2025

edsiper merged commit 5c68495 into fluent:master May 8, 2025

BrewTestBot mentioned this pull request May 14, 2025

fluent-bit 4.0.2 Homebrew/homebrew-core#223434

Merged

Navigation Menu

Uh oh!

systemd: build: fix service startup failure when using IMDSv2 - #10221

systemd: build: fix service startup failure when using IMDSv2#10221
edsiper merged 1 commit into
fluent:masterfrom
PedigreeTechnologies:charltonstanley-systemd-fix-sethostname-on-amazonlinux2

charltonstanley commented Apr 16, 2025 •

edited

Loading

Uh oh!

charltonstanley commented Apr 21, 2025 •

edited

Loading

Uh oh!

cosmo0920 left a comment

Uh oh!

cosmo0920 commented Apr 30, 2025

Uh oh!

charltonstanley commented Apr 30, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

charltonstanley commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charltonstanley commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cosmo0920 left a comment

Choose a reason for hiding this comment

Uh oh!

cosmo0920 commented Apr 30, 2025

Uh oh!

charltonstanley commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charltonstanley commented Apr 16, 2025 •

edited

Loading

charltonstanley commented Apr 21, 2025 •

edited

Loading

charltonstanley commented Apr 30, 2025 •

edited

Loading