Skip to content

Do not warn on target allocator config load for non-TA Prometheus configs#2157

Open
musa-asad wants to merge 2 commits into
aws:mainfrom
musa-asad:fix/ta-warn-when-disabled
Open

Do not warn on target allocator config load for non-TA Prometheus configs#2157
musa-asad wants to merge 2 commits into
aws:mainfrom
musa-asad:fix/ta-warn-when-disabled

Conversation

@musa-asad

@musa-asad musa-asad commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

A plain Prometheus deployment of the CloudWatch Agent (no Target Allocator) logs a misleading WARN ... Could not load config for target allocator from file at startup. No functional impact, but confusing because the user never requested a Target Allocator.

Root cause: createTargetAllocatorManager unconditionally tries to parse the Prometheus config as a prometheusreceiver.Config. A plain config (global:/scrape_configs: at top level) fails strict unmarshal and fires a spurious WARN on every non-TA deployment.

This change gates the warning on whether the config contains a target_allocator: section: warn only when TA config was expected, otherwise log at Debug.

Changes

plugins/inputs/prometheus/target_allocator.go:

  • New helper hasTargetAllocatorSection(filename) — checks if config has a target_allocator: key
  • Gated the warning: Warn when target_allocator: section present, Debug otherwise

plugins/inputs/prometheus/target_allocator_test.go (new):

  • TestCreateTargetAllocatorManager_NoTASection_NoWarn — no Warn for non-TA config
  • TestCreateTargetAllocatorManager_MalformedTASection_Warns — genuine TA errors still warn
  • TestHasTargetAllocatorSection — helper unit test

Test Output

Before (stock agent image):

level=WARN source=target_allocator.go:97 msg="Could not load config for target allocator from file" filename=/etc/prometheusconfig/prometheus.yaml err="decoding failed due to the following error(s):

'prometheusreceiver.Config' has invalid keys: global, scrape_configs"
level=INFO source=start.go:138 msg="Target Allocator status" enabled=false

WARN grep count            = 1

After (this branch):

level=INFO source=start.go:138 msg="Target Allocator status" enabled=false

WARN grep count            = 0
enabled=false line count   = 1
pod phase                  = Running   restarts=0

Unit tests:

$ CGO_ENABLED=0 go test ./plugins/inputs/prometheus/ -run 'TestCreateTargetAllocatorManager|TestHasTargetAllocatorSection' -count=1 -v
--- PASS: TestCreateTargetAllocatorManager_NoTASection_NoWarn (0.00s)
--- PASS: TestCreateTargetAllocatorManager_MalformedTASection_Warns (0.00s)
--- PASS: TestHasTargetAllocatorSection (0.00s)
PASS
ok  github.com/aws/amazon-cloudwatch-agent/plugins/inputs/prometheus  0.034s

Verified on both Helm and EKS Add-On paths

The same agent binary runs on both paths. Verified the spurious WARN is absent on both:

# Helm path:
$ kubectl logs -n <test-ns-helm> cloudwatch-agent-0 | grep -c 'Could not load config for target allocator'
0

# EKS Add-On path:
$ kubectl logs -n <test-ns-addon> cloudwatch-agent-0 | grep -c 'Could not load config for target allocator'
0

# Both still log the enabled=false status at INFO:
$ kubectl logs -n <test-ns-helm> cloudwatch-agent-0 | grep 'Target Allocator status'
ts=... level=info msg="Target Allocator status" enabled=false

$ kubectl logs -n <test-ns-addon> cloudwatch-agent-0 | grep 'Target Allocator status'
ts=... level=info msg="Target Allocator status" enabled=false

Related

@musa-asad musa-asad force-pushed the fix/ta-warn-when-disabled branch from cad69ec to e4be65d Compare June 16, 2026 18:37
@musa-asad musa-asad self-assigned this Jun 16, 2026
@musa-asad musa-asad requested review from okankoAMZ and sky333999 June 16, 2026 18:38
@musa-asad musa-asad added the ready for testing Indicates this PR is ready for integration tests to run label Jun 16, 2026
@musa-asad musa-asad marked this pull request as ready for review June 17, 2026 14:53
@musa-asad musa-asad requested a review from a team as a code owner June 17, 2026 14:53
@musa-asad musa-asad force-pushed the fix/ta-warn-when-disabled branch 3 times, most recently from 4cc7024 to c944da0 Compare June 17, 2026 19:48
Comment thread plugins/inputs/prometheus/target_allocator.go
@musa-asad musa-asad force-pushed the fix/ta-warn-when-disabled branch from c944da0 to 65c52cc Compare June 17, 2026 23:59
@musa-asad musa-asad requested a review from okankoAMZ June 18, 2026 00:02
…figs

A plain Prometheus config (top-level global:/scrape_configs:, no
target_allocator: section) fails to parse as a prometheusreceiver.Config,
which previously surfaced a misleading WARN at startup even though the
Target Allocator was never requested. Gate the warning on the presence of a
target_allocator: section: warn only when TA config was expected, otherwise
log at DEBUG. Add regression tests covering the no-TA (no warn), malformed-TA
(still warns) and section-detection cases.
@musa-asad musa-asad force-pushed the fix/ta-warn-when-disabled branch from 65c52cc to f601100 Compare June 18, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants