PoC: EKS Auto Mode enhanced logging — CloudWatch Vended Logs for managed components

[nguforw-moj]

Parent

Part of #8415 (PoC: Logging and Log Management Architecture Validation)

Objective

Configure and validate EKS Auto Mode's enhanced logging capability, which delivers logs from Auto Mode's managed Kubernetes components via CloudWatch Vended Logs. This gives platform engineers visibility into the behaviour of infrastructure components that are otherwise opaque in Auto Mode.

Background

Announced Feb 2026, EKS Auto Mode can now emit logs from its managed capabilities as CloudWatch Vended Logs delivery sources:

• AUTO_MODE_COMPUTE_LOGS — Karpenter (compute autoscaling decisions, node provisioning)
• AUTO_MODE_BLOCK_STORAGE_LOGS — EBS CSI (volume attach/detach, provisioning failures)
• AUTO_MODE_LOAD_BALANCING_LOGS — AWS Load Balancer Controller (target group registration, health checks)
• AUTO_MODE_IPAM_LOGS — VPC CNI IP Address Management (IP allocation, ENI attachment)

These are configured via CloudWatch Logs APIs (PutDeliverySource, PutDeliveryDestination, CreateDelivery) — not via in-cluster agents like Fluent Bit. Logs can be delivered to CloudWatch Logs, S3, or Amazon Data Firehose.

Vended Logs pricing applies (reduced cost vs standard CloudWatch Logs ingestion).

Tasks

• Enable all four Auto Mode log delivery sources on PoC cluster using CloudWatch APIs
• Configure delivery destination: CloudWatch Logs (per-component log groups, 30-day retention)
• Configure delivery destination: S3 (same bucket as Fluent Bit output, separate prefix /auto-mode/)
• Validate logs are flowing for each component under normal cluster operations
• Trigger log-generating events and confirm visibility:
  • Compute: scale-up event (deploy a workload that requires new nodes)
  • Block storage: PVC creation and attachment
  • Load balancing: create a Service type LoadBalancer or Gateway HTTPRoute
  • IPAM: pod scheduling that requires IP allocation
• Assess log content: are they useful for troubleshooting? What fields are available?
• Test Firehose delivery (CloudWatch Vended Logs → Firehose → S3/Cortex XSIAM) for SOC integration
• Document IAM permissions required for log delivery configuration
• Document Terraform/IaC approach for enabling these log sources (reproducible across clusters)
• Measure cost: Vended Logs ingestion volume under normal PoC cluster operation
• Evaluate overlap with existing EKS control plane logging (api, audit, authenticator, controllerManager, scheduler)

Evaluation Criteria

Criterion	What to measure
Operational value	Do the logs help diagnose real issues (e.g. why a node didn't scale, why a volume attach failed)?
Setup complexity	Number of API calls / Terraform resources to enable across all 4 sources
Cost	GB/day ingested under normal operation; projected cost at 22 clusters
SOC compatibility	Can these logs be forwarded to Cortex XSIAM via the same Firehose pattern as control plane logs?
Overlap	Do these replace or supplement any logs currently collected by Fluent Bit or ADOT?

Output

• All four Auto Mode log sources enabled and delivering on PoC cluster
• Assessment of operational value (with real examples of useful log entries)
• Terraform module or configuration snippet for repeatable enablement
• Cost projection at full scale (22 clusters)
• Recommendation: which log sources to enable by default vs opt-in
• Input to comparison report (PoC: Logging and Log Management Architecture Validation #8415) and SOC integration (PoC: SOC log forwarding — Cortex XSIAM in multi-account #8421)

References

• AWS Announcement: EKS Auto Mode Enhanced Logging (Feb 2026)
• EKS Documentation: Access AWS-Managed Component Logs
• CloudWatch Vended Logs Permissions
• CloudWatch Vended Logs Pricing
• Related: PoC: SOC log forwarding — Cortex XSIAM in multi-account #8421 (SOC log forwarding — includes Vended Logs → Firehose → Cortex validation)
• Related: PoC: Fluent Bit multi-output pipeline (S3 + CloudWatch + OpenSearch) #8419 (Fluent Bit pipeline — different mechanism, complementary coverage)