Skip to content

Downgrade Lister Errors for Deprecated/Unavailable Services to Warning #1013

Description

@aws-khargita

Feature Request: Downgrade Lister Errors for Deprecated/Unavailable Services to Warning

Summary

When aws-nuke encounters a service API that is blocked, deprecated, or unavailable in newer AWS accounts, the lister failure is logged at ERROR level. This is misleading — the nuke run completes successfully (exit code 0), but operators reviewing logs see error-level messages and incorrectly believe something failed. These should be logged at WARN level to indicate a non-fatal condition that does not affect the outcome of the run.

Problem

Many AWS services have been deprecated and their APIs are blocked in accounts created after certain dates. When aws-nuke attempts to list resources for these services, the AWS API returns various errors (deprecation notices, DNS failures, connection timeouts, subscription-required responses).

In libnuke's scanner, any lister error that isn't an ErrSkipRequest or ErrUnknownEndpoint is logged at error level:

logger.WithError(err).Errorf("listing failed:\n%s", dump)

The nuke run still completes successfully — these failures don't affect the exit code or prevent other resources from being processed. But the log output is filled with ERROR entries that are entirely expected and non-actionable.

This causes real operational problems:

  1. False alerts — log monitoring that triggers on ERROR-level messages fires unnecessarily
  2. Operator confusion — people reviewing nuke run logs see errors and investigate, wasting time on a non-issue
  3. Signal-to-noise degradation — actual errors (like permission issues on services that should work) get lost in the noise of expected deprecation failures
  4. Automation uncertainty — systems parsing logs to determine run health can't distinguish "deprecated service, expected" from "real failure, needs attention"

Proposed Solution

Introduce a way for lister errors to be classified by severity so the scanner can log them at the appropriate level. Today, libnuke's scanner has two non-error paths (ErrSkipRequest and ErrUnknownEndpoint, both logged at DEBUG) and one catch-all (everything else → ERROR). There's no middle ground for errors that are expected and non-fatal but still worth surfacing.

The ask is to add a WARN-level classification for errors that indicate a service is unavailable due to deprecation, lack of subscription, or account-level restrictions — conditions where the lister cannot function but the overall nuke run is healthy. This keeps ERROR reserved for genuinely unexpected failures that warrant investigation.

  • ERROR means something is wrong and may need attention
  • WARN means something couldn't be reached but the run is unaffected
  • DEBUG means something was intentionally skipped (as today)

Note: the ErrDeprecatedResourceType error type already exists in libnuke's errors package but does not appear to be used for this purpose today.

Real-World Findings

The following error-level log entries were observed in an actual aws-nuke run on a newer AWS account. Every single one is a non-fatal condition caused by a deprecated or unavailable service — yet they all log at ERROR level, producing 31 misleading error entries in a single run.

Category 1: Explicit Deprecation Messages

Services that return a clear "this is deprecated" response:

Resource Type Error Message
MachineLearningMLModel "Thank you for your interest in Amazon Machine Learning. AmazonML is no longer available to new customers."
MachineLearningDataSource (same)
MachineLearningEvaluation (same)
MachineLearningBranchPrediction (same)
AWS::Timestream::Database "Only existing Timestream for LiveAnalytics customers can access the service."
AWS::Timestream::Table (same)
AWS::Timestream::ScheduledQuery (same)
CloudSearchDomain "New domain creation not supported on this account."
Cloud9Environment "This account does not have access to the Cloud9 service"

Category 2: DNS Resolution Failures (endpoint decommissioned)

The service endpoint no longer exists in DNS:

Resource Type Error
OpsWorksApp dial tcp: lookup opsworks.us-east-1.amazonaws.com: no such host
OpsWorksInstance (same)
OpsWorksLayer (same)
OpsWorksUserProfile (same)

Category 3: Connection Timeouts (deprecated service black-holing connections)

The DNS entry still exists but the endpoint drops connections silently. These are all deprecated services:

Resource Type Service EOL Date
InspectorAssessmentRun Inspector Classic Dec 2024 (new accounts); May 2026 (full EOL)
InspectorAssessmentTarget Inspector Classic Dec 2024 (new accounts); May 2026 (full EOL)
InspectorAssessmentTemplate Inspector Classic Dec 2024 (new accounts); May 2026 (full EOL)
ElasticTranscoderPipeline Elastic Transcoder Nov 13, 2025
ElasticTranscoderPreset Elastic Transcoder Nov 13, 2025
OpsWorksCMServer OpsWorks for Chef Automate May 5, 2024
OpsWorksCMBackup OpsWorks for Chef Automate May 5, 2024
OpsWorksCMServerState OpsWorks for Chef Automate May 5, 2024
CodeStarProject CodeStar Jul 31, 2024

These are especially problematic because each timeout takes ~30 seconds. With 9 resource types timing out, this adds ~4.5 minutes of wasted time per region scanned, purely from deprecated services that will never return resources.

Category 4: Service Not Subscribed/Activated

The service is active on AWS but requires an explicit subscription or admin configuration that this account does not have:

Resource Type Error
ShieldProtection "The subscription does not exist" (Shield Advanced not enabled)
ShieldProtectionGroup (same)
FMSNotificationChannel "No default admin could be found for account" (FMS not configured)
FMSPolicy (same)

Category 5: Decommissioned API (503 with empty response)

The service endpoint returns an HTTP 503 with no body — fully decommissioned:

Resource Type Error
LexBot status code: 503, request id: (Lex V1)
LexIntent (same)
LexSlotType (same)
LexModelBuildingServiceBotAlias (same)

Category 6: Throttling (Legitimate Transient Error)

Resource Type Error
OSVPCEndpoint "ThrottlingException: Rate exceeded"

This is the only error in the entire run that represents an active service with a real failure. Unlike the others, throttling means the service IS functional but the request was rate-limited. This is a potential source of silently missed resources if not retried, and should remain at ERROR level (or better, be retried automatically).

Summary of Proposed Log Level Changes

Error Category Count in Sample Run Current Level Proposed Level Reason
Explicit deprecation message 9 ERROR WARN Service says it's unavailable
DNS resolution failure 4 ERROR WARN Endpoint decommissioned
Connection timeout (deprecated) 9 ERROR WARN Deprecated service black-holing; also wastes ~4.5 min
Subscription not found 4 ERROR WARN Service not activated for account
503 empty response 4 ERROR WARN Service fully decommissioned
Throttling 1 ERROR ERROR (+ retry) Active service, transient failure

Total: 30 of 31 error-level entries should be warnings. Only 1 represents an actual actionable failure.

Performance Impact of Timeouts

Beyond the log noise issue, the connection timeout category has a real performance cost. Deprecated services that black-hole connections force aws-nuke to wait for the TCP timeout (typically 30s) per resource type, per region. In the observed run, this added approximately 4.5 minutes of idle wait time for a single region. In multi-region configurations, this multiplies significantly. A mechanism to detect and fast-fail these (e.g., a curated list of known-dead endpoints, or a shorter configurable timeout for initial connection) would provide both log clarity and performance improvement.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions