consider: item queue revenant state

[ekristen]

See the implementation https://github.com/rebuy-de/aws-nuke/pull/838/files

[bdellegrazie]

Hi,

I definitely think this is worthwhile.

Areas where it may prove useful include things like IAM Service Linked roles which can't be deleted until the consumers of the service have been.

Side-note - thank you for all of your efforts thus far!

[ekristen]

@bdellegrazie based on the implementation from 838, correct me if I'm wrong, but a DAG based ordered deletion would solve this as well?

It looks like the implementation assumes in this case that all cloud watch log groups are revenant irrespective if they actually are.

I assume the goal is to make it so that multiple runs of the tool is not required, and that ordered deletion can be done.

[bdellegrazie]

@ekristen to be fair I haven't looked at the implementation for a while.

This is what I think is happening.
EKS uses a Service Linked Role for the cluster which contains permission to create CloudWatch Log groups.

I think a race condition exists, consider:

1. EKS deleted
2. CloudWatch Log Groups deleted
3. EKS Service Linked Role deleted

Between 2 & 3 some logs may not have been processed so because the role still exists the log groups are recreated.

I suspect the correct order should be:

1. EKS deleted
2. EKS Service Linked Role deleted
3. CloudWatch Log Groups deleted

I think there needs to be an implicit dependency between the service linked role(s) and CloudWatch log groups.
This same implicit dependency might also exist with other resources.

Semi-related:
I've seen issues deleting AWS IAM Service Linked Role for EC2 Autoscaling although that may relate to CloudWatch metrics rather than CloudWatch log groups.

Side-note that I'm sure you're already aware of and the code probably already handles (I haven't checked)
When deleting Service Linked roles, if the role fails to delete it will tell you what resources are blocking its deletion. This may provide more info for the resource graph in terms of direct dependencies.