I've encountered 2 race conditions recently that I believe have been caused by terraform's management of the aws-auth configmap here
In a recent CI-initiated terraform destroy, the following error was encountered:
...
module.eks.aws_security_group.aws-node: Still destroying... [id=sg-069b01ce27e2bffd3, 30m1s elapsed]
Error: Error deleting security group: DependencyViolation: resource sg-069b01ce27e2bffd3 has a dependent object
status code: 400, request id: 11278f08-4336-4f13-b335-bf0dc38e1ed8
During debugging, I noticed that kube couldn't be accessed with the kubeconfig generated by terraform output, which led to the discovery that this configmap had already been deleted. This cluster is now in a state where terraform can't destroy it, due to dependencies still existing in that security group, which @colstrom has speculated is due to cleanup of an eni expected to be managed by the worker node failing, because the worker node could no longer manage cluster resources after deletion of this configmap.
This configmap has also led to race conditions during deployment, where terraform apply has been failing with alarming frequency, with the following error:
module.services.kubernetes_config_map.aws_auth: Still creating... [30s elapsed]
Error: Post https://6A7BFC843CD4C7578DCB503446548A17.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp 34.218.122.126:443: i/o timeout
on modules/services/aws_auth_cm.tf line 7, in resource "kubernetes_config_map" "aws_auth":
7: resource "kubernetes_config_map" "aws_auth" {
Though I believe this is due to attempting to overwrite/patch it before the kube api is up.
I've encountered 2 race conditions recently that I believe have been caused by terraform's management of the
aws-authconfigmap hereIn a recent CI-initiated
terraform destroy, the following error was encountered:During debugging, I noticed that kube couldn't be accessed with the kubeconfig generated by
terraform output, which led to the discovery that this configmap had already been deleted. This cluster is now in a state where terraform can't destroy it, due to dependencies still existing in that security group, which @colstrom has speculated is due to cleanup of aneniexpected to be managed by the worker node failing, because the worker node could no longer manage cluster resources after deletion of this configmap.This configmap has also led to race conditions during deployment, where
terraform applyhas been failing with alarming frequency, with the following error:Though I believe this is due to attempting to overwrite/patch it before the kube api is up.