Not deploying to EKS

[mckunkel]

Describe the bug

Using EKS templates provided in the fix-eks-issue branch.
Using the main branch to deploy cf-for-k8.

kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Error
12:10:19PM: fail: reconcile deployment/eirini-api (apps/v1) namespace: cf-system
12:10:19PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-api-85c488b6f5" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/eirini-api (apps/v1) namespace: cf-system:
Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-api-85c488b6f5" has timed out progressing.))

Tried again
12:09:21PM: fail: reconcile deployment/log-cache-backend (apps/v1) namespace: cf-system
12:09:21PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "log-cache-backend-c5675b68b" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/log-cache-backend (apps/v1) namespace: cf-system:
Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "log-cache-backend-c5675b68b" has timed out progressing.))

Third attempt:
12:11:06PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-instance-index-env-injector-6b7f479874" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/eirini-instance-index-env-injector (apps/v1) namespace: cf-system:
Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-instance-index-env-injector-6b7f479874" has timed out progressing.))

Everytime a different failure.

To Reproduce*

Steps to reproduce the behavior:

1. go to deploy/eks/terraform and create eks cluster. Tested with versions 1.19, 1.20, and 1.21
2. Create template file as prescribed in walkthrough, mine ended as
   `app_registry:
   hostname: https://index.docker.io/v1/
   repository_prefix: "mckunkel"
   username: "mckunkel"
   password: "XXX"

add_metrics_server_components: true
enable_automount_service_account_token: true
load_balancer:
enable: false
metrics_server_prefer_internal_kubelet_address: true
remove_resource_requirements: true
use_first_party_jwt_tokens: true3. Use kapp to deploykapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)`
4. See error

It's helpful to include snippets of the error response or logs output

Expected behavior

A clear and concise description of what you expected to happen.
cf-for-k8 to deploy to eks sucessfully

Additional context

cf-for-k8s SHA

Please paste cf-for-k8s SHA hyperlink

Deploy instructions

Please include the kapp deploy... command, including all config-optional files
kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Cluster information

Please provide IaaS name(s) here (GKE, AKS, EKS, minikube, kind, etc.)
EKS

CLI versions

paste output of the following commands

1. ytt --version: 0.40.0
2. kapp --version:0 .46.0
3. kubectl version: 1.21, 1.20, 1.19
4. cf version: 7.4.0+e55633fed.2021-11-15

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/181856385

The labels on this github issue will be updated when the story is started.

[mckunkel]

Further into the investigation it seems that all failures point to isitio.
This is output that all
kubectl describe ReplicaSet/<from replica list> -n <replica namespace>

`
Type Reason Age From Message

---

Warning FailedCreate 15m (x8 over 15m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": service "istiod" not found
Warning FailedCreate 15m (x4 over 15m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": no endpoints available for service "istiod"
Warning FailedCreate 8m41s replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.0.3.190:15017: i/o timeout
Warning FailedCreate 3m4s (x5 over 14m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded
`

I do not see any resolutions in the issues page on this. Can I be guided in the correct path to fix this please?