What happened:
Trying to deploy Portworx on vanilla Kubernetes cluster using Helm charts but deployment is failing...
Internet is configured via a local squid proxy server ...
helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
portworx default 1 2023-03-16 11:25:50.056516791 +0000 UTC deployed portworx-2.11.4
kubectl get pods -A |grep -v Running
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system portworx-lz9qm 0/1 CrashLoopBackOff 17 (4m24s ago) 78m
kube-system portworx-phmhj 0/1 CrashLoopBackOff 18 (22s ago) 78m
kube-system portworx-ths29 0/1 CrashLoopBackOff 17 (4m2s ago) 78m
kubectl -n kube-system describe pod/portworx-lz9qm | grep -A100 Events
Events:
Type Reason Age From Message
Normal Pulled 58m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 4m37.523590901s
Normal PortworxMonitorImagePullInPrgress 58m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 58m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 57m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal Pulled 57m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 1m1.985070185s
Warning PortworxMonitorImagePullFailed 57m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:46154->192.168.239.36:53: i/o timeout
Normal Created 56m (x3 over 58m) kubelet Created container portworx
Normal Pulled 56m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 3.554413785s
Normal PortworxMonitorImagePullInPrgress 56m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal Started 56m (x3 over 58m) kubelet Started container portworx
Warning PortworxMonitorImagePullFailed 56m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:56111->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 56m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal PortworxMonitorImagePullInPrgress 54m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 54m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:54644->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 53m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 52m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:44818->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 50m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 49m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:41941->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 44m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 44m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:39193->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 39m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 38m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:37853->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 33m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning Unhealthy 33m (x41 over 58m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Normal Pulling 28m (x11 over 63m) kubelet Pulling image "portworx/oci-monitor:2.11.4"
Normal PortworxMonitorImagePullInPrgress 28m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 28m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 22m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal PortworxMonitorImagePullInPrgress 17m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 17m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:52790->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 11m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 11m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 6m15s portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 5m59s portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Warning BackOff 3m24s (x234 over 57m) kubelet Back-off restarting failed container
Normal PortworxMonitorImagePullInPrgress 53s portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 37s portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:54572->192.168.239.36:53: i/o timeout
Environment:
- Container Orchestrator and version: Kubernetes v1.24.3
- Cloud provider or hardware configuration: Ubuntu 22.04 VM
- OS (e.g. from /etc/os-release): Ubuntu 22.04
- Kernel (e.g.
uname -a): 5.4.0-137-generic
- Install tools: NA
- Others: NA
What happened:
Trying to deploy Portworx on vanilla Kubernetes cluster using Helm charts but deployment is failing...
Internet is configured via a local squid proxy server ...
helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
portworx default 1 2023-03-16 11:25:50.056516791 +0000 UTC deployed portworx-2.11.4
kubectl get pods -A |grep -v Running
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system portworx-lz9qm 0/1 CrashLoopBackOff 17 (4m24s ago) 78m
kube-system portworx-phmhj 0/1 CrashLoopBackOff 18 (22s ago) 78m
kube-system portworx-ths29 0/1 CrashLoopBackOff 17 (4m2s ago) 78m
kubectl -n kube-system describe pod/portworx-lz9qm | grep -A100 Events
Events:
Type Reason Age From Message
Normal Pulled 58m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 4m37.523590901s
Normal PortworxMonitorImagePullInPrgress 58m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 58m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 57m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal Pulled 57m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 1m1.985070185s
Warning PortworxMonitorImagePullFailed 57m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:46154->192.168.239.36:53: i/o timeout
Normal Created 56m (x3 over 58m) kubelet Created container portworx
Normal Pulled 56m kubelet Successfully pulled image "portworx/oci-monitor:2.11.4" in 3.554413785s
Normal PortworxMonitorImagePullInPrgress 56m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal Started 56m (x3 over 58m) kubelet Started container portworx
Warning PortworxMonitorImagePullFailed 56m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:56111->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 56m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal PortworxMonitorImagePullInPrgress 54m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 54m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:54644->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 53m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 52m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:44818->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 50m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 49m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:41941->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 44m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 44m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:39193->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 39m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 38m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:37853->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 33m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning Unhealthy 33m (x41 over 58m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Normal Pulling 28m (x11 over 63m) kubelet Pulling image "portworx/oci-monitor:2.11.4"
Normal PortworxMonitorImagePullInPrgress 28m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 28m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 22m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Normal PortworxMonitorImagePullInPrgress 17m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 17m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:52790->192.168.239.36:53: i/o timeout
Normal PortworxMonitorImagePullInPrgress 11m portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 11m portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Normal PortworxMonitorImagePullInPrgress 6m15s portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 5m59s portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: server misbehaving
Warning BackOff 3m24s (x234 over 57m) kubelet Back-off restarting failed container
Normal PortworxMonitorImagePullInPrgress 53s portworx Portworx image docker.io/portworx/px-enterprise:2.11.4 pull and extraction in progress
Warning PortworxMonitorImagePullFailed 37s portworx Could not pull docker.io/portworx/px-enterprise:2.11.4: Could not pull docker.io/portworx/px-enterprise:2.11.4: could not pull docker.io/portworx/px-enterprise:2.11.4: error fetching image: failed to resolve reference "docker.io/portworx/px-enterprise:2.11.4": failed to do request: Head "https://registry-1.docker.io/v2/portworx/px-enterprise/manifests/2.11.4": dial tcp: lookup registry-1.docker.io on 192.168.239.36:53: read udp 10.10.40.202:54572->192.168.239.36:53: i/o timeout
Environment:
uname -a): 5.4.0-137-generic