PVC in pending status-no other error

[suadhika]

My configuration:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
eu-data-k8s-k8s-master-nf-1 Ready master 46h v1.16.2 172.21.97.113 CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://18.9.7
eu-data-k8s-k8s-node-nf-1 Ready 46h v1.16.2 172.21.97.104 CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://18.9.7
eu-data-k8s-k8s-node-nf-2 Ready 46h v1.16.2 172.21.97.101 CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://18.9.7
eu-data-k8s-k8s-node-nf-3 Ready 46h v1.16.2 172.21.97.105 CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://18.9.7

Kubectl get pods
default glusterfs-7r6fb 1/1 Running 0 41h
default glusterfs-gj8qd 1/1 Running 0 41h
default glusterfs-pvwfz 1/1 Running 0 41h
default heketi-588777bbc6-tr47f 1/1 Running 0 41h
kube-system calico-kube-controllers-669f75dc99-226xw 1/1 Running 0 46h
kube-system calico-node-2xbv7 1/1 Running 1 46h
kube-system calico-node-5p79k 1/1 Running 1 46h
kube-system calico-node-5sxsp 1/1 Running 1 46h
kube-system calico-node-878kg 1/1 Running 1 46h
kube-system coredns-58687784f9-pztkh 1/1 Running 0 46h
kube-system coredns-58687784f9-tbbxj 1/1 Running 0 46h
kube-system dns-autoscaler-79599df498-qq8st 1/1 Running 0 46h
kube-system kube-apiserver-eu-data-k8s-k8s-master-nf-1 1/1 Running 2 46h
kube-system kube-controller-manager-eu-data-k8s-k8s-master-nf-1 1/1 Running 1 46h
kube-system kube-proxy-5q45v 1/1 Running 0 46h
kube-system kube-proxy-fqwfq 1/1 Running 0 46h
kube-system kube-proxy-gh8kv 1/1 Running 0 46h
kube-system kube-proxy-qzhhd 1/1 Running 0 46h
kube-system kube-scheduler-eu-data-k8s-k8s-master-nf-1 1/1 Running 1 46h
kube-system kubernetes-dashboard-556b9ff8f8-kms9q 1/1 Running 0 46h
kube-system nginx-proxy-eu-data-k8s-k8s-node-nf-1 1/1 Running 0 46h
kube-system nginx-proxy-eu-data-k8s-k8s-node-nf-2 1/1 Running 0 46h
kube-system nginx-proxy-eu-data-k8s-k8s-node-nf-3 1/1 Running 0 46h
kube-system nodelocaldns-59g74 1/1 Running 0 46h
kube-system nodelocaldns-6jhvg 1/1 Running 0 46h
kube-system nodelocaldns-8r2m9 1/1 Running 0 46h
kube-system nodelocaldns-xtgs2 1/1 Running 0 46h

Kubectl get SVC:

heketi ClusterIP 10.233.36.157 8080/TCP 41h
heketi-storage-endpoints ClusterIP 10.233.18.189 1/TCP 41h
kubernetes ClusterIP 10.233.0.1 443/TCP 46h

Kubectl get EP:

heketi 10.233.83.7:8080 41h
heketi-storage-endpoints 172.21.97.101:1,172.21.97.104:1,172.21.97.105:1 41h
kubernetes 172.21.97.113:6443 46h

Kubectl storageclass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mysc
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.233.83.7:8080"
restuser: "admin"
secretName: "heketi-secret"
secretNamespace: "default"

Kubectl PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc001
annotations:
volume.beta.kubernetes.io/storage-class: "mysc"
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: mysc
EndPoints:
[centos@eu-data-k8s-k8s-master-nf-1 ~]$ kubectl describe svc/heketi | grep "Endpoints:" | awk '{print $2}'
10.233.83.7:8080

Tests:
kubectl exec -i heketi-588777bbc6-tr47f -- curl http://10.233.83.7:8080/hello 2>/dev/null
Hello from Heketi[centos@eu-data-k8s-k8s-master-nf-1 ~]$

Kubectl get sc:

NAME PROVISIONER AGE
mysc (default) kubernetes.io/glusterfs 17s

Kubectl get pvc:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc001 Pending mysc 68s

Kubectl describe pvc:

Name: pvc001
Namespace: default
StorageClass: mysc
Status: Pending
Volume:
Labels:
Annotations: volume.beta.kubernetes.io/storage-class: mysc
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By:
Events:

Could you please tell me what are the other areas I could check ? I am new to Kubernetes.

[suadhika]

kubectl exec -i heketi-588777bbc6-tr47f -- heketi-cli -s http://localhost:8080 --user admin --secret MyAdminKey cluster list
Clusters:
Id:8b7a3eaf217ffad9b3745674a1dde4ce [file][block]

kubectl exec -i heketi-588777bbc6-tr47f -- heketi-cli -s http://localhost:8080 --user admin --secret MyAdminKey cluster info 8b7a3eaf217ffad9b3745674a1dde4ce
Cluster id: 8b7a3eaf217ffad9b3745674a1dde4ce
Nodes:
60024741870616bd76d5495c236b7778
92de23f267859937d96a7d879300cd73
fa2b5200daa4fe4ddcf8441aa00b1c96
Volumes:
02735c0ca3e74d4f2251b9cb0af6d59d
11d5ea2532fe6194afe6afa0d066d55d
37105d07970f4834ddb743b0fa156990
37d0d4dbf22a4d520f817c3dd320b363
3815ab8237c7350317e0ac196667eef6
fc20d028ee2fe2919e03afd863a93613

kubectl exec -i heketi-588777bbc6-tr47f -- heketi-cli -s http://localhost:8080 --user admin --secret MyAdminKey volume info fc20d028ee2fe2919e03afd863a93613
Name: vol_fc20d028ee2fe2919e03afd863a93613
Size: 1
Volume Id: fc20d028ee2fe2919e03afd863a93613
Cluster Id: 8b7a3eaf217ffad9b3745674a1dde4ce
Mount: 172.21.97.105:vol_fc20d028ee2fe2919e03afd863a93613
Mount Options: backup-volfile-servers=172.21.97.104,172.21.97.101
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 3
Snapshot Factor: 1.00

[suadhika]

Hi, can someone put in some thoughts?