kubectl cmd from kritis preinstall auth failure

[perezjasonr]

Expected Behavior

successful kubectl call from within kritis preinstall container

Actual Behavior

kubectl logs kritis-preinstall -c kritis-preinstall
time="2020-11-27T20:37:54Z" level=info msg="contents of /var/run/secrets/kubernetes.io/serviceaccount/namespace: default"
time="2020-11-27T20:37:54Z" level=info msg="running preinstall\nversion v0.2.2\ncommit: bea073f2a2f299af94363dc399b7780fde8f2afc"
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
The connection to the server 10.96.0.1:443 was refused - did you specify the right host or port?
time="2020-11-27T20:37:54Z" level=info msg="[cfssl genkey -]"
time="2020-11-27T20:37:54Z" level=info msg="{\"csr\":\"-----BEGIN CERTIFICATE REQUEST-----\\n<redacted>-----END CERTIFICATE REQUEST-----\\n\",\"key\":\"-----BEGIN EC PRIVATE KEY-----\\n<redacted>\\n-----END EC PRIVATE KEY-----\\n\"}\n"
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
    name: tls-webhook-secret-cert
    labels:
        kritis.grafeas.io/install: ""
spec:
    groups:
    - system:authenticated
    request: <redacted>
    usages:
    - digital signature
    - key encipherment
    - server auth
time="2020-11-27T20:37:54Z" level=info msg="[cfssljson -bare server]"
time="2020-11-27T20:37:54Z" level=info
time="2020-11-27T20:37:54Z" level=info msg="[kubectl apply -f -]"
time="2020-11-27T20:37:54Z" level=info
time="2020-11-27T20:37:54Z" level=error msg="error: unable to recognize \"STDIN\": Get https://10.96.0.1:443/api?timeout=32s: dial tcp 10.96.0.1:443: connect: connection refused\n"
time="2020-11-27T20:37:54Z" level=fatal msg="exit status 1"

Steps to Reproduce the Problem

Environment, commands

I used the standalone docs, im really interested in this toolchain so demo-ing for my team.

./setup_grafeas.sh
./setup_kritis.sh

Additional info

basically, it seems like the kubectl call isn't authorized. Do see the service account exists:

 kubectl get sa
NAME                               SECRETS   AGE
...
...
kritis-preinstall-serviceaccount   1         27m

as is its clusterrolebinding:

kubectl get clusterrolebinding
NAME                                                   ROLE
                  AGE
...
...
kritis-preinstall-clusterrolebinding                   ClusterRole/cluster-admin

but I cant tell by looking at the golang (im a beginner/intermediate at golang at best) where its getting its kubeconf from or if its trying to use the service account token. It is noteworthy that other things I'm running that have an SA and require kube-api access arent being rejected. Any ideas?

[perezjasonr]

Some followup material, after testing things out, now I really don't know where that kubectl command is getting its credentials, it doesn't appear to be its own service account. I am fortunate enough to have an istio proxy sidecar running, which mounts the same SA token:

    image: <our private registry>/istio-1.7/proxyv2:1.7.2
    imagePullPolicy: Always
    name: istio-proxy
    <other properties removed for brevity>
    volumeMounts:
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
    - mountPath: /var/lib/istio/data
      name: istio-data
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/istio/pod
      name: istio-podinfo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kritis-preinstall-serviceaccount-token-ttgdp
      readOnly: true

note the last mount path is the kritis preinstall token. So I exec into there and using that token I am authorized just fine:

curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes/apis/certificates.k8s.io -vvv
*   Trying 10.96.0.1...
* TCP_NODELAY set
* Connected to kubernetes (10.96.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Nov 27 18:06:37 2020 GMT
*  expire date: Nov 27 18:07:39 2021 GMT
*  subjectAltName: host "kubernetes" matched cert's "kubernetes"
*  issuer: CN=kubernetes
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x5651b9a13a70)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /apis/certificates.k8s.io HTTP/2
> Host: kubernetes
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Bearer <redacted>
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200
< cache-control: no-cache, private
< content-type: application/json
< content-length: 297
< date: Mon, 30 Nov 2020 13:55:12 GMT
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "certificates.k8s.io",
  "versions": [
    {
      "groupVersion": "certificates.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "certificates.k8s.io/v1beta1",
    "version": "v1beta1"
  }
* Connection #0 to host kubernetes left intact

Same IP is resolved and everything, I even went to the certificates api group just to list the one that the kubectl call was trying to make. So I guess my question becomes, where is this kubectl call from within pre-install getting its credentials from?