About

To cache images, Kubernetes Image Puller creates a Daemonset on the desired cluster, which in turn creates a pod on each node in the cluster consisting of a list of containers with command sleep 720h. This ensures that all nodes in the cluster have those images cached. The sleep binary being used is golang-based (please see Scratch Images). We also periodically check the health of the daemonset and re-create it if necessary.

The application can be deployed via Helm or by processing and applying OpenShift Templates. Also, there is a community supported operator available on the OperatorHub.

Configuration

Configuration is done via env vars pulled from ./deploy/helm/templates/configmap.yaml, or ./deploy/openshift/configmap.yaml, depending on the deployment method. The config values to be set are:

Env Var	Usage	Default
CACHING_INTERVAL_HOURS	Interval, in hours, between checking health of daemonsets	"1"
CACHING_MEMORY_REQUEST	The memory request for each cached image when the puller is running	10Mi
CACHING_MEMORY_LIMIT	The memory limit for each cached image when the puller is running	20Mi
CACHING_CPU_REQUEST	The CPU request for each cached image when the puller is running	.05 or 50 millicores
CACHING_CPU_LIMIT	The CPU limit for each cached image when the puller is running	.2 or 200 millicores
DAEMONSET_NAME	Name of daemonset to be created	kubernetes-image-puller
DAEMONSET_ANNOTATIONS	Annotations applied to the daemonset, provided in this format '{"key":"value"}'	'{}'
NAMESPACE	Namespace where daemonset is to be created	kubernetes-image-puller
IMAGES	List of images to be cached, in this format <name>=<image>;...	Contains a default list of images, but should be configured when deploying
NODE_SELECTOR	Node selector applied to pods created by the daemonset, provided in this format '{"key":"value"}'	'{}'
IMAGE_PULL_SECRETS	List of image pull secrets, in this format pullsecret1;... to add to pods created by the DaemonSet. Those secrets need to be in the image puller's namespace and a cluster administrator must create them.	""
AFFINITY	Affinity applied to pods created by the daemonset, in this format '{"nodeAffinity":{ ... }}'	'{}'
KIP_IMAGE	The image puller image to copy the sleep binary from	quay.io/eclipse/kubernetes-image-puller:next
TOLERATIONS	Tolerations applied to pods created by the daemonset, (escaped JSON) '[{\"key\":\"foo\",\"operator\":\"Equal\",\"value\":\"bar\"}]'	'[]'

Configuration - Helm

The following values can be set:

Value	Usage	Default
deploymentName	The value of DAEMONSET_NAME to be set in the ConfigMap, as well as the name of the deployment	kubernetes-image-puller
deploymentAnnotations	The value of annotations to be set for the Deployment	"{}"
image.repository	The repository to pull the image from	quay.io/eclipse/kubernetes-image-puller
image.tag	The image tag to pull	next
serviceAccount.name	The name of the ServiceAccount to create	k8s-image-puller
resources	Resource requests and limits for the image puller Deployment	requests: {cpu: 50m, memory: 64Mi}, limits: {cpu: 200m, memory: 256Mi}
tolerations	The value of tolerations to be set for the Deployment	"[]"
nodeSelector	The value of nodeSelector to be set in the Deployment	"{}"
updateStrategy.type	The updateStrategy type to use when restarting the Deployment	Recreate
priorityClassName	The updateStrategy type to use when restarting the Deployment	""
configMap.name	The name of the ConfigMap to create	k8s-image-puller
configMap.images	The value of IMAGES to be set in the ConfigMap	// TODO create a reasonable set of default containers
configMap.cachingIntervalHours	The value of CACHING_INTERVAL_HOURS to be set in the ConfigMap	"1"
configMap.cachingMemoryRequest	The value of CACHING_MEMORY_REQUEST to be set in the ConfigMap	"10Mi"
configMap.cachingMemoryLimit	The value of CACHING_MEMORY_LIMIT to be set in the ConfigMap	"20Mi"
configMap.cachingCpuRequest	The value of CACHING_CPU_REQUEST to be set in the ConfigMap	.05
configMap.cachingCpuLimit	The value of CACHING_CPU_LIMIT to be set in the ConfigMap	.2
configMap.daemonsetAnnotations	The value of DAEMONSET_ANNOTATIONS to be set in the ConfigMap	"{}"
configMap.nodeSelector	The value of NODE_SELECTOR to be set in the ConfigMap	"{}"
configMap.imagePullSecrets	The value of IMAGE_PULL_SECRETS	""
configMap.affinity	The value of AFFINITY to be set in the ConfigMap	"{}"
configMap.tolerations	The value of TOLERATIONS to be set in the ConfigMap (escaped JSON) '[{\"key\":\"foo\",\"operator\":\"Equal\",\"value\":\"bar\"}]'	"[]"

Configuration - OpenShift

The following values can be set:

Parameter	Usage	Default
SERVICEACCOUNT_NAME	Name of service account used by main pod	k8s-image-puller
IMAGE	Name of image used for main pod	quay.io/eclipse/kubernetes-image-puller
IMAGE_TAG	Tag of image used for main pod	next
DAEMONSET_NAME	The value of DAEMONSET_NAME to be set in the ConfigMap	"kubernetes-image-puller"
DEPLOYMENT_NAME	The name of the image puller deployment	"kubernetes-image-puller"
CACHING_INTERVAL_HOURS	The value of CACHING_INTERVAL_HOURS to be set in the ConfigMap	"1"
CACHING_MEMORY_REQUEST	The value of CACHING_MEMORY_REQUEST to be set in the ConfigMap	"10Mi"
CACHING_MEMORY_LIMIT	The value of CACHING_MEMORY_LIMIT to be set in the ConfigMap	"20Mi"
CACHING_CPU_REQUEST	The value of CACHING_CPU_REQUEST to be set in the ConfigMap	.05
CACHING_CPU_LIMIT	The value of CACHING_CPU_LIMIT to be set in the ConfigMap	.2
NAMESPACE	The value of NAMESPACE to be set in the ConfigMap	k8s-image-puller
NODE_SELECTOR	The value of NODE_SELECTOR to be set in the ConfigMap	"{}"
IMAGE_PULL_SECRETS	The value of IMAGE_PULL_SECRETS	""
AFFINITY	The value of AFFINITY to be set in the ConfigMap	"{}"
TOLERATIONS	The value of TOLERATIONS to be set in the ConfigMap	"[]"
DEPLOYMENT_CPU_REQUEST	CPU request for the image puller deployment	"50m"
DEPLOYMENT_MEMORY_REQUEST	Memory request for the image puller deployment	"64Mi"
DEPLOYMENT_CPU_LIMIT	CPU limit for the image puller deployment	"200m"
DEPLOYMENT_MEMORY_LIMIT	Memory limit for the image puller deployment	"256Mi"

Installation - Helm

The Helm chart is published as an OCI artifact to quay.io on each release.

Install from the OCI registry (recommended):

kubectl create namespace k8s-image-puller
helm install kubernetes-image-puller -n k8s-image-puller oci://quay.io/eclipse/kubernetes-image-puller --version <version>

Or install from a local checkout:

kubectl create namespace k8s-image-puller
helm install kubernetes-image-puller -n k8s-image-puller deploy/helm

To set values, use --set property.name=value or provide a custom values file with -f values.yaml.

Installation - OpenShift

Openshift special consideration - Project Quotas

OpenShift has a notion of project quotas to limit the aggregate resource consumption per project/namespace. The namespace that the image puller is deployed in must have enough memory and CPU to run each container for each node in the cluster:

(memory/CPU limit) * (number of images) * (number of nodes in cluster)

For example, running the image puller that caches 5 images on 20 nodes, with a container memory limit of 5Mi, your namespace would need a quota of 500Mi.

Installing the image puller

oc new-project k8s-image-puller

oc process -f deploy/openshift/serviceaccount.yaml | oc apply -f -

oc process -f deploy/openshift/configmap.yaml | oc apply -f -

oc process -f deploy/openshift/app.yaml | oc apply -f -

To change parameters, add -p PARAM=value to the oc process command, before piping to oc apply.

Security

Pod Security

DaemonSet pods created by the image puller run with a restricted security profile:

• Non-root execution: The copy-sleep initContainer (which runs KIP_IMAGE) executes as UID/GID 65532 with runAsNonRoot: true. Cached-image containers do not inherit this UID and run as whatever user their image defines. If you use a custom KIP_IMAGE, ensure it supports running as UID 65532.
• Seccomp profile: RuntimeDefault seccomp profile is applied to all pods. This requires Kubernetes 1.19+.
• Read-only filesystem: All containers run with a read-only root filesystem.
• No privilege escalation: allowPrivilegeEscalation is set to false and all capabilities are dropped.
• Bounded ephemeral storage: The shared kip volume is an emptyDir with a 50Mi size limit.

RBAC

The image puller service account uses least-privilege RBAC:

• DaemonSets: create, delete, list, watch, get
• Deployments: get only (used for owner reference lookup)

Building

Makefile

# Build Go binary:
make build
# Make docker image:
make docker
# The above:
make
# Clean:
make clean

The provided Makefile has two parameters:

• DOCKERIMAGE_NAME: name for docker image
• DOCKERIMAGE_TAG: tag for docker image

Manual

Build:

CGO_ENABLED=1
BINARY_NAME=kubernetes-image-puller

GOOS=linux go build -v -o ./bin/${BINARY_NAME} ./cmd/main.go
GOOS=linux go build -a -ldflags '-w -s' -a -installsuffix cgo -o ./bin/sleep ./sleep/sleep.go

Make docker image:

DOCKERIMAGE_NAME=kubernetes-image-puller
DOCKERIMAGE_TAG=next

docker build -t ${DOCKERIMAGE_NAME}:${DOCKERIMAGE_TAG} -f ./build/dockerfiles/Dockerfile .

Testing

Once built and published to a registry, you can test FIPS compliance using https://github.com/openshift/check-payload#scan-a-container-or-operator-image

To run the unit tests:

make test

End to end tests require kind. Note that kind should not be installed with go get from this repository's directory.

cd $HOME && GO111MODULE="on" go get sigs.k8s.io/kind@v0.7.0 && cd ~-

./hack/run-e2e.sh

Will start a kind cluster and run the end-to-end tests in ./e2e. To remove the cluster after running the tests, pass the --rm argument to the script, or run kind delete cluster --name k8s-image-puller-e2e.

Scratch Images

The image puller also supports pre-pulling the scratch images. Previously the image puller was not able to pull scratch images, as they do not contain a sleep command.

However, the daemonset created by the Kubernetes Image Puller now:

1. creates an initContainer that copies a golang-based sleep binary to a common kip volume.
2. creates containers volumeMounts set to the kip volume, and with command set to /kip/sleep 720h

As a result, every container (including scratch image containers) uses the provided golang-based sleep binary.

Release

The release process consists of the following steps:

1. Run the make-release.sh script from main:

   ./make-release.sh <version>

   This creates a <version>-release branch.

2. Wait until the build-release and helm-publish workflows complete successfully.

3. Check out the release branch and test the image puller on both Kubernetes and OpenShift by deploying with Helm:

   git checkout <version>-release
   kubectl create namespace k8s-image-puller
   helm install kubernetes-image-puller -n k8s-image-puller deploy/helm

   Verify that the daemonset is created and pods are running on all nodes in both environments.

Trademark

"Che" is a trademark of the Eclipse Foundation.