pvc-autoscaler

pvc-autoscaler is a Kubernetes controller which periodically monitors persistent volumes and resizes them, if the available space or number of inodes drops below a certain threshold.

Requirements

• Kubernetes cluster
• Storage class with enabled volume expansion
• Metrics source (currently only Prometheus is supported)
• minikube or KinD (for local development)

Installation

There are a few ways to install pvc-autoscaler.

In order to install pvc-autoscaler within an existing Kubernetes cluster, please refer to the included dist/install.yaml bundle.

kubectl apply -f dist/install.yaml

Or you can install it this way instead.

kubectl apply -f https://raw.githubusercontent.com/gardener/pvc-autoscaler/master/dist/install.yaml

Note, that the provided install bundle uses cert-manager for issuing TLS certificates for the admission webhook server. If you intend on using another approach for issuing TLS certificates, make sure to adjust the default kustomization overlay instead.

In addition to that you could also install pvc-autoscaler using the default kustomization overlay located in config/overlay/default. Note that the default kustomization overlay is meant to be used in dev setups. For non-dev setups it is recommended that you create a different overlay that uses config/base as base.

kustomize build config/overlays/default | kubectl apply -f -

pvc-autoscaler uses Prometheus as a metrics source in order to monitor the persistent volumes usage. By default it will use the http://prometheus-k8s.monitoring.svc.cluster.local:9090 endpoint.

If your Prometheus instance is running at a different endpoint, make sure that you configure the --prometheus-address option for the pvc-autoscaler-controller-manager deployment.

Usage

In order to start monitoring and automatically resize a persistent volume, when the available space or inodes drop below a certain threshold you need to create a new PersistentVolumeClaimAutoscaler resource, e.g.

---
apiVersion: autoscaling.gardener.cloud/v1alpha1
kind: PersistentVolumeClaimAutoscaler
metadata:
  name: my-pvca
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-sts
  volumePolicies:
  - maxCapacity: 3Gi
    scaleUp:
      utilizationThresholdPercent: 80
      stepPercent: 10
      minStepAbsolute: 1Gi
      cooldownDuration: 10m

The following properties must be specified when creating a new PersistentVolumeClaimAutoscaler resource.

Property	Description	Default
.spec.targetRef.name	Name of the controller or PVC to monitor and autoscale	N/A
.spec.volumePolicies[].maxCapacity	Max capacity up to which a PVC can be resized	N/A
.spec.volumePolicies[].scaleUp.utilizationThresholdPercent	Threshold percentage for used space/inodes that triggers a resize	80
.spec.volumePolicies[].scaleUp.stepPercent	Percentage by which to increase the PVC during resize	10
.spec.volumePolicies[].scaleUp.minStepAbsolute	Minimum absolute increase in capacity during scale-up	1Gi
.spec.volumePolicies[].scaleUp.cooldownDuration	Duration to wait before another scale-up operation for the targeted PVC objects	N/A

In order to watch the status of the autoscaler you can kubectl describe your PersistentVolumeClaimAutoscaler resource, where you will find information about the latest observed state, last and next scheduled check, status conditions, etc.

Local development

The local environment of pvc-autoscaler supports both minikube and KinD as the local Kubernetes cluster. Once a cluster is running, deployment is handled exclusively via Skaffold — the same commands work regardless of which cluster type you chose.

Setting up the cluster

minikube

make minikube-up

This installs OpenEBS, the LVM-LocalPV CSI driver, and configures the openebs-lvm storage class with volume expansion support. It also installs kube-prometheus, which provides Prometheus, Grafana and Alert Manager.

The default minikube driver is set to qemu, which can be configured via the MINIKUBE_DRIVER env variable.

For example, if you are on GNU/Linux and have KVM you would set the driver to kvm2, e.g.

MINIKUBE_DRIVER=kvm2 make minikube-up

KinD

make kind-up

Note

On KinD, real PVC volume usage metrics are not available for e2e testing. KinD's default local-path-provisioner backs PVCs with plain subdirectories on the node's root filesystem rather than separate block devices. Because of this, kubelet's kubelet_volume_stats_capacity_bytes and kubelet_volume_stats_available_bytes reflect the capacity and free space of the node's root disk, not the requested PVC size. Writing files into the PVC therefore changes node-level disk stats that are shared across all PVCs and pods, making it impossible to simulate per-PVC utilisation thresholds reliably. To work around this, the KinD e2e setup uses a fake metrics server that serves pre-configured Prometheus metrics at each test stage instead of relying on real kubelet-reported usage — see config/overlays/kind/fake-metrics.

Deploying pvc-autoscaler

After the cluster is up (with either make minikube-up or make kind-up), deploy pvc-autoscaler and all required resources using Skaffold.

make pvc-autoscaler-up

For iterative development with automatic redeploy on code changes, use the dev target instead.

make pvc-autoscaler-dev

Tearing down

To remove the deployed resources without destroying the cluster.

make pvc-autoscaler-down

When done with the local environment entirely, tear down the cluster.

make minikube-down   # if using minikube
make kind-down       # if using KinD

Also, make sure to check the help information about each Makefile target by executing this command.

make help

Tests

Run the unit tests.

make test

In order to run the end-to-end tests we need a clean test environment. After you have created a development cluster with make minikube-up or make kind-up and deployed the necessary resources with make pvc-autoscaler-up, you can run the e2e tests using the following command.

make test-e2e-local

If you want to automate the entire creation of the KinD dev cluster and execution of the e2e tests, you can directly run the following command.

make ci-e2e-kind

Contributing

pvc-autoscaler is hosted on Github. Please contribute by reporting issues, suggesting features or by sending patches using pull requests.

License

This project is Open Source and licensed under Apache License 2.0.